Endpoint Protection

This is not an issue with constant high CPU usage, lets get that clear now.  We are seeing disk and CPU spikes on a regular cycle that can be tracked back to the Symantec Service.

In our VM environment this is easy to see on both CPU and Disk due to ability to look at current stats for the past hour/day/month on a VM.

SAV 10 clients:  (10.1.6.6000, 10.1.7.7000 and 10.1.8.8000, on 2003 and XP clients)

Every 3 mins and 20 seconds you'll see a CPU and disk spike.  Disk usage will be 4-8 KB/s and then jump to between 3-12 MB/s usage for about 20-40 seconds.  Upon investigating with filemon we see that SAV is going out and rereading in its definition files again.

SEPM 11 clients (latest for sure, know we saw it in earlier clients also, 2008, 2003, xp, etc)

Similiar to SAV 10, except every 5 mins and 40 seconds on the machine I was just looking at.

Simple fix is to go in and restart the Symantec service, if it is a SAV 10 box, this fixes it until the next AV update, once it has updated its definitions it starts this lovely cycle over again.  A full restart of the system will normally take care of the problem for a few days to a week, but when they are servers this isn't normally an option.  A complete removal of the client, deleting all old def files out there in common folders, etc, will sometimes fix it for a week to a month, but typically comes back again sooner or later.

SEPM clients if you restart the service, this normally fixes them for a week or more, does not normally appear to come back the next time the defs are updated.

Out of 100+ servers we normally see this happening on a few a month, this week I'm seeing it on about 8 machines.  I've opened a ticket in the past on it and basically got told to send them to liveupdate instead of our local managed server, not really an option nor did it fix it, but after trying repeatedly to get them to understand the problem (over a few days) I gave up and just went back to the "simple fix" mentioned above.

The CPU spike is annonying, but not my biggest concern, with shared storage on 8 VMs misbehaving, we see our average MB/s go from about 6 MB/s for all our VMs to spikes in the 24 MB/s, just depends on how many are misbehaving at the same time!

Anyone else seeing any similiar issues?   Any idea why it is constantly rereading in its def files like this on some systems, but not others and why restarting the service fixes it until the next definition update?

Symantec Endpoint Protection Client configuration changes for performance optimization

Is there something specific if there that you think may be applicable to what I'm seeing?

The only thing I can see that deals with time issues is under:  Modify the default communication settings:

There is a 5 minute interval for the heartbeat, but that is checking in with the parent server, not the client checking its definitions on disk. 

This also doesn't effect all servers, so while it could be a setting tweak for the policy may help, I'm not seeing it in the doc you linked to, but I'll look through it again.

the communication setting will help you out with disk space too please check this

https://www-secure.symantec.com/connect/forums/heart-beat

I'll give it a try and see my disk spikes go at a different interval with this setting, but I'm not worried about disk space usage, it is an issue of definition files getting read in on the local system every 5 mins and 40 seconds on SEPM boxes and 3 mins and 20 seconds on SAV clients.  And this reading in of the files causing excessive disk access on the SAN that they reside on.

Maybe a better question is:  How often should definition files be getting read back in by SEPM or SAV?  And why do some systems seem to have no major spikes when it re-reads the definition files in and others seem to hammer the system resources when it does?

I seem to recall a hit on this type of search before that pointed to a registery fix, but this was in SAV 8.  Will try to find that article again.

Interesting, I changed it from 5 mins to 10 mins and at least one of the SEPM boxes appears to be showing spikes at every 11 mins now.  Just tweaked the heartbeat for 20 mins, and I assume I'll see spikes every 22 mins or so then.  Will report back what I find.

Now to go look and see if there is a setting like this for SAV and then find out exactly what the heartbeat setting is used for and how long I can make it.

 it works, you can check this link for more info , this issue is from a long time with respect to SAN 

http://communities.vmware.com/message/1210918;jsessionid=27730EC892B2D572D41A8078E9EA7242

Rafeeq,

Thanks, been down the road of trying to stagger when definition updates happen already, that is a whole different headache, but thanks for the link.  That is typically a once a day problem vs an every X number of mins.

I need to give it another 60 mins or so to verify, but it appears it is a heartbeat issue with SEP.   At least on the interval of it hammering the disk.

This does not answer why when I restart the Symantec service that it only "slightly" hammers the disk, say at 200 KB/s vs when things have gone bad of hammering the disk at 11 MB/s when it reports back to the SEPM server.  Is checking for new policy and uploading logs really that hard hitting?

Still looking for info on SAV 10 for a heartbeat setting since most of my servers are still on it.

just a thought

Configuring the IpSubnetMask value when managing updates by subnet

http://service1.symantec.com/SUPPORT/ent-security.nsf/docid/2001040511373248?Open&seg=ent


Scalability and performance guidelines for Symantec AntiVirus Corporate Edition 10.x and Symantec Client Security 3.x

http://service1.symantec.com/SUPPORT/ent-security.nsf/ppfdocs/2005072910171148?Open&dtype=corp&src=&seg=&om=1&om_out=prod

Rafeeq, I looked at the reporting server settings (again thanks for the links) thinking that may be part of it, but we don't have reporting server setup, so that isn't it.  Is there any time settings in SAV 10 that anyone is aware of for a basic setup?  Looking for something that would be ~3 mins.

And I marked the first reponder as correct/answer solved since it was indeed a heartbeat setting on SEP clients, at least for the timeframe things are happening, but as mentioned above, that still doesn't explain why it is spiking so bad at those time intervals on some clients.  Oh well, one mystery at a time.

VMWare sessions running on an ESX Server with Symantec AntiVirus or Symantec Endpoint Protection installed are performing poorly during definition updates.

http://service1.symantec.com/support/ent-security.nsf/docid/2008031411460648?Open&seg=ent

same setup?

This is an issue, but one we've learned to deal with by force client updates during the night (or attempting to since it still seems to update during the day sometimes.  I like the info in there about changing from Push to Pull.  I had found that in a different thing a bit ago.

Good info on what happens with SEP and the heartbeat is here:
https://www-secure.symantec.com/connect/videos/about-communication-between-sep-sepm

For now I think we're good, or at least good enough.  I may have to open a new thread down the road asking why during the heartbeat and uploading of logs I get such high disk spike usages, but that will have to wait a few weeks!

thanks for all the links Rafeeq!  Got me looking at a few other things in our setup.