Endpoint Protection

 View Only
Back to discussions
Expand all | Collapse all

Symantec Endpoint Protection slows file transfer and network speed on VMs in VMware environment by a factor of four

  • 1.  Symantec Endpoint Protection slows file transfer and network speed on VMs in VMware environment by a factor of four

    Posted Jul 25, 2013 01:19 PM

    We are experiencing significant reduction in file transfer rate and network speed with in our LAN between Virtual Machines running Windows 7 and Windows Server 2008 R2 guest OSs, Symantec Endpoint Protection (SEP) installed is 12.1.2015.2015.

    VMs with all features of SEP installed have file transfer speed of about 30 MB/sec vs 120 MB/sec with no SEP installed.

    Network speeds measured using the iperf utility shows a similar speed degradation of 4 times, 350 Mb/sec vs 1400 Mb/sec.

    To simplify and exclude all extraneous factors we performed file transfer and network speed test where all VMs are hosted on the same VMware ESXi virtualization hosts (Version ESXi 5.1.0 Build 1117900). All VMs are x64 and the ethernet adapters are VMXNET 3, VMWare tools are installed and updated to the latest versions. Virtualization Host CPU usage is 20% and Memory Usage is 40% during the test.

    The only article I found on the subject was http://www.symantec.com/connect/forums/sep-121-ru2-windows-server-2012-vm-singnificantly-reduced-performance. We already had the power setting to high performance so the solution did not help our case.

    We tried enabling only the relevant features of SEP, it did not result in any significant improvement. Only installing SEP Core or unistalling SEP completely seem to be the only solution.

    This seems to be a much bigger trade off between Security and Network Speed than anticipated. Any suggestions and comments are welcome.

     



  • 2.  RE: Symantec Endpoint Protection slows file transfer and network speed on VMs in VMware environment by a factor of four

    Posted Jul 25, 2013 01:25 PM

    What happens with the firewall disabled?

    Have you tried 12.1 RU3?

     



  • 3.  RE: Symantec Endpoint Protection slows file transfer and network speed on VMs in VMware environment by a factor of four

    Posted Jul 25, 2013 02:14 PM

    Brian, Thanks for your recommendation, I did try the test earlier with Firewall feature uninstalled.

    Virus, Spyware and Basic Download Protection: All features Installed and On

    Proactive Threat Protection: All features Installed and On

    Network Threat Protection: Installed and On
    Sub-feature - Intrusion Prevention On
    Sub-feature - Firewall uninstalled.

    Results:

    File transfer rate at 36 MB/sec vs 130 MB/sec
    Iperf network speed 913 Mb/sec vs 2385 Mb/sec

    As expected Updating to 12.1.3 might be an initial suggestion and perhaps a viable Hit and Trial. But SEP 12.1.2015 being a stable release has anyone else faced similar issues? Does SEP 12.1.3 addresses these issues?

     



  • 4.  RE: Symantec Endpoint Protection slows file transfer and network speed on VMs in VMware environment by a factor of four

    Posted Jul 25, 2013 02:20 PM

    Support will be able to determine this.

    You will need to enable WPP logging via the SymHelp tool and provide Wireshark traces. They can than make the determination as to what's going on.

    I only suggest Ru3 because it's the latest (and this may have been addressed) but if it still occurs than there may be a previously undiscovered bug.



  • 5.  RE: Symantec Endpoint Protection slows file transfer and network speed on VMs in VMware environment by a factor of four

    Trusted Advisor
    Posted Jul 26, 2013 08:40 AM

    Hello,

    Please check these 2 recent Articles:

    SEP 12.1.2 Best Practices on Citrix Virtual Desktops ( Provisioning Services) -Part 1-

    SEP 12.1.2 Best Practices on Citrix Virtual Desktops ( Provisioning Services) -Part 2-

    I would suggest you to check the Virtual Image Exception (VIE) tool - 

    The Symantec Endpoint Protection (SEP) 12.1 client checks for this attribute before scanning files and skips scanning any files that are marked as "known good" by the VIE tool. Scans on VDI clients created with images processed by the VIE tool will experience lower I/O load, CPU usage, and network bandwidth usage during scheduled and manual scans.

    The Virtual Image Exception (VIE) tool was created specifically for VDI environments deployed using shared base images. The VIE tool provides the ability to exempt the files in a base image from SEP client scans once the image is deployed. If the files are updated or changed in any way, the updated/changed files will be scanned as usual.

    It is suggested that VM admins either record their VIE exceptions list prior to their VM template machine being added to the domain, or place the computer account for the VM template machine into an OU with no GPOs applied.  Once the VIEtool's exceptions list has been created, GPOs can then be applied to the system as normal. 

    Please see the following article for more information on use of the VIE tool:

    http://www.symantec.com/business/support/resources/sites/BUSINESS/content/staging/DOCUMENTATION/4000/DOC4335/en_US/2.0/sep_virtual_image_exception.pdf

    Here are the Steps and Action:

    Step 1: On the base image, perform a full scan all of the files to ensure that the files are clean. If the Symantec Endpoint Protection client quarantines infected files, you must repair or delete the quarantined files to remove them from quarantine.

    Step 2: Ensure that the client's quarantine is empty. 

    Step 3: Run the Virtual Image Exception tool from the command line to mark the base image files. Check the Article:

    Step 4: Enable the feature in Symantec Endpoint Protection Manager so that your clients know to look for and bypass the marked files when a scan runs.

    Step 5: Remove the Virtual Image Exception tool from the base image.

    The Virtual Image Exception tool supports fixed, local drives. It works with the files that conform to the New Technology File System (NTFS) standard.

    Reference: 

    Symantec Endpoint Protection Virtual Image Exception User Guide 12.1

    http://www.symantec.com/docs/DOC4335

    About the Symantec Virtual Image Exception tool

    http://www.symantec.com/docs/TECH172218

    Symantec Endpoint Protection 12.1 - Virtualization Best Practices

    http://www.symantec.com/docs/TECH173650

    SEP 12.1 & Virtualization

    https://www-secure.symantec.com/connect/articles/sep-121-virtualization

    Hope that helps!!



  • 6.  RE: Symantec Endpoint Protection slows file transfer and network speed on VMs in VMware environment by a factor of four

    Posted Jul 26, 2013 10:21 AM

    Brian, thanks for your suggestion. I have support to assist me with SymHelp and Wireshark.

    Mithun, I looked up the Best Practices documents and articles you provided. Though valuable,I am not sure if they will apply to this case because I am not in Citrix environment and I am not trying to optimize scans. I can create exceptions for certain files but I could not find the files for VMware environment.

    The slow network and file transfer speed that we are experiencing are during normal operation of machine and not during the scans.



  • 7.  RE: Symantec Endpoint Protection slows file transfer and network speed on VMs in VMware environment by a factor of four

    Posted Jul 26, 2013 10:27 AM

    Please update this thread with your progress if you can.



  • 8.  RE: Symantec Endpoint Protection slows file transfer and network speed on VMs in VMware environment by a factor of four

    Posted Jul 26, 2013 03:51 PM

    cor089f702

    		 N iperf -s cor089f701 N iperf -c 172.29.67.55 -P 10 -t 240 -w 130000 [SUM]  0.0-240.0 sec   149 GBytes  5.32 Gbits/sec Same host, same vLAN, Same vSwitch
    cor089f702 N iperf.exe -s -P 0 -i 1 -p 5001 -f k cor089f701 N iperf.exe -c 172.29.67.55 -P 1 -i 1 -p 5001 -f k -t 10 -T 1 [164]  0.0-10.0 sec  0.96 GBytes  0.83 Gbits/sec Same host, same vLAN, Same vSwitch
    cor089f702 N iperf.exe -s -P 0 -i 1 -p 5001 -f k cor089f701 N iperf.exe -c 172.29.67.55 -P 10 -i 1 -p 5001 -f k -t 10 -T 1 [SUM]  0.0-10.3 sec  3.72 GBytes  3.10 Gbits/sec Same host, same vLAN, Same vSwitch
    cor089f702 N file copy cor089f701 N file copy 75MB/s (both directions) Same host, same vLAN, Same vSwitch
    cor089f702 N iperf -s cor089f701 Y iperf -c 172.29.67.55 -P 10 -t 240 -w 130000 [SUM]  0.0-274.2 sec  35.7 GBytes  1.12 Gbits/sec Same host, same vLAN, Same vSwitch
    cor089f702 N file copy cor089f701 Y file copy 28MB/s (F702-F701) 21MB/s (F701-F702) Same host, same vLAN, Same vSwitch
    cor089f702 N iperf -s cor089f701 Y iperf -c 172.29.67.55 -P 10 -t 240 -w 130000 [SUM]  0.0-240.5 sec  5.08 GBytes   181 Mbits/sec Same host, same vLAN, Same vSwitch
    cor089f702 N file copy cor089f701 Y file copy 16MB/s (both directions) Same host, same vLAN, Same vSwitch
    cor089f702 NTP removed iperf -s cor089f701 NTP removed iperf -c 172.29.67.55 -P 10 -t 240 -w 130000 [SUM]  0.0-240.3 sec  27.4 GBytes   978 Mbits/sec Same host, same vLAN, Same vSwitch
    cor089f702 NTP removed file copy cor089f701 NTP removed file copy 30-60MB/s (F702-F701) Same host, same vLAN, Same vSwitch
    cor089f702 AV only iperf -s cor089f701 AV only iperf -c 172.29.67.55 -P 10 -t 240 -w 130000 [SUM]  0.0-240.0 sec  33.8 GBytes  1.21 Gbits/sec Same host, same vLAN, Same vSwitch
    cor089f702 AV only file copy cor089f701 AV only file copy 50-60MB/s (F702-F701) Same host, same vLAN, Same vSwitch
    cor089f702 N iperf -s cor089f701 N iperf -c 172.29.67.55 -P 10 -t 240 -w 130000 [SUM]  0.0-241.3 sec   128 GBytes  4.56 Gbits/sec Same host, same vLAN, Same vSwitch

     

    I am most curious about your results as well, because after some exhaustive testing, I can say SEP definitely has a major impact on my systems.



  • 9.  RE: Symantec Endpoint Protection slows file transfer and network speed on VMs in VMware environment by a factor of four

    Trusted Advisor
    Posted Jul 29, 2013 05:01 AM

    Hello,

    Could you please let us know if this file transfer issue occurying from Server to client machine or vice versa or both ways?

    Secondly, could you try installing the AV/AS component only and disable the symtdi.sys driver from the machines and check if that helps.

    The SEP firewall components will not protect a VMware guest operating system. 

    If the VMware guest operating system requires SEP protection, it must be installed directly to the VMware guest Operating System.

    For Vmware Environment, check these Articles:

    Guidelines for installing and running the Symantec Endpoint Protection Manager (SEPM) in a VMware image.

    http://www.symantec.com/docs/TECH132456

    Best Practices for Symantec Endpoint Protection in Virtual Environments

    http://www.symantec.com/docs/TECH95300

    Using Symantec Endpoint Protection in virtual infrastructures

    http://www.symantec.com/docs/HOWTO81060

    Best Practice for Symantec Endpoint Protection Scheduled Scans in VMWare

    http://www.symantec.com/docs/TECH95928

    SEPM: poor database performance

    http://www.symantec.com/docs/TECH155046

    Hope that helps!!



  • 10.  RE: Symantec Endpoint Protection slows file transfer and network speed on VMs in VMware environment by a factor of four

    Trusted Advisor
    Posted Jul 30, 2013 05:47 AM

    Hello,

    In your case, I would request you to please open a Case with Symantec Technical Support Team and PM me the Case #.

    How to create a new case in MySupport

    http://www.symantec.com/docs/TECH58873

    Phone numbers to contact Tech Support:-

    Regional Support Telephone Numbers:

    United States: https://support.broadcom.com (407-357-7600 from outside the United States)
    Australia: 1300 365510 (+61 2 8220 7111 from outside Australia)
    United Kingdom: +44 (0) 870 606 6000

    Additional contact numbers: http://www.symantec.com/business/support/contact_techsupp_static.jsp

    Hope that helps!!



  • 11.  RE: Symantec Endpoint Protection slows file transfer and network speed on VMs in VMware environment by a factor of four

    Posted Jul 30, 2013 04:32 PM

    Mithun, I did create a case with Symantec Tech Support back in June but don't have a resolution yet. I have sent you the case # in PM.

    Posting my issue in this community forum is an alternate attempt to find a solution. Or perhaps understand that file transfer and network speed being four times slower is normal and expected behavior on Virtual Machines with Symantec Endpoint Protection installed in VMware environment.



  • 12.  RE: Symantec Endpoint Protection slows file transfer and network speed on VMs in VMware environment by a factor of four

    Posted Jul 30, 2013 08:14 PM

    Actually, that version is NOT a stable release.  It causes problems with the teefer driver and slows down your network.  I have turned on all the firewalls and have verified that 12.1.3 fixes the known issue found in 12.1. RU2.  Your choices, just after RU2 came out were documented as either turn off the firewall, wait for RU3 or revert to the previous version.  Reverting was awful.  Everything OK with RU3.



  • 13.  RE: Symantec Endpoint Protection slows file transfer and network speed on VMs in VMware environment by a factor of four

    Posted Jul 31, 2013 12:37 AM

    Hi, 

    On your base VM is network utilization is more than 300 MB?

    Regards

    Ajin