Endpoint Protection

Updated July 16th, 2012 10:34 AM PST:  Additional details provided on confirmed examples where we have seen the blue screen issue.

Updated July 31st, 2012 1:36 PM PST: Issue was also experienced on Windows Server 2003 and has been confirmed by Symantec.

What Happened?

On July 11, 2012 at approximately 10:30 PM PT, Security Response started receiving reports of customers experiencing blue screens on Windows XP and Windows Server 3003 machines after applying definitions July 11^th revision 18 and SONAR definitions July 11^th rev11. This update only contained signature updates and no change to the SONAR driver.

The problem has been identified as a compatibility issue in SONAR definitions released July 11^th at 6:25PM PT. Once the cause of the issue was discovered, the signature was removed from the definition set and an updated definition set was published.  This “rollback” of signatures was done on July 12^th at 2:51AM PT. Once the signature was rolled back, no new issues were reported from the field.

Why Did the Blue Screen Issue Happen? (Updated July 16th, 2012 10:34 AM PST)

After a full evaluation and root cause analysis of the issue, we have determined that the issue was limited to machines running a combination of Windows XP or Windows Server 2003, the latest version of the SONAR technology, the July 11^th rev11 SONAR signature set, and certain software.  Only customers running this combination of technologies and who downloaded the July 11^th rev11 SONAR signature set via LiveUpdate between 6:25PM PT and 2:51AM PT on July 12^th were affected.

The root cause of the issue was an incompatibility due to a three-way interaction between software that implements a file system driver using kernel stack-based file objects.  The three-way interaction is between the software that implements a file system driver (using kernel stack-based file objects), the SONAR signature and the Windows XP Cache manager.  The SONAR signature update caused new file operations that create the conflict and led to the system crash. 

We have confirmed examples of this interaction with the following products:

• Novell ZenWorks
• PGP Whole Disk Encryption
• Sophos LanCrypt
• SlySoft Virtual Clone Drive

How Will Symantec Prevent this from Happening Again?

Symantec understands the consequences of this type of issue to our customers and goes to great lengths to prevent them.  The quality assurance process for SONAR signatures is extensive.  The process includes:

• Peer review and vetting of all signatures
• True positive testing
• False positive testing
• Functional testing of all signature content
• Compatibility testing

The compatibility testing part of the quality assurance process for SONAR signatures missed catching this compatibility issue.  It is this part of our process that we will be improving to avoid future issues.  We are currently restructuring our testing process to improve compatibility testing and will not be releasing new SONAR signatures until this new process is in place.

Which Enterprise Products are Impacted?

Based on our root cause analysis, we determined the problem is isolated to some Windows XP machines with file system drivers (usually encryption) running:

• Symantec Endpoint Protection Small Business Edition (SEP SBE) 12.1
• Symantec Endpoint Protection (SEP) 12.1
• Symantec Endpoint Protection.cloud (SEP.cloud)

Mac customers are not impacted.  Additionally, Symantec Endpoint Protection 11 is not impacted.

This issue has not been reported on any other operating system or other version of Symantec’s Enterprise security products.

How do Customers Know if they are at Risk?

If customers have not experienced a blue screen it is highly unlikely they are at risk.  If customers have any concerns, they should make sure they are running the latest definitions and, if they are, they should have no risk of experiencing blue screens.

How do Customers Remediate this Situation?

Customers running Symantec Endpoint Protection (SEP) 12.1 or Symantec Endpoint Protection Small Business Edition 12.1, should refer to the knowledge based (KB) article for specific details on how to resolve this issue.   If customers continue to experience issues or have further questions, they should contact technical support via their regular support channels. 

http://www.symantec.com/business/support/index?page=content&id=TECH192811

How can Symantec Endpoint Protection.cloud Customers Resolve this Issue?

If a Symantec Endpoint Protection.cloud customer is experiencing this issue they should contact Symantec.cloud technical support.

http://www.symanteccloud.com/supportcentre/information/contact_global_support

Symantec Endpoint Security Team

thanks for the detailed information.

Thanks for Sharing

What is the third-party software which implements the system driver using encryption ?

The blog post mentioned "typical of encryption drivers, the SONAR signature and the Windows XP Cache manager."   The encryption drivers described are for products implementing "whole disk encryption" which manage encryption of all the data on the hard disk.  Products in this category include Microsoft BitLocker, PGP WDE, WinMagic SecureDoc and others.  Several different WDE product drivers are correlated to the incident, and investigation is continuing on the root cause to understand the specific interaction.

Perhaps providing a list with the WDE software found on the BSDO related so far will clarify the topic for other customers.

Maybe also attach the list to the KDB.....

I have updated the original post with confirmed examples where we have seen this interaction occur.  These include:

• Novell ZenWorks
• PGP Whole Disk Encryption
• Sophos LanCrypt
• SlySoft Virtual Clone Drive

Hi,

Just out of curiosity then, can you tell me how this issue managed to get hrough testing, get through quality control (assuming there is some) and get released to end users to cause the issues it did?

Also, can you tell me why your support seemed to have no idea what was going on, until i went through to the "technical Support" who seemed to know all about it and rather annoyingly said "Oh so your aware of the issue then?" and then went on to explain it.

Surely it would have made sense to have the first line aware of the issue, because the first person i spoke to didnt have a clue, yet it seemed more important for her to take details that were not relevant to the issue at all, thus wasting more time and then saying the job is a certain priority and someone will contact me in 24 hours...It was actually a major issue with most of the site down...so I had to argue with for a higher priority and got put through to someone.

Thanks for the details! Good to know for consulting services too!

<reposting from the main thread for visibility>

We do understand the consequences of this type of issue to our customers and go to great lengths to prevent them.  The quality assurance process for SONAR signatures is extensive.  The process includes:

• Peer review and vetting of all signatures
• True positive testing
• False positive testing
• Functional testing of all signature content
• Compatibility testing

The compatibility testing part of the quality assurance process for SONAR signatures missed catching this compatibility issue.  It is this part of our process that we will be improving to avoid future issues.  We are currently restructuring our testing process to improve compatibility testing and will not be releasing new SONAR signatures until this new process is in place.

On the subject of support, I'm sorry you had problems with our support lines, but as I am sure you can imagine, when a quick spreading issue like this hits, there is always potential for people to miss communications or notifications (e.g. first line just came in for the day and started answering calls before checking their email, etc.).  As soon as we knew what was happening we took steps to make sure everyone was aware and cases were dealt with as quickly as possible (looking at some cases, I see customers being responded to within 30 minutes of logging their call regardless of what level of severity it was logged at).

Hi

We've submitted a crashdump from Windows 2003 SP2 + SEP 12.1 to Symantec Technical Support. The result : SEP with this update and Sonar activated, then the OS finish with blue screen ... How a virus def. can crash OS ???

Why can't we found this information on the Symantec Web Site ?

Regards.