Endpoint Protection

On Friday, I posted a query relating to the accumulation of .tmp files within the xfer directory and the identification of these files as being infected by the JS.Alescurf malicious code:

http://www.symantec.com/connect/forums/endpoint-protection-detection-tmp-files-within-symantec-endpoint-protection-xfer-directory-in

Several responded to my post identifying articles about similar problems with the suggestion that I should upgrade my version of EP to ver 12.1, etc.

I read the references and followed the suggestions, susequently posting some updates about my progress.  I upgraded to ver. 12.1.  I downloaded all updates to definitions and ran a complete scan of the C: drive, which identified a single file in the DefWatch.DWH directory as being infected.  This much, I related in a follow up post.

*

Sunday morning, things went from bad to worse.  When I restarted my Dell laptop from hibernation, I began to get new reports of infection by the JS.Alescurf malicious code within this directory:

c:\programdata\symantec\DefWatch.DWH

I have some meticulous notes about my experiences over the past 30 hours which are only abstracted in this post.  I will post some additional details in a follow up.

The most important thing to relate is that there seems to persist some problem with accumulation of .tmp files which are detected as being infected, albeit in a new directory.

Overall, I have come to the tentative conclusion that the files are NOT, in fact, infected but rather are being falsely REPORTED as being infected due to some unresolved BUG in the Symanantec software.

Amongst the bizarre behavior I experienced was EP continuing to report that it was finding and Quarantining files purported to be from this directory even AFTER all of the files were completely DELETED from this directory and there were NO FILES remaining in this directory at all.

In fact, I believe that the bug involves some sort of circular problem with RP identifying a file as infected, reporting such file as "Pending Analysis" or Quarantined and then rediscovering the very same file within its Quarantine and reporting it again.  Alternatively, or possibly in combination with this, EP seems to be detecting its own downloaded Virus definition files as being infected.

One reason to suspect the latter is that I began receiving a new set of messages regarding possible infections Sunday night at about 10:45 right after LiveUpdate downloaded and installed new virus definitions.

I am NOT talking about just a few files.  My system, which was reported to be free of malicious code on Saturday after installation of the newer version of EP had identified for remediation almost 3,100 files ALL in the DefWatch.DWH directory by 3:49 AM this morning but it wasn't through yet. 

To assure that I wasn't receiving any newly infected files (or EP Virus Definition files) while I continued to troubleshoot and remediate this problem, I disconnected from the Internet by physically disabling the WIFI on my system.  Even so, NEW files continued to appear within the DefWatch.DWH directory faster than the computer was able to Quarantine these files.

For this reason, finally at about 3:50 AM, I went into the Command window, maneuvered to the DefWatch.DWH directory and manually DELETED ALL the .tmp and .js files (one) in that directory.  Even so, EP continued to both identify and assert that it was Quarantining .tmp files from this directory even though both a Dir command from the Command window or viewing this directory using Explorer showed that there were no remaining files there.

By about 10:15 this morning, I had received a total of 3,532 file notifications from EP ALL asserting that it was Quarantining files (or marking them "Pending Analysis") since Sunday morning.  EP continued to report and Quarantine about 1 new infection per minute for more than six hours after ALL of the files in this directory had been deleted.

Finally, I simply re-started the machine.  When it came back up, I was presented with an EP alert window, but NO FILES were identified as infected.

I ran an Active Scan of the computer.  NO FILES were identified in this abbreviated scan. 

Then I logged off completely and logged back on under an Administrator account and ran another Active Scan under that administrator account with similar results.  Next, I ran a scan of the ProgramData directory, which was also detected no infections.

Thereafter, I reconnected to the Internet and ran LiveUpdate to obtain the latest AntiVirus definitions.  Now that the definitions have been updated, I am AGAIN running a scan of the ProgramData directory.

Thus far, this scan has detected ONLY two tracking cookies.

Interestingly, while this scan continues to run, the DefWatch.DWH directory, which was EMPTY initially, is showing the successive appearance of a single .tmp file, apparently as each new file is scanned.  But when running properly, it appears that these files make a momentary appearance and then are deleted with only a single file in the directory at a time.  It would appear that as EP was running, it was detecting these very temporary files and reporting them to be infected.

I have already spent far more time than I could afford troubleshooting what appears to be an obscure bug in the Endpoint Protection program.  I have several theories about the circumstances that might cause this problem to present.  I will post some additional comments later after I catch up on my work.

I would encourage those familiar with the inner workings of EP to carefully read and assess what I have posted above.  I would actually be interested to see whether someone else perceives the likely circumstances that would present this bizarre problem.

Even though the problem does NOT appear to occur often, I believe that there IS A BUG in EP that requires remediation.  I am reasonably confident that the problem is replicable and thus it CAN and WILL occur to other customers.

From what I can recall, here is one of (if not the first) thread related to this problem:

https://www-secure.symantec.com/connect/forums/generic-trojan-dwhtmp-temp-folder

On the last page there is a post by Ryan_Dasso (Symantec employee) which explains why it occurs and following that post is another by Mithun Sanghavi (Symantec employee) on how to remediate.

With that being said, I have seen this issue since the very early days of 11.x and while it has been "improved" in subsequent versions, there has not been a true fix for it.

Not really sure where else to point you though since it seems here to stay until a true fix is applied.

Brian:

Thanks for the link to the older thread.  It certainly seems that this problem has been around for a while!

I see several very insightful posts already that, particularly an early post from Ryan Dasso that seem to me to be well thought out and also consistent with my own experience.

Let me mention that I have been using EP on this machine under version 11 for more than three years incident free and have been quite satisfied.  I have been using Symantec AntiVirus and later EndPoint Protection for more than a decade and have been very pleased with the product.  It has been so long since I had an issue that these Forums came into existence since I last experienced any trouble!

I do NOT have time right now to carefully read every post in the old thread, but I think that I have two suggestions as to possible conditions that might cause precisely the problem presented and ought to be considered in furture patches to the software.

First, realizing that Endpoint Protection is creating these files in the ProgramData directory (for Windows 7) or other similar directories for other operating systems, it is intuitively obvious that one way that at least orphaned .tmp file might be created is when the user's computer crashes during a routine scheduled or manual scan!

This might have been one of the sources of my original problem.  I have been having some battery problems.  I unplugged my laptop to move to a different power outlet and before I could plug back in Windows detected a low power condition and went into a less than totally graceful shutdown.  I do not recall whether I was running any scan at the time, but if I was, does EP MONITOR whether a system was shut down unexpectedly and expressly go back and CLEAN UP files under such a condition??  This is something that should be checked!

Similarly, what happens in terms of EP temp files if there is an unexpected shutdown of a system while in the process of running LiveUpdate?  Upon restart, does EP CLEAN UP the temp files created prior to the crash or does it leave these in place, possibly creating a condition where these same files might be subject to a false identification as malicious code?

I do NOT have any specific reason to believe that the single unexpected shutdown of my system during the past week had anything to do with the problem, but it seems to me that good programming practice ought to include some code to address and remediate temp files in EXPECTED PLACES when there is an unexpected shutdown of the OS, especially if in coincident with either a LiveUpdate installation of software patches or definitions OR of a scheduled or manual scan underway at the time the crash occured.  Setting and clearing some data flag showing that these processes are underway or finished would be pretty elementary and if it isn't being done it should be.

*

More likely, in my case the problem may relate to having a system logged onto multiple computer accounts.

I  do NOT do this often, but part of my regular security discipline on this computer is to use an unprivileged account for routine work.  I log onto a separate administrator account when I do Windows OS patches (which I select manually), do other software updates or perform similar tasks that require additional privileges.

I usually download and install the new EP virus definitions using the unprivileged account.  The computer is configued to run LiveUpdate daily, but very often, I run Live Update one or more additional times each day just to always be on the very latest definitions.

One thing that CHANGED this past week was that when I was downloading and updating the latest Windows OS patches, I was online doing some research and had various browser windows open, etc.  Usually, when I update the Windows patches, I also manually check for other software updates.  When I was interrupted in this process, I switched users, leaving the session under the administrator OPEN, while logging back onto my regular account sans privileges.  I usually would expressly LOG OUT of the privileged account.

Thus, I had two separate Windows sessions open, one active and the other suspended.  Each of these sessions has a separate scheduled LiveUpdate run daily.  SInce I usually explicitly log off each account before logging onto the other account, these would not ordinarily interfere with one another.

But this past week, as the problem discussed in this and my other related thread reveals, I mostly had both accounts logged on and was switching between these two accounts.

When the problem with the tmp files first manifested itself, there was at least some REASON to do this.  The first diagnostic messages I received from EP stated "Access Denied" in respect of remediating the problem!  Thus, it seemed that my regular account lacked the privileges to actually Quarantine these files.  (I now think that this may actually be a misnomer in the diagnostic message and that this reflects only the fact that "Analysis Pending" was what precluded the Quarantine.)

After getting these messages, I logged onto the administrator account and ran a scan of the very directory where the infections were being reported.  EP reported finding and Quarantining the infected files and I had the sensation that the problem was solved.

But it appears that EP was probably still running (but suspended) in the other ordinary user session from which I had switched.

Basically, I think that the two sessions were running essentially competing instances of EP.  My guess is that the possibilitiy that users might be running more than one instance of EP in different simultaneous sessions is not well thought out!

Bear in mind that under older models of user/program data storage where certain data is stored in a user specific and privilege segregated directory each user might never see the other user's data.  I think if your programmers put on their thinking caps and ask themselves how EP might perform and would perform given different scenarios of simultaneous sessions, etc., I think they can probably replicate this problem and fashion a durable solution!

The experience is a reminder for ME that no mater how convenient it might be to switch between user accounts, it is probably a better idea to simply close out of processes and do a full log out before embarking upon a new session.  Just because I can switch, it doesn't follow that I should.  Even so, there are a number of instances where switching between user accounts is necessary and it should work.  Moreover, especially in a corporate setting, there may be cases where corporate IT types need or desire to run processes on end user machines.  Simultaneous sessions or processes, which could introduce competing instances of EP, ought to behave more gracefully with one another.

I agree with Ryan's point that there are likely to be multiple origins of the symptoms of this problem.  Still, two of these would seem to me to be unexpected shutdowns and EP operating across competing sessions.  My suspicion is that the latter was implicated in my experiences.  I will seek to AVOID this problem by being even more sparing in switching rather than logoff/logon, etc.

What would be helpful would be answers to the following:

1) Is this a managed or self-managed client?

2) What is the exact version of your SEP client?

3) What is your OS version?

MJD

Michael:

I addressed some of these questions in my previous thread, but let me summarize here.  I am running Windows 7 64-bit edition on a Dell Studio.  My version of Symantec EP is self-managed.

When I posted the original thread, I was running Symantect Endpoint Protection (ver. 11.0.5002.333), installed initially on my Dell machine when new (2010).

At the suggestion of those responding to my prior thread, I upgraded to Symantect Endpoint Protection (ver. 12.1.2015.2015, as shown in EP Help).  The Common Client version shown within the Troubleshooting dialogue box is now ver. 12.1.1.5.

Let me add that while I have been using Symantec AntiVirus and EP for more than a decade and a half, I have not been involved in managing clients in a corporate setting for almost a decade.  I keep my machine fairly well locked down.  Thus, my ongoing experience with Symantec EP troubleshooting is actually quite limited despite being a longstanding customer.

Also, our most recent posts crossed in the Ether.  You might want to take a look at some of the additional elaborating detail given in my previous post.

I might add that I have a vague recollection of experiencing problems of anti-virus software detecting software or virus definition files of a competing anti-virus product as malicious code some two decades or so ago.  This is a rather obvious challenge of putting more than one antivirus product on a machine.  It had not occured to me until this past weekend that running multiple sessions (in switched mode) using the same product on a single machine could present a similar problem.

In troubleshooting, I have often found that one of the best questions to ask oneself is what changed.  I started off thinking that Symantec had changed something.  It this was the case, many others would probably be experiencing the symptoms I described.  Thus, I needed to be a bit more introspective.  In asking myself what I was doing differently, the switching between user accounts and the unexpected crash of the system are the two issues that stand out.  Some of the notes, screen shots and log entires I recorded are at least consistent with the theory that switching between user sessions might be cloasely associated with or implicated in this problem.

Brian:

I am still working through the thread you identified above.  I want to call to the express attention of others your post of 01 Feb 2011:

It was just a suggestion. It worked for me when I had to deal with it. I uninstalled RU6a completely and installed RU6 MP2. Problem solved. As a workaround, assuming you're using a managed SEP client, you can go into the SEPM and under AV Policy >> Quarantine >> When New Definitions Arrive >> select the radio button for Do Nothing

Otherwise, call Symantec and open a case for further assistance.

The reason you hear upgrade or uninstall/re-install is because the majority of the time it works. If you are a user of Microsoft products, you should be more than familar with this.

If it still doesn't work, call Symantec so they can review. You can also ask them about their QA process if so inclined.

See:  https://www-secure.symantec.com/connect/forums/generic-trojan-dwhtmp-temp-folder#comment-5125751

There seem to be implicitly two things that can be done at least to improve performance and to possibly minimize the possibility of the problem.  I still think the core accumulation problem is likely to be related to a poor cleanup of files after an OS or hardware crash and/or conflicts between multiple sessions.

But even now after overcoming the most conspicuous effects of the problem, Symantec EP seems to be doing the comparison of the newly downloaded definitions and the Quarantine File.  This seems to explain the sequential appearance of individual dwh*.tmp files within the DefWatch.DWH directory since I did the most recent download of new definitions via LiveUpdate several hours ago.

My Quarantine file is now cluttered with upwards of 7,500 Quarantined files arising out of the experiences discussed above.  Your post calls attention to the fact that the end user can choose whether these files are rescanned after new Virus definitions are downloaded.

I am a little confused as to how to implement your suggestion above in SEP 12.1.  The closest thing I see is the optin outlined below: 

Within Virus and Spyware Protection Settings (accessed through the Change Settings button), select the Auto-Protect tab. 

On the Auto-Protect tab, choose the Advanced Option.  Within the Auto-Protect Advanced Options Dialogue Box, there are check boxes within the File Cache section providing for "Enable the File Cache" and "Rescan the cache when new definitions load".

Have I correctly identified the current method of disabling the re-scanning?  Or is there some other option that I haven't found yet?

I am a bit of a data packrat and, generally speaking, my preference would be to maintain the Quarantine files to have better documentation of problems encountered.  But as I now understand that SEP is going to be running the new virus definitions against my Quarantine files, I now have an almost endless chore of comparisons! 

SEP seems to be rechecking the Quarantine now and seems to be taking about five (5) seconds per file, so it looks as though it will take almost ten hours to re-scan these files every time LiveUpdate downloads new virus definitions!

There is an interesting implication of this rechecking process to be considered within the context of my observation about multiple user sessions and instances.  IF LiveEdit downloads new definitions and it takes this long to perform the re-scanning (on a reasonably fast dual processor machine with a light application load), the chances that this rescanning might be underway when a random crash occurs OR when a user switches between accounts on the same computer, etc., would be quite high.

Thus, it seems that I need to either clear my Quarantine directory of these specious false positive .tmp files OR to disable the rescanning against newly installed virus definitions or both.  It is probably most sensible to go ahead and DELETE the Quarantined files!  Otherwise, I create both an ongoing performance issue, as well as a much higher probability that I might encounter a recurrence of the problem I already expereinced by switching users.

I therefore ahead and CLEARED the Quarantine, leaving only a handful of exemplary files showing instances of the problems described.  I had only four files Quarantined in the first three years I used SEP.  Now, I had more than 7,500 in the past ten days.

It is noteworthy that the process of single .tmp files appearing within the DefWatch.DWH directory immediately stopped when I had cleared the bulk of the Quarantine files, implicitly confirming that the appearance of these files one at a time was being driven by the comparisons discussed above!

I also set the retention period for Quarantine files at 45 days.  I have never really had to go back and look at older Quarantine files.

*

I would therefore identify clearing the Quarantine files as one of the alternative remediation strategies to resolve both the accumulation of .tmp files and any performance issues presented after a prior accumulation.    

You may not like the suggestion but I would do full removal of that client with the latest version of CleanWipe and then install the newer 12.1 RU3 (12.1.3001) self-managed client.

Just my 2 cents worth...

MJD

Michael:

Thanks for your suggestion.  However, a careful reading of the various references posted by others in this and the earlier thread reveals two critical insights. 

First, reinstallation has mostly been effective when it accomplishes the deletion of both the temporary files as well as the Quarantined files.  If this has been done, the symptoms of the problem should be resolved as to the tmp files and the warnings of infection.

Second, despite repeated assertions by Symantec (going back more than four years) that the problem has been resolved in each successive update, there seems to be no actual evidence whatsoever that this was ever true.

My implementation of SEP is now working just fine without any furher reinstallation.  I suspect that it would have worked fine with some remediation without ever upgrading from SEP ver. 11, IF I had known about the other discussion and remediation steps.

I would still hope that someone from Symantec will carefully read my posts and forward the information to those who actually fashion patches, because I believe that my additional insights might lead to an actual FIX for the underlying problem.

As has been noted in many, many patches over the past four years, telling users that they can work around this known defect in the SEP software is not really a solution, especially for corporate users managing hundreds or thousands of systems!