Endpoint Protection

I have encountered stop error 0x7f on several W2K3 boxes randomly and repeatedly in my environment these days. According to the memory dump, it looks symevent.sys is related to this issue.

Basically, this stop error means that there was kernel stack over flow. And I saw registry entry named "KStackMinFree" to prevent lack of kernel stack on Symantec KB TECH118984 (seems Japanese ONLY).

But I'm just wondering if there is any other workaround or resolution for this phenomenon...

 

Software : SAVCE 10.1.7.7000

Symevent.sys version : 12.2.1.1

OS version : Windows Server 2003 Standard SP2 JP, Windows Server 2003 R2 Standard SP2 JP/EN

 

Does anyone have any idea on this ?

Cheers,

Shinsaku

 

1: kd> !analyze -v

*******************************************************************************

*                                                                             *

*                        Bugcheck Analysis                                    *

*                                                                             *

*******************************************************************************

 

UNEXPECTED_KERNEL_MODE_TRAP (7f)

This means a trap occurred in kernel mode, and it's a trap of a kind

that the kernel isn't allowed to have/catch (bound trap) or that

is always instant death (double fault).  The first number in the

bugcheck params is the number of the trap (8 = double fault, etc)

Consult an Intel x86 family manual to learn more about what these

traps are. Here is a *portion* of those codes:

If kv shows a taskGate

        use .tss on the part before the colon, then kv.

Else if kv shows a trapframe

        use .trap on that value

Else

        .trap on the appropriate frame will show where the trap was taken

        (on x86, this will be the ebp that goes with the procedure KiTrap)

Endif

kb will then show the corrected stack.

Arguments:

Arg1: 00000008, EXCEPTION_DOUBLE_FAULT

Arg2: f7727fe0

Arg3: 00000000

Arg4: 00000000

 

Debugging Details:

------------------

 

 

BUGCHECK_STR:  0x7f_8

 

TSS:  00000028 -- (.tss 0x28)

eax=00000000 ebx=b908110c ecx=85ed9880 edx=85aa8008 esi=b9081088 edi=b9080f74

eip=8088c80d esp=b9081000 ebp=b9081004 iopl=0         nv up ei pl zr na pe nc

cs=0008  ss=0010  ds=0023  es=0023  fs=0030  gs=0000             efl=00010246

nt!KiTrap0E+0xc9:

8088c80d 50              push    eax

Resetting default scope

 

DEFAULT_BUCKET_ID:  DRIVER_FAULT

 

PROCESS_NAME:  csrss.exe

 

CURRENT_IRQL:  1

 

TRAP_FRAME:  b9081318 -- (.trap 0xffffffffb9081318)

ErrCode = 00000000

eax=dc5ff000 ebx=00000001 ecx=0000000f edx=00000000 esi=85ed9880 edi=00000000

eip=808b64a6 esp=b908138c ebp=b90813c8 iopl=0         nv up ei ng nz ac po cy

cs=0008  ss=0010  ds=0023  es=0023  fs=0030  gs=0000             efl=00010293

nt!CcMapData+0x8c:

Page a7012 not present in the dump file. Type ".hh dbgerr004" for details

808b64a6 8a10            mov     dl,byte ptr [eax]          ds:0023:dc5ff000=??

Resetting default scope

 

LAST_CONTROL_TRANSFER:  from 80881438 to 8088c80d

 

STACK_TEXT: 

b9081004 80881438 badb0d00 85aa8008 00000000 nt!KiTrap0E+0xc9

b908110c 8081df85 8acff718 85aa8008 85aa8008 nt!_alloca_probe+0x1c

b9081120 f723fd28 8a2930b8 8acb82a8 85aa81e0 nt!IofCallDriver+0x45

b908114c 8081df85 8b4bf730 85aa8008 85aa8204 fltmgr!FltpDispatch+0x152

b9081160 ba0bb8e1 89c39970 8aa16410 89c39568 nt!IofCallDriver+0x45

WARNING: Stack unwind information not available. Following frames may be wrong.

b9081174 8081df85 89e522c0 85aa8008 85aa8008 SYMEVENT+0x78e1

b9081188 f723fd28 0083f000 8acb82a8 00000000 nt!IofCallDriver+0x45

b90811b4 8081df85 89c39970 85aa8008 85aa8008 fltmgr!FltpDispatch+0x152

b90811c8 8081e50d 85ed9880 8b4bd450 c06e2ff8 nt!IofCallDriver+0x45

b90811e0 80851198 8b5a120b 8b4bd488 8b4bd468 nt!IoPageRead+0x109

b908127c 8085eac0 00000001 dc5ff000 8b4bd450 nt!MiDispatchFault+0xece

b9081300 8088c820 00000000 dc5ff000 00000000 nt!MmAccessFault+0x89e

b9081300 808b64a6 00000000 dc5ff000 00000000 nt!KiTrap0E+0xdc

b90813c8 f7b90f2d 8b5a12a0 b90813f8 00000400 nt!CcMapData+0x8c

b90813e8 f7b8e494 b9081a88 8b530468 0083f000 Ntfs!NtfsMapStream+0x4b

b908145c f7b90df0 b9081a88 8acff7f8 e3d21330 Ntfs!NtfsReadMftRecord+0x86

b9081494 f7b90fac b9081a88 8acff7f8 e3d21330 Ntfs!NtfsReadFileRecord+0x7a

b90814cc f7b4f8a8 b9081a88 e3d21328 e3d21330 Ntfs!NtfsLookupInFileRecord+0x37

b90815dc f7b50674 b9081a88 e3d213f0 0000000f Ntfs!NtfsLookupAllocation+0xdd

b90817ac f7b5082c b9081a88 86512008 e3d213f0 Ntfs!NtfsPrepareBuffers+0x25d

b9081988 f7b51156 b9081a88 86512008 e3d213f0 Ntfs!NtfsNonCachedIo+0x1ee

b9081a74 f7b51079 b9081a88 86512008 00000001 Ntfs!NtfsCommonRead+0xaf5

b9081c20 8081df85 8acff718 86512008 86512008 Ntfs!NtfsFsdRead+0x113

b9081c34 f723fd28 8a2930b8 8acb82a8 865121e0 nt!IofCallDriver+0x45

b9081c60 8081df85 8b4bf730 86512008 86512204 fltmgr!FltpDispatch+0x152

b9081c74 ba0bb8e1 89c39970 8aa16410 89c39568 nt!IofCallDriver+0x45

b9081c88 8081df85 89e522c0 86512008 86512008 SYMEVENT+0x78e1

b9081c9c f723fd28 0000f000 8acb82a8 00000000 nt!IofCallDriver+0x45

b9081cc8 8081df85 89c39970 86512008 86512008 fltmgr!FltpDispatch+0x152

b9081cdc 8081e50d 85ed9880 8b47d838 c0629e78 nt!IofCallDriver+0x45

b9081cf4 80851198 89b1fa0b 8b47d870 8b47d850 nt!IoPageRead+0x109

b9081d90 8085eac0 00000001 c53cfe00 8b47d838 nt!MiDispatchFault+0xece

b9081e14 808592de 00000000 c53cfe00 00000000 nt!MmAccessFault+0x89e

b9081e50 808b5724 c53cfe00 00000000 b9081fa8 nt!MmCheckCachedPageState+0x4f8

b9081ee0 f7b836ce 8541ef90 0000fe00 00000200 nt!CcFastCopyRead+0x1a2

b9081f38 f723eca2 8541ef90 b9081fa8 00000200 Ntfs!NtfsCopyReadA+0x1c1

b9081f6c f724b8b3 00000003 00000000 b9081fa0 fltmgr!FltpPerformFastIoCall+0x230

b9081fc0 ba0b7cca 8541ef90 b9082064 00000200 fltmgr!FltpFastIoRead+0xa9

b9081ff4 f723eca2 8541ef90 b9082064 00000200 SYMEVENT+0x3cca

b9082028 f724b8b3 00000003 00000000 b908205c fltmgr!FltpPerformFastIoCall+0x230

b908207c 808f239d 8541ef90 b90820b8 00000200 fltmgr!FltpFastIoRead+0xa9

b9082118 808897ec 80003f20 00000000 00000000 nt!NtReadFile+0x2c5

b9082118 8082f501 80003f20 00000000 00000000 nt!KiFastCallEntry+0xfc

b90821b4 ba11c083 80003f20 00000000 00000000 nt!ZwReadFile+0x11

b9082244 ba0edcff ee79ce20 e6292548 00000200 savrt+0x46083

b9082268 ba10a403 e6292548 00000200 ba10aa59 savrt+0x17cff

b9082274 ba10aa59 e7441b9d e6292548 00000200 savrt+0x34403

b908229c ba10aba1 e6292001 e6292210 00000000 savrt+0x34a59

b90822c4 ba10b0eb e6292008 01000000 00017eff savrt+0x34ba1

b90822f0 ba10b143 0000ffff 01000000 e6292008 savrt+0x350eb

b908230c ba115655 0000fffe 00000000 00000001 savrt+0x35143

00000000 00000000 00000000 00000000 00000000 savrt+0x3f655

 

 

STACK_COMMAND:  .tss 0x28 ; kb

 

FOLLOWUP_IP:

SYMEVENT+78e1

ba0bb8e1 5f              pop     edi

 

SYMBOL_STACK_INDEX:  5

 

SYMBOL_NAME:  SYMEVENT+78e1

 

FOLLOWUP_NAME:  MachineOwner

 

MODULE_NAME: SYMEVENT

 

IMAGE_NAME:  SYMEVENT.SYS

 

DEBUG_FLR_IMAGE_TIMESTAMP:  4551513d

 

FAILURE_BUCKET_ID:  0x7f_8_SYMEVENT+78e1

 

BUCKET_ID:  0x7f_8_SYMEVENT+78e1

 

Followup: MachineOwner

---------

There is an English version of the document:

http://www.symantec.com/business/support/index?page=content&id=TECH99708

If that document will not help I would suggest to open the case with Symantec TechSupport and provide the dump for the analysis.

Hello,

Please have a look at the Microsft Article on the same:

You receive a "Stop 0x0000007F" error message or your computer unexpectedly restarts

http://support.microsoft.com/?kbid=822789

 

Hope that might help you!!

The latest build of SAV for Win 2k, 2k3, and XP is 10.1.9 & for Win Vista, 7, 2008, and 2008R2 would be SAV 10.2.

I would recommend to upgrade to the latest build for respective operating systems as there have been many enhancements in the product. In addition many of our customers have moved from SAV to SEP as SEP provides more features and is better designed to handle the current threat landscape.

w-d
Mithun Sanghavi
Kurt G.

hi all, thank you so much for your advice !!

I've contacted local symantec support and they told me that upgrading to SEP might be one of resolutions.

according to their prior cases, they said that this phenomenon may be caused by auto-protect kernel driver (not depend on symevent.sys). and major design change was made on it in SEP.

the driver in SAV is savrt.sys which is normal kernel mode driver. and now in SEP, it is srtsp.sys which is act as filter driver.

i've confirmed it in my test environment.

- SAV (savrt.sys 9.7.2.3)
C:\>fltmc

Filter Name                     Num Instances Frame
------------------------------  ------------- -----
eeCtrl                                  3       1
SymEvent                                     <Legacy>
DfsDriver                                    <Legacy>

- SEP (srtsp.sys 10.3.0.14)
C:\>fltmc

Filter Name                     Num Instances    Altitude    Frame
------------------------------  -------------  ------------  -----
eeCtrl                                  3       329010         0
SRTSP                                   4       329000         0
luafv                                   1       135000         0

so i would like to upgrade SAV to SEP at first and see how it works.

will update when i get the result :-)

 

thanks again for your help !!

Shinsaku

Hello,

Since you have now decided to Migrate from SAV 10 to SEP 11, here are few Symantec Knowledgebase Articles which may help you.

1) How To Migrate From Symantec Antivirus System Center Console To Symantec Endpoint Protection Manager

http://service1.symantec.com/support/ent-security.nsf/854fa02b4f5013678825731a007d06af/eda2ac3faa9a31b0882574f90001484e?OpenDocument

2) Obtaining an upgrade or update for Symantec Endpoint Protection 11.x or Symantec Network Access Control 11.x
http://service1.symantec.com/SUPPORT/ent-security.nsf/docid/2007121216494948?Open&docid=2007121216360648&nsf=ent-security.nsf&view=docid

3) 'Installing and configuring Symantec Endpoint Protection 11 for the first time'
http://service1.symantec.com/SUPPORT/ent-security.nsf/docid/2007082915561148?Open&src=w

4) Creating custom Client Installation packages in the Symantec Endpoint Protection Manager Console
http://service1.symantec.com/support/ent-security.nsf/854fa02b4f5013678825731a007d06af/c741ec26fa674b1e8825738a0076abf3?OpenDocument

5) How to Deploy Symantec Endpoint Protection to your client computers using the Migration and Deployment Wizard.
http://service1.symantec.com/SUPPORT/ent-security.nsf/docid/2007111409432848

Hi Mithun,

Thank you so much for your advice !

Those infomation you provided are very helpful for me :-)

I would like to share it with local support team.

 

I think it will take a long time to complete SEP installation because there are over 300 servers to be upgraded, but I will get back to here when I finished it.

 

thanks,

Shinsaku

Hi all,

I'm now performing upgrading to SEP. But... as you may know, Japan is now under critical situation due to huge earthquake and it is not easy to performe upgrading tasks smoothly.

I would like to get back to here when I complete.

 

Thanks,

Shinsaku

this is just an update.

i have performed upgrading to SEP over 180 servers. and on those servers, the phenomenon has never happened again at this moment.

i will proceed upgrading on remaining servers in my environment.

thx,

Shinsaku