Endpoint Protection

We've been upgrading from SAV 10.1 to SEP 11 across our company, and I've successfully upgraded around 250 client computers so far. Almost all of them are working just fine, but on a very small number (say 5 or 10, tops) the client installs but I never get the green dot. No red dot, just the yellow shield with no dots at all. In these cases, the client never picks up definitions and seems as if it's talking to the server at all.

In all cases I'm using the same install pack, which is for a managed client. I've tried reinstalling, repairing, uninstall + reinstall, but I get the same thing every time -- yellow shield, no dots at all, and apparently no ability to talk to the server or get updates.

For the record, all clients are running XP SP3. All clients have the firewall turned off via group policy. I can't remember at the moment, but I'm pretty sure that from the client I can ping the AV server, so in theory other communications ought to be working.

Anybody have any suggestions on what might be blocking the communication, or how to fix this?

This document will show you which ports the SEP client communicates on. I would start there.

Title: 'Which communication ports does Symantec Endpoint Protection 11.0 use?'
Document ID: 2007090614430148
> Web URL: http://service1.symantec.com/SUPPORT/ent-security.nsf/docid/2007090614430148?Open&seg=ent

Hello,

Can you verify that the client is not pointing to an old group that no longer exists?

1. Open the SEP client
2. Click on "Help and Support"
3. Select "troubleshooting" from the drop down
4. Check  that the Group name is valid

You can also run the SEP Support Tool on one of the clients that experiences the issue.

http://www.symantec.com/techsupp/home_homeoffice/products/sep/Sep_SupportTool.exe


Thomas

This is a new install/migration of SAV to  SEP, there would be no old group that it could be pointing to in this case.

You could also try placing the appropriate sylink.xml file from your SEPM install onto each of the affected clients.  There are numerous ways of doing so, including the SylinkDrop tool on CD2 of the install media or the SylinkReplacer tool available from Support, but with so few having issues you probably don't need a complex automated method.  Basically the process is to stop the Symantec Management Client (SMC.exe), drop the sylink.xml file into the SEP install directory, then restart the management client.  

Hi

You could try downloading SylinkMonitor and check what happens when the client tries to communicate with the server.
http://service1.symantec.com/SUPPORT/ent-security.nsf/docid/2007456519454798

/Lasse

Thanks for the suggestions. As Ted noted, it's a new install/upgrade, so there's no other group to point to, and since the install process is the same for all computers, I don't think that would apply. I checked anyway, and in that section it did have the right group listed. More notably, it listed the server as being "offline" -- talking to the server is definitely the problem.

I tried replacing the sylink.xml file in conjunction with the stop/start command, and that didn't seem to help. Just to make sure: I took the version that was in the install pack on the SEPM server and put that on the client -- was that the right one?

So I ran the debug logger tool, and got the following. There's a lot of info in there, and I'm not sure what to make of most of it. The most noteable thing I can see in there is the "unknown error" line, which doesn't tell me much, but maybe it'll mean something to someone else. I've trimmed out a lot of the stuff at the top because it had a lot of keys and user names and computer names, and I'm not sure I should include them. If this isn't enough, I can put the whole log in there.

05/01 08:57:54 [4564] <CheckHeartbeatTimer>====== Heartbeat loop starts at 08:57:54 ======
...lots of registration stuff goes in here ...

05/01 08:57:55 [4564] <SendRegistrationRequest:>http://gfd-av-03:8014sÜ‹Ê*\ïús½{ >¥IàÐRQÚtN•<‰Óþ´ÖT´–•š Óò ĤCíÔR 
²ØÏ÷3(Ô Jz 
à,ã<E
óöÓ® ‹¤²’’´v¼ƒŸÊËZSÄ°ìÎ9^oýŽ¬îï_ÿ¨—‰ÓÐ 8"Ít@ÓOIðþF€kÎzU÷ 4*Á`Kr|1ñ¬KлX/Ï2b~Z ~Ö  ª O¡S—/ðþ$p9#  <æ°¶ÂBi’ ª  ;¬í#ˆô‹H û-ã•|3{3NgÝšËt"¸Ó¡¥( ¬  
³œ^ÛššƒPC5 RóÓ GÞVÏY
05/01 08:57:55 [4564] <ParseErrorCode:>0=>Unknown error code.
05/01 08:57:55 [4564] <SendRegistrationRequest:>SMS return=0
05/01 08:57:55 [4564] <ParseHTTPStatusCode:>0=>Uninterpreted Status
05/01 08:57:55 [4564] <SendRegistrationRequest:>ERR to query content length
05/01 08:57:55 [4564] <SendRegistrationRequest:>Content Lenght => 
05/01 08:57:55 [4564] HTTP returns status code=0
05/01 08:57:55 [4564] <SendRegistrationRequest:>RECEIVE STAGE COMPLETED
05/01 08:57:55 [4564] <SendRegistrationRequest:>COMPLETED
05/01 08:57:55 [4564] HEARTBEAT: Check Point 5.1
05/01 08:57:55 [4564] <RegHeartbeatProc>switch to another server
05/01 08:57:55 [4564] HEARTBEAT: Check Point 9
05/01 08:57:55 [4564] HEARTBEAT: Check Point 8
05/01 08:57:55 [4564] <PostEvent>going to post event=EVENT_SERVER_DISCONNECTED
05/01 08:57:55 [4564] <PostEvent>done post event=EVENT_SERVER_DISCONNECTED, return=0

Take a backup and delete the following registry keys:

HKEY_USER\.DEFAULT\SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings\Connections\DefaultConnectionSettings

HKEY_USER\.DEFAULT\SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings\Connections\SavedLegacySettings



Reboot the machine.

">http://gfd-av-03:8014sÜ‹Ê*\ïús½{ >¥IàÐRQÚtN•<‰Óþ´ÖT´–•š Óò ĤCíÔR
²ØÏ÷3(Ô Jz
à,ã<E
óöÓ® ‹¤²’’´v¼ƒŸÊËZSÄ°ìÎ9^oýŽ¬îï_ÿ¨—‰ÓÐ 8"Ít@ÓOIðþF€kÎzU÷ 4*Á`Kr|1ñ¬KлX/Ï2b~Z ~Ö ª O¡S—/ðþ$p9# <æ°¶ÂBi’ ª ;¬í#ˆô‹H û-ã•|3{3NgÝšËt"¸Ó¡¥( ¬
³œ^ÛššƒPC5 RóÓ GÞVÏY
"

clearly indicates that the sylink.xml is corrupt.

Make sure you pick up and fresh sylink and try replacing it. The procedure to do so is as mentinoed in http://service1.symantec.com/SUPPORT/ent-security.nsf/docid/2008101707165348

......Barkha

Tried updating the sylink.xml file as instructed in the article, but it didn't seem to help. Ran the debug again and got identical results.

I did notice one thing that might be related, though. Right before it sends the registration request with all the strange characters, it looks like there's malformed XML when it builds the request. I've trimmed out some of the details in the middle, but note that the XML being generated appears to simply cut off in the middle of a value, with no quote mark or /> to close out the tag. Could this be part of the problem?

05/01 10:38:54 [764] <mfn_GenPostData (for Registration):>Request is: s_origin_length: 1264
s_session_id: 334093192953C5CC8D9F9762FBB6C53D
Sygate-SSN: 1
<?xml version="1.0" encoding="UTF-8" ?><SSARegData NameSpace="rpc">

... specific computer details cut from here ....

<SSAProduct Version="11.0.3001.2224" />
<SSAOS Version="5.1.2600" Desc="Windows 4C0A940PProfessional" Type="17105154" ServicePack="ServicePack"/>
<Processor ProcessorType="x86 0x1.0aa600p-947mily%20Model%20Stepping" ProcessorClock="1997" ProcessorNum="2"/>
<Memory Size="1071693824"/>
<BIOS Version="DELL%20-

Interesting.

Does this computer name or desciption contain any special character?

No, just letters, numbers, and a couple of dashes in the name (same as all our other computers), and the description is just the owner's name.

Think you should call in support simultaneously as well.

This requires some active troubleshooting and the theory right now is that you can't get cleanwipe unless the tech analyzes and determines that you really need it. I would suggest cleanwiping the system and install afresh once again.

I've heard about the cleanwipe program, so that sounds like a good thing to try. Thanks for the suggestion.

If the gold shield in the system tray does not have a green dot (meaning it is connected to the Policy Manager), obviously, this means that the agent is not connected to the policy manager. Some of the things you might want to check are as follows:

1) Open the SyLink.xml file and verify that the ip address and port number are correct for the Policy Manager.

2) If the ip address and port number are correct, can you ping the ip address of the Policy Manager? If not, then you have a network issue and the client will never be able to connect to the Policy Manager to be updated.

3) Are there other computers on the same subnet that belong to the same group able to connect to the Policy Manager successfully? If so, this would more or less validate that you don't have a network issue because other clients can access the Policy Manager from that particular subnet.

Those are good suggestions, networkccna, but I've checked all of those angles. The sylink.xml file is pointing to the right policy manager, and I can indeed ping that policy manager's IP. When it didn't seem to be talking to the server, that seemed like one of the first things to test.

As for 3), the results are completely erratic. We've got ~250 computers with SEP installed, and all but 5 are fine. The 5 that aren't fine are scattered among 4 separate offices, but all other computers in those offices are upgraded and working properly. There doesn't seem to be any real connection based on physical location or portion of the network.

The only possible correlation we've been able to come up with is we think 4 of the 5 stubborn computers may have each had a virus sometime in the last half year, but our memory's a little fuzzy. Pretty sure it wasn't the same virus on each, though. But possibly the cleanup tools we used did something to the computers that is causing a problem. On the other hand, we've had another 10 or so computers that had viruses and aren't experiencing these issues, so that's not looking like particularly consistent evidence.

At this point I'm waiting until Friday when I can get my hands on one of the computers for a few hours and call in to tech support. Hopefully they'll let me try the cleanwipe tool.

Just wanted to follow up on this, in case someone else with a similar problem comes along. Spent about 4 hours on the phone with Symantec support, and got one of the computers fixed after extensive testing. Unfortunately there's no short and sweet answer; it only started working after extensive registry cleanup (plus possibly some earlier tests that didn't appear to help until the registry was updated).

So the real answer is call Symantec support and let them figure it out for you.

With some further testing, I think I've narrowed down the fix to a single registry edit. I can't guarantee that this will work for anyone else, and I'm always hesistant to recommend registry edits to other people, but this single item has fixed 7 out of 7 computers it's been tried on. The following is the full instructions as I shared with the pc techs at my company:
--------------------------

Sometimes after install, SEP simply won't establish communications with the server. This is often indicated by having the yellow shield without any dots at all (no green, no red) or sometimes/eventually transitioning to a red dot. You can also verify by opening up the client, click Help and Support > Troubleshooting. The first line on the window will indicate the Server. If the connection is good, it will show a server name or IP, but if it says "offline" then this should fix it.

Open regedit. Go to:

HKEY_USERS\.DEFAULT\Software\Microsoft\Windows\CurrentVersion\Internet Settings

If there's an item called GlobalUserOffline, delete it.

Reboot.

You can verify if the fix worked by opening the client and checking the server status -- where it used to say offline it should now show the server name. Note that it may take 10 minutes or so for the client to actually download updates. Until it does, it may still be showing the red dot or missing the green one. Also, sometimes running the Intelligent Updater from the Symantec web site gets things activated more quickly.

 Just writing this to say thanks rpatty. It is very helpful for us in support, as well as the rest of the community when users follow up on successful fixes. It makes this community a better resource for those who are experiencing similar issues.

Thanks Again
Grant

Quite welcome. Once I got the fix narrowed down, I sent it to the phone tech who had helped me with this, too. After he'd spent 3+ hours on the phone with me testing things, I figured he'd want to know which step really did the trick. And of course I'm hopeful there will be a way for the fix to eventually make its way into future versions so that it's not even an issue for others.

So what exactly is this registry key used for and why does deleting it fix the problem?

Seems like this key came back on a few workstations I deleted them on previously. I will have to delete the key again to re-establish the commnunication back to SEP manager. I guess I have to put in a group policy. Anyone else having this issue?

This sounds like the "offline" setting in Internet Explorer. On the macines where it keeps comming back, check the File --> Work Offline setting.

rpatty: Was your "no dot/no communication" issue accompanies by high CPU usage? I'm trying to determine if these two are related.
Thanks! 

Nope, I didn't notice any unusual CPU usage with my symptoms. The client just thought the server was offline.

Only time I've run into high CPU was when the full version of SEP was accidentally installed on a Win2K3 server -- I forget which one (network threat protection?), but one of the components isn't supported on servers, and was causing the processor to max out at 100%. But that's a known issue, and didn't really apply to the missing dots in my case.

Whenever the green dot disappears (Usually after a rebuild of the SEPM server for upgrade) It also mean that the group policy number as mentioned previously is different for the client. It can also become corrupted. 
Using the Sylink drop has had mixed results for me, but a sure fire method that always seems to work is this:

1. I delete the clients and folder having the problem from the SEPM.
2. I recreate the group (which gives a new policy number.)
3. I export a package and give it the policy from that new group.
4. I push the package to all clients recognized in AD for that subnet (or I simply seach for all clients on that subnet using the "Find unmanaged computers" tool.

Perhaps a bit overkill to reinstall the whole package, but it does ensure that the clients will be communicating properly with the Policy manager.

Hello,

I would like to extend a thank you for this solution.
I have had several clients that have not had comms for a few months now and the reg key fix worked.
Sym support had me send them images of the affected systems, and didn't provide a solution.
Only occasionally I had success with cleanup/registry cleanups, but some wouldn't connect no matter what.
I had tried nearly everything I could think of but reimage their systems.


Thanks again,
nubanx

Hi,

I have a problem whereby only a few clients have a green dot.  I have about 15pcs (on a workgroup) and they all show the green dot but at different times. Is there a maximum number of connections permitted at a time?

Basically, everytime I refresh, a different client shows the green dot status but they are updating...

@ razin8stor, Is your management server running an XP OS? XP only allows 10 concurrent connections.

Hi Cycletech,

Yes it is on an XP machine.

After running that Syslink update tool none of the machines in the manager have the green dot, but when I look on the machines themselves they are fully updated :/

Hi, if anyone runs into this issue, can you send me a copy of the registry that had to be removed?
This issue has been dificult to track down, and a copy of the problem registry key would be helpful.
Thanks.

Hi there,
I know you're probably going to say, "No that's not my problem... no that's not it either", but since you're running XP it may be worth a shot.

1) Put your clients in PULL mode.
You set this under Clinets button --> Policies Tab --> Settings --> Communication Settings link
If you're clients are in PULL mode, XP can support more than 10 clients. If you still have trouble with the 10 connection limit, raise the heartbeat interval (set in the same menu) to an half an hour or more.

2) Verify your XP server has a static IP address. If you have a dynamic IP the clients may not be able to connect whenever the IP address changes.

3) From the server first, and then from the clients try to go to this web page:
http://Server_address:ServerPort/secars/secars.dll?hello,secars
Normal the address looks something like:
http://192.168.1.1:8014/secars/secars.dll?hello,secars

If you get a page that give a big OK then the test passed. Otherwise there is a problem.

4) Run the "Support Tool". The Support Tool is design to find connection issues as well as other known issues that might be present.
You can download the support tool from this support page:
http://service1.symantec.com/SUPPORT/ent-security.nsf/docid/2008071709480648
Run the tool on both the server an the client computers.

5) Enable the Sylink log files on the clients and get the connection error(s).
This page shows how to enable the Sylink log files.
http://service1.symantec.com/support/ent-security.nsf/docid/2008041812561948
After enabling the logs, I recommend using a program such as BareTail ( http://www.baremetalsoft.com/baretail/ ) to follow the log files. Look for any link that contains "http" and below, not to far away, there should be the result of the http request. It may say something like SMS return=500 (for an HTTP 500 error) or Send Request failed.. Error Code = 12007 (for a timeout error).

And lastly, which perhaps I should have mentioned first, check the Windows Firewall. I'm sure you did, but once in a while the simple things are over looked.

The full details are already here, in my post marked as the solution:

Open regedit. Go to:

HKEY_USERS\.DEFAULT\Software\Microsoft\Windows\CurrentVersion\Internet Settings

If there's an item called GlobalUserOffline, delete it.

Reboot.

I understand the solution -- I'm actually trying to recreate the issue. What I'd like to try to do is take known-to-be-a-problem registry values and put them in my client to see if I can recreate the issue. So what I would like is a registry export file of the the whole "Internet Settings" hive from computer that had this specific problem.

I have seen this on laptops. it seems to happen when they go off the network and select Work offline within Internet Explorer. This key gives me trouble as well. I'm tempted to put in a GPO to stop it because I get many clients that don't connect bck to SEPM when coming back on on the network and it is annoying.

Hi Ghent, thanks for your assistance.

I could not access the website you provided so I ran the support tool. I have attached a screenshot of what it has found.

I setup the SEPM on port 8443

So this indicates that:
1) You have 2 SEPM entries in your connection list (the Sylink.xml file)
2) One server, the one with a DNS address, seems to be working fine (assuming the DNS can be resolved by your client, in this case it can)
2) One server, the one refered to by IP is not responding.

So half the clients may work because the connect to the 'working' SEPM. Depending on how your connection list is structured, the clients may have the downed SEPM server as Priority 1 -- they from time to time they try to switch to it. In an error condition, clients will wait up to 34 minutes before trying to reconnect.

So start troubleshooting the 'down' SEPM server. I believe the error code zero implies that the server did not reply. So I would guess one of the following issues:
1) There is a firewall in the network between the remote server and the clients
2) There is a firewall on the SEPM server itself
3) The IIS services is not running.
4) The Port Number of the IIS server may of been changed somehow (I doubt it, but double check)

My first step would be to try to log into the broken server and try to access go to http://localhost:8014/secars/secars.dll?hello.secars . Then try http://<External_IP>:8014/secars/secars.dll?hello,secars

If the LocalHost works, but the external address does not, then there may be an IIS permissions issue. Check the IIS logs.
If the external address works from the Server, but not from any external host, then it's likely to be a firewall issue.

This should return SOMETHING regardless if the SEPM service is running because Secars runs inside the IIS process.
So first step, let's get IIS working.
Another page you can try to hit is http://<Server_Addr>:8014/Reporting . Client's don't use this address, but it may be useful to verify is basic IIS functionality is working.

Another useful link to test a little bit end-to-end (from Secars to SEPM-Tomcat) is http://localhost:8014/secars/secars.dll?action=36 . Note, this only works from the localhost.

As soon as you get IIS working, and can connect to if using a browser from an external host, run the test again and see if you client's don't connect after half and hour.

Hi again,

Thanks for your response yet again..

1) We have a RADIUS server on the network and a proxy
2) The firewall on the SEPM is on but using the default policy, should I turn it off?
3) The service is running on the machine the SEPM is installed on
4) I dont beleive it has changed but how can I confirm?

I think I may have browser issues, my IE keeps running with addons turned off and when I load that link into firefox, the page displays as blank. I will try get that working in the meantime

If you are having issues with your browser, you may want to figure those out -- but for the moment, there is a workaround.
In any case, you should be able to just run the Support Tool on the downed server -- it might even tell you exactly what's wrong.

Next, let me address the Firewall, Point 2.
Note that in SEP 11.x and 12.x products, unlike the legacy SAV products, the server itself contains no protection technologies. In SEP the Manager Manages. The client protects. If you want to protect your server you need to (and should) install a "client" on your server.
So, setting the firewall policy settings on your manager won't make a difference unless a client is installed on the manager. If you have a client on the manager, I would recommend disabling the firewall, or giving it an "Allow All" rule until you know what the problem is.
Here is a good clue if you think a client on the server is blocking other clients from connecting. If the locally installed client connect, but remote clients to not connect, it is likely to be a firewall issue (It could be a SEP, Microsoft, or other firewall).

Point 3, the service is installed and on. This is a good thing. However, note that the SEP 11.x manager is made of 2 services. There is the Symantec Endpoint Protection Management service, and there is IIS. Check the managers main log file located at: %SEPM%\tomcat\logs\scm-sever-0.log
If there are no severe issues then I would turn my attention towards IIS. IIS is used as the communication link between SEPM and all the clients -- so even if the SEPM service is running, if there is some issue with IIS, clients can't communicate.
Once again, I suggest running the Support Tool on the problem server. It will check for common IIS issues.

Point 1,
If the RADIUS server is running on the same machine as SEPM, it's possible there is a port conflict. The SEPM product has a feature that uses the RADIUS port, but you're probably not using it. Anyhow, I'm not going to go into that topic unless the RAIDUS server is on the same machine as the SEPM server.
And there is a proxy. Verify whether the proxy is required for clients to connect to SEPM. If clients are requried to connect to the proxy, then you may need to see if they have an issue with the proxy. Generally networks only require a Proxy server if your computer is going on the internet -- if you think the clients have an issue with the proxy, you'll need to fill in some more details on how it's setup.

Bottom line, run the Support Tool on the downed server.
Verify there are no severe logs errors in the SEPM's log file.
Log into the SEPM console and check for errors under the Monitors --> Logs page.
I'm not going to worry about point 4 until we think IIS working properly.