Endpoint Protection

 View Only
  • 1.  Complete Behavior of Install Packages and Upgrade Schedules

    Posted Feb 10, 2011 12:55 PM

    So there is a lot of incomplete and contradicting information out there about how the Install packages and upgrade schedules work.  I would like a definitive answer about exactly how SEP handles these things.  Please be patient, I am in a ranting mood this morning.

    We have a decent sized environment with around 15,000 client connecting in.  When we do an upgrade, it nice to distribute the upgrade over several days to keep the servers from getting overwhelmed.  But its unclear exactly what happens when we use these settings, specifically regarding the "Distribute Upgrades Over"setting.  After two weeks of trying to upgrade our clients to RU6 we barely have 50% who have received it.

    This information if provided in the help.

    Distribute upgrades over

    Specifies the number of days or hours over which the packages can be distributed. The following formula calculates time:

    (Package Size\Server Transfer Rate) * Number of Computers

    as

    MB / MB per second * Num = seconds

    Can someone please explain how this formula is used by Symantec (or if)?  Sure, this tells us the total time it should take to distribute the clients.  But is Symantec using this equation somehow to determine how it schedules the updates?  Where are the number of days I specified taken into consideration?  If I put in a 5 hour block over 2 days and the server thinks it will take 15 hours, does it ignore what I put in?  How does it calculate "Server Transfer Rate", etc.?  

    Or is all of this speculation irrelevant and Symantec is just kindly giving us the formula for us to manually calculate how much time we should allow for in the schedule?

    And then there is the question as to what happens to older clients that connect in after the period is over. Does Symantec let it get the update or, since it is out of the allowed window, does that client never get updated (this is our observation)?  Also, can I just specify 0 days (similar to setting the times equal in the other setting) and just let the updates run indefinitely during the specified time window?

    Which bring up another question.  If the time window if set to a couple hours in the middle of the night, and a client is never turned on in the middle of the night, will it never get an update?  Or does it behave like the definitions and get it as soon as it connects in?

    Basically the behavior of these upgrades are sorely under documented which has left us with a lot of questions.  I shouldn't have to spend my time experimenting with these features to figure out how they work when Symantec could just have someone look at the algorithm that they wrote and just tell us what it does and what we should expect so that we can plan appropriately for our environments.  If this document exists then I apologize (but may start a new rant about not making it easy to find).

    Rant Over....



  • 2.  RE: Complete Behavior of Install Packages and Upgrade Schedules

    Posted Feb 10, 2011 03:10 PM

    Also, if I tell SEP to perform the updates over 7 days, does it try to load balance who gets the updates over that time period?  Or can all the clients be updated that first night if there are plenty of resources.  And if it is load balancing, how is it doing it?  Does it calculate how many it needs to do per day and then stop after it hits that limit. Or, even worse, is each client assigned a day and if they miss it, then oh well?

    Obviously some of these scenarios would be ridiculous if Symantec actually implemented it that way.  But based on what we have seen so far (and since Symantec has not told us otherwise), none of these scenarios can be ruled out.



  • 3.  RE: Complete Behavior of Install Packages and Upgrade Schedules

    Posted Feb 22, 2011 11:07 AM

    So it would appear that everyone that frequents these forums is as much in the dark as I am.  Hopefully at least one of the Symantec employees I see out and about on these forums has at least passed on this information to the right people and we will see much improved documentation in the next version release.



  • 4.  RE: Complete Behavior of Install Packages and Upgrade Schedules
    Best Answer

    Posted Feb 22, 2011 01:38 PM

    Sooo... there's a good reason there's a lot of different information on this... mostly related to the fact that there's some misunderstanding and due to a defect that's never been addressed. 

    Out of the gates, I should recommend not using the Schedule Updates and Distribute Updates features at the same time... they simply don't play well together. I'm fully aware of the obsurdity of that, but it's just the way it is right now... and probbaly the way it always will be.

    Can someone please explain how this formula is used by Symantec (or if)?  Sure, this tells us the total time it should take to distribute the clients.  But is Symantec using this equation somehow to determine how it schedules the updates?  Where are the number of days I specified taken into consideration?  If I put in a 5 hour block over 2 days and the server thinks it will take 15 hours, does it ignore what I put in?  How does it calculate "Server Transfer Rate", etc.?  

    We do use a formula (I think it's listed if you click the Help button) to calculate a schedule for all the clients, but I don't know if it's the one that's listed. As far as I'm aware, the formula you've given is not used by the SEPM... it's supposed to help you figure out how long to set the "Distribute Updates over X" setting. We take whatever time you put there and divide the clients evenly along that time line. Example: If you have 2 clients and set it to distribute over 30 days, the first one will go immediately and the second will go after 15 days.

    Using Distribute Clients over X days will assign a time to each client and only allows the client to get the update at that time. If the client misses it, it doesn't get the update until the cycle starts over. Example: That client that had to wait 15 days is offline on Day 15. On Day 16, the SEPM denies the update. On Day 17, the SEPM denies again... this will continue until Day 31 when the SEPM will assign new times to the clients. Even though this is an extreme example, you can see how this could make clients take a long time to get their updates. Even on a more reasonable 2-3 day distribution, a client could miss a number of consecutive update cycles.

    An important note on the 'Distribute over X' feature: It sets an update timeframe for the clients. Once a client gets this time, it cannot be changed or reset. Example: Using the same 2 clients over 30 days... on Day 2 you realize 30 days was excessive and set it to distribute over 1 day. The second client has already been assigned to Day 15 and won't check for the new 1 day distribution until after Day 15. Note that these times are assigned by the SEPM and stored client-side. Once the client has a time, it will not check for updates until the timer is expired. 

    The Update Schedule (setting a window of time for updates to occur) has always been buggy. Even on the lastest builds, I've heard of people having trouble with clients that never upgrade as long as a schedule is set. The bug, as it was explained to me, is that the client will check-in and find the update schedule properly, but the bug makes it read the schedule as if it were for tomorrow. I don't know all the details, but I can say that removing the Schedule has worked for a lot of customers. It's worth noting that this bug doesn't happen all the time... sometimes clients update just fine... sometimes all but a few update... sometimes all but a few don't. YMMV, as they say.

    Using the two features together yields even more interesting results. Assuming the schedule works, the 'Distribute over X' feature isn't aware of the schedule, so it sets updates to occur off-schedule. So your 5pm-5am window may never hit some clients, no matter how many times they go through the update process.

    All-in-all, my recommendation is to not use either feature... or, at the most, use a smaller 'Distribute over X' time frame (2 days). If you're worried the SEPM will slam the network, look into throttling IIS. The client packages are served from the "ClientPackages" virtual directory and you can throttle it separate from the rest of the server to keep communication intact.

    The other workaround, which is a lot more manual, is to create a sub-group with your Install Package setup to update clients that move into the group. Move batches of clients that haven't updated into the sub-group, let them update, move them back. For 15,000 clients, that's probably not a viable option (and not at all possible if you sync with AD), but for smaller environments, it can be an acceptable solution.



  • 5.  RE: Complete Behavior of Install Packages and Upgrade Schedules

    Posted Mar 09, 2011 10:43 AM

    Thank you so much for sharing this.  Hopefully your notes can somehow be included in the documentation in the future.  Knowing this beforehand would have saved us a lot of time/stress.

    It is unfortunate to hear the issues of using these things in conjunction with each other.  We have scenarios where both of these are important and one without the other could cause us potential issues.

    Also, do you know if there are issues with setting a schedule that spans to multiple days (i.e., 6:00pm to 6:00am).  that would at least increase our window and allow a higher percentage of machine to get updated until the issue is resolved.