CUIT Systems Patching Overview

Overview

Patching at CUIT is scheduled and based around Microsoft Patch Tuesdays, which is the second Tuesday* of every month.  This methodology and naming scheme is used for all systems, including Linux servers.  Though there is no outside standard which requires that Linux patching conforms to Patch Tuesday scheduling, we use it for both Windows and Linux platforms to manage any applications that have cross-platform requirements and dependencies.

*NOTE: Contrary to any documentation you may find on the internet, we do not use or designate any fourth Tuesday as a "Patch Tuesday", though patching does sometimes occurs on fourth Tuesdays.  Patch Tuesdays are used as an organizing and naming scheme for scheduling and managing systems patches.  Click here for more information on Patch Tuesdays.

The week of Patch Tuesday is always considered to be Week 0, with each week beginning on a Sunday.  Sunday, Monday, and Tuesday are considered embargoed blackout dates for patching.  The Sunday and Monday of Week 0 occurs before any patches are released on Patch Tuesday.  Patch Tuesday is when patches are actually released, so are also considered an embargo date which we use to release new code and/or content views to manage patching.  For Linux, Patch Tuesdays are when content views (a list of patches available to systems) are frozen so that any new patches released since Patch Tuesday are not included until the next Patch Tuesday.  Freezing content Linux views in this way ensures that if a patch is released after test and development systems are patched, it will not be deployed to production systems first.  That patch will be first deployed on test and development during the next Patch Tuesday cycle.

Schedules are referred to by the week number, then the day, then the time (in 24 hour format).  E.g., 2_Sun_1000 refers to the 10 am on Sunday of the second week.  A preceding '0' is added to any time that occurs before 10am, e.g., 2_Wed_0800.  This scheme means that the first day that any system can patch is 0_Wed, and the first Sunday that any system can patch is 1_Sun.  It also means that not every month will have a week 4.  Onetime patching jobs can be scheduled for a fourth week, if necessary and when they occur, but we avoid scheduling regular patching during week 4 for this reason.

In addition, regular patching schedules must be restricted to one of the patch windows below

NOTE: Schedules in red are reserved for critical infrastructure.

* 3_Thu_1800 is not technically restricted, but its use is discouraged because of its potential to run into the 3_Thu_1900 schedule.
   No new hosts should be added and existing hosts should be moved a different schedule or a new schedule on a different day.
   When 3_Thu_1800 contains no more hosts, this schedule will be deleted.

⁺ 2_Mon_1900 is the default patching schedule set when no other schedule has been set, but should be considered temporary.
   Because it is default and the schedule of last resort, Development and Production servers are likely to patched at the same times.
   Servers automatically added to this schedule should be moved to a schedule more considered and appropriate to their environment.
   Ideally, this schedule has no servers using this window, but it still remains as the default case when no other window has been chosen.

Patch Sequencing and Linux

Within Linux, we are able to arbitrarily sequence patching of servers within the same patch window.  This is recommended and strongly encouraged.  Sequencing allows the minimization and sometimes even elimination of application downtime, and allows for automated validation before proceeding to the next server.  When applications require a particular order (e.g., patching and rebooting a database before an application server), this allows the work to proceed in order without specifying an arbitrary time to wait which could be exceeded depended on the scope of patching.  We have scripts available that only allow patching upon setting a specific trigger to allow for a finer grain of control should an application want to release code during their scheduled patch windows.  Most ordering is simply sequential (either patching immediately after patching and reboot succeeds on a prerequisite host or the same requirement, plus 10 minutes to allow all services to come up.

However, any application requirement can also be coded as a patch prerequisite.  When prerequisites are not met, after a 90 minute timeout, patching stops.  This ensures that if patching introduces a problem, further patching on an application does not proceed.  For full functionality, we recommend coding up application requirements as prerequisites rather than simply using time up since successful patching and reboot as a proxy for health.  E.g., a web server could be coded to check a database for functionality before patching and rebooting.  These can also be used to program maintenance tasks and requirements, e.g., one application uses these requirements to shut down the application process (but not the server), patches and reboots the database, then patches and reboots the application server.  Oracle database servers apply routine quarterly patch sets using prerequisite scripts.

Prerequisite scripts and scheduling an application within the same window is recommended over staggering hosts within the same tier of an application over several days.  In practice, we have found that spreading out patching only leads to a delayed failure if application groups are not proactive about checking the first servers in a sequence.  Automated checking and sequencing requires more upfront coding but provides a more robust way of maintaining our infrastructure.

For more details on how we implement this, including an actual use case, see: SPOG Linux Patch Prerequisites.