It’s not the best moment of the week when your change fails after hours of preparation and hard work during an implementation window. Plus it breaks other things too. Sometimes it’s obvious right after implementation, but other times you don’t find out until hours or days later. If you’re lucky, you can make an emergency change to fix a fairly obvious root cause, e.g. B. Missing an item on the implementation checklist. However, in many cases this is not possible and you will have to undo the change.
A key to a successful backout is a plan. Yes, a backout plan is often overlooked. After all, you want your retreat to be an honorable surrender, not a panic escape. To limit the damage to the company and your reputation, you need to stay in control of the situation. To do this, the engineering team needs to know what to do and the service desk needs to keep the business informed.
A backout plan is designed to keep you in control. It’s your insurance policy against Murphy’s law. Let’s be honest with ourselves: we don’t insure everything. Don’t prepare a formal backout plan for every change. Just make sure the team can verbally describe how to go back if things get messy.
However, for more complex changes, you need a more formal plan. Creating such a plan is one of the most inconvenient activities of many technicians. For this reason, the change manager should be responsible for the implementation. It should be included with the rest of the change documentation for use when needed.
A good backout plan should include:
- low-level, technical instructions,
- specific communication instructions with contact names.
The list of technical instructions is created by reversing the order of activities from your implementation plan and describing how to back out of each of the steps performed. It can be relatively easy when most of the work could be done by restoring the latest backup. Consider an example backout plan for such a scenario:
- Notify the service desk of initiating the backout plan. (Call them, email them, or raise a ticket—be explicit about it.)
- Disable user access to the system. (How? List the actions.)
- Restore the backup that was created before the change was implemented. (List the required actions.)
- Conduct system integrity checks. (List them all.)
- Enable user access.
- Notify the service desk of the successful reset.
The plan is often more complex than it appears. There may be many more recovery steps that affect different databases, file systems, and other areas of the IT infrastructure. The base template still applies. It needs to be detailed and tailored to each organization and each change. Of course, every action should have an owner, so make sure it’s clear who’s doing what.
Communication with the service desk is very important. Communication in general needs to be part of the plan to stay in control of the situation. In addition, the business needs to know that IT is in control. The service desk should take care of projecting the image of control onto the company. They can do this by issuing periodic communications when the business impact is severe enough. They also take calls from dissatisfied users and update them on the solution status.
A backout plan is your insurance policy. It’s up to you to have it or not. It is recommended to have it for any complex change as business continuity and IT credibility are at stake. Start preparing such a plan for the most complex change coming up in your pipeline. Then you build on it and over time you are prepared for any risky changes.