Managing Cutover Issues - 4 tips to help you succeed
by Andy Smith, on 10/10/18 12:06
With a decade or more experience of running technology cutovers, one key learning point from every cutover is that the effective management of issues will have a very significant impact on the success or otherwise of your technology cutover.
In this blog, I’ll explore how this topic is typically approached, how it should be approached and what the impact of effective issue management can be.
To paraphrase something Bill Clinton once said, “It’s the issues, stupid!”
The Definition of an Issue
The first thing to do is define an Issue and make a clear distinction between that other well-known “I” in the technology and business world, the Incident. I’ll cover the relationship between issues and incidents in a future blog.
As issue is simply the occurrence of any unexpected problem that inhibits the planned progress of the cutover plan according to the details held within the runbook. It is a challenge that should be identified and managed by the cutover team. It is vital to recognise that an issue is owned by and needs to be resolved by the cutover team.
Issues are known as snags, problems, defects and by any number of other names. An issue could be a technical problem. It could be a logistics problem. It could be a human problem. It could be an external event that’s out of your control altogether.
Whatever it is, the single and critical feature of an issue is that it is likely to cost you time. Time in the cutover context is the most precious and finite of commodities. So how you manage issues really matters.
The Inevitability of Issues
With technology cutovers, detailed and diligent planning is undergone to make the best possible cutover plan. The aim is to have every activity proven in advance and for the entire exercise to run like clockwork. If only it were that simple.
Rehearsals are extensively used to create robust, comprehensive and proven cutover plans. Runbooks are refined over time to identify gaps, confirm timings and validate dependencies.
Despite all this good practice and disciplined effort, I have never been involved in a live cutover of any scale that did not experience several issues.
Cutovers are, in that sense, a bit like life itself – accept that you’ll get knocked down from time to time but know that what really matters is how you deal with these setbacks when they do arise. That’s not an invitation for complacency or short-cuts in the rehearsal and refinement space. Far from it.
It’s simply a mature response to accept the reality that issues will arise during cutovers; Prepare for them and be ready to effectively manage and overcome them.
Tip #1 - Capturing Issues
The first step in managing issues is to capture the fact that you have one to address. This should be done quickly, accurately and with a high degree of visibility i.e. everyone who needs to know should know as soon as possible. Issues in one area of the plan often impact other aspects of the plan - so sharing is vital.
How it's done today
In my experience, I’ve seen a wide range of solutions to this critical first step:
- Liberal use of highlighter pens on the A3 printout of the Excel spreadsheet runbook, with hand-written notes
- A separate tab on an Excel spreadsheet, updated with a new row for each issue
- A whiteboard in a control room somewhere, with a list of all the issues written on it with marker pens
- An instant messaging chat room with a conversation thread dedicated to recording issues
How it should be done
- Issues should be created within the overall cutover management solution
- The date and time that the issue emerged should be automatically recorded
- Issues should be accurately described by the person experiencing the issue
- The initial view of the potential impact of the issue should be added immediately - it can be refined after further assessment
- The associated activity should be clearly identified
- The issue should be visible to everyone who is participating in the cutover event
- The issue should have a clearly assigned owner and potentially a separate issue manager
Tip #2 - Assessing Issues
The key theme of this blog is that issues are important. That doesn’t mean that all issues are “born equal” because the impact of every issue will be unique. Some issues will have no effect on the duration of the associated activity. Others will have a significant impact on that duration. Some issues will affect the overall timeline of the entire runbook.
For every issue, an impact assessment needs to be made at the earliest opportunity. That impact should initially be assessed against the associated activity itself. The broader impact on the overall runbook must then be evaluated.
How it's done today
This is perhaps the area where I believe most cutovers start to go off the rails.
- A generally poor common understanding of the nature of the issue, based on a vague description
- A universal tendency to under-estimate of the impact on the duration of the associated activity
- No contemporary means of having some “expert eyes” review the issue as currently defined
- An intuition-based assessment of the impact on the overall runbook schedule
How it should be done
- Every issue should be documented in full by the individual experiencing the issue – and updated as the level of understanding evolves
- Every raised issue should be visibly recognisable against the associated activity
- Every issue should be impact assessed as soon as possible – initially by the person raising the issue then by the lead cutover manager
- Each issue should be assigned a severity – typically High, Medium or Low
- Each issue should have an estimated resolution time or duration
- The impact of that estimated resolution time should be immediately reflected in the forecast end time of the associated item (activity, runbook, overall event)
- Any issue that results is a significant change in the forecast end time of the cutover should trigger immediate communication
Tip #3 - Managing Issues
How it's done today
- Issue are raised and the challenge of resolving the issue is typically left to the person who raised the issue
- Issues are logged using one of the methods mentioned earlier and thereafter processed manually
- Updates on issue status require conference calls, IMs, emails or phone calls
- Issues can remain open for an inordinate amount of time as the “fix-time” slides further and further to the right
- The actual impact of any given issue on the overall timeline is lost – and what started as a minor issue with minimal impact can quickly become a major issue that’s hurting the overall timeline
- Updates can fall off the radar – especially when the most recent set of issues land on the table and attention is diverted elsewhere
- After the event, issue reporting is manually collated, with considerable manpower effort and is based on part-fact, part-recollection, part-anecdote.
How it should be done
- Managers should be easily assigned to any issue that represents a threat to the cutover success
- Status updates should be in real-time, integrated in the cutover solution
- Any issue that has the potential to impact the overall runbook should be assessed by a panel of experts, or “triaged”
- All necessary resources should be assigned to issues that have this potential to impact the overall runbook
- Late resolution of known issues that impact the critical path should automatically push the forecast end time out
- Late updates to issues should be flagged automatically so that appropriate status queries can be made
- Every issue, no matter how minor, should be logged and automatically reported during the cutover at Checkpoints
- Every issue should be encapsulated in the Post Implementation Review report for review after the event
Tip #4: Reducing the cost of issues
On a smaller-scale cutover, being inefficient in this regard will mean your cutover simply takes more time to complete. Your team will unfortunately work longer hours and the overall cost of your cutover will be marginally higher than it needs to be.
Let’s also assume that you have 15 participants involved over a planned 10-hour period. Let’s assume that the average cost per participant is US$50 per hour. If your small-scale change runs for 2 additional hours due to ineffective management of the issues encountered, that’s an additional cost of USD$1500.
On large-scale cutovers, the cost of time overruns really starts to ramp up. Let’s say you have 100 people involved at any given moment in time. That’s USD$5,000 to be saved or lost with every hour that you gain or lose. Over a weekend-long cutover, you could easily lose, say, 6 hours due to the inefficient management of issues. That’s USD$30,000 of unnecessary cost – not to mention the drain on your staff and partners. That experience could be repeated several times if you include rehearsals.
Even at this less painful end of the spectrum, these are outcomes you really want to avoid. There are better ways to spend your money and a much better way to take care of your staff.
Whatever way you look at this topic, it's clear that the effective management of issues will have a very significant impact on the success or otherwise of your technology cutover.
Demonstrating before the live cutover that you have the processes, tools and people in place to handle the inevitable issues is a fundamental boost to the confidence that your Exec team have in your ability to get the job done. When you have an effective and proven issue management solution, you can maintain a high degree of control, even in unpredictable circumstances.
One large-scale cutovers, managing issues effectively can quite literally mean the difference between success and total failure. This is the other end of the pain spectrum.
Inefficient management of issues could mean that your cutover timeline substantially suffers, to the extent that critical business services aren’t back up and running when they are needed. The dreaded service outage occurs.
In this context, failure means customer impact, brand and reputational damage. The overall cost impact to your business will dwarf the direct costs of the cutover team running on much longer than expected. Recent examples of major failures confirm that USD$100m isn’t an exaggeration of the cost implication and for some it’s much higher.
With ICEFLO, we've designed, built and refined our solution that handles all aspects of issue management in the cutover context. If you'd like to learn more about how ICEFLO can help, please get in touch and let's explore some options.