The story of how one financial services company:
- went from 4 releases a year, with a 40% failure rate and an average 1.5 day MTTR (Mean Time To Recovery) to bi-weekly releases with a 3% failure and a 2 hour MTTR
- has saved over $500k and
- has generated an estimated $4.2m in additional revenue in the process.
The company was following a fairly standard delivery process: developers spent 3-4 months implementing a set of features and fixes, they were tested and cleaned up for 1-2 months, and a release was prepared. A long meeting was held with developers, QA, database administrators and operators to create the release plan – an Excel spreadsheet sent out to all participants.
During the release itself, every participant worked off their copy of the release plan (which were all supposed to be the same version, but that hardly ever happened). After every step, the person carrying out the task communicated the result back to the release manager, who was occupied full time keeping track of the emails and chat messages from different team members. The only person with the up-to-date version of the spreadsheet was the release manager, and quite often team members had to check with each other, or with the release manager, how another task that they were waiting on was going.
Of course, sometimes tasks happened in the wrong order, or one team member ended up waiting for another task to complete, only to discover that it had actually been completed a while ago. And, of course, it’s not as though the technical team only had one release to worry about: with multiple releases taking place at the same time, it frequently happened that someone like the database admin was busy with a task for one release, when they were actually urgently needed by a different team.
In other words, they were facing common issues many organizations have and still are facing in their release process.
Making the Move to Continuous Delivery
When a new CIO arrived, accelerating time to market and cutting down on release failures was one of the first items on the new agenda. The goal was to move to at least monthly releases by adopting techniques from Continuous Delivery. But, of course, the existing business needed to continue.
Where to start? What to tackle first? The project team quickly discovered that, whilst team members remembered individual problems in some releases, there was no really reliable or consistent data to properly identify the most useful areas to address first.
What they learned about XL Release
Tackling this missing data problem became much easier with XL Release. They were able to turn their Excel sheet/Wiki page/Word document/MS Project plan into release templates in minutes. They discovered that modeling the different phases and tasks of a release is easy and intuitive. XL Release also allowed them to add conditional gates and common tasks such as notifications to their release, eliminating the need tediously correspond with email templates and arbitrarily CC’ed members of the team.
Time-sensitive releases were easily created, tracked and fine-tuned using the planner, with intuitive ease compared to Word, Excel or Project. By using planner they were better able to identify idle-time, bottlenecks and more.
Introducing Automation Step-by-Step
One thing the project team also immediately noticed was that many tasks in the release were simply instructions for a team member to run a script, create a change ticket, or manually carry out one of a number of common, easily-automated actions. By replacing the manual actions with automated tasks featured out-of-the-box in XL Release, the team was quickly able to avoid many of the usual issues they typically faced. Issues such as rushed typos, logging into the wrong server, being stuck on a different task, or simply forgetting to tell the release manager straight away. The team was even able to add their own automated tasks to XL Release.
Another seemingly small thing that delivered an immediate improvement to the company was the ability to assign tasks to teams, not just individuals. Previously, a task such as updating the database had been assigned to a specific DBA. If that DBA was off sick or busy with other tasks, the release managers, already busy tracking multiple releases, had to identify someone else at the last minute.
In XL Release, they simply assigned to the DBA team, allowing the task to be picked up or handed off from one DBA to another. Additionally, DBAs were able to leave notes for each other, attach files to tasks, or ‘flag’ tasks with alerts that were visible to the entire release team. This helped the company’s release teams focus their attention on the most important tasks, even when juggling multiple releases.
Tracking with Multiple Releases and Pipelines
With a couple releases in the air, having a higher-level overview is a necessity. Previously, at any given time only the release manager had a full and true perspective, but with XL Release any of their team members were able to check in any time at a high-level. Any member could see their own tasks, everyone elses tasks, and also a summary view of every release to see what’s been done, what’s in progress, and where any problems might have been.
A challenge the organization had also faced was that any given release manager was really only able to manage one or two releases at a time with all of the manual tracking, communicating, reporting, and firefighting. By running XL Release and using the Release Dashboard all of their release managers were able to scale up to 4-5 releases at a time, and focus their efforts on value-added activities instead of simple yet tedious manual tasks.
Although potentially confusing with 4-5 releases in the air, release managers were able to track with ease by using their pipeline view and calendar. Each view provides an intuitive, searchable and configurable view of releases, their progress and dependencies. With alerts and “flagged” tasks highlighted, the company found that during critical release periods, teams would use flags to highlight problems and release managers would focus on flagged releases where they could contribute the most value.
Handling Failures and Deviations from the Plan
Obviously, reality has a way of deviating from the best-laid plans. In the past when releases encountered fire-drills, the release teams at the organization would do what most do: panic a bit, and then work double-time to establish a fix. Once fixed, even the release managers rarely had the energy to re-work planning documents retroactively.
Contrarily, with XL Release they were able to more conveniently re-work & document firedrills. The release manager could modify their release plan on the fly by restarting an entire phase (e.g. the integration tests fail first time around and needed to be re-run with a fixed application version) or if that wasn’t actually necessary: simply skipping the steps that had been planned and adding a single task along the lines of “we went into panic mode”. By doing so, they were even able to use data at the end of releases to better plan and automate for the next!
Analysing and Improving the Release Process
After they had run a couple of releases, XL Release allowed them to find and quantify troubled spots (all of their “we went into panic mode” tasks) in a way they previously could not. By looking at planner data, the release value stream and other reports, they were able to identify idle time, long running tasks, unreliable tasks and phases where the plan had a tendency to “go off the rails” in a systematic way. In other words, XL Release was able to add value for them in process, and then compound that value after running through a few releases by providing more accurate & usable data.
Integrating XL Release with CI Tools
As they expanded use of XL Release out to more groups, the project teams discovered that some parts of the organization had already created delivery pipelines in various CI tools such as Jenkins. These typically stopped at the Test or QA environment, however, since that was typically the last environment developers had access to before entering into the realm of the “controlled release process”.
There was some initial resistance to adoption as the groups feared they would have to recreate what they already had. What they found is that it is certainly possible to build a fully-automated pipeline in XL Release, BUT it is also just as easy to integrate XL Release and Jenkins to support a hybrid, “two-tier” release strategy that starts with an automated pipeline in the CI tool run for each commit or once a day, and continues in XL Release with a release-per-sprint OR per month.
Providing a True End-to-end View for IT and the Business
In the end, by including XL Release the business was able to visualize the entire flow form Development through to Production. Whereas, previously combining information about the pipeline to QA versus the “real” release to Production was a manual aggregation task (if it ever happened at all). XL Release also provided a convenient “progress tracker” for business stakeholders, for whom the information in Jenkins was too technical and low-level to be relevant.
All in all, the company was able to generate $500k in cost savings due to reduced manual overhead during the release process, as well as cut down time to release substantially from end-to-end, AND as they progress through each new release and commit more tedious manual processes to automations their savings only increase!
If you’re looking to achieve what this organization was able to accomplish, go ahead and take a look at XL Release today!