Dante's Inferno described his journey through the nine rings of hell, which included gluttony, violence, and treachery. IT managers might include another: application updates. Patching a complex application portfolio on thousands of endpoints is a challenge that could vaporize your weekend. As it turns out, though, another type of ring can save you from this curse. Welcome to the world of ring deployments.
How software development evolved
Application patching was once a relatively simple process. Software teams handled tasks sequentially in large chunks, creating infrequent, predictable updates. From the early 2000s, we saw four changes that transformed software development.
First, agile development techniques broke those larger tasks down into smaller, more manageable ones that developers could complete more quickly in a series of software 'sprints'. These enabled developers to deliver software more quickly.
Second, the architecture of the software changed. Programs moved away from millions of lines of monolithic code, where a single change could have unpredictable effects across the whole code base. Developers broke them into small software services that could interact in well-defined ways. Multiple teams could then update parts of the overall program simultaneously without worrying about breaking things. Suddenly, software development could happen concurrently, rather than sequentially.
Developers' tooling also became more sophisticated. CI/CD pipelines emerged that automated large parts of the software development and testing process. All developers committed their changes to a single pipeline that applied automated, gated tests before automatically sending updates to deployment.
Finally, these three advances empowered users. They could now offer more regular feedback to developers, who were more accountable. Agile development and advanced tooling created a continuous improvement loop in which user issues make it back to developers quickly for rapid handling and redeployment.
Victims of our own success
These advances led to smaller software updates - including security patches - that are released more often. Today's software is released in micro-cycles measured not in months, but in weeks, days, or even hours.
That's great news for security teams and end users alike, who get the most secure, up-to-date software. It's terrible news for the engineers and operations staff who have to deploy those updates across large fleets of endpoints. In the past, they'd go through rigorous testing with each update to ensure that they worked as expected in the field. More frequent updates make those tests more difficult. The rising number of applications per endpoint has compounded this problem. In short, engineers face a crisis in updating velocity and volume.
Ringing in the changes
Companies responded to this problem by introducing ring deployments. This concept collapses the testing and the deployment phases by rolling out code updates gradually to an ever-increasing number of users on live systems. It gets its name from the concentric rings of users that it uses as test subjects.
Some might worry about employing users as test subjects, but ring deployments include measures to minimize any damage from problematic software rollouts. One of the most important is to limit the 'blast radius' of any problems by easing into the deployment one group of users at a time.
Ring deployments enable engineers to constantly monitor the success of their software roll-out. With each ring, they deploy, gather feedback, and then decide what to do next.
Deployment
A common approach to these deployments involves three pre-production rings, followed by a production deployment. In the software developers' spirit of zero-based numbering, we tend to call the first ring P0 (P standing for phase), or the zero ring. That's especially appropriate given that this is ground zero for the software update because users in the zero ring get it before anyone else.
In some circles, zero-ring users are known as canaries. They don't simply get the software and get on with their jobs. They have a responsibility to work collaboratively with IT, sounding the alarm if they run into any problems with the updated software. These canaries should contain only very technical employees. Consider IT groups and engineers for this small subset and be sure they know how to provide feedback early and often when they see any potential problems.
While we're looking for unexpected hiccups in the zero ring, it's important to communicate the expected too. With this in mind, be sure to thoroughly read the release notes for the update and pass these on to canary users. This gives them context, while also enabling you to ask better questions.
The second ring is known as the P1 ring, which contains a wider base of early adopters that sit somewhere between canaries and average end-users. These users can be outside engineering and IT, but they should still be a technical audience that is friendly to the IT department.
Early adopters are often willing to try an update because they want the latest version of the software and are often less bothered by potential glitches. They only see the software after you're satisfied that the first set of users is happy with the update. There are more of these early adopters, reflecting your growing confidence in the update.
Third is the P2 ring. This targets a wider audience, but it is still part of the pre-production deployment. Engineers often consider this to be the user acceptance test ring, where representatives from business user groups get to use the software. By this stage, though, most if not all of the major problems should have been worked out.
After these three rings, it's time to deploy the software into production. While some companies do that all at once, others will continue with a ring-based deployment that gradually expands the blast radius throughout the rest of the organization.
Production rollouts have different kinds of groups based on geographies, departments, or the overall number of endpoints. Other variables affecting this phased deployment are technical. For example, companies might be limited by the network bandwidth available, and staging deployments to avoid traffic bottlenecks.
Gathering feedback
Collecting user feedback is something that should happen in each pre-production ring, but methods often vary throughout the process. For earlier rings, it is useful to hunt for issues by giving tech-savvy canaries and early adopters a user feedback questionnaire. In later stages, it's often better to take a passive approach by waiting for problem reports, especially with business users who assume that you've already tested the software using traditional methods.
When taking a passive approach, it is important to work closely with your help desk staff. Alert them of potential tickets around the software update and ensure that there's a channel for you to see those easily.
Deciding what to do next
After collecting feedback for a set period in each ring, you enter the decision phase. Here, you deal with any problems that have occurred, and then either proceed by deploying to the next ring, or in the worst scenario, abort the deployment.
Your approach to resolving deployment problems will depend on several variables. For example, the vendor might be aware of the problem and have issued a follow-up patch to fix it. In some cases, a reboot is enough to fix the issue. Or a simple configuration change to the application or another part of the system could resolve the problem, giving you the green light for the next ring.
If none of these options are appropriate, you might have to admit defeat for the time being and restore the system state until another solution emerges. Some patches have hotfixes that support rollbacks, making it easier to revert user endpoints. Other updates are entirely new product versions that require a reinstall, although engineers can often automate that task.
Managing multiple updates
Even with the ring method, companies will still face complex deployments involving multiple patches. Unless your solution is capable of deploying multiple patches at once to prevent repeated endpoint restarts and downtime like Adaptiva’s OneSite Patch can, avoid deploying multiple patches to the zero ring at once, as these can make root cause analysis more difficult. The caveat is that different parts of the organization might use different tools, enabling you to handle simultaneous rollouts to different sets of canaries and early adopters without creating conflicts.
In all cases, a deployment calendar can help you to prioritize and schedule updates based on their importance to the organization. Naturally, security updates will often bubble to the top of the list. In extreme cases, where you're racing to deploy a critical zero-day patch before attackers take advantage, you can expedite the process by compressing the time between deployment rings.
Ring deployment has become the smart response for more organizations that longer find controlled testing logistically plausible. Done well, it can minimize the up-front work involved in software updates and accelerate the deployment process. However, it means handling user feedback responsibly, which in turn means building robust relationships between engineering, operations, and support teams.
This is the latest entry in our Adaptiva Practitioner Blog Series. In these blog posts, we share what we know about managing endpoints. Stop by to hear from our own in-house subject matter experts. We are excited to discuss best practices, technical how-tos, and other topics we think you'll find valuable. Our solution architects, product experts, and own IT practitioners have seen and done it all. We are adding new content regularly and are happy to have you here.
