Pipeliner: How Upside Manages Spinnaker Pipeline-Templates

Everything is at its best when it flows properly — photo by Mike Lewis HeadSmart Media

Here at Upside, our Site Reliability Engineering (SRE) team has gone all-in on Spinnaker for our software deployment strategy. As SREs, we’re responsible for our AWS cloud infrastructure, DevOps pipelines, and really just automating all the things. On the DevOps front, Spinnaker has helped us tremendously.

Taking a step back, Spinnaker is an open-source deployment platform developed by Netflix. It allows for building out complex delivery pipelines that would have otherwise been either too tedious or just too impossible to do via a common deployment tool such as Jenkins. I don’t want to get too into why we ultimately chose Spinnaker, because that can be found in an earlier post on exactly this topic. Instead, I’d like to talk about how our internal SRE team built out a custom tool called Pipeliner that allows us to manage our Spinnaker pipelines and their relevant pipeline-templates.

The Problems

Spinnaker is a wonderful tool because it provides pre-built “stages” such as “bake helm chart”, “deploy to Kubernetes”, “rollback a deployment”, and so on. To define a deployment pipeline, you have to create a big JSON file that specifies some config such as what Docker Registry to look at, and then provide an array of stages with their varying configurations. While it’s extremely convenient to use these pre-built stages, these JSON files can grow very large and hard to manage. These files are called “pipeline-templates” and, at Upside, we define a few pipeline-templates that all of our 150+ microservices use for their deployment pipelines. In short, deployment pipelines use pipeline-templates as their source of truth. Because one pipeline-template touches a lot of our microservices, we need to be sure that when we roll out new changes to these few pipeline-templates, that they work exactly as we intend. Needless to say, the process behind building out new functionality can slow due to this sensitivity.

Our old process for iterating new pipeline-template functionality looked like:

1. Create a new Spinnaker application to test new pipeline-template functionality.
2. Create a few new (but nearly identical) pipeline-templates to use just for this sandbox Spinnaker application. This is to be sure that we don’t “step on the toes” of our in-use pipeline-templates as we test functionality.
3. Manually edit the big JSON pipeline-template(s) and test in the sandbox application until desired functionality is met.
4. Copy and paste the new snippets of JSON from the sandbox pipeline-template file(s) to our in-use pipeline-templates locally.
5. Use the spin CLI to save the pipeline-template, which propagates the change to all of our in-use pipelines.
6. Hope nothing breaks!

To sum up the above, the problems we faced were:

• Manually, (and tediously!) editing and keeping track of about 10 JSON pipeline-templates.
• These 10 or so JSON pipeline-templates were very similar to each other, only differing by a few stages. We needed reusability.
• No clear way of comparing a pipeline-template that we’re about to apply to its in-use equivalent. In short, confidence was not high.
• Our team prefers to use YAML when possible, as we think it’s more readable.

The Solution — Pipeliner

We internally built a tool called Pipeliner that we use for all things related to pipeline-template management. It’s our “one stop shop” for:

• Creating sandbox Spinnaker applications for testing
• Compiling our logical YAML pipeline-templates into their JSON equivalents that Spinnaker can read (more on this later)
• Diff’ing our local pipeline-templates with their in-use versions so that we can see exactly the changes that will be made when the pipeline-templates are applied.
• Applying local pipeline-templates to Spinnaker.

Here is the man page for Pipeliner. The different functionalities will be explained shortly.

The man page for Pipeliner that explains usage

Did you just say YAML pipeline-templates?

Yes! Our SRE team finds YAML to be a bit more readable, but unfortunately Spinnaker pipeline-templates have to be written in JSON. We wanted to create YAML pipeline-template equivalents of our JSON pipeline-templates, and have Pipeliner “compile” the YAML version down into the JSON version, one that Spinnaker knows how to read. Our new pipeline-template concept consists of transforming our old JSON pipeline-templates into “logical” YAML pipeline-templates. The benefit of this, aside from increased readability, is that we can store our stages as separate and individual files and reference them by file in our YAML pipeline-templates. Because a lot of our different pipeline-templates use the same stages, we parameterized each stage enough so that they can be reused, which is as simple as just referencing the stage’s file in our YAML pipeline-templates!

Let’s see an example…

A snippet of our “deploy-dev” YAML pipeline-template

This is a snippet of one of our YAML pipeline-template. You can see it’s broken up into 2 sections:

1. Config — if you’re familiar with JSON pipeline-templates, this is the section where Docker Registries, pipeline triggers, template variables, Helm info, etc are defined. These usually differ throughout our pipeline-templates and aren’t as reusable, but we still nonetheless separate them out into their own files so that they can be referenced in the YAML pipeline-templates.
2. Stages — this is an array of stage objects and it is pretty minimalistic. Each object references its relevant stage file and can dynamically pass in variables to that stage, so that stages can be unique enough to be reused. Of these variables passed in, one includes being able to define what stages depend on others, so that we can ensure correct ordering, and it just makes it more readable being able to see the flow of the pipeline.

We really value the simplicity of this YAML pipeline-template strategy because you can more easily view exactly what is happening in the deployment pipeline. It gives a good high level view that doesn’t overwhelm the reader/engineer.

What exactly does a stage file look like?

Here is the stage file for our “set-initial-variables” stage. In the above screenshot you can see it’s the first stage that executes in our pipeline, and it sets variables to be used downstream. Notice that it looks almost identical to how it would look in a JSON pipeline-template. Where it differs is with the values that are encoded like so: $((value)) . These are values that are passed in by the YAML pipeline-template at compile time. They’re defined in a way that, for example, this “set-initial-variables” stage can be reused for our dev, stg, and prod pipelines. Our Pipeliner script uses an open-source tool called Kexpand that can dynamically seed values that are enclosed by $(( )).

Our “set-initial-variables” stage

Most of our pipeline-templates have ~20 stages, with ~18 of them being used across all pipeline-templates. Because we parameterize each stage, we’re able to reuse them simply by referencing the file in the YAML pipeline-template, and pass in the relevant variables. This has saved us a lot of time when it comes to developing new pipeline-templates and/or adding new functionality.

Pipeliner’s Apply Functionality

As stated above, we use Pipeline to “compile” these YAML pipeline-templates into the Spinnaker-readable JSON equivalents. When we want to actually apply our local changes to Spinnaker, we simply run ./pipeliner.py apply --file location-of-yaml-pt.yaml. What this does is takes that YAML pipeline-template, compiles it into the JSON version, and then uses the spin CLI to save it to our in-use pipelines.

Pipeliner’s Diff Functionality

This is how we increase our confidence when it comes to applying new pipeline-template functionality. Before running the apply, we run ./pipeliner.py diff --file location-of-yaml-pt.yaml. What this does is it compiles the local pipeline-template into JSON, uses the spin CLI to get its equivalent in-use pipeline-template, and then diff’s the two JSON files.

Here’s an example output of a diff command if I locally change our “deploy dev” YAML pipeline-template to inject env: foo into the “set-initial-variables” stage, instead of env: dev

Changing it in the YAML pipeline-template

The output of the diff function shows the env variable changing to foo

Having the ability to diff gives us a huge sense of assurance that we know exactly what will be applied before actually applying.

Pipeliner’s Sandbox Functionality

The last area that Pipeliner has really helped us with is the automation behind spinning up a new Spinnaker application with separate and new pipeline-templates. As stated earlier, we have to create new pipeline-templates for each sandbox application so that as we build out functionality, we aren’t applying the sandbox pipeline-template changes to the in-use pipeline-templates and potentially breaking them.

We replicate each sandbox application and its relevant pipeline-templates exactly how they are for our in-use spinnaker applications. By having them nearly identical, we further increase our confidence that if new pipeline-template functionality works in a sandbox application, it should work in the real applications.

To fully mimic an in-use pipeline-template, Pipeliner’s Sandbox functionality creates our in-use pipeline-templates and modifies only what it has to in order to make the pipeline-template unique enough from their in-use equivalents. For example changing the pipelineId, and creating Kubernetes ConfigMaps with unique names so that our pipeline stages that use a RunJobManifest are pointing to unique and isolated Kubernetes ConfigMaps that contain the job scripts. It then uses the spin CLI to save these pipeline-templates and creates the Spinnaker application along with its isolate pipelines.

Sandbox has 2 actions:

• apply — takes all local changes and saves them in the Sandbox app.
• destroy — tears down all deployed Kubernetes infrastructure related to this sandbox app, and then tears down the Spinnaker sandbox app along with its pipeline-templates.

An example command to create a new sandbox application called jeremy-sandbox would be: ./pipeliner.py sandbox --name jeremy-sandbox --action apply

Having this Sandbox functionality has saved us countless hours. Previously, it’d take anywhere from 30 minutes to an hour to properly mimic an application in a sandbox application. Now it takes Pipeliner about 20 seconds to fully spin up (or spin down) a sandbox application. This allows us to test changes quickly which leads to more confidence in when we decide to actually apply these local changes to our in-use pipeline-templates.

The typical flow of development of new pipeline-template functionality

1. Create a Spinnaker sandbox application: ./pipeliner.py sandbox --name jeremy-sandbox --action apply
2. Edit the YAML pipeline-template, add a new stage, change a ConfigMap script, etc. and then apply those local changes to the sandbox application: ./pipeliner.py sandbox --name jeremy-sandbox --action apply
3. Test the pipeline in the sandbox app in the Spinnaker UI to confirm everything works as expected.
4. When satisfied with the changes, use the diff command to see exactly what will be changing when applied: ./pipeliner.py diff --file path-to-yaml-pt.yaml
5. Apply the pipeline-template change so that it can be used by our in-use pipelines: ./pipeliner.py apply --file path-to-yaml-pt.yaml
6. Tear down the sandbox application: ./pipeliner.py sandbox --name jeremy-sandbox --action destroy

Summing up

The creation of our custom-built utility called Pipeliner has enabled our SRE team to iterate new Spinnaker pipeline-template functionality faster, more efficiently, and with more confidence. Shifting to using “logical” YAML pipeline-templates has helped not only with readability of the pipeline-template, but also enabled us to “reuse” stages that are stored in separate stage files. When creating a new pipeline-template (that most likely has many of the same stages as other pipeline-templates), we simply create a new YAML pipeline-template, have it point to the appropriate config and stage files, and pass in the relevant variables.

Also, the ability to quickly spin up or down a sandbox application to test local changes, combined with being able to diff these local changes with the in-use variant has led to a massive increase in confidence in rolling out new changes. Pipeliner has transformed the once tedious task of iterating new pipeline-template functionality into an actual enjoyable process, and allows us to focus more on adding business value with Spinnaker.

Interested in a fast-paced development and deployment culture? Check out our open positions here at Upside. Also feel free to shoot Jeremy an email about Spinnaker, his SRE experience, or really anything in general!

jeremy.deppen@upside.com