Secret Versioning in Hashicorp’s Vault
One of the major hurdles WeWork has had to overcome with Hashicorp’s Vault project is its lack of versioning of secrets you write to and from its API.
When you change a value, the previous iteration is lost to the great Vault gods in the kernel.
At WeWork, we use a combination of Vault and ConfD to load our environment secrets into Kubernetes containers through the use of the Init Container API. So the incorrect update of even a single variable could cause minutes of downtime as humans try to hunt down the issue, correct the variable with the previous iteration, and restart the application.
This is no bueno, and yes, it has happened. Twice.
Configuration changes, as much as we wish to automate all things away, is still very much a human endeavor. So what can we do? We love Vault, so was there a way we could keep it?
Sure!
Understanding The Problem
Appending Values…
Vault utilizes the concept of Vault mounts as endpoints for holding your secrets. Concerning the generic mount type, you can specify a single, or multiple, keys to a given endpoint.
Let’s assume from here on out we have an application called single, and another called multi, stored in Vault.
Example:
~ ❯❯❯ vault mounts
Path Type Default TTL Max TTL Description
cubbyhole/ cubbyhole n/a n/a per-token private secret storage
dev/ generic system system Secrets for Developers
~ ❯❯❯ vault read /dev/single
Key Value
--- -----
refresh_interval 720h0m0s
FOO BAR
~ ❯❯❯ vault read /dev/multi
Key Value
--- -----
refresh_interval 720h0m0s
BAR FOO
FOO BARNow let’s say we want to change /dev/single to also hold the value BAR=FOO.
~ ❯❯❯ vault write /dev/single BAR=FOO
Success! Data written to: dev/single
~ ❯❯❯ vault read /dev/single
Key Value
--- -----
refresh_interval 720h0m0s
BAR FOOFor those who have used Vault previously, the conundrum is most certainly not new to you: Vault endpoints are not directly appendable from the command line itself.
There are ways to edit and append values to Vault, for example, by simply using the -format=json flag to pull down JSON, and then uploading the appended data to Vault. As an example of this, let's try adding FOO=BAR back to the /dev/single endpoint:
~ ❯❯❯ vault read -format=json /dev/single \
| jq --arg foo BAR '.data + {FOO: $foo}' \
> /tmp/single.json
~ ❯❯❯ cat /tmp/single.json
{
"BAR": "FOO",
"FOO": "BAR"
}
~ ❯❯❯ vault write /dev/single @/tmp/single.json
Success! Data written to: dev/single
~ ❯❯❯ vault read /dev/single
Key Value
--- -----
refresh_interval 720h0m0s
BAR FOO
FOO BARSo this solves the first issue: how to append values to an already created endpoint.
Retrieving Deleted Values…
However, what happens if one of our kind developers whom, only trying to make a simple edit to a configuration for one of our applications, were to overwrite an endpoint? You can imagine this would be incredibly unfortunate. Many of our applications have 10s of variables stored to a single endpoint!
You could mitigate this problem by, for example, having each variable stored as a single endpoint (i.e. /dev/single/BAR, /dev/single/FOO) and then having the tool you use to decode the vars (such as ConfD) iterate over a list of endpoints. However, this does not solve the underlying problem: the previous version of our variable is gone into the void! The Vault Audit log can not save you here...save to tell you who did it (...Steve.).
How We Solved This
At WeWork, we have a proprietary command line interface that we have written for our developers dubbed the we tool, which is based on the open source project Robo. It has allowed us to do pretty interesting things through shell loopbacks, and is quick to iterate on.
Because our developers use the we tool to communicate directly to both Kubernetes and Vault, which abstracts away much of the nuances of both the kubectl and vault command line tools, we have the advantage of being able to solve this directly from the tool.
Obviously, this is not viable for everyone out there; our hope is to long-term merge these ideas directly into the Vault binary. As of now, doing direct writes to Vault with the vault tool simply does not provide the functionality we need, so a wrapper CLI was necessary.
For now, let us discuss our solution in its current implementation.
The /versions mount
Take a look at our updated Vault mount points:
~ ❯❯❯ vault mounts
Path Type Default TTL Max TTL Description
cubbyhole/ cubbyhole n/a n/a per-token private secret storagedev/ generic system system Secrets for Developersversions/ generic system system Mount for storing Config Versions
You will notice that there is a new versions/ mount added to Vault. This mount point is a simple generic type.
Now, let us say our developer wants to make an update to the single application's /dev/single endpoint once again. In practice at WeWork it would look something akin to this:
~ ❯❯❯ we dev single config set HELLO=YOU
Saving current config state...
Success! Data written to: dev/singleRestart single with updated configs? [y/n]
What just happened?
Behind the scenes, our command line tool has done the following:
- Pulled down the previously set configurations as JSON
- Wrote the previous config JSON to
/versions/dev/single/$(date +"%Y%m%d%H%M")and/versions/dev/single/previous - Added
FOO=BARto the pulled down configuration (using JQ), and uploaded the changes to the given/dev/singleendpoint
This is actually relatively simple Bash:
Result
Now, if we return to vault, we would see something similar to this:
~ ❯❯❯ vault read dev/single
Key Value
--- -----
refresh_interval 720h0m0s
BAR FOO
FOO BAR
HELLO YOU
~ ❯❯❯ vault list versions/dev/single
Keys
----
20180110184521
previous
~ ❯❯❯ vault read versions/dev/single/previous
Key Value
--- -----
refresh_interval 720h0m0s
BAR FOO
FOO BAROur we tool users can then very easily rollback a configuration if necessary:
# A user can rollback to 'previous' by default
# or provide a specific timestamp to rollback to
~ ❯❯❯ we dev single config rollback
Rolling back application config to "previous"...
Saving current config state...
Success! Data written to: dev/single
~ ❯❯❯ vault read dev/single
Key Value
--- -----
refresh_interval 720h0m0s
BAR FOO
FOO BARThis bash is pretty straight forward:
Minimizing Bloat
Likely, the first question that comes to mind after seeing an implementation such as this would be: "wait, would your Vault pile up with hundreds/thousands of stale secrets?" You would be correct, and unfortunately, Vault under the generic mount does not provide us the ability to have TTLs set for our secrets themselves, only the tokens that create them.
Luckily, there are many tools out there that can provide cron abilities to ensure this will not be the case. At WeWork, we use Jenkins extensively, and have a very straight forward job that runs once per week to ensure that any versioned secrets older than a certain amount of time are expunged.
Below is a bash representation of how that job looks based on our examples above (it should be tailored to fit your needs):
With this bash method, we ensure that versioned secrets older than a week are deleted on a weekly basis (in this case).
However, note that the /previous secret version is not destroyed.
Let’s look back at our single app's versioned vault:
~ ❯❯❯ vault list versions/dev/single
Keys
----
20180110184521
previousprevious here is a special versioned secret which is permanently maintained. This ensures that we can always rollback, at least, to our last set configurations and/or retrieve them.
Permissioning
Permissions with the versioned Vault match 1-for-1 to the permissions a user/group receives for the original Vault it is based on. So if a user has write access to the/dev/single endpoint for example, it would look something like this:
path "versions/dev/single" {
policy = "write"
}path "dev/single" {
policy = "write"
}
Obviously, the opposite would be true if they were denied access. More information on Vault Policies can be found here.
What comes next?
As has been shown here, this implementation currently requires a wrapper CLI around Vault, be it an edited binary or personal CLI tool, to implement properly. Currently, through WeWork’s use of Robo, this is primarily written in Bash, and relies on the Vault CLI tool to implement. Our team is currently working on a design document to have this kind of mechanism added directly into the Vault binary and project.
An example of one way to handle such a mechanism may be to have a versioned mount type, very similar to generic, that would automatically handle creating versioned secrets.
It’s worth noting that while this is one way to handle this issue, there are others as well:
- Making consistent backups of your Vault backend and using them as restore points
- Using an API Proxy in front of Vault that would ensure that all writes are versioned
- …never make a mistake?
