Honeycomb in Rails without hard-coding the API key

Today we hooked up Honeycomb to an app for the first time! It was easy and fun!

Then we configured it so that we don’t have to commit our apikey. It’s a public repo. Anyone could grab our API key and send spurious events to our Honeycomb datasets, screwing up our information and costing us money.

Remember that if your MVP isn’t secure, it is neither viable nor a product.

For reference, this is how we configured Honeycomb in Rails (developing in Docker, deployed to Heroku) while keeping our apikey private.

Get the API key from the environment (safely)

The recommended integration was easy-peasy. A rails generate created config/initializers/honeycomb.rb. This contains:

Honeycomb.configure do |config|
  config.write_key = 'YOUR_API_KEY'
  config.dataset = "mydatasetname"
  # ... bunch of other sensible things...
end

We want to get the key from an environment variable. If that variable is not defined, we want the app to go about its business without sending to Honeycomb. So we changed it to:

Honeycomb.configure do |config|
  honeycomb_key = ENV["HONEYCOMB_WRITEKEY"]
  if honeycomb_key.blank? && Rails.env.production?
    raise "Unable to send events. Not OK. Define HONEYCOMB_WRITEKEY to our Honeycomb API key"
  end
  if honeycomb_key.blank? 
    warn "Not sending data to Honeycomb. Define HONEYCOMB_WRITEKEY to enable tracing."
    next
  end
  config.write_key = honeycomb_key
  ... bunch of other sensible things ...
 end

This gets the environment variable and then checks it for blankness. If it’s blank in production, not OK. It’s a security risk to run in prod without recording events; we need to know that things are happening.

If it’s empty in development or test, no bother. Maybe someone cloned my repo to contribute to it, and they don’t have my Honeycomb API key. In that case, we break out of the block, and Honeycomb doesn’t get configured at all. No events, no big deal, for development.

(Ruby gotcha: it’s next and not return here because we’re in a block, not a function. return will do bad things, returning prematurely from the function that called this block. So will break)

Get the API key into the environment (in Docker)

Now, how do I set that HONEYCOMB_WRITEKEY environment variable? I could set it in my Dockerfile or in docker_compose.yml, but both of those get committed to the repository. That would defeat the purpose.

I could set it globally on my PC and reference that in docker_compose.yml. But I want to keep it with this project. I want to have it all the time when I run this one project, and never when I run any other project.

Docker has a solution to this: a file called .env. When I run docker compose up, it reads that file and makes the definitions available. So we added it in .env:

HONEYCOMB_WRITEKEY=<lots of letters and numbers>

and referenced it in docker_compose.yml:

services:
  app:
    # ... stuff ...
    environment:
      # ... stuff ...
      HONEYCOMB_WRITEKEY: "${HONEYCOMB_WRITEKEY}"

Now my Rails app will have it.

Save the file, don’t commit it

But that .env is in my project too! If I’m not careful, that’ll end up in GitHub as well!

We told git to ignore that file… which didn’t work, because it had already been committed in our project. So we used git rm --cached` to take it out of the commit without deleting it.

Don’t forget to commit those two operations together. Here’s the spell:

echo ".env" >> .gitignore
git rm --cached .env
git add .gitignore
git commit -m "Keep sensitive data in .env out of the repository"

Leave a clue what to do

When we don’t commit .env, how will people know to populate it? How will other people figure out where to put HONEYCOMB_WRITEKEY?

The README, sure, but we can give them additional help.

We copied .env to .env.example and replaced our sensitive data with a placeholder. That file we can commit and reference in the README.

Don’t forget production

We’re deploying to Heroku, so that meant

heroku config:set HONEYCOMB_WRITEKEY=<letters and numbers>

When that worked, I documented it in the README.

If the MVP is not secure, it’s neither viable nor a product.

Kudos to the Honeycomb team for making it fast and easy to see events, in development, after one gem install and one rails generate. Poof! We saw data from our local machine show up in our dataset.

Honeycomb contributes to security by including a bunch of configuration in the generated code to avoid sending sensitive data in events. (The events include the SQL from Active Record traces, but not the data retrieved. I want intimate knowledge of my system, not of my customers.)

It’s my job as a developer to complete the transition from maximally easy to secure.

Seeing that API key in a git diff was enough to set off alarm bells in my head. Rails, Docker, and Heroku all provide mechanisms for keeping data like this safe. Hopefully this post makes it easier for you to use them, too.

Fail gracefully, except when continuing would be the greater failure.

It’s also my job as a developer to handle the unhappy paths smoothly, giving people the information they need to get to the happy path.

Every environment variable is an opportunity to not have it. Or have some garbage in it. (I’m counting on Honeycomb to handle the garbage, because I don’t know for sure and for perpetuity what makes a valid key.)

In development, graceful failure means: no key, no events, no big deal.

In production, graceful failure means: never start up in ghost-mode without events, and loudly scream the instructions of how to do it right. The message “Unable to send events. Not OK. Define HONEYCOMB_WRITEKEY to our Honeycomb API key” tries to convey why this is a problem and what to do about it. This is a pain, but nothing like the pain of “I’m hitting that endpoint right now! Why do I see zero calls to it?” or worse, “Nah, our load is normal” when load is 200% and half our instances aren’t reporting events.

Sometimes loud failure is the kindest failure.