Never Go Alone to Production
I wrote before about fessing up anything funny in production - this time, I want to share another piece of advice in production:
Never go alone to Production
In other words, always pair when you’re in production or have a reliable help line.
Why?
Just like fessing up when there’s something funny in production, the problem is when make an unexpected mistake in production, having another person there helps immensely when trying to fix the production problem under pressure.
When you go alone into production and have an error, your mind races quickly and anything that remotely sounds like a hint of an idea to solve the problem becomes a great idea quickly.
This is where having another human being helps. Either by validating ideas, coming up with more ideas, or just have a helpful buddy.
My Situation
Once when I went into production alone on the weekend - I thought I was fixing a routine issue, one that I’ve fixed before, so I was over-confident.
Well, the issue wasn’t routine and I thought I fixed it by applying another patch.
When I checked through, the solution looked good. I fixed the original issue.
Monday comes around and everyone is asking:
Who touched this server?! It’s misconfigured!
Well, what I didn’t realize: yes, I fixed the original issue - I put the server into an unintended state that required some configuration not documented.
🤦
Since it was Monday and everyone was there, the solution came quickly - because all the knowledge was there.
On the weekend alone, when I felt something funny, I just focused on that one issue, not the bigger system, and once I fixed things, I just got out.
Buddy System
This is a case where even having a buddy would alert me that I need to call in for more help and/or the my hack of a fix wasn’t as simple as I thought it would be.
In this case, I also violated the rule of not fessing up when there was a problem in production.
Let’s just say, that was a rough Monday.
😅