The 7th edition of DevOpsDays Amsterdam in ‘Pakhuis de Zwijger’ was my first experience with DevOpsDays. I’d like to share some talks and my impressions about the conference, because sharing ideas worth spreading is what we aim for at JWorks.

Table of contents

• Observability for Emerging Infra: What Got You Here Won’t Get You There, by Charity Majors
• How Convenience is Killing Open Standards, by Bernd Erk
• Come Listen to me, I’m a fraud, by Joep Piscaer
• Fight, Flight or Freeze – Releasing Organizational Trauma, by Matty Stratton

Observability for Emerging Infra: What Got You Here Won’t Get You There, by Charity Majors

Charity Majors is a co-founder and engineer at Honeycomb.io, a monitoring and analyzer tool that can gather information from various sources to provide an overview of what’s happening in your (production) environment.

Most of you have experience with sieving information through logs and analyzing all kinds of dashboards to find and fully understand what went wrong in your application. Finding and understanding the data can be really hard. You lack the bigger picture because the tools for doing metrics, logging and tracing (what she calls the “three pillars of observability”) have their own benefits and drawbacks.

Logs can be useful for compliance reasons but they are not so great to picture the context. Metric tools might be more visual than logs, but they are less suitable for debugging purposes or to completely understand why something failed.

We as human beings like visual representation to easily perceive why certain situations occurred. For example during and after deploying new versions because the chance is higher that something will go wrong. According to Charity, deployments are not like ‘flipping a binary switch’. It’s more like a step in a long process of being confident in your code that it will work on other environments rather than just on your local machine or development environment.

Especially deployments to production will not always go as planned and it will be even more difficult to do so, since infrastructure has become more complex over the recent years. We see our infrastructure changing from simple LAMP stacks to replica sets, nodes and micro-service architectures.

It would be very difficult to quickly grasp what’s going on with your applications and its underlying systems without good observability.

Observability is a measure of how well internal states of a system can be inferred from knowledge of its external outputs. It has the advantages of logging, tracing and metrics, and could be achieved with the right tools, if set up correctly. This is what Honeycomb is trying to achieve.

Charity states that observability is not the same as monitoring. That is because monitoring is more like a ‘post-talk’ approach in her eyes. You’re typically too late and the damage has already been done upon discovery. Monitoring systems have not changed significantly in twenty years and have fallen behind the way we build software. Software nowadays consists of large distributed systems that are made up of many non-uniform interacting components, while the core functionality of monitoring systems has stagnated.

How Convenience is Killing Open Standards, by Bernd Erk

You may know that migrating configurations from one cloud provider to another can be challenging. Cloud providers are mostly lacking an ‘open standard’ to make it easier. Bernd’s talk wants us to think about what Open Standards and Open Source could bring to us and how they have led to the success of Linux and Open Source communities we have today.

In his early career, Bernd was a Solaris system engineer who maintained more than just Solaris systems within his company such as Unix-like systems AIX and HP-UX. Managing unfamiliar platforms was hard, but luckily, most *NIX systems are relying on the POSIX standard. POSIX made it possible to develop other libraries such as GNU tools so they weren’t forced to use Sun’s (proprietary) freeware for instance. Fast-forwarding to 2019, we see that open source is becoming more popular than ever before. However, regarding Open Standards, there is still some work to do.

The market has changed over the recent decade. Public and hybrid cloud are emerging and on-premise deployment is declining. We see that especially AWS and Azure have been expanding in the recent years as more and more customers are using their services. All those cloud providers like AWS, Google Cloud, Azure,… have their own APIs and those are not interchangeable since they are lacking a standard. The fact that an API is open does not imply that it is a standard, it’s only something you have access to. That is why tools like Terraform are becoming popular. People have a need for a unified standard to manage their resources. Bernd thinks that Open Source is very important to have open standards although those are not the same.

There is also a problem if a certain cloud provider uses Open Software. If they use a certain piece of open source software, there is no direct connection between the customer and the original developer of that software, because that layer was cut. No matter what business model, the creator of the software has to earn money.

Open Source tools in the cloud make sure that you have direct contact to the creators of the software to help you out. AWS, on the other hand, has their own versions of Open Source Software like Open Distro which is basically their own distribution of ElasticSearch. This removes the connection with the original developer, because you can only contact them for support. It brings fear and protection to the developers of the software because they are afraid that money and intellectual property goes away. That’s the reason why they came up with own licenses like Server-Side Public License (SSPL) that MongoDB uses, for instance. This tells you that if you alter it to sell it as a SaaS offering, you have to contribute to that code. And Redis prohibits you to sell their enterprise model as their cloud solution.

Bernd doesn’t like that people come up with new licenses, but he also understands that they are afraid that their money stream would go away, and they see this as a possible solution. We have to see that we support enough Open Source and Open Standards, because they need it to improve and maintain their software or service.

In today’s oligopoly of cloud providers, it’s really difficult to new players to emerge their new business, although this will be good to have a more diverse internet. Most companies are focusing on the more human side of things when it comes to diversity, which is great, but there is a lot of work on the technology side as well. He compares it to when there are only a few companies that control the availability of Insulin because there’s no market law to reduce prices. It would be a lot cheaper if there were more players that provide Insulin. “Abuse of Power comes as no surprise”.

At the end of his talk, he draws some conclusions:

Be reasonable

• Think about what you are doing.
• It must be easier for new players to enter the cloud market, with the help of open standards.

Interoperability is important

• You can easily work with different services from different providers so you can work together.

Support variety

• Try to be out there and have a look on the industry on how it is. Think about what your next move will be.

Demand contribution

• Demand as a customer what you want to obtain from your provider.

Diversity is up to you

• Demand more diversity on the technical level.

Come Listen to me, I’m a fraud, by Joep Piscaer

“Do you tend to chalk your accomplishments up to luck or timing”; “do you hate making a mistake, being less than fully prepared or not doing things perfectly or do you even fear feedback?”

Joep’s talk started with an exercise to see if the people in the audience have some form of Impostor Syndrome by asking several questions like the one mentioned above. People had to raise their hand if the question applied to them and to keep them up during the next questions. At the end of his question round, more than half the audience had raised their hand.

A collection of feelings of individual doubts that overwhelmed their successes and accomplishments which makes them think they are a ‘fraud’. He described Impostor syndrome as an internal measuring stick that is broken or Pluralistic ignorance. Doubting yourself privately, believing you are alone in thinking that way because no one voices their doubts. It is very likely that one or more of your colleagues have Impostor Syndrome although they will not admit they suffer from it. Some are more open about that topic than others. No matter how successful they are in their personal life and career, they ‘suffer’ Impostor Syndrome in some way or another. Mike Cannon-Brookes, CEO of one of the larges DevOps companies in the world is very open about it during his TED talk.

“Other people feel like this, too. And apparently it doesn’t go away with more success.” - Mike Cannon-Brookes, Atlassian

Atlassian started, like most big tech companies, from a garage. It grew bigger and bigger and at some point, they needed an HR-person. While Mike was interviewing his potential HR-person, he felt a bit unsure, because he has never worked in a company with an HR-person before. So how will he know if the potential HR-person has the right skills and attitude?

Like with many things, there is always a beginning where you have to figure out what way is the right way to do things. There is mostly not one best practice since everyone has their own. It is sometimes hard to know if your way is the right way, the good way or just bad. That will mostly lead to discussions and insecurity with people, because they don’t know or they are not really sure if their statement is well grounded.

Especially technology can change rapidly over time. By the time you have learned a new thing it’s already obsolete.

Joep learned about Novell back when it was the latest and the greatest server Operating System you could have. By the time Joep became a guru, it was already obsolete and surpassed by other Operating Systems.

The tech landscape changes every couple of years and so sometime, you have to learn things again. That doesn’t really help when the Dunning-Kruger effect applies when learning new stuff. Most people don’t really discuss that certain things are new to them and that they are not really good at it yet. They feel insecure sometimes because they are unsure or afraid that their idea or statement is incorrect or a best practice.

There are certain things you can do to reduce the feeling of the Impostor Syndrome (or to recalibrate your ‘internal ruler’ so to speak).

Nothing compares to you.
Every time you say something negative about yourself, say two things that are positive about yourself. It is as easy as that to retrain your internal NLP (Neuro Linguistic Programming) filter. At some point, your negative thoughts will diminish, and the positive ones will stay. Make sure to try this in front of a mirror!

I will not compare myself to strangers on the internet.
You shouldn’t compare yourself to others, especially to people on the internet, because they are the worst.

Practice on giving compliments to others. And learn to receive them.
They can come from different perspectives. For a kid it’s an accomplishment if he or she is able to tie his/her own shoelaces because it is new to him/her.

Learn how others make stuff and let others learn from you.
Other people, no matter how professional they are, make mistakes as well and can learn things from you and you can learn from them.

• Pair programming
• Pair review
• Celebrate your failure
• Speak publicly

Find something you’re passionate about and keep being tremendously interested in it.
Like a hobby where you can screw things up without consequences.

Fight, Flight or Freeze – Releasing Organizational Trauma, by Matty Stratton

Imagine a zebra grazing in the savanna of the African wilderness when suddenly a predator goes after the zebra. Of course, the zebra will run for its life (the flee state). When the zebra is caught by the predator, it ‘freezes’ because the zebra’s nervous system is overwhelmed and it has no further solutions anymore, hoping the predator will drop him and go on. After the zebra has managed to get away, it literally shakes all his stress off and goes back to grazing like nothing happened. Humans can react the same way zebras do, although we are, of course, not a zebra. We humans on the other hand can experience traumas long after a dreadful situation has occurred.

Animals are not traumatized by routine threats to their lives, where humans on the other hand, are really overwhelmed and often subject to traumatic symptoms of hyper arousal, shutdown and dysregulation, as Peter Levine’s calls it. During day two, Matty Stratton, DevOps advocate at PagerDuty, was talking about crisis management and traumatic events within organizations and teams based on his own experience with post-traumatic stress.

Let’s start on how traumas can occur, although it’s rather complex to explain.

• Traumas occur when one’s solutions (active response to threat) does not work. The case where your nervous system can’t handle it. Traumas can result from both real or perceived threats. Even a perception of a ‘threat’ can also cause or recall traumatic situations.

• Trauma is subjective and relative. Everyone experiences and handles stress in a different way. People from the military, for example, experience stress and traumas in a different way than people in software projects. But that doesn’t mean that your experience isn’t real, even though the experience from a soldier is much worse.

But how does that apply to an organization? You can deal with common situations and everything will go back to normal. It’s always business as usual (the gray line in the normal range), but severe outages or a similar event, where the team’s capacity for a solution does not work, can lead to non-discharged stress (the red line).

Hyperarousal organizations are hyper-vigilant (like they are fighting Voldemort). They are hyper-aware of threats which causes a slowdown in moving forward. It’s not necessarily bad for an organization to be vigilant but it is bad when it loses a sane balance between vigilant and the will to move forward and innovate.

Organizations can also be in a ‘freeze’ state when they believe that they are ‘immune’ to a standstill so they won’t make any changes on how they operate.

Humans like pattern recognition. We see signals that might remind us of events that happened in the past and motivate us to respond the way we did last time. But the thing is that nothing is the same as it ever was back in the day as systems are becoming more complex as they grow.

Today’s challenge is to make organizations more resilient against dreadful events. Resilient organizations are not traumatized by routine threats to their mission or business. Non-resilient organizations are readily overwhelmed and often subject to the symptoms of overreaction, shutdown and lack of regulated effort.

What you can do to be more resilient is to learn from your mistakes. Organize game days where you can practice an incident or outage in a relax and safe way.

Conclusion

DevOpsDays has a different but interesting format. There are talks in the morning and lightning talks and open spaces during the afternoon/evening where you can bring your own topics that you want to discuss with others. What I also like is the fact that there’s a good variety between the amount of technical and soft-skills talks.

The badges that DevOpsDays handed out were actually seed mats that you can plant afterwards (which I’ve done already)! UPDATE: Unfortunately nothing came out of it.

Overall, it was a good and informative conference that I can recommend to anyone who is interested in DevOps and teamwork optimizations within organizations.

More conference pictures can be found here.