When I first started my software development career, deploying to production meant connecting to the server using FTP, uploading all the new build files and copy and pasting into the application.
Naturally this process wasn’t without error. A file wouldn’t get updated, some new package needed to be installed on the server or just good old human error.
Thankfully, this is a solved problem these days when it comes to deploying applications, and we are spoilt for choice.
We have docker containers that aim to solve the “it works on my machine” problem by giving you a container image that can be deployed on any machine and works as expected.
We have cloud native applications such as AWS Lambdas and Azure Functions that just work, and you don’t need to worry about the underlying software that is being used.
Server Environments
This is all good for applications, but it doesn’t help you with the server the application is running on.
For example, if you are using docker inside a VM, you need to make sure that docker and likely docker-compose are installed. Maybe you need kubernetes or you need to install an Octopus tentacle on the server. When you come to set up a new server you end up manually running different commands to get the box set up.
I recently set up a free Oracle VM to use with boringproxy which involved running various commands to pull down the binary as well set up iptables. None of this was automated and should I need to do it again, I am going to be scrambling around trying to find the various blog posts and StackOverflow answers that got me to a working system.
When I started my self-hosting journey I was using a Raspberry Pi. Unfortunately on a Pi the OS is installed on a SD card which isn’t the most reliable of media. After a few too many dead cards and wasted days setting everything back up again, I moved all my applications to docker containers and put the configuration in git. This helped a lot, but I still needed to waste time setting up docker and git for a new server (I should have just backed up the SD card).
Various tools have been developed to try and solve this problem with the typical arbitrary names you get in tech:
- Ansible
- Puppet
- Chef
- Vagrant
None of these systems are foolproof and even though they try and be repeatable that isn’t always the case.
Anytime you are installing packages with these systems you have two options:
- Install the latest package available, but you can’t guarantee the latest version doesn’t contain a breaking change.
- Install a fixed version but have the process fail when that version is no longer available in the package repository.
Anytime your scripts or playbooks rely on a third party, you can’t guarantee it is going to work again when you need it to.
Dev Environments
This brings me on to development environments. We may deploy using cloud native technologies or docker containers, but the development still needs to be done locally on your machine.
In most cases, project dependencies are localised to the project. This might be installing node modules for a frontend application or pulling down NuGet packages. But what about the dependencies you need to install before getting to that point?
For node, you will often need npm installed and maybe yarn. If you have multiple projects that use different versions of node then you need nvm installed to be able to switch between them.
For .NET you need the SDK installed. Then there is python and all the pip packages that get installed globally on your system.
Most of these packages developers just install globally using homebrew or some other package manager. A quick check running brew list on my systems shows I have 160 packages installed, all for various projects I have tried over the last year since getting this machine.
Again there are various solutions that have been developed to overcome this:
- Run your development environment inside a docker container and use docker-compose watch to monitor files.
- Develop entirely in the cloud using GitHub Codespaces or GitPod. Good luck if you have a bad internet connection!
- Run your dev environment inside a virtual machine.
None of these are great solutions which is why we end up installing packages globally and just dealing with the fallout from conflicting versions.
Nix and NixOS
I am not just fashionably late, but embarrassingly late to the party when it comes to Nix.
Nix has been round for over 20 years. It is a package manager, language and an OS. It has only been from listening to podcasts such as Self-Hosted that has got me looking into Nix and NixOS a bit more.
NixOS is what we call a declarative operating system. Essentially the whole OS is configured in one big file. Nix uses nix packages for everything which solves the third-party issue which plague other tools.
This is great for servers as it means you can define all the requirements your server needs and commit it to your git repository.
If you really wanted to, you can use NixOS as your main operating system as well and have a repeatable system that can be deployed and rolled back.
The Nix package manager can also be used on Linux and MacOS to install packages instead of using homebrew.
The benefit really comes when you use nix-shell. This is a separate shell which you can spin up with preinstalled packages. If you specify a shell.nix file in your project you can configure all the external dependencies your project needs without installing them on your system.
There are a few tools that try and simplify this process a bit more. I am not sure the benefit of these yet over vanilla Nix, but they are all using Nix under the hood:
- Devbox (7.1k stars) - simplified
devbox.jsonfile instead ofshell.nix - devenv (3k GitHub stars) - this one uses a
devenv.nix. - Flox (1.2k GitHub stars) - similar but uses a
manifest.tomlfile. - devshell (1k GitHub stars) - simplified
devshell.tomlfile but can fall back on theshell.nix.
Nix seems to have quite a steep learning curve hence all these projects to simplify things for the user.
When I have time I am going to try and replace all my homebrew and pip installed packages with project specific nix configurations. I will let you know how I get on!
These have been helpful for learning nix:
If you are already an avid Nix user and have some tips then let me know!
❤️ Picks of the Week
🛠️ Tool - Devin (the AI Software Developer) - developers are expensive, so naturally companies are looking for ways to replace us. If you watch the video you can see some of its capabilities. It is essentially what AutoGPT promised to be. Devin scored 13.86% on the SWE Bench which is a lot higher than other models, but it still failed at 86.14% of them, so I think we are safe for a little while.
📝 Article - Any Technology Indistinguishable From Magic is Hiding Something - The big 4 Google, Amazon, Microsoft and Meta control most of the internet. Even my website is hosted on AWS, it is simple and cheap. At least I can move it at anytime. It does make you wonder though how much control we should be giving these mega-corporations.
📝 Article - Are We Watching The Internet Die? - I am sure AI has some generally useful applications, but the current fascination with generative AI isn’t it. All the top companies seem to be trying to find a way to profit using AI and their users’ data.
📝 Article - The internet isn’t dying, it’s changing - Cory’s response to the above. It does look like we are seeing a bust cycle for social media. It is definitely becoming more fragmented. I am looking forward to being able to connect with Bluesky and Threads users via my own Mastodon server provided it doesn’t expose me too much to all the AI generated, non-authentic, fake news that seem to flood this platform.
📝 Article - More options for apps distributed in the European Union - I am pretty sure Apple are getting annoyed by EU mandates, but they are pulling the “genie card” when comes to implementing them.
Distribute your apps to iPhone from your own website, no problem … (only available for those who have been on the Apple Developer Program for 2 years and have more than 1 million downloads in the EU).
🛠️ Tool - Trangram - create motion graphics and animated SVGs for free. “Think Adobe Illustrator but for animation”
🛠️ Tool - JSON Canvas — An open file format for infinite canvas data. - I am big fan of Obsidian for taking notes and one of the features I have been using for planning out my new website redesign is there canvas feature. You can add images, websites and text to a canvas similar to a Miro board. Obsidian have just open-sourced this as a JSON canvas, so you can use it in your own apps.
🛠️ Tool - LocalSend - Apart from my servers, I am now 100% in the Apple ecosystem (don’t judge me). As a result I can copy, paste and airdrop text and files as much as I want. This isn’t so easy if you have a mix of different non-Apple OSs. LocalSend looks like a great alternative.
🛠️ Tool - bruno - I have been looking for good Postman alternative. I am using Insomnia at the moment, but I am not sold on the UI. Bruno looks like a good open-source option.
🛠️ Tool - Tauri - Build smaller, faster, and more secure desktop applications with a web frontend - Electron gets a bad reputation for being slow and bulky. Tauri aims to fix that but reusing the web view already available on every system instead of bundling in most of Chrome into your application.
💬 Quote of the Week
“None of us know what will happen. Don’t spend time worrying about it. Make the most beautiful thing you can. Try to do that every day. That’s it.”
- Laurie Anderson