Databases: The Future Buggy Whips of Software
This article has been excerpted from my book Avoiding IT Disasters. It is a continuation of the articles What I’m Telling Business People About Why Relational Databases Are So Bad and The Relational Database Perfect Storm: How Such a Bad Technology Became So Ubiquitous
We don’t use buggy whips today. The idea of inflicting pain on a four legged animal in order to get a vehicle to go faster is such an antiquated concept that it is hard to imagine the kind of world that used these things. The technology moved on.
Databases are a technique that has only arisen out of the primitive nature of our current computer systems. This article looks at what the future of databases when the hardware technology improves to the next level. Like buggy whips they are going to become a largely forgotten concept that was necessary to deal with crude technology.
The whole success of computers is that they can do an incredible number of small operations in a very short time. This means that it is necessary that the data in the computer’s memory be accessible in incredibly short times, and that any of these memory locations can be accessed in the same time as any other. We are talking nanoseconds — billionths of a second. This allows us to build up highly complex operations out of an incredible number of really simple ones, and to do it in a reasonable time.
Unfortunately, to get this speed, the memory technologies we use today are called “volatile.” This is the word engineers borrowed from English to express the concept that its contents only remain as long as the power is on.
If you imagine computer memory as a big line of light switches imagine that all those switches that are “on” are held in position by the power being on, and when the power is cut, they all flip back to “off.” That is volatile memory.
For the opposite of volatile, engineers chose the word “persistent.” Persistent memory is any kind of memory that holds its data even when the power is turned off.
For most of the twentieth century we had recorded music, or other content, on magnetic tapes. They kept their information even when the power was off. If you go to a vacation cabin and find an old tape player and some tapes that haven’t been used in decades, they will probably still play. The magnetic information lasts for a long time. It is very persistent.
Initially, data was kept on magnetic tape. Those large reels of half-inch tape looked great in movies as they spun this way and then that. The problem was that they were slow. If you wanted to get to data at the end you had to wind through the whole tape.
Tape was replaced by disks. This was the same magnetic coating but instead of having it covering a long plastic tape, it covered the surface of a spinning disk. Think of the read/write head moving in and out like the arm on a vinyl record; it can read the data as the disk spins below it. Reading data requires the head to move (called seek) to the track it wants to read or write which takes around 5/1000 of a second (5ms). Then there is a wait until the information spins around to come under the head (called latency). Nowadays disks tend to spin at 7200rpm, that is just over 8/1000 of a second (8 milliseconds or 8ms) for one turn. That means, on average, it takes 4ms for the data you want to come under the head. So, to find a piece of data you have to move the head to the right position and wait for it to rotate beneath — this is around 9 milliseconds. Note that all these times are measured in milliseconds, which is a million times slower than the nanoseconds that main memory takes to get to data.
This means that you can’t treat disk memory the same as the main computer memory. It is too slow to handle random accesses to any part of its data — you have to prepare data to be packaged in blocks to write to the disk, and to read you have to read fixed-sized chunks of data at a time and restructure it for main memory.
This problem, that the main memory of a computer (all those switches) is volatile, has bedeviled my career and that of most programmers. Whenever I am storing any kind of data I have to move it to a persistent store. If I don’t, it will be lost if the power fails. To do this I have to convert it into a group of bytes that I can write as a block to the disk.
Having to deal with block-structured persistent memory adds a considerable amount of entropy into a computer system. It consumes a lot of code, and it is the source of a lot of problems. When you add incredibly awkward relational databases as a way of persisting data, it becomes a giant amount of entropy.
The irony of all this trouble caused by block storage devices and databases is that it is a detail of systems that will disappear as soon as hardware technology advances.
The breakthrough in computers that will make a huge difference to the success rate of enterprise systems will be high-speed persistent memory. If we had fast computer memory that was not volatile, that kept its data between power shutdowns, then we would not need a database. We would just keep the data in the convenient format that we use in volatile memory rather than going to all the trouble to translate it into forms suitable for storing in slow, magnetic storage. Even if the power failed the information would be there, exactly as we left it, when the computer restarted.
There are various high-speed persistent memory technologies being worked on. I’m sure that they will be available sometime in the next decade. As soon as that happens, the need for databases will disappear. The whole architecture of systems will change to saving-memory structures that are immediately usable by programs.
At that point there will be no point in criticizing relational databases because they will become a relic of past technology, and enterprise systems will be much improved because of it. In fact, any kind of database, including the newer NoSQL databases, will become obsolete. I for one can’t wait. For programmers it will be totally liberating, and for users it will make systems, including enterprise systems, faster to implement, more reliable and much, much cheaper.
In the not too distant future Databases will become as useful as buggy whips are today — an unknown appendage that came about because of primitive technology.
This article has been excerpted from my book Avoiding IT Disasters
✉️ Subscribe to CodeBurst’s once-weekly Email Blast, 🐦 Follow CodeBurst on Twitter, view 🗺️ The 2018 Web Developer Roadmap, and 🕸️ Learn Full Stack Web Development.
