Test Like You Fly - The World Revolves Around You
In Test Like You Fly - Part 2 - Basics, we discussed the fundamentals of how you practice a TLYF strategy. No systems operate in such isolation that they are not reliant on some other code foundations. And those foundations rely on other foundations.
Environments and Configurations - Oh My!
Our applications don't live in a vacuum. They take advantage of the operating systems, databases, caches, networks, and other external services available today in the cloud deployment environment. Let's see what we should think about in regard to Testing Like We Fly.
Predefine and Test
- Provisioning – Ensure existence/creation of directories
- Validate that dependencies (gems, npm packages, etc.) state compatibility with the environment, not just assume future compatibility
- Consistency – Verify language (Ruby, Rails, JavaScript, Database, etc.) versions and their dependencies being tested match those in our deployed environment.
- Environment variables (and defaults for lack thereof)
- IP usage and Port conflicts
Get Specific
Specificity is an invaluable ally in operating environments that have been confirmed through testing. Do not make the fatal inference: “It should work”.
- Bugs exist in all code.
- Bugs existed in code in older versions then you are using
- Bugs (especially unknown bugs) can be introduced in code you rely on if it is beyond your tested version
- Upgrading is not only a requirement but a risk
- All environmental changes require retesting
Cautionary Tale - MySQL Version Upgrade
Let's say we have a database running MySQL 8.0.18. We are tasked with upgrading this database to MySQL 8.0.19. Based on semantic versioning, this looks like an innocent patch update. Unfortunately, this is far from the truth. In this update, the code was changed to assume that TINYINT(1) should be treated as a boolean. However, this was NOT the case any longer if the column was UNSIGNED when the schema was extracted. When schemas from MySQL 8.0.18 were dumped and loaded into another database running MySQL 8.0.19, these boolean columns became TINYINT(4) columns. Existing databases were unaffected, but any new databases loading the old schemas signalled to applications that these columns were no longer boolean.
Moral: Always test the full cycle with any change including downstream practices like data transfer/replication
Networking - The Outside World Awaits
One of the most difficult contexts to emulate in testing is interfacing with networks. APIs with credentials that serve us in production should be secured from testing access. So how do we sufficiently TLYF while still abiding by network security best practices?
Start Yanking Cables
If network access is crucial, then taking the network down before and randomly during test execution provides critical insight into how your application handles these inevitable scenarios.
- Adjust response to degrade gracefully where possible
- Use gentle throttling to handle peaks
- Test on a variety of networks to detect hardcoded expectations
Log In as Production
If possible, use or at least test using the same cloud credentials as will be used in production. This will ensure the accessible “user” context is tested. Testing Like You Fly in this respect gives us the confidence in knowing that if we can hit production cloud services in the test environment, the odds are good that we can hit them in production as well. Make absolutely sure that any test system you establish cannot access the production system directly - protect the production environment from mutation during tests.
Get a Bigger Measuring Tape
In the age of cloud computing, physical distance between your servers is often overlooked. Consider the availability needs and data processed by the servers physically closest to the user locations they service.
- Failover to other regions within same provider
- Failover to other providers
- Alternate DNS servers for DNS lookup
Short but important considerations for all parties involved in launching a successful application. Next up is Timing.
