Test Like You Fly - Timing is (Almost) Everything

So far in the TLYF series we have covered the Principles, the Basics, and the Impact of the external components that support your mission. Lets expand on that with the one dimension we have little ability to stop - Time.

It's about time, it's about space, About two men in the strangest place. It's about time, it's about flight. Traveling faster than the speed of light.*

In the real world, not everything happens at once. Probably a good thing. And some activities take much longer than others. Like going to the office could mean a drive to work in heavy morning traffic, or putting on your slippers and walking into the other room.

Applications are no different. Requests take different amounts of time to respond. Operations occur in some order; not all at once nor always in parallel. And sometimes we only need to perform a task once - like swapping one database column for another during a migration.

Let's explore considerations from the Test Like You Fly perspective.

Time and Timing and Time Again

Speed

Performance means speed. Not only do tasks need to complete within some finite amount of time, but they need to meet reasonable expectations supporting user processes. Cascading and serial operations must complete in an overall duration that supports the business objectives. These may be difficult to reproduce in a test environment - especially if this is a recently inaugurated application without a reservoir of built up experience in real production data. However, it is absolutely necessary that the first time real production volumes of data are experienced NOT be in production.

• Test using hardware resources comparable to production
  • A migration may take 2 minutes on your own desktop and 10 minutes in the production environment
• Be clear about expected operation cycle times
  • Explore any and all deviations from anticipated activity durations
    • Do different user supplied parameters cause wide swings in operation cycle time?
  • For operations that are deemed too slow what sort of synchronicity is required?
  • Sometimes too fast an operation can induce race conditions
  • Variability can be the enemy
• Consider how long people will wait for inline actions and defer to background whenever sub-optimal
  • Ensure that background actions are idempotent and are failure resistant/tolerant or degrade robustly
  • If background queues build - will you become aware? How? What tolerance is there for this?

Timezones

Most applications today are used in more than a single time zone. In truth most applications of any size are globally accessible. From the perspective of the app's user they are in their own zone and most likely want to see all their information in that context. Databases typically are served in a consistent single time zone, with internal data stored with that time zone basis. This is usually UTC, but may be a different zone depending on the evolution of the app.

Ensure that different timezones are considered when determining dates from timestamps and date+time values. Time zone awareness, conversion between zones, and handling of notions like Summertime and Daylight Saving Time are critical as time is portrayed in the app. Calculations which cross DST boundaries (April and November currently) need to acknowledge the loss or gain of an hour. Transitioning across Year boundaries is another concern. The Space Shuttle Team was concerned about launching in one year and being in space when New Year's Eve transpired. It's critical to simulate all of this before putting the application in production. Be aware that not all timezones change their clocks on the same date. Is that handled? Is it even tested?

Weekends/Beginning and End of Hour/Day/Month/Year - crossover boundaries

• Specs may fail on boundaries where factory generated data crosses over a time "milestone"
• Test differing size months - span Jan/Feb/Mar with horizon of 30 days from Jan 31 to early March
  • Note: there is no June 31st.
  • Does your application assume a standard size month e.g. 30 days? If so errors may accumulate.
• Consider leap years - and for certain applications, leap seconds
  • What happens when the clock gets set back/forward?
  • What happens on Feb 29th?
  • How accurate do you have to be about time durations that span daylight saving time changes?
• Business days vs weekends (non-business days)
• Holidays - are they special? to be avoided? Are they locale centric?

Throttling

Applications that serve other applications, such as APIs, often have throttling built in to ensure they are not overwhelmed or artificially susceptible to Denial of Service attacks. When we rely on those APIs, it is imperative that we test our application's behavior when throttled by these external "forces".

• Controlling response rates can create queue build-up
• Does the throttling serve your app or your customer?

Sequencing

Getting ahead of ourselves, especially in the world of multi-core apps and serverless/microservices is a critical concern. Everything has an order. Order may be implicit or explicit. The more explicit the ordering - such as via creating pipelines of activities - the easier to ensure the sequences are maintained.

Test Independent and Sequential Operations

Many test frameworks randomize tests, which is important. But purposely running tests in a planned sequence that proves graceful processing in an out-of-sequence manner is also important. Do not trust that randomness will generate all combinations.

• Test calling same method repetitively to ensure they handle context properly without side effects
• Use immutability to ensure state is retained - improves correctness and makes for easier reasoning about local behavior

Test Parallel and Concurrent Operations

System complexity is at the heart of why TLYF is critical yet difficult. Isolating components from each other is the antithesis of this type of testing. This could lead to exponential combinatorial test scenarios. That could be a smell, or it could be a requirement.

Evaluate Reentrancy and Repeated Use Considerations

These are some of the most difficult areas to simulate, however perhaps also the ones that can unexpectedly be the most problematic. Methods that work successfully once, may fail on subsequent invocations merely because of prior state. Methods that alter their input parameters invisibly can behave fine, then fail the next time.

• Long-lived instances like background workers vs. short-lived objects from request/response controller actions
  • These long-lived workers need simulation runs over a long period of time or a focused stress test to ensure they will perform without side effects.
• Repeat methods calls with mutable state vs. fresh or immutable objects
  • Calling a method once is almost never real world behavior
  • Incorporate a test mechanism that ensures that your tests run multiple times in a random order with the modules still loaded into memory
  • Memory leakage/object accumulation can overwhelm process resources, but only if sufficiently exercised in the test environment
    • Monitor resource usage over time in your test environment to identify incremental growth

Migrations and Other One-Off Events

Single occurrence events are not just single first-time activities, but also single last-time activities. You may only get one chance in the production environment to get these correct. Much like single-execution missions (i.e. expendable rockets), migrations and other corrective actions often dissuade time spent on testing. After all, we only need this to be correct once.

• Use copies of real-world data for confirming prior to the “mission” that the migrations will process properly
• Rename database columns being retired and run specs to ensure they are not indirectly referenced before removal
• Add foreign key constraints wherever possible to mandate data integrity
• Provide logged evidence that migrations performed as expected
• Where critical one-time execution is required independently design inline confirmation testing
• Dry runs are great as long as they emulate the task completely

I hope you have found Part 4 on Time and Timing valuable. Next up in our TLYF series - "What Was Your Intent?"