Permissive, then restrictive: concrete solutions and examples

May 03, 2020
« Previous post   Next post »

This post is a short addendum to my last post about how a better way to learn to design Haskell is to make your code permissive first, restrictive later. If you haven't already seen that, read that first.


So now you're convinced (or at least, pretending to be for this post) that starting permissive is the way to go. But even if you get the gist of the philosophy, what does it actually mean? Concretely, what are you supposed to do in this or that situation? And how do you even recognize when you're working with a problem that's best solved by moving up or down the permissiveness 'scale'?

Relax. We're going to take a look at those specific problems where the philosophy applies, list out what some possible solutions are, and get a 'progression' of how to gradually improve both our code and our understanding.

Each situation is presented as a specific technical issue you might have, followed by a list of potential fixes. To ease lookup, each issue is a general, high-level problem like "I need to add logging." The fixes underneath each issue are ordered by restrictiveness; at the top are the most permissive choices that will get you to working code fast, but leave holes for bugs to crawl in, or make your code less maintainable, or both. As you go down, the choices either become more restrictive and give you more guarantees about your code, or give you the same guarantees but with more maintainability, with the 'ideal' solution(s) at the bottom.

The idea is that, as a learner, you should start with the topmost solution for each problem, and gradually work your way downwards. That might mean refactoring your existing projects to use a different choice of solution, or it might mean trying a lower-down solution in some new code or project.

Many of the solutions will look very similar to each other; this is intentional, as a lot of these issues tend to shake out as just special cases of side effects, for which Haskell has many competing choices.


  1. You need a global config that everything can access
  2. You need mutability
  3. You need to handle errors/exceptions
  4. You need to add logging (to a file, to a third-party service, etc.)
  5. You need to access an external service or database
  6. You need to interleave IO/impurity with your logic
  7. You need to work with data that can be in different shapes
  8. All your functions are in IO now. How do you test anything?

You need a global config that everything can access

[toc]
  • Have all your functions return IO. Create a toplevel IORef to hold your config using unsafePerformIO.

    import Data.IORef
    import System.IO.Unsafe ( unsafePerformIO )
    
    {-# NOINLINE config #-}
    config :: IORef Text
    config = unsafePerformIO (newIORef "")
    
    main :: IO ()
    main = do
      writeIORef config "olvers"
      ...
    
    someOtherFunction :: Int -> IO Int
    someOtherFunction x = do
      cfg <- readIORef config
      ...
  • Instead of having your functions return IO, have them return a Reader instead. Use that to pass your config around.

    import Control.Monad.Trans.Reader as Reader
    
    main :: IO ()
    main = do
      let result = runReader (someOtherFunction 5) "olvers"
      ...
    
    someOtherFunction :: Int -> Reader Text Int
    someOtherFunction x = do
      cfg <- Reader.ask
      ...
  • Upgrade to using the ReaderT monad transformer + MonadReader instead, or use some sort of algebraic effect system like freer-simple, fused-effects, or polysemy. This way you can combine with other side effects that you need more easily. Be warned that this can be a huge endeavour.

  • OR, just pass your config around as a normal function parameter.

You need mutability

[toc]
  • Have all your functions return IO. Create an IORef somewhere and pass it between your functions. Update said IORef when you need to.

  • Instead of having your functions return IO, have them return a State monad instead. Use that to hold your mutable variable.

    import Control.Monad.Trans.State.Strict as State
    
    main :: IO ()
    main = do
      let result = evalState (someOtherFunction "olvers") 5
      ...
    
    someOtherFunction :: Text -> State Int Text
    someOtherFunction str = do
      count <- State.get
      State.put (count + 1)
      if even count
        then pure str
        else pure (Text.reverse str)
  • Upgrade to using the StateT monad transformer + MonadState instead, or use some sort of algebraic effect system like freer-simple, fused-effects, or polysemy. This way you can combine with other side effects that you need more easily. Be warned that this can be a huge endeavour.

  • OR, ask yourself very carefully whether you need mutability at all. See if you can't solve the problem by just passing around updated data structures.

You need to handle errors/exceptions

[toc]

That's it. That's the only solution I'll provide here for this problem. The main thing I wanted to make clear is that runtime exceptions are not something to avoid. There's no need to bang your head trying to thread Maybe throughout your entire program. You can throw exceptions from anywhere in your code without any modifications necessary, so if you have something that's best handled in main or some higher-level function, just throw an exception now and worry about designing things better later.

You need to add logging (to a file, to a third-party service, etc.)

[toc]
  • Have all your functions that need instrumentation return IO. Use the fast-logger package and just stick log statements wherever you need.

  • Create a record of logging functions for different log levels at the toplevel of your program. Pass it to all your functions. Instead of having your functions return IO, have them return an abstract monad m.

    data LogFunctions m = LogFunctions
      { logInfo :: Text -> m ()
      , logWarn :: Text -> m ()
      , logError :: Text -> m ()
      }
    
    main :: IO ()
    main = do
      info <- mkInfoLogFunction :: IO (Text -> IO ())
      warn <- mkWarnLogFunction :: IO (Text -> IO ())
      error <- mkErrorLogFunction :: IO (Text -> IO ())
    
      let logger = LogFunctions info warn error
    
      someInstrumentedFunction logger 4
    
    -- we could also use a Reader to hold our record of functions
    -- if we didn't want to have to explicitly pass the record
    -- through every function in our stack
    someInstrumentedFunction :: Monad m => LogFunctions m -> Int -> m Text
    someInstrumentedFunction logger x = do
      logInfo logger "entering someInstrumentedFunction"
      if x > 5 then
        logWarn logger "x too large"
      ...

    How does this help? Notice someInstrumentedFunction has no knowledge of what specific monad it's working in. This prevents your code from having any side effects other than the ones available in your record of functions.

  • Use either monad transformers + monad-logger, or define your own logging effect in some sort of algebraic effect system like freer-simple, fused-effects, or polysemy. This way you can combine with other side effects that you need more easily. Be warned that this can be a huge endeavour.

You need to access an external service or database

[toc]
  • Have all your functions that need access to said external service return IO. Go wild with API calls, database queries, whatever.

  • Figure out what API calls/DB queries/etc. your code needs, and make specific functions for them. Create a record of those functions at the toplevel of your program. Pass it to all your core logic. Instead of having your core logic functions return IO, have them return an abstract monad m. See the section on solving logging for an example.

  • Use either monad transformers + monad-logger, or define your own effect for your external service in some sort of algebraic effect system like freer-simple, fused-effects, or polysemy. This way you can combine with other side effects that you need more easily. Be warned that this can be a huge endeavour.

You need to interleave IO/impurity with your logic

[toc]

An example of something like this might be an open-world game; you need to constantly load in new level data as the player moves around, but you want the world update logic, like collision, movement etc. to be pure so that you can test it easily.

  • Have all your functions return IO. Just put side-effects wherever you need them.

  • Figure out what side-effecting abilities your code needs constantly, and make specific functions for them. Create a record of those functions at the toplevel of your program. Pass it to all your core logic. Instead of having your core logic functions return IO, have them return an abstract monad m. See the section on solving logging for an example.

  • Break your code up into pure "handlers" that are orchestrated by impure "drivers." This is a little abstract, but let's look at the example of an open-world game. Your first thought might be to have a core function like:

    updateWorld :: Action -> GameState -> GameState
    updateWorld action state = ...

    But if this is the only thing responsible for updating the game state, then we have the issue that we can't do I/O in response to things happening in-game.

    Instead, we can have our update function also return data structures representing what side effects it needs to have happen, and then take in any external data produced.

    data Action
      = LoadLevelChunk ChunkID
      | AutoSave
    
    updateWorld :: [ChunkData] -> Action -> GameState -> (GameState, [Action])
    updateWorld levelData action state = ...

    Now our updateWorld can signal when it needs to load some level data by returning a list containing LoadLevelChunks. Notice that this function is still pure: it's not actually doing anything, just saying what it needs done.1

    Then we'd write some kind of impure "driver" that just does what updateWorld tells it to: feed the current gamestate to updateWorld, run any actions that get returned, feed that data into the next loop of updateWorld, and repeat.

    Our impure side-effecting code is completely separated from our pure game logic. We get all the benefits of pure code, in particular the ability to run tests on our game logic very easily, without sacrificing lazy loading.

    An example of this pattern in action is the Elm architecture; Elm webapps are built entirely through pure logic that simply signals what inputs it needs from the browser.

You need to work with data that can be in different shapes

[toc]

Like I mentioned in the previous article, we'll be using JSON as the main example here, but the general idea applies equally well to other formats like key-value pairs or XML.

  • Just parse into and pass around the raw data type for that format. JSON Values, a raw XML type, a Map String String for key-value pairs. When you need to use the data, do runtime checks to pull out the field you need.

  • As you gain more knowledge of the constraints on the data, pull out well-understood fields into a custom datatype; leave things that you have less confidence in in the raw type.

    Note that the goal is not necessarily to pull all the fields you use out into a custom type, only to add as much type safety as is appropriate to your specific domain. If you find that an API changes its format frequently, or the requirements for your code get shifted, it's perfectly reasonable to leave things relatively 'untyped'.

Okay, I took all this advice and all my functions are in IO. How do I test anything?!

[toc]
  • Figure out what side-effects your code is using, and make specific functions for them (specific DB queries, API calls, file access etc.). Create a record of those functions at the toplevel of your program. Pass it to all your core logic. Instead of having your core logic functions return IO, have them return an abstract monad m. See the section on solving logging for an example.

    Once you've done this, create stub or mock versions of your side-effect functions. Use those in your function record instead in your test harness.

  • Go through your code and remove IO from as many functions as you can. Push side effects up the stack, pass down needed data as parameters. Refactor out as much pure logic as you can, and test that pure logic the normal way.

  • For everything that still has to be impure, refactor to use some sort of algebraic effect system like freer-simple, fused-effects, or polysemy. Write separate testing interpreters for your effects. Be warned that this can be a huge endeavour; it's also not at all necessary. The previous two solutions together are good enough for 95% of cases.


Just this set of situations alone can form a pretty decent process for writing good Haskell: when starting a project, use the topmost solutions, then walk down each list's solutions and refactor to use them. As you gain experience, you'll start to see when it's not necessary to go all the way to the extreme of maximum permissiveness; oh, I can just use a Reader in this situation; oh, I know enough about what this data looks like to model it in a type right from the start. And, you'll be able to go the opposite direction: to see when you've accidentally made your code too restrictive and know exactly how much to loosen things.

As you understand the language better, one thing to note is that these permissiveness scales aren't just for learning; they're also a useful design tool in their own right. A good way to design libraries is to build a maximally permissive 'internal' or 'low-level' interface first, then build more typed interfaces on top of that to give you more guarantees. Having the low-level interface makes sure you don't get stuck and have to redesign everything, while having the high-level interface makes sure you don't make mistakes while actually using your library.2 It turns out knowing how to move around on the permissiveness scale permeates Haskell.

Have a problem that you think fits this post, but not sure how to approach it? Think there's a common permissiveness vs. restrictiveness situation that's missing from this post? Found this useful, or otherwise have comments or questions? Talk to me!

« Previous post   Next post »

Before you close that tab...


Footnotes

↥1 Since we’re using video games as an example, it’s worth mentioning two other sources of impurity that come up often in games: randomness and time.

Both of these can be solved in exactly the same way: just represent the “need” for a random number or getting the current time as an action like GetRandomNumberBetween 5 10 or GetCurrentTime.

↥2 Some examples of libraries that embody this:

  • WAI vs Servant

    Where WAI provides an extremely loose definition of a webapp as anything that takes in an HTTP request and produces an HTTP response, Servant builds on top of WAI and locks down that looseness to better model typical real-world usage in APIs and websites.

    WAI says: “Want to make a URL endpoint that decides whether it wants to serve HTML or JSON based on the time of day? Want to use underscores as your URL path separator? Yeah, that’s totally doable.”

    Servant says: “No, that’s stupid, we make normal endpoints that have deterministic content types here.”

  • postgresql-simple vs Opaleye/Beam/Esqueleto

    All the typed database access libraries build on top of the lower-level libraries like hasql, postgresql-simple and friends. In doing so, they provide more compile-time guarantees, like guaranteeing that your query results can be correctly parsed into your Haskell types and vice versa, that you’re not sending malformed SQL to the database, and so on.