The letter A styled as Alchemists logo. lchemists
Brooke Kuhlmann's Avatar Brooke Kuhlmann
Published March 3, 2024 Updated April 23, 2024
Pipeable Icon

Pipeable

0.3.0

Circe CI Status. Alchemists Style Guide. Git Badge Sponsor Badge

A DSL for workflows built atop native Function Composition which leverages the Railway Pattern. This allows you to write a sequence of steps that cleanly read from left-to-right or top-to-bottom which results in a success or a failure without having to rely on exceptions which are expensive and should not be used for control flow.

Features

Requirements

  1. Ruby.

  2. A strong understanding of Function Composition.

Setup

To install with security, run:

# 💡 Skip this line if you already have the public certificate installed.
gem cert --add <(curl --compressed --location https://alchemists.io/gems.pem)
gem install pipeable --trust-policy HighSecurity

To install without security, run:

gem install pipeable

You can also add the gem directly to your project:

bundle add pipeable

Once the gem is installed, you only need to require it:

require "pipeable"

Usage

You can turn any object into a transaction by requiring and including this gem as follows:

require "csv"
require "pipeable"

class Demo
  include Pipeable

  def initialize client: CSV
    @client = client
  end

  def call data
    pipe data,
         check(/Book.+Price/, :match?),
         :parse,
         map { |item| "#{item[:book]}: #{item[:price]}" }
  end

  private

  attr_reader :client

  def parse result
    result.fmap do |data|
      client.instance(data, headers: true, header_converters: proc { |key| key.downcase.to_sym })
            .to_a
            .map(&:to_h)
    end
  end
end

The above allows Demo#call to be a sequence steps which may pass or fail due to all step being Dry Monads. This is the essence of the Railway Pattern.

To execute the above example, you’d only need to pass CSV content to it:

Demo.new.call <<~CSV
  Book,Author,Price,At
  Mystics,urGoh,10.50,2022-01-01
  Skeksis,skekSil,20.75,2022-02-13
CSV

The computed result is a success with each book listing a price:

Success ["Mystics: 10.50", "Skeksis: 20.75"]

Pipe

Once you’ve included the Pipeable module within your class, the #pipe method is available to you and is how you build a sequence of steps for processing. The method signature is:

pipe(input, *steps)

The first argument is your input which can be a Ruby primitive or a monad. Regardless, the input will be automatically wrapped as a Success — but only if not a Result to begin with — before passing to the first step. From there, all steps are required to answer a monad in order to adhere to the Railway Pattern.

Behind the scenes, the #pipe method is syntactic sugar on top of Function Composition which means if this code were to be rewritten:

pipe csv,
     check(/Book.+Price/, :match?),
     :parse,
     map { |item| "#{item[:book]}: #{item[:price]}" }

Then the above would look like this using native Ruby:

(
  check(/Book.+Price/, :match?) >>
  method(:parse) >>
  map { |item| "#{item[:book]}: #{item[:price]}" }
).call Success(csv)

The problem with native function composition is that it reads backwards by passing your input at the end of all sequential steps. With the #pipe method, you have the benefit of allowing your eye to read the code from top to bottom in addition to not having to type multiple forward composition operators.

Steps

There are several ways to compose steps for your pipe. As long as all steps succeed, you’ll get a successful response. Otherwise, the first step to fail will pass the failure down by skipping all subsequent steps (unless you dynamically attempt to turn the failure into a success). The following sections detail how to mix and match steps for building a robust implementation.

Basic

The following are the basic (default) steps for building for more advanced functionality.

As

Allows you to message the input as different output. Example:

pipe :a, as(:inspect)                  # Success ":a"
pipe %i[a b c], as(:dig, 1)            # Success :b
pipe Failure("Danger!"), as(:inspect)  # Failure "Danger!"
Bind

Allows you to perform operations on a successful result only. You are then responsible for answering a success or failure accordingly. This is a convenience wrapper to native Dry Monads #bind functionality. Example:

pipe %i[a b c], bind { |input| Success input.join("-") }           # Success "a-b-c"
pipe %i[a b c], bind { |input| Failure input }                     # Failure [:a, :b, :c]
pipe Failure("Danger!"), bind { |input| Success input.join("-") }  # Failure "Danger!"
Check

Allows you to check if the input and messaged object evaluate to true or Success. When successful, input is passed through as a Success. When false, input is passed through as a Failure. Example:

pipe :a, check(%i[a b], :include?)                  # Success :a
pipe :a, check(%i[b c], :include?)                  # Failure :a
pipe Failure("Danger!"), check(%i[a b], :include?)  # Failure "Danger!"
Fmap

Allows you to unwrap a successful operation, make a modification, and rewrap the modification as a new success. This is a convenience wrapper to native Dry Monads #fmap functionality. Example:

pipe %i[a b c], fmap { |input| input.join "-" }           # Success "a-b-c"
pipe Failure("Danger!"), fmap { |input| input.join "-" }  # Failure "Danger!"
Insert

Allows you to insert an element after the input (default behavior) and wraps native Array#insert functionality. If the input is not an array, it will be cast as one. You can use the :at key to specify where you want insertion to happen. This step is most useful when needing to assemble arguments for passing to a subsequent step. Example:

pipe :a, insert(:b)                  # Success [:a, :b]
pipe :a, insert(:b, at: 0)           # Success [:b, :a]
pipe %i[a c], insert(:b, at: 1)      # Success [:a, :b, :c]
pipe Failure("Danger!"), insert(:b)  # Failure "Danger!"
Map

Allows you to map over an enumerable and wraps native Enumerable#map functionality.

pipe %i[a b c], map(&:inspect)           # Success [":a", ":b", ":c"]
pipe Failure("Danger!"), map(&:inspect)  # Failure "Danger!"
Merge

Allows you to merge the input with additional attributes as a single hash. If the input is not a hash, then the input will be merged with the attributes using step as the key. The default step key can be renamed to a different key by using the :as key. Like the Insert step, this is most useful when needing to assemble arguments and/or data for consumption by subsequent steps. Example:

pipe({a: 1}, merge(b: 2))             # Success {a: 1, b: 2}
pipe "test", merge(b: 2)              # Success {step: "test", b: 2}
pipe "test", merge(as: :a, b: 2)      # Success {a: "test", b: 2}
pipe Failure("Danger!"), merge(b: 2)  # Failure "Danger!"
Orr

Allows you to operate on a failure and produce either a success or another failure. This is a convenience wrapper to native Dry Monads #or functionality.

ℹ️ Syntactically, or can’t be used for this step since or is a native Ruby keyword so orr is used instead.

Example:

pipe %i[a b c], orr { |input| Success input.join("-") }          # Success [:a, :b, :c]
pipe Failure("Danger!"), orr { Success "Resolved" }              # Success "Resolved"
pipe Failure("Danger!"), orr { |input| Failure "Big #{input}" }  # Failure "Big Danger!"
Tee

Allows you to run an operation and ignore the response while input is passed through as output. This behavior is similar in nature to the tee program in Bash. Example:

pipe "test", tee(Kernel, :puts, "Example.")

# Example.
# Success "test"

pipe Failure("Danger!"), tee(Kernel, :puts, "Example.")

# Example.
# Failure "Danger!"
To

Allows you to delegate to an object — which doesn’t have a callable interface and may or may not answer a result — for processing of input. If the response is not a monad, it’ll be automatically wrapped as a Success. Example:

Model = Struct.new :label, keyword_init: true do
  include Dry::Monads[:result]

  def self.for(...) = Success new(...)
end

pipe({label: "Test"}, to(Model, :for))    # Success #<struct Model label="Test">
pipe Failure("Danger!"), to(Model, :for)  # Failure "Danger!"
Try

Allows you to try an operation which may fail while catching the exception as a failure for further processing. Example:

pipe "test", try(:to_json, catch: JSON::ParserError)     # Success "\"test\""
pipe "test", try(:invalid, catch: NoMethodError)         # Failure "undefined method..."
pipe Failure("Danger!"), try(:to_json, catch: JSON::ParserError)  # Failure "Danger!"
Use

Allows you to use another transaction which might have multiple steps of it’s own, use an object that adheres to the Command Pattern, or any function which answers a Dry Monads Result object. In other words, you can use use any object which responds to #call and answers a Dry Monads Result object. This is great for chaining multiple transactions together.

function = -> input { Success input * 3 }

pipe 3, use(function)                   # Success 9
pipe Failure("Danger!"), use(function)  # Failure "Danger!"
Validate

Allows you to use an operation that will validate the input. This is especially useful when using Dry Schema, Dry Validation, or any operation that can respond to #call while answering a result that can be converted into a hash.

By default, the :as key uses :to_h as it’s value so you get automatic casting to a Hash. Use nil, as the value, to disable this behavior. You can also pass in any value to the :as key which is a valid method that the result will respond to.

schema = Dry::Schema.Params { required(:label).filled :string }

pipe({label: "Test"}, validate(schema))           # Success label: "Test"
pipe({label: "Test"}, validate(schema, as: nil))  # Success #<Dry::Schema::Result{:label=>"Test"} errors={} path=[]>
pipe Failure("Danger!"), validate(schema)         # Failure "Danger!"

Advanced

Several options are available should you need to advance beyond the basic steps. Each is described in detail below.

Procs

You can always use a Proc as a custom step. Example:

include Pipeable
include Dry::Monads[:result]

pipe :a,
     insert(:b),
     proc { Success "input_ignored" },
     as(:to_sym)

# Yields: Success :input_ignored
Lambdas

In addition to procs, lambdas can be used too. Example:

include Pipeable

pipe :a,
     insert(:b),
     -> result { result.fmap { |input| input.join "_" } },
     as(:to_sym)

# Yields: Success :a_b
Methods

Methods — in addition to procs and lambdas — are the preferred way to add custom steps due to the concise syntax. Example:

class Demo
  include Pipeable

  def call input
    pipe :a,
         insert(:b),
         :join,
         as(:to_sym)
  end

  private

  def join(result) = result.fmap { |input| input.join "_" }
end

Demo.new.call :a  # Yields: Success :a_b

All methods can be referenced by symbol as shown via :join above. Using a symbol is syntactic sugar for Object#method so the use of the :join symbol is the same as using method(:join). Both work but the former requires less typing than the latter.

Custom

If you’d like to define permanent and reusable steps, you can register a custom step which requires you to:

  1. Define a custom step as a new class.

  2. Register your custom step along side the existing default steps.

Here’s what this would look like:

module CustomSteps
  class Join < Pipeable::Steps::Abstract
    def initialize(delimiter = "_", **)
      super(**)
      @delimiter = delimiter
    end

    def call(result) = result.fmap { |input| input.join delimiter }

    private

    attr_reader :delimiter
  end
end

Pipeable::Steps::Container.register :join, CustomSteps::Join

include Pipeable

pipe :a, insert(:b), join, as(:to_sym)
# Yields: Success :a_b

pipe :a, insert(:b), join(""), as(:to_sym)
# Yields: Success :ab

Containers

Should you not want the basic steps, need custom steps, or a hybrid of default and custom steps, you can define your own container and provide it as an argument to .[] when including pipeable behavior. Example:

require "containable"

module CustomContainer
  extend Containable

  register :echo, -> result { result }
  register :insert, Pipeable::Steps::Insert
end

include Pipeable[CustomContainer]

pipe :a, echo, insert(:b)

# Yields: Success [:a, :b]

The above is a hybrid example where the CustomContainer registers a custom echo step along with the default insert step to make a new container. This is included when passed in as an argument via .[] (i.e. include Pipeable[CustomContainer]).

Whether you use default, custom, or hybrid steps, you have maximum flexibility using this approach.

Composition

Should you ever need to make a plain old Ruby object functionally composable, then you can include the Pipeable::Composable module which will give you the necessary #>>, #<<, and #call methods where you only need to implement the #call method.

Development

To contribute, run:

git clone https://github.com/bkuhlmann/pipeable
cd pipeable
bin/setup

You can also use the IRB console for direct access to all objects:

bin/console

Architecture

The architecture of this gem is built on top of the following concepts and gems:

  • Function Composition: Made possible through the use of the #>> and #<< methods on the Method and Proc objects.

  • Containable: Allows related dependencies to be grouped together for injection as desired.

  • Dry Monads: Critical to ensuring the entire pipeline of steps adhere to the Railway Pattern and leans heavily on the Result object.

  • Marameters: Through the use of the .categorize method, dynamic message passing is possible by inspecting the operation method’s parameters.

Style Guide

  • Transactions

    • Use a single method (i.e. #call) which is public and adheres to the Command Pattern so transactions can be piped together if desired.

  • Steps

    • Inherit from the Abstract class to gain monad, composition, and dependency behavior. This allows subclasses to have direct access to the base positional, keyword, and block arguments. These variables are prefixed with base_* in order to not conflict with subclasses which might only want to use non-prefixed variables for convenience.

    • All filtered arguments — in other words, the unused arguments — need to be passed up to the superclass from the subclass (i.e. super(*positionals, **keywords, &block)). Doing so allows the superclass (i.e. Abstract) to provide access to base_positionals, base_keywords, and base_block for use if desired by the subclass.

    • The #call method must define a single positional result parameter since a monad will be passed as an argument. Example: def call(result) = # Implementation.

    • Each block within the #call method should use the input parameter to be consistent. More specific parameters like argument or operation should be used to improve readability when possible. Example: def call(result) = result.bind { |input| # Implementation }.

    • Use implicit blocks sparingly. Most of the default steps shy away from using blocks because it can make the code more complex. Use private methods, custom steps, and/or separate transactions if the code becomes too complex because you might have a smaller object which needs extraction.

Debugging

If you need to debug (i.e. Debug) your pipe, use a lambda. Example:

pipe data,
     check(/Book.+Price/, :match?),
     -> result { binding.break },  # Breakpoint
     :parse

The above breakpoint will allow you inspect the result of the #check step and/or build a modified result for passing to the subsequent :parse method step.

Troubleshooting

The following might be of aid to as you implement your own transactions.

Type Errors

If you get a TypeError: Step must be functionally composable and answer a monad, it means:

  1. The step must be a Proc, Method, or some object which responds to #>>, #<<, and #call.

  2. The step doesn’t answer a result monad (i.e. Success some_value or Failure some_value).

No Method Errors

If you get a NoMethodError: undefined method `success? exception, this might mean that you forgot to add a comma after one of your steps. Example:

# Valid
pipe "https://www.wikipedia.org",
     to(client, :get),
     try(:parse, catch: HTTP::Error)

# Invalid
pipe "https://www.wikipedia.org",
     to(client, :get)  # Missing comma.
     try(:parse, catch: HTTP::Error)

Tests

To test, run:

bin/rake

Benchmarks

To view/compare performance, run:

bin/benchmark

💡 You can view current benchmarks at the end of the above file if you don’t want to manually run them.

License

Security

Code of Conduct

Contributions

Versions

Community

Credits