Railway Pattern | Alchemists

 This pattern is coined from the Railway Oriented Programming presentation — and subsequent book: Domain Modeling Made Functional — by Scott Wlaschin which explains how to pipe functions together in a fault tolerant manner. The good news is Ruby is great at blending objects with functions for maximum effect. In order to learn how to apply this pattern within your own code, we’ll leverage the following foundational gems that make this pattern shine: - Dry Monads: Provides low level monad functionality. There are several types of monads included in this gem but we’ll focus on the Result monad. - Pipeable: Implements this pattern by building upon Ruby’s native function composition capabilities via the forward (#>>) and backward (#<<) composition methods. When coupled with Dry Monads, we get a clean and succinct Domain Specific Language (DSL) for piping (composing) multiple functions together as a single fault tolerant pipeline. In order to illustrate how powerful this pattern is, we’ll focus on building an implementation that can obtain information from an external third-party API service. The problem with external APIs is that they can be faulty, suffer from outages, network delays, have ugly data structures, and a slew of other problems which makes this pattern perfect for implementing an elegant solution to this common problem. - Fundamentals - Setup - Advantages - Disadvantages - Guidelines - Conclusion ## Fundamentals At a high level, one way to conceptualize this pattern is to think of train tracks which can either fork to the left (success) and right (failure). Alternatively, you can think of this pattern as a series of functional steps in a pipeline. The result of each step is either a success or failure. Even better, each success or failure can be transformed further before producing the final result. This is what makes a fault tolerant pipeline powerful because you have the opportunity to transform (heal) a failure by turning it into a success. To illustrate, here’s a visualization of a fault tolerant pipeline using Dry Monads Result syntax:  You don’t need to know what each method of a Result does to understand the above. You only need to see that, for each step within the pipeline, you’ll get a success or failure which can be transformed or left as is. This means there are multiple ways to yield a success or failure along with the opportunity to turn a failure into a success (i.e. fault tolerance). If it helps, you can also think of the Result monad as a specialized boolean: true (success) and false (failure). Unlike a boolean, a Result carries information which can be unwrapped for further processing and then rewrapped to produce an updated result. Definitely more powerful than a simple boolean yet not losing basic boolean functionality! ## Setup Let’s say we want to obtain Studio Ghibli films by implementing an API client. To pull this off, we need to: - Use the Studio Ghibli API, specifically the films endpoint. - Use an HTTP client to request and parse API requests. - Validate the JSON response. - Model the response as a new records for immediate interaction. Visually, this is:  As you can see, several steps are required for pulling this off. Additionally, we can have bad data, network issues, and potential exceptions along the way. In order to tackle these steps, we can start by building up a Bundler Inline script with the dependencies and objects we need for quick experimentation. Here are the dependencies: ```plain text #! /usr/bin/env ruby # frozen_string_literal: true # Save as `demo`, then `chmod 755 demo`, and run as `./demo`. require "bundler/inline" gemfile true do source "https://rubygems.org" gem "amazing_print" gem "debug" gem "http" gem "dry-monads" gem "dry-schema" gem "pipeable" end ``` Now we can add a few objects to the above script to implement our solution: - The following provides a low-level HTTP requester which is built upon the HTTP gem for making basic HTTP requests specifically tailored for the Ghibli API. We only need to make HTTP GET requests but, in practice, you’d have methods for POST, PUT, DELETE, and so forth. ```plain text module Ghibli module API class Requester def initialize uri: "https://ghibliapi.vercel.app", http: HTTP @uri = uri @http = http end def get(path) = http.get("#{uri}/#{path}") private attr_reader :uri, :http end end end ``` - Contract The following provides a Dry Schema contract for validating the HTTP response while also casting attributes to the their appropriate type. Due to the poor design of the Ghibli API which doesn’t account for additional metadata in the API response, the contract ensures all records be an array (value) of the :data key. The other reason is Dry Schema can’t handle a raw array without a key. ```plain text module Ghibli module API module Contracts Film = Dry::Schema.JSON do required(:data).array(:hash) do required(:id).filled :string required(:title).filled :string required(:original_title).filled :string required(:original_title_romanised).filled :string required(:description).filled :string required(:director).filled :string required(:producer).filled :string required(:release_date).filled :string required(:running_time).filled :string required(:rt_score).filled :string required(:people).filled :array required(:species).filled :array required(:locations).filled :array required(:vehicles).filled :array required(:url).filled :string end end end end end ``` - The following models validated data from our contract so we can immediately interact with record objects instead of a primitive hash. We use the .for method as a convenient solution map over the array of data entries and create new records. ```plain text module Ghibli module API module Models Film = Data.define( :id, :title, :original_title, :original_title_romanised, :description, :director, :producer, :release_date, :running_time, :rt_score, :people, :species, :locations, :vehicles, :url ) do def self.for(**attributes) attributes[:data].map { |entry| new(**entry) } end end end end end ``` - Client The following ties all of the above into our primary API client so we can make the HTTP GET request, parse the response, validate the response, and model the response as records we can immediately interact with. ```plain text module Ghibli module API class Client def initialize requester: Requester.new, contract: Contracts::Film, model: Models::Film @requester = requester @contract = contract @model = model end def films requester.get("films") .parse .then { |payload| contract.call(data: payload).to_h } .then { |attributes| model.for(**attributes) } end private attr_reader :requester, :contract, :model end end end ``` - The last thing we need to is initialized the client and message the #films method: ```plain text client = Ghibli::API::Client.new films = client.films ``` You should find that films is an array of film objects. Example (truncated for brevity): ```plain text [ #<data Ghibli::API::Models::Film>, #<data Ghibli::API::Models::Film>, #<data Ghibli::API::Models::Film> ] ``` These objects are the core building blocks when designing an API client because you need to: - Make HTTP requests to the API server. - Validate the HTTP responses since you sometimes can’t rely on the data being correct or always in the same structure. - Model the data so you have a quick way to interact with structured objects instead of raw hashes which avoids the Primitive Obsession code smell. As you follow along, take the time to copy and paste the above code into a Bundler Inline script so you can quickly run the code. This’ll come in handy as we tweak and evolve each of these objects into our final solution. ## Approaches Now that you have all of the objects necessary for implementing our API client we can discuss the various approaches to solving the problem. ### Naive The naive approach is exactly what you see in the Ghibli::API::Client#films implementation which has several problems. Here’s the code, again, for convenience: ```plain text requester.get("films") .parse .then { |payload| contract.call(data: payload).to_h } .then { |attributes| model.for(**attributes) } ``` If any of the above steps fail, we’ll get an exception with a stack dump. Worse, each step assumes the previous step was successful with no way to deal with errors, invalid data, etc. This approach only solves the happy path at the cost of being brittle due to not being fault tolerant. ### Traditional The traditional approach is to use conditional and exception logic. Here’s what this looks like after updating our Ghibli::API::Client#films implementation: ```plain text def films result = validate parse case result in data: Array then model.for(**result) else result end end private attr_reader :requester, :contract, :model def parse response = requester.get "films" body = response.parse response.status.success? ? body : {error: body} rescue HTTP::Error, JSON::ParserError => error {error: error.message} end def validate payload return payload if payload.is_a? Hash result = contract.call data: payload result.success? ? result.to_h : {error: result.errors.to_h} end ``` If you apply the above changes and run the code again, you’ll get the same result as before. Conditionals and rescues are introduced to make the code more robust so we get a films array or an error hash. Definitely an improvement at the cost of becoming more complex due to extra conditional logic and use of array (films) or hash (error) type checking. Definitely not ideal. There is one major improvement with this implementation which is we are starting to see the beginning of a fault tolerant pipeline. This means each step (i.e. request, parse, validate, and model) is robust enough to either produce the desired result of film records or an error. If an error is encountered in any of the steps, then the error bubbles up so we can do further processing of the result as desired. We can do better by switching to monads which brings us to a more functional approach. ### Functional For this approach, we need to make a few modifications to our original objects. Here’s the changes: - The following modifies the original implementation so that we include the Dry Monads Result type and modify #get so it answers either a Success or Failure. This includes capturing any HTTP::Error as a failure as well. ```plain text class Requester include Dry::Monads[:result] def initialize uri: "https://ghibliapi.vercel.app", http: HTTP @uri = uri @http = http end def get path http.get("#{uri}/#{path}").then do |response| response.status.success? ? Success(response) : Failure(response) end rescue HTTP::Error => error Failure error.message end private attr_reader :uri, :http end ``` - For our contract, we only need to teach it how to enable monad support by loading the monads extension as follows: ```plain text Dry::Schema.load_extensions :monads ``` - The following updates the #films method to use the Fluent Interface of Dry Monads while deleting all of the previously added private methods. ```plain text class Client def initialize requester: Requester.new, contract: Contracts::Film, model: Models::Film @requester = requester @contract = contract @model = model end def films requester.get("films") .fmap(&:parse) .bind { |payload| contract.call(data: payload).to_monad } .fmap { |result| model.for(**result.to_h) } rescue JSON::ParserError => error Failure error: error.message end private attr_reader :requester, :contract, :model end ``` In case you are not familiar with how the Result monad works, here’s a breakdown of each step: 1. requester.get("films"): Answers either a Success or Failure because we taught the Requester to always answer a monad. 2. fmap(&:parse): The #fmap method is short for function map and means the block will only be called if the result is a Success, otherwise any Failure passes on to the next step. So when we have a Success, that success will be automatically unwrapped to pass the HTTP response to the block which we can then parse. Once parsed, the result (i.e. JSON body) will automatically be wrapped back up as a Success to pass on to the next step. 3. bind { |payload| contract.call(data: payload).to_monad }: The #bind method only executes the block if the result is a Success. On Success, the contract will be called to answer a monad via the #to_monad message and — this is important — bind doesn’t automatically wrap the result of the block as a Success which means that when #to_monad is called we’ll answer back either a Success or Failure. We still have to account for JSON parsing errors which is why the rescue is still in the #films method. In this case, we wrap the hash as a Failure with an error key and string message. You could also wrap the entire error as a failure (i.e. Failure error:) in order for you to use more advanced Pattern Matching due to knowing the type of error that caused this failure instead of a simple string as the error value. For the purposes of this implementation, we’ll stick with a string. If you apply the above changes and run the code, you’ll see films is wrapped as a Success. Example (truncated for brevity): ```plain text Success( [ #<data Ghibli::API::Models::Film>, #<data Ghibli::API::Models::Film>, #<data Ghibli::API::Models::Film> ] ) ``` This means we can use Pattern Matching to process the result: ```plain text include Dry::Monads[:result] client = Ghibli::API::Client.new case client.films in Success(*films) then puts films in Failure(error: message) then puts "ERROR: #{message}" else puts "Unknown type." end ``` We now have the beginnings of a functional flow where each step expects either a Success or Failure. If any of the steps fail, then the failure gracefully passes through all of the subsequent steps for further processing or skipped altogether. At this point, we could stop here because we have an implementation that adheres to the Railway Pattern. Each step will either continues down the Success track of the railway or forks to end up as a failure. We can do better, which leads us to our final approach that fully implements the Railway Pattern. ### Railway We only need to update our Client to use the Pipeable gem which allows us to use Function Composition. Here’s the updated implementation: ```plain text class Client include Pipeable def initialize requester: Requester.new, contract: Contracts::Film, model: Models::Film @requester = requester @contract = contract @model = model end def films pipe requester.get("films"), try(:parse, catch: JSON::ParserError), merge(as: :data), validate(contract, as: :to_h), to(model, :for) end private attr_reader :requester, :contract, :model end ``` Two important changes to call out are: - We include Pipeable functionality at the top of the class. - The #films method has been updated to pipe all steps as one continuous pipeline. As with the monad approach, if any of the steps fails, the Failure will pass through all subsequent steps without further operation or exception. If you’ve not used the Pipeable gem before, the gem allows you to pipe (compose) a series of functional steps together as a single operation. In this case, we are using the default steps that come with the gem but you can build your own custom steps too. Here’s what each step is doing (which is similar to what we did in the monads approach only with a concise syntax): 1. We start by using pipe to supply all steps. The first step expects a Result. If not a Result it’ll be automatically wrapped as a Success in order to kick off processing (handy when you have a raw value as input without having to wrap it first). 2. The next step tries to parse the HTTP result. Any errors will be caught and wrapped as Failures. Otherwise, the result is a wrapped as a Success. 3. The next step merges the array of films into a hash with data as the key because the contract requires a hash to validate. 4. The next step validates the payload using the injected contract. Upon success, it’ll be cast as a Hash. 5. Finally, the last step will delegate the result of the previous step (i.e. the validated payload) to the injected model using the .for method to convert the payload of data into actual Film records. For Elm fans, the pipe method becomes the pipe operator (i.e. |>) which would yield this blended Elm/Ruby pseudo code: ```plain text requester.get("films") |> try(:parse, catch: JSON::ParserError) |> merge(as: :data) |> validate(contract, as: :to_h) |> to(model, :for) ``` Make no mistake, the above is syntactically incorrect Elm code but the shape is similar to how’d you implement a pipeline in Elm. With Ruby, we don’t have a pipe operator so, by using Pipeable, we can achieve the desired effect using the pipe method. Each step ends up being composable using Function Composition courtesy of the forward composition method (#>>) which is available to procs, lambdas, methods, and — with Pipeable — classes. 🎉 ## Advantages There are several advantages to this pattern: - You can pipe multiple steps together as a single, fault tolerant, operation. - Each step can produce a success or failure for which you can transform and/or repair further. - You can build super pipes which allows you to chain multiple pipes together with their own series of steps. - You can interchangeably compose procs, lambdas, methods, and classes together which makes blending functions with objects seamless. - You can apply Pattern Matching to process the final result to powerful effect. ## Disadvantages There can be a few disadvantages to this pattern depending on your situation: - Requires gem dependencies since Ruby doesn’t have native support. - Each pipeline answers a monad which you may or may not always want. That said, Dry Monads does provide convenience methods, such as #value!, to get around this. - Can degrade performance especially in hot code paths so monitor, measure, and adjust accordingly. ## Guidelines When using this pattern, there are a few guidelines worth noting: - Use to gracefully bubble up errors, problems, and/or data you can’t repair or massage into a success. There are times, like capturing a JSON parse error, and bubbling it up as a failure where the failure is not catastrophic but worth informing the user so they can take corrective action. Other times, you might want to capture an HTTP timeout exception as a failure but then retry the HTTP request with an exponential backoff after three attempts to make your pipeline more fault tolerant. - Use to build more advanced architectures where you can mix and chain multiple pipes together as super pipes. - Avoid when needing to fail fast. Letting an exception through or even exiting the program can be useful for catastrophic errors that need immediate attention. - Avoid when not needing to process any failures, don’t need to chain pipelines together (i.e. super pipes), or don’t need to unwrap the result each time. - Avoid this pattern if you need to be highly performant. Both the Pipeable and Dry Monads gems introduce some overhead which may not be desired in all code paths. ## Conclusion If you take a moment to look at where we started with the naive approach and compare that to the final railway solution, you see the steps are, roughly, the same: