ExState: Database-backed statecharts for Elixir and Ecto

ExState: Database-backed statecharts for Elixir and Ecto

Kevin Buchanan

May 20, 2020

On a recent project, working with Elixir, Ecto, and Phoenix, we were encountering difficulty modeling and managing a complex business workflow. This is a fairly typical problem to have on any project—there have been many times I've known that I'm building an implicit finite state machine but don't take the time to make it explicit—but here it was becoming particularly unmanageable. It became clear that we needed to implement a more formal state machine to encode this logic.

Eventually, we implemented a solution that allows us to easily model these workflows in code, in a simple enough way that it also facilitates talking through these workflows and statecharts with non-technical project stakeholders. Read on to learn more about ExState, or try it out to see if it's useful to you.

The Problem

Tracking any complex process or workflow inevitably requires conditional logic based on the state of entities involved in the workflow. This can easily lead to a proliferation of fields and states used to track this process. Take an example Order process, for instance:


defmodule Order do
  use Ecto.Schema

  schema "orders" do
    field :created_at, :utc_datetime
    field :confirmed, :boolean
    field :confirmed_at, :utc_datetime
    field :shipped, :boolean
    field :shipped_at, :utc_datetime
    field :shipment_status, :string
    field :cancelled, :boolean
    field :cancelled_at, :utc_datetime
    has_many :items, Item
  end
end

In the above schema, we have four fields related to tracking the state of the order. These fields can change independently and it's unclear how they should relate to one another. Is %{confirmed: false, shipped: true, cancelled: true} a valid state? Probably not, but it would require a case by case validation to enforce that.

The order workflow and state becomes much easier to understand and validate if we collapse those fields into one state field:

defmodule Order do
  use Ecto.Schema

  schema "orders" do
    field :state, :string
    field :created_at, :utc_datetime
    field :confirmed_at, :utc_datetime
    field :shipped_at, :utc_datetime
    field :cancelled_at, :utc_datetime
    has_many :items, Item
  end
end

defmodule Orders do
  alias Ecto.Multi

  def cancel(order) do
    if order.state in ["pending", "confirmed"] do
      order
      |> Order.changeset(%{status: "cancelled", cancelled_at: DateTime.utc_now()})
      |> Repo.update()
    else
      {:error, "can't cancel this order"}
    end
  end

  def ship(order) do
    if order.state == "confirmed" do
      Multi.new()
      |> Multi.update(:order, Order.changeset(order, %{status: "shipped", shipped_at: DateTime.utc_now()}))
      |> Multi.run(:ship, fn _repo, %{order: order} -> Shipments.ship(order) end)
      |> Repo.transaction()
    else
      {:error, "order must be in the confirmed state"}
    end
  end
end

We can then use the states ["created", "confirmed", "shipped", "delivered", "cancelled"] to represent our order workflow. But we still need to replace the shipment_state field and to express sub-states of the "shipped" state. We're also still required to do a lot of explicit checking of the current state before taking any action.

Statecharts

A lot of the workflows and processes we build in our software can be modeled as finite state machines. The system, or an entity in the system, is in one given state at any time. In that state, the system handles events and transitions to a next state with logic that is dependent upon the current state.

Elixir and Erlang already provide tools for implementing state machine behavior through gen_statem and its predecessor gen_fsm. These are great for implementing stateful processes, but aren't well-suited for state that's stored in a database with a request and response lifecycle.

Our order process resembles a finite state machine, but with a few additional details. In the "shipped" state, we want to track multiple shipment_status states. So the "shipped" state itself represents multiple states:

[
 {"shipped", "pending"},
 {"shipped", "in_progress"},
 {"shipped", "arriving_soon"},
 {"shipped", "out_for_delivery"}
]

Rather than a simple finite state machine, we have a hierarchical state machine, with certain states like "shipped" representing a lower level of states and transitions.

ExState

ExState was designed to simplify the definition of states, events, and transitions for simple or hierarchical state machines in Elixir. ExState can also persist these states and metadata to the database through Ecto for use in web apps or other database-backed applications.

Using ExState begins with a definition module. ExState.Definition defines the workflow macro, which builds a data representation of the state chart and binds it to the module and the associated functions for transitioning the workflow with a subject and other context data.


defmodule OrderWorkflow do
  use ExState.Definition

  workflow "order" do
    subject :order, Order

    initial_state :pending

    state :preparing do
      initial_state :pending

      on :cancel, :cancelled

      state :pending do
        step :confirm
        on_completed :confirm, :confirmed
      end

      state :confirmed do
        step :ship
        on_completed :ship, {:<, :shipped}
      end
    end

    state :shipped do
      initial_state :in_transit

      on_entry :update_shipped_at

      state :in_transit do
        on :arriving_soon, :arriving_soon
      end

      state :arriving_soon do
        on :out_for_delivery, :out_for_delivery
      end

      state :out_for_delivery do
        on :delivered, {:<, :delivered}
      end
    end

    state :delivered do
      final
    end

    state :cancelled do
      on_entry :update_cancelled_at
      final
    end
  end

  def update_shipped_at(%{order: order}) do
    order
    |> Order.changeset(%{timestamp => DateTime.utc_now()})
    |> Repo.update()
  end

  def update_cancelled_at(%{order: order}) do
    order
    |> Order.changeset(%{timestamp => DateTime.utc_now()})
    |> Repo.update()
  end

  defp update_timestamp(order, timestamp) do
    order
    |> Order.changeset(%{timestamp => DateTime.utc_now()})
    |> Repo.update()
  end
end

The subject of the workflow is an Ecto model that defines a workflow association using the has_workflow macro.


defmodule Order do
  use Ecto.Schema
  use ExState.Ecto.Subject

  schema "orders" do
    field :created_at, :utc_datetime
    field :confirmed_at, :utc_datetime
    field :shipped_at, :utc_datetime
    field :cancelled_at, :utc_datetime
    has_workflow OrderWorkflow
    has_many :items, Item
  end
end

The associated context module can then update the model and its workflow through ExState:


defmodule Orders do
  alias Ecto.Multi

  @doc """
  Create an order and workflow in a Ecto.Multi transaction.
  """
  def create_order(attrs) do
    {:ok, %{order: order, workflow: workflow}} =
      Multi.new()
      |> Multi.create(:order, Order.new(attrs))
      |> ExState.Multi.create(:order)
      |> Repo.transaction()
  end

  @doc """
  Load, complete step, and persist.
  """
  def confirm_order(order) do
    execution = ExState.load(order)
    {:ok, execution} = ExState.Execution.complete(execution, :confirm)
    {:ok, order} = ExState.persist(execution)
  end

  @doc """
  Use `ExState.event/3` convenience function to load, transition, and persist.
  """
  def cancel_order(order) do
    {:ok, order} = ExState.event(order, :cancelled)
  end

  @doc """
  Load, complete, ship, and persist in a transaction.
  """
  def ship(order) do
    {:ok, %{ship: shipped_order, shipment: shipment}} =
      Multi.new()
      |> ExState.Ecto.Multi.complete(order, :ship)
      |> Multi.run(:shipment, fn _repo, %{order: order} ->
        Shipments.ship(order)
      end)
      |> Repo.transaction()
  end

  def arriving_soon(order) do
    {:ok, order} = ExState.transition(order, :arriving_soon)
  end
end

States

States are defined in four main forms.

Atomic States

Atomic states have no child states. The following three states are atomic states:


workflow "example" do
  initial_state :atomic_a

  state :atomic_a do
    on :next, :atomic_b
    on :done, :atomic_done
  end

  state :atomic_b do
    on :back, :atomic_a
  end

  state :atomic_done
end

Compound States

Compound states contain child states and specify an initial state (one of the child states). The :preparing and :shipped states in the order workflow are compound states.

Final States

A final state represents a state of either a child state or the entire workflow where the state should be considered "complete."


workflow "example" do
  initial_state :a

  state :a do
    on_final :b

    state :one do
      on :did_one, :two
    end

    state :two do
      final
    end
  end

  state :b
end

Transient States

Transient states are used for dynamically resolving the next state based on conditions defined in the definition module. The transient state immediately handles a "null" event, :_, and transitions to the first state in the list that a guard allows.


defmodule SetupWorkflow do
  use ExState.Definition

  workflow "setup" do
    initial_state :unknown

    state :unknown do
      on :_, [:accept_terms, :working]
    end

    state :accept_terms

    state :working
  end

  def guard_transition(:unknown, :accept_terms, context) do
    if context.user.has_accepted_terms? do
      :ok
    else
      {:error, :accepted}
    end
  end
end

Actions

Actions are useful for triggering side effects on certain events. Actions are called as functions on the definition module, and should return :error | {:error, reason} if transactional execution should be halted when an action cannot be completed. Actions can also return an {:updated, subject} tuple to replace the updated subject in the execution state.


defmodule ExampleWorkflow do
  use ExState.Definition

  workflow "example" do
    state :working do
      on_entry :send_entry_email
      on :cancel, :canceled, actions: [:send_cancelled_email]
      on_final :done, actions: [:send_final_email]
    end
  end

  def send_entry_email(_context), do: :ok
  def send_final_email(_context), do: :ok
  def send_cancelled_email(context) do
    {:updated, Map.put(context, :document, %{context.document | cancellation_email_sent_at: DateTime.utc_now()})}
  end
end

Guards

Guards can ensure that transitions are only made when certain conditions are met. The guard_transition/3 function on the definition module will be called during workflow execution with the current state, next state, and the context as arguments. A guard returns :ok to allow the transition, or {:error, reason to prevent the transition.

ExState doesn't rescue exceptions in guards or actions, so exception handling behavior is dependent upon the current database transaction context, if any.

Steps

Steps are a convenient way to collapse an implicitly linear set of states and events into an explicitly ordered list of events. This is useful for exposing required steps to UI components or API consumers. Steps can also be ignored through the use_step/2 callback.


state :working do
  initial_state :adding_name

  on_final :reviewing

  state :adding_name do
    on :name_added, :adding_email
  end

  state :adding_email do
    on :email_added, :adding_phone_number
  end

  state :adding_phone_number do
    on :phone_number_added, :confirming
  end

  state :confirming do
    on :confirmed, :done
  end

  state :done do
    final
  end
end

state :reviewing

The above workflow could be rewritten using four explicit steps:


defmodule SetupWorkflow do
  use ExState.Definition

  workflow "setup" do
    subject :account, Account

    initial_state :working

    state :working do
      step :add_name
      step :add_email
      step :add_phone_number
      step :confirm

      on_completed :confirm, :reviewing
    end

    state :reviewing
  end

  def use_step?(:add_phone_number, context) do
    context.account.phone_number_required?
  end

  def use_step?(_, _), do: true
end


{:ok, account} = ExState.create(account)
{:ok, account} = ExState.complete(account, :add_name)
{:ok, account} = ExState.complete(account, :add_email)
{:error, _reason} = ExState.complete(account, :confirm)
{:ok, account} = ExState.complete(account, :add_phone_number)
{:ok, %{workflow: %{state: "reviewing"}}} = ExState.complete(account, :confirm)

Decisions

Similar to steps, decisions are required events that transition based on the value provided in the decision.


defmodule ReviewWorkflow do
  use ExState.Definition

  workflow "review" do
    initial_state :rating

    state :rating do
      step :rate
      on_decision :rate, :good, :done
      on_decision :rate, :bad, :feedback
    end

    state :feedback do
      step :provide_feedback
      on_completed :provide_feedback, :done
    end

    state :done do
      final
    end
  end
end

{:ok, review} = ExState.create(review)
{:ok, review} = ExState.decision(review, :rate, :good)

Querying State

You'll likely want to use the workflow state in queries as well. ExState has some builtin queries to help with this:

import ExState.Ecto.Query

shipped_orders =
 Order
 |> where_any_state(:shipped)
 |> Repo.all()

in_transit_orders =
 Order
 |> where_state([:shipped, :in_transit])
 |> Repo.all()

confirmed_orders =
 Order
 |> where_step_complete(:confirm)
 |> Repo.all()

Without Ecto

ExState can be used without persisting the workflow to the database, either for testing or in memory use cases. Just use the functions defined on the workflow definition module itself:


%{state: %{name: "delivered"}, context: %{order: order}} = execution =
  OrderWorkflow.new(%{order: order})
  |> OrderWorkflow.transition_maybe({:completed, :confirm})
  |> OrderWorkflow.transition_maybe({:completed, :ship})
  |> OrderWorkflow.transition_maybe(:out_for_delivery)
  |> OrderWorkflow.transition_maybe(:arriving_soon)
  |> OrderWorkflow.transition_maybe(:delivered)

{:error, _reason, execution} = OrderWorkflow.transition(execution, :cancel)

Notes

Try ExState on hex, read the docs, or check out the code on GitHub for additional documentation and examples.

Credit to David Khourshid and xstate for excellent documentation, examples, and API inspiration on this topic.

Kevin Buchanan

Principal Crafter

Kevin Buchanan writes code and rides bikes in California. He is an experienced software professional who has led development teams to deliver complex systems that solve unique business challenges — like pharmaceuticals, telemedicine, medical devices, insurance, private equity, and more.