ExState: Database-backed statecharts for Elixir and Ecto

ExState: Database-backed statecharts for Elixir and Ecto

Kevin Buchanan
Kevin Buchanan

May 20, 2020

On a recent project, working with Elixir, Ecto, and Phoenix, we were encountering difficulty modeling and managing a complex business workflow. This is a fairly typical problem to have on any project—there have been many times I've known that I'm building an implicit finite state machine but don't take the time to make it explicit—but here it was becoming particularly unmanageable. It became clear that we needed to implement a more formal state machine to encode this logic.

Eventually, we implemented a solution that allows us to easily model these workflows in code, in a simple enough way that it also facilitates talking through these workflows and statecharts with non-technical project stakeholders. Read on to learn more about ExState, or try it out to see if it's useful to you.

The Problem

Tracking any complex process or workflow inevitably requires conditional logic based on the state of entities involved in the workflow. This can easily lead to a proliferation of fields and states used to track this process. Take an example Order process, for instance:

defmodule Order do
 use Ecto.Schema

 schema "orders" do
 field :created_at, :utc_datetime
 field :confirmed, :boolean
 field :confirmed_at, :utc_datetime
 field :shipped, :boolean
 field :shipped_at, :utc_datetime
 field :shipment_status, :string
 field :cancelled, :boolean
 field :cancelled_at, :utc_datetime
 has_many :items, Item
 end
end

In the above schema, we have four fields related to tracking the state of the order. These fields can change independently and it's unclear how they should relate to one another. Is %{confirmed: false, shipped: true, cancelled: true} a valid state? Probably not, but it would require a case by case validation to enforce that.

The order workflow and state becomes much easier to understand and validate if we collapse those fields into one state field:

defmodule Order do
 use Ecto.Schema

 schema "orders" do
 field :state, :string
 field :created_at, :utc_datetime
 field :confirmed_at, :utc_datetime
 field :shipped_at, :utc_datetime
 field :cancelled_at, :utc_datetime
 has_many :items, Item
 end
end

defmodule Orders do
 alias Ecto.Multi

 def cancel(order) do
 if order.state in ["pending", "confirmed"] do
 order
 |> Order.changeset(%{status: "cancelled", cancelled_at: DateTime.utc_now()})
 |> Repo.update()
 else
 {:error, "can't cancel this order"}
 end
 end

 def ship(order) do
 if order.state == "confirmed" do
 Multi.new()
 |> Multi.update(:order, Order.changeset(order, %{status: "shipped", shipped_at: DateTime.utc_now()}))
 |> Multi.run(:ship, fn _repo, %{order: order} ->
 Shipments.ship(order)
 end)
 |> Repo.transaction()
 else
 {:error, "order must be in the confirmed state"}
 end
 end
end

We can then use the states ["created", "confirmed", "shipped", "delivered", "cancelled"] to represent our order workflow. But we still need to replace the shipment_state field and to express sub-states of the "shipped" state. We're also still required to do a lot of explicit checking of the current state before taking any action.

Statecharts

A lot of the workflows and processes we build in our software can be modeled as finite state machines. The system, or an entity in the system, is in one given state at any time. In that state, the system handles events and transitions to a next state with logic that is dependent upon the current state.

Elixir and Erlang already provide tools for implementing state machine behavior through gen_statem and its predecessor gen_fsm. These are great for implementing stateful processes, but aren't well-suited for state that's stored in a database with a request and response lifecycle.

Our order process resembles a finite state machine, but with a few additional details. In the "shipped" state, we want to track multiple shipment_status states. So the "shipped" state itself represents multiple states:

[
 {"shipped", "pending"},
 {"shipped", "in_progress"},
 {"shipped", "arriving_soon"},
 {"shipped", "out_for_delivery"}
]

Rather than a simple finite state machine, we have a hierarchical state machine, with certain states like "shipped" representing a lower level of states and transitions.

ExState

ExState was designed to simplify the definition of states, events, and transitions for simple or hierarchical state machines in Elixir. ExState can also persist these states and metadata to the database through Ecto for use in web apps or other database-backed applications.

Using ExState begins with a definition module. ExState.Definition defines the workflow macro, which builds a data representation of the state chart and binds it to the module and the associated functions for transitioning the workflow with a subject and other context data.

defmodule OrderWorkflow do
 use ExState.Definition

 workflow "order" do
 subject :order, Order

 initial_state :pending

 state :preparing do
 initial_state :pending

 on :cancel, :cancelled

 state :pending do
 step :confirm
 on_completed :confirm, :confirmed
 end

 state :confirmed do
 step :ship
 on_completed :ship, {:<, :shipped}
 end
 end

 state :shipped do
 initial_state :in_transit

 on_entry :update_shipped_at

 state :in_transit do
 on :arriving_soon, :arriving_soon
 end

 state :arriving_soon do
 on :out_for_delivery, :out_for_delivery
 end

 state :out_for_delivery do
 on :delivered, {:<, :delivered}
 end
 end

 state :delivered do
 final
 end

 state :cancelled do
 on_entry :update_cancelled_at
 final
 end
 end

 def update_shipped_at(%{order: order})
 update_timestamp(order, :shipped_at)
 end

 def update_cancelled_at(%{order: order})
 update_timestamp(order, :cancelled_at)
 end

 defp update_timestamp(order, timestamp) do
 order
 |> Order.changeset(%{timestamp => DateTime.utc_now()})
 |> Repo.update()
 end
end

The subject of the workflow is an Ecto model that defines a workflow association using the has_workflow macro.

defmodule Order do
 use Ecto.Schema
 use ExState.Ecto.Subject

 schema "orders" do
 field :created_at, :utc_datetime
 field :confirmed_at, :utc_datetime
 field :shipped_at, :utc_datetime
 field :cancelled_at, :utc_datetime
 has_workflow OrderWorkflow
 has_many :items, Item
 end
end

The associated context module can then update the model and its workflow through ExState:

defmodule Orders do
 alias Ecto.Multi

 @doc """
 Create an order and workflow in a Ecto.Multi transaction.
 """
 def create_order(attrs) do
 {:ok, %{order: order, workflow: workflow}} =
 Multi.new()
 |> Multi.create(:order, Order.new(attrs))
 |> ExState.Multi.create(:order)
 |> Repo.transaction()
 end

 @doc """
 Load, complete step, and persist.
 """
 def confirm_order(order) do
 execution = ExState.load(order)
 {:ok, execution} = ExState.Execution.complete(execution, :confirm)
 {:ok, order} = ExState.persist(execution)
 end

 @doc """
 Use `ExState.event/3` convenience function to load, transition, and persist.
 """
 def cancel_order(order) do
 {:ok, order} = ExState.event(order, :cancelled)
 end

 @doc """
 Load, complete, ship, and persist in a transaction.
 """
 def ship(order) do
 {:ok, %{ship: shipped_order, shipment: shipment}} =
 Multi.new()
 |> ExState.Ecto.Multi.complete(order, :ship)
 |> Multi.run(:shipment, fn _repo, %{order: order} ->
 Shipments.ship(order)
 end)
 |> Repo.transaction()
 end

 def arriving_soon(order) do
 {:ok, order} = ExState.transition(order, :arriving_soon)
 end
end

States

States are defined in four main forms.

Atomic States

Atomic states have no child states. The following three states are atomic states:

workflow "example" do
 initial_state :atomic_a

 state :atomic_a do
 on :next, :atomic_b
 on :done, :atomic_done
 end

 state :atomic_b do
 on :back, :atomic_a
 end

 state :atomic_done
end

Compound States

Compound states contain child states and specify an initial state (one of the child states). The :preparing and :shipped states in the order workflow are compound states.

Final States

A final state represents a state of either a child state or the entire workflow where the state should be considered "complete."

workflow "example" do
 initial_state :a

 state :a do
 on_final :b

 state :one do
 on :did_one, :two
 end

 state :two do
 final
 end
 end

 state :b
end

Transient States

Transient states are used for dynamically resolving the next state based on conditions defined in the definition module. The transient state immediately handles a "null" event, :_, and transitions to the first state in the list that a guard allows.

defmodule SetupWorkflow do
 use ExState.Definition

 workflow "setup" do
 initial_state :unknown

 state :unknown do
 on :_, [:accept_terms, :working]
 end

 state :accept_terms

 state :working
 end

 def guard_transition(:unknown, :accept_terms, context) do
 if context.user.has_accepted_terms? do
 :ok
 else
 {:error, :accepted}
 end
 end
end

Actions

Actions are useful for triggering side effects on certain events. Actions are called as functions on the definition module, and should return :error | {:error, reason} if transactional execution should be halted when an action cannot be completed. Actions can also return an {:updated, subject} tuple to replace the updated subject in the execution state.

defmodule ExampleWorkflow so
 use ExState.Definition

 workflow "example" do
 state :working do
 on_entry :send_entry_email
 on :cancel, :canceled, actions: [:send_cancelled_email]
 on_final :done, actions: [:send_final_email]
 end
 end

 def send_entry_email(_context), do: :ok
 def send_final_email(_context), do: :ok
 def send_cancelled_email(context) do
 {:updated, Map.put(context, :document, %{document | cancellation_email_sent_at: DateTime.utc_now()})}
 end
end

Guards

Guards can ensure that transitions are only made when certain conditions are met. The guard_transition/3 function on the definition module will be called during workflow execution with the current state, next state, and the context as arguments. A guard returns :ok to allow the transition, or {:error, reason to prevent the transition.

ExState doesn't rescue exceptions in guards or actions, so exception handling behavior is dependent upon the current database transaction context, if any.

Steps

Steps are a convenient way to collapse an implicitly linear set of states and events into an explicitly ordered list of events. This is useful for exposing required steps to UI components or API consumers. Steps can also be ignored through the use_step/2 callback.

state :working do
 initial_state :adding_name

 on_final :reviewing

 state :adding_name do
 on :name_added, :adding_email
 end

 state :adding_email do
 on :email_added, :adding_phone_number
 end

 state :adding_phone_number do
 on :phone_number_added, :confirming
 end

 state :confirming do
 on :confirmed, :done
 end

 state :done do
 final
 end
end

state :reviewing

The above workflow could be rewritten using four explicit steps:

defmodule SetupWorkflow do
 use ExState.Definition

 workflow "setup" do
 subject :account, Account

 initial_state :working

 state :working do
 step :add_name
 step :add_email
 step :add_phone_number
 step :confirm

 on_completed :confirm, :reviewing
 end

 state :reviewing
 end

 def use_step?(:add_phone_number, context) do
 context.account.phone_number_required?
 end

 def use_step?(_, _), do: true
end
{:ok, account} = ExState.create(account)
{:ok, account} = ExState.complete(account, :add_name)
{:ok, account} = ExState.complete(account, :add_email)
{:error, _reason} = ExState.complete(account, :confirm)
{:ok, account} = ExState.complete(account, :add_phone_number)
{:ok, %{workflow: %{state: "reviewing"}}} = ExState.complete(account, :confirm)

Decisions

Similar to steps, decisions are required events that transition based on the value provided in the decision.

defmodule ReviewWorkflow do
 use ExState.Definition

 workflow "review" do
 initial_state :rating

 state :rating do
 step :rate
 on_decision :rate, :good, :done
 on_decision :rate, :bad, :feedback
 end

 state :feedback do
 step :provide_feedback
 on_completed :provide_feedback, :done
 end

 state :done do
 final
 end
 end
end
{:ok, review} = ExState.create(review)
{:ok, review} = ExState.decision(review, :rate, :good)

Querying State

You'll likely want to use the workflow state in queries as well. ExState has some builtin queries to help with this:

import ExState.Ecto.Query

shipped_orders =
 Order
 |> where_any_state(:shipped)
 |> Repo.all()

in_transit_orders =
 Order
 |> where_state([:shipped, :in_transit])
 |> Repo.all()

confirmed_orders =
 Order
 |> where_step_complete(:confirm)
 |> Repo.all()

Without Ecto

ExState can be used without persisting the workflow to the database, either for testing or in memory use cases. Just use the functions defined on the workflow definition module itself:

%{state: %{name: "delivered"}, context: %{order: order}} = execution =
 OrderWorkflow.new(%{order: order})
 |> OrderWorkflow.transition_maybe({:completed, :confirm})
 |> OrderWorkflow.transition_maybe({:completed, :ship})
 |> OrderWorkflow.transition_maybe(:out_for_delivery)
 |> OrderWorkflow.transition_maybe(:arriving_soon)
 |> OrderWorkflow.transition_maybe(:delivered)

{:error, _reason, execution} = OrderWorkflow.transition(execution, :cancel)

Notes

Try ExState on hex, read the docs, or check out the code on GitHub for additional documentation and examples.

Credit to David Khourshid and xstate for excellent documentation, examples, and API inspiration on this topic.