ExState: Database-backed statecharts for Elixir and Ecto

ExState: Database-backed statecharts for Elixir and Ecto

Kevin Buchanan

May 20, 2020

On a recent project, working with Elixir, Ecto, and Phoenix, we were encountering difficulty modeling and managing a complex business workflow. This is a fairly typical problem to have on any project—there have been many times I've known that I'm building an implicit finite state machine but don't take the time to make it explicit—but here it was becoming particularly unmanageable. It became clear that we needed to implement a more formal state machine to encode this logic.

Eventually, we implemented a solution that allows us to easily model these workflows in code, in a simple enough way that it also facilitates talking through these workflows and statecharts with non-technical project stakeholders. Read on to learn more about ExState, or try it out to see if it's useful to you.

The Problem

Tracking any complex process or workflow inevitably requires conditional logic based on the state of entities involved in the workflow. This can easily lead to a proliferation of fields and states used to track this process. Take an example Order process, for instance:


defmodule Order do
		use Ecto.Schema

		schema "orders" do
				field :created_at, :utc_datetime
				field :confirmed, :boolean
				field :confirmed_at, :utc_datetime
				field :shipped, :boolean
				field :shipped_at, :utc_datetime
				field :shipment_status, :string
				field :cancelled, :boolean
				field :cancelled_at, :utc_datetime
				has_many :items, Item
		end
end

In the above schema, we have four fields related to tracking the state of the order. These fields can change independently and it's unclear how they should relate to one another. Is %{confirmed: false, shipped: true, cancelled: true} a valid state? Probably not, but it would require a case by case validation to enforce that.

The order workflow and state becomes much easier to understand and validate if we collapse those fields into one state field:

defmodule Order do
		use Ecto.Schema

		schema "orders" do
				field :state, :string
				field :created_at, :utc_datetime
				field :confirmed_at, :utc_datetime
				field :shipped_at, :utc_datetime
				field :cancelled_at, :utc_datetime
				has_many :items, Item
		end
end

defmodule Orders do
		alias Ecto.Multi

		def cancel(order) do
				if order.state in ["pending", "confirmed"] do
						order
						|> Order.changeset(%{status: "cancelled", cancelled_at: DateTime.utc_now()})
						|> Repo.update()
				else
						{:error, "can't cancel this order"}
				end
		end

		def ship(order) do
				if order.state == "confirmed" do
						Multi.new()
						|> Multi.update(:order, Order.changeset(order, %{status: "shipped", shipped_at: DateTime.utc_now()}))
						|> Multi.run(:ship, fn _repo, %{order: order} -> Shipments.ship(order) end)
						|> Repo.transaction()
				else
						{:error, "order must be in the confirmed state"}
				end
		end
end

We can then use the states ["created", "confirmed", "shipped", "delivered", "cancelled"] to represent our order workflow. But we still need to replace the shipment_state field and to express sub-states of the "shipped" state. We're also still required to do a lot of explicit checking of the current state before taking any action.

Statecharts

A lot of the workflows and processes we build in our software can be modeled as finite state machines. The system, or an entity in the system, is in one given state at any time. In that state, the system handles events and transitions to a next state with logic that is dependent upon the current state.

Elixir and Erlang already provide tools for implementing state machine behavior through gen_statem and its predecessor gen_fsm. These are great for implementing stateful processes, but aren't well-suited for state that's stored in a database with a request and response lifecycle.

Our order process resembles a finite state machine, but with a few additional details. In the "shipped" state, we want to track multiple shipment_status states. So the "shipped" state itself represents multiple states:

[
 {"shipped", "pending"},
 {"shipped", "in_progress"},
 {"shipped", "arriving_soon"},
 {"shipped", "out_for_delivery"}
]

Rather than a simple finite state machine, we have a hierarchical state machine, with certain states like "shipped" representing a lower level of states and transitions.

ExState

ExState was designed to simplify the definition of states, events, and transitions for simple or hierarchical state machines in Elixir. ExState can also persist these states and metadata to the database through Ecto for use in web apps or other database-backed applications.

Using ExState begins with a definition module. ExState.Definition defines the workflow macro, which builds a data representation of the state chart and binds it to the module and the associated functions for transitioning the workflow with a subject and other context data.


defmodule OrderWorkflow do
		use ExState.Definition

		workflow "order" do
				subject :order, Order

				initial_state :pending

				state :preparing do
						initial_state :pending

						on :cancel, :cancelled

						state :pending do
								step :confirm
								on_completed :confirm, :confirmed
						end

						state :confirmed do
								step :ship
								on_completed :ship, {:<, :shipped}
						end
				end

				state :shipped do
						initial_state :in_transit

						on_entry :update_shipped_at

						state :in_transit do
								on :arriving_soon, :arriving_soon
						end

						state :arriving_soon do
								on :out_for_delivery, :out_for_delivery
						end

						state :out_for_delivery do
								on :delivered, {:<, :delivered}
						end
				end

				state :delivered do
						final
				end

				state :cancelled do
						on_entry :update_cancelled_at
						final
				end
		end

		def update_shipped_at(%{order: order}) do
				order
				|> Order.changeset(%{timestamp => DateTime.utc_now()})
				|> Repo.update()
		end

		def update_cancelled_at(%{order: order}) do
				order
				|> Order.changeset(%{timestamp => DateTime.utc_now()})
				|> Repo.update()
		end

		defp update_timestamp(order, timestamp) do
				order
				|> Order.changeset(%{timestamp => DateTime.utc_now()})
				|> Repo.update()
		end
end

The subject of the workflow is an Ecto model that defines a workflow association using the has_workflow macro.


defmodule Order do
		use Ecto.Schema
		use ExState.Ecto.Subject

		schema "orders" do
				field :created_at, :utc_datetime
				field :confirmed_at, :utc_datetime
				field :shipped_at, :utc_datetime
				field :cancelled_at, :utc_datetime
				has_workflow OrderWorkflow
				has_many :items, Item
		end
end

The associated context module can then update the model and its workflow through ExState:


defmodule Orders do
		alias Ecto.Multi

		@doc """
		Create an order and workflow in a Ecto.Multi transaction.
		"""
		def create_order(attrs) do
				{:ok, %{order: order, workflow: workflow}} =
						Multi.new()
						|> Multi.create(:order, Order.new(attrs))
						|> ExState.Multi.create(:order)
						|> Repo.transaction()
		end

		@doc """
		Load, complete step, and persist.
		"""
		def confirm_order(order) do
				execution = ExState.load(order)
				{:ok, execution} = ExState.Execution.complete(execution, :confirm)
				{:ok, order} = ExState.persist(execution)
		end

		@doc """
		Use `ExState.event/3` convenience function to load, transition, and persist.
		"""
		def cancel_order(order) do
				{:ok, order} = ExState.event(order, :cancelled)
		end

		@doc """
		Load, complete, ship, and persist in a transaction.
		"""
		def ship(order) do
				{:ok, %{ship: shipped_order, shipment: shipment}} =
						Multi.new()
						|> ExState.Ecto.Multi.complete(order, :ship)
						|> Multi.run(:shipment, fn _repo, %{order: order} ->
								Shipments.ship(order)
						end)
						|> Repo.transaction()
		end

		def arriving_soon(order) do
				{:ok, order} = ExState.transition(order, :arriving_soon)
		end
end

States

States are defined in four main forms.

Atomic States

Atomic states have no child states. The following three states are atomic states:


workflow "example" do
		initial_state :atomic_a

		state :atomic_a do
				on :next, :atomic_b
				on :done, :atomic_done
		end

		state :atomic_b do
				on :back, :atomic_a
		end

		state :atomic_done
end

Compound States

Compound states contain child states and specify an initial state (one of the child states). The :preparing and :shipped states in the order workflow are compound states.

Final States

A final state represents a state of either a child state or the entire workflow where the state should be considered "complete."


workflow "example" do
		initial_state :a

		state :a do
				on_final :b

				state :one do
						on :did_one, :two
				end

				state :two do
						final
				end
		end

		state :b
end

Transient States

Transient states are used for dynamically resolving the next state based on conditions defined in the definition module. The transient state immediately handles a "null" event, :_, and transitions to the first state in the list that a guard allows.


defmodule SetupWorkflow do
		use ExState.Definition

		workflow "setup" do
				initial_state :unknown

				state :unknown do
						on :_, [:accept_terms, :working]
				end

				state :accept_terms

				state :working
		end

		def guard_transition(:unknown, :accept_terms, context) do
				if context.user.has_accepted_terms? do
						:ok
				else
						{:error, :accepted}
				end
		end
end

Actions

Actions are useful for triggering side effects on certain events. Actions are called as functions on the definition module, and should return :error | {:error, reason} if transactional execution should be halted when an action cannot be completed. Actions can also return an {:updated, subject} tuple to replace the updated subject in the execution state.


defmodule ExampleWorkflow do
		use ExState.Definition

		workflow "example" do
				state :working do
						on_entry :send_entry_email
						on :cancel, :canceled, actions: [:send_cancelled_email]
						on_final :done, actions: [:send_final_email]
				end
		end

		def send_entry_email(_context), do: :ok
		def send_final_email(_context), do: :ok
		def send_cancelled_email(context) do
				{:updated, Map.put(context, :document, %{context.document | cancellation_email_sent_at: DateTime.utc_now()})}
		end
end

Guards

Guards can ensure that transitions are only made when certain conditions are met. The guard_transition/3 function on the definition module will be called during workflow execution with the current state, next state, and the context as arguments. A guard returns :ok to allow the transition, or {:error, reason to prevent the transition.

ExState doesn't rescue exceptions in guards or actions, so exception handling behavior is dependent upon the current database transaction context, if any.

Steps

Steps are a convenient way to collapse an implicitly linear set of states and events into an explicitly ordered list of events. This is useful for exposing required steps to UI components or API consumers. Steps can also be ignored through the use_step/2 callback.


state :working do
		initial_state :adding_name

		on_final :reviewing

		state :adding_name do
				on :name_added, :adding_email
		end

		state :adding_email do
				on :email_added, :adding_phone_number
		end

		state :adding_phone_number do
				on :phone_number_added, :confirming
		end

		state :confirming do
				on :confirmed, :done
		end

		state :done do
				final
		end
end

state :reviewing

The above workflow could be rewritten using four explicit steps:


defmodule SetupWorkflow do
		use ExState.Definition

		workflow "setup" do
				subject :account, Account

				initial_state :working

				state :working do
						step :add_name
						step :add_email
						step :add_phone_number
						step :confirm

						on_completed :confirm, :reviewing
				end

				state :reviewing
		end

		def use_step?(:add_phone_number, context) do
				context.account.phone_number_required?
		end

		def use_step?(_, _), do: true
end


{:ok, account} = ExState.create(account)
{:ok, account} = ExState.complete(account, :add_name)
{:ok, account} = ExState.complete(account, :add_email)
{:error, _reason} = ExState.complete(account, :confirm)
{:ok, account} = ExState.complete(account, :add_phone_number)
{:ok, %{workflow: %{state: "reviewing"}}} = ExState.complete(account, :confirm)

Decisions

Similar to steps, decisions are required events that transition based on the value provided in the decision.


defmodule ReviewWorkflow do
		use ExState.Definition

		workflow "review" do
				initial_state :rating

				state :rating do
						step :rate
						on_decision :rate, :good, :done
						on_decision :rate, :bad, :feedback
				end

				state :feedback do
						step :provide_feedback
						on_completed :provide_feedback, :done
				end

				state :done do
						final
				end
		end
end

{:ok, review} = ExState.create(review)
{:ok, review} = ExState.decision(review, :rate, :good)

Querying State

You'll likely want to use the workflow state in queries as well. ExState has some builtin queries to help with this:

import ExState.Ecto.Query

shipped_orders =
 Order
 |> where_any_state(:shipped)
 |> Repo.all()

in_transit_orders =
 Order
 |> where_state([:shipped, :in_transit])
 |> Repo.all()

confirmed_orders =
 Order
 |> where_step_complete(:confirm)
 |> Repo.all()

Without Ecto

ExState can be used without persisting the workflow to the database, either for testing or in memory use cases. Just use the functions defined on the workflow definition module itself:


%{state: %{name: "delivered"}, context: %{order: order}} = execution =
		OrderWorkflow.new(%{order: order})
		|> OrderWorkflow.transition_maybe({:completed, :confirm})
		|> OrderWorkflow.transition_maybe({:completed, :ship})
		|> OrderWorkflow.transition_maybe(:out_for_delivery)
		|> OrderWorkflow.transition_maybe(:arriving_soon)
		|> OrderWorkflow.transition_maybe(:delivered)

{:error, _reason, execution} = OrderWorkflow.transition(execution, :cancel)

Notes

Try ExState on hex, read the docs, or check out the code on GitHub for additional documentation and examples.

Credit to David Khourshid and xstate for excellent documentation, examples, and API inspiration on this topic.

Kevin Buchanan

Principal Crafter

Kevin Buchanan writes code and rides bikes in California. He is an experienced software professional who has led development teams to deliver complex systems that solve unique business challenges — like pharmaceuticals, telemedicine, medical devices, insurance, private equity, and more.