Publishing a multi-tenant ETS index without partial state

When a hot path receives many telemetry readings, querying PostgreSQL for the same asset mapping on every event is wasteful. ETS is an attractive answer: concurrent reads, constant-time lookup, and no network hop.

But replacing a database query with ETS does not remove correctness work. It moves that work into publication semantics.

The actual requirement is not “cache this table.” It is:

Every reader should see either the old complete index or the new complete index, never a partially rebuilt mixture.

Start with the real identity

Suppose a provider supplies an external_asset_id. Looking it up by that value alone is only safe if it is globally unique. In integration platforms it often is not. Two organizations may use the same provider-side identifier, or two providers may issue values from separate namespaces.

The key should encode the isolation boundary:

{tenant_id, provider_id, external_asset_id}

The value can stay narrow:

%{asset_id: asset_id, organization_id: organization_id}

A composite key is not a cosmetic detail. It makes cross-tenant collision impossible by construction and turns the read path into a direct lookup rather than a scan plus authorization filter.

One writer, many readers

ETS can accept concurrent writes, but “can” is different from “should.” If updates combine inserts, deletes, metadata changes, and publication, allowing every caller to mutate the table makes ordering difficult to reason about.

A small GenServer can own mutation while readers access ETS directly:

def lookup(key) do
  generation = :persistent_term.get({__MODULE__, :generation})

  case :ets.lookup(table_for(generation), key) do
    [{^key, value}] -> {:ok, value}
    [] -> :error
  end
end

The GenServer is not in the read path. It serializes refreshes and incremental mutations, giving the cache one place where state transitions are defined.

Build, then publish

Clearing the active table and inserting rows one batch at a time creates a partial-state window. During that window, a valid asset can look absent. In an ingestion system, “absent” may mean reject, dead-letter, or perform an expensive fallback.

A safer rebuild uses generations:

1. Allocate a new ETS table.
2. Load and validate the full snapshot into it.
3. Record its generation and basic counts.
4. Atomically publish the new generation pointer.
5. Retire the old table only after readers have moved on.

The atomic operation is the pointer swap, not thousands of individual inserts. Readers see one complete snapshot or the other.

This is a read-copy-update pattern in small clothes. It also gives rollback a cheap shape: keep the previous generation briefly and restore the pointer if post-publication checks fail.

Incremental changes need a sequence

Full snapshots are easy to reason about but can be expensive. Incremental updates are efficient but introduce a race:

• a snapshot begins at database state S;
• a row changes to S+1 while the snapshot is loading;
• an incremental update applies the change to the currently active table;
• the snapshot publishes and silently restores the old row.

The fix is sequencing. Capture a version or change position for the snapshot, queue later mutations, publish the snapshot, then apply mutations newer than its position. The exact mechanism can be a database sequence, updated-at cursor with a tie-breaker, an outbox id, or a monotonic generation managed by the owning process.

Without that ordering, “refresh occasionally plus listen for changes” is not a coherent protocol.

Warm-up is a system state

Startup matters. If the pipeline begins consuming before the index is ready, the first events can be falsely rejected. Make readiness explicit:

• :cold — no usable index;
• :loading — snapshot is being assembled;
• :ready — a complete generation is published;
• :degraded — a previous generation is usable but refresh failed.

Consumers can delay demand, use a bounded fallback, or fail closed according to the product. What they should not do is infer readiness from whether one lookup happened to succeed.

Observe the contract

Useful cache telemetry belongs in log messages and metrics:

• active generation;
• entries loaded versus expected;
• refresh duration;
• incremental mutations replayed;
• lookup hits and misses by provider, not by sensitive identifier;
• refresh failures while an old generation remains active.

The cache is healthy when readers get correct answers, not merely when the ETS process exists.

ETS can make an Elixir hot path extremely fast. Composite identity makes it safe across tenants. A single mutation owner makes transitions understandable. Generation publication makes snapshots atomic from the reader's perspective. Together, those choices turn an optimization into infrastructure I can trust.