field note / elixir · telemetry · data-quality
GPS can lie: designing telemetry validation before side effects
A practical ordering for duplicate, time, coordinate, rate, and movement checks in an ingestion pipeline.
A telemetry record looks harmless: an asset identifier, a coordinate, a timestamp, perhaps speed and ignition. The danger is not in parsing that record. It is in allowing a bad record to become state.
Once an impossible point updates “last known position,” triggers a geofence, advances a route, or enters a customer-facing stream, rejecting it later is no longer enough. The event has acquired consequences.
The most useful rule I have found is simple:
Validation is not a cleanup stage. It is the boundary before side effects.
Put cheap certainty first
A good validation chain is ordered by both cost and meaning. Mine usually resembles this:
- Can the payload be decoded and normalized?
- Can its tenant and asset be resolved?
- Have we already processed the provider's event identity?
- Are latitude, longitude, and timestamp structurally valid?
- Is the event within an acceptable time window?
- Does its reported rate agree with basic physical limits?
- Is movement from the previous accepted point plausible?
- Only then: persist, cache, publish, and derive downstream events.
The first checks are deterministic and cheap. Later checks need context, such as the last accepted reading. This ordering prevents an expensive database or geospatial operation from becoming the default answer to malformed input.
In an Elixir pipeline, I like the control flow to make rejection explicit:
with {:ok, reading} <- normalize(payload),
{:ok, asset} <- resolve_asset(reading, tenant),
:ok <- reject_duplicate(reading, asset),
:ok <- validate_coordinates(reading),
:ok <- validate_timestamp(reading),
:ok <- validate_movement(reading, asset) do
accept(reading, asset)
else
{:error, reason} -> reject(payload, reason)
end
The important part is not the with. It is that accept/2 is the first place allowed to perform irreversible work.
Time is not one dimension
Telemetry has at least three times:
- when the device says the observation happened;
- when a provider received or emitted it;
- when our system processed it.
Treating these as interchangeable creates subtle bugs. A reading can arrive late but still be historically valid. A device can also jump into the future because of a broken clock. Retries can reorder observations even when every component is behaving correctly.
For movement checks, the elapsed time must use observation timestamps. For operational latency, compare observation time with ingestion time. For retry behavior, retain provider identity or an explicit idempotency key.
A negative time delta deserves a named policy—not an accidental arithmetic path. Depending on the product, that policy might be “store as historical but do not update current state,” “reject as out of order,” or “re-evaluate a bounded window.” Any of those can be correct. Letting it fall through as plausible movement is not a policy.
The last point must mean last accepted point
Impossible-movement detection usually compares distance with elapsed time. The arithmetic is easy; the state semantics are hard.
If a rejected point replaces the cache entry used by the next check, one bad point can poison the validator. The next real reading is measured from fiction. If an old reading replaces the current one, the validator can also move backward in time.
The cache contract should therefore say:
- only accepted readings can advance current position;
- historical readings do not replace newer accepted state;
- rejection records may be observed elsewhere, but never become the validation baseline;
- cache keys include tenant identity, not only an external asset id.
That final clause matters in multi-tenant systems. External identifiers are rarely globally unique, no matter how convenient it would be if they were.
Rejections are first-class telemetry
A rejected reading should not disappear. Record a bounded, privacy-conscious envelope containing the reason, provider, tenant, asset resolution result, relevant timestamps, and enough normalized fields to reproduce the decision.
Make the reason machine-readable:
invalid_coordinates
future_timestamp
stale_timestamp
duplicate_event
impossible_movement
unknown_asset
rate_out_of_range
This gives operations something more useful than “processing failed.” It also turns production behavior into a dataset: which providers produce stale timestamps, where duplicate bursts begin, whether a new rule is too aggressive.
Test invariants, not only examples
Example tests are necessary: zero coordinates, a future timestamp, the same event twice, a jump across a country in thirty seconds. The stronger tests protect invariants:
- a rejected reading creates no publishable event;
- a rejected reading does not advance last accepted state;
- two tenants with the same external asset id never share state;
- replaying an accepted provider event is idempotent;
- processing late data cannot move the live position backward;
- the reason stored for rejection is stable enough to query.
The shape of the pipeline may change—Broadway stages, queue boundaries, caches—but these statements should remain true.
A GPS pipeline is not correct because most dots look sensible on a map. It is correct when bad inputs have a small, observable blast radius and cannot quietly become truth.