Cold-Chain Telemetry and Compliance Platform

Problem

A European pharmaceutical logistics provider runs about 20,000 refrigerated vehicles and 300 depots across 25 countries, moving vaccines and biologics under Good Distribution Practice (GDP). Each vehicle reports position, two temperature probes, door status and humidity every 10 seconds. Three device vendors each have their own data format and portal. Dispatchers switch between three portals. Excursions are usually found after delivery, and each shipment’s compliance report is assembled by hand. Vehicles lose connectivity for up to 72 hours and later upload a backlog. Nothing may be lost, and a report must be verifiable ten years later.

Approach

One tension shapes the design. Vehicles keep losing connectivity, yet no measurement may be lost. An auditor must also be able to confirm the record is complete and unaltered. The loss problem is solved on the device. The proof problem is solved in storage. The device keeps every measurement until the broker acknowledges it, and tags each one with its own timestamp and an increasing sequence number. Everything else builds on that sequence number.

Architecture

• Ingestion: EMQX for MQTT devices and an HTTPS batch API for the vendor that only offers exports. A downlink topic lets the platform ask a device to resend a sequence range.
• Backbone: Kafka with Avro contracts. Reference data from the transport system arrives by Debezium.
• Processing: Flink deduplicates by device and sequence, detects gaps in real time, evaluates excursion and geofence rules, and accepts late data for 72 hours.
• Stores by access pattern: ClickHouse for 90 days of hot analytics. Iceberg on Ceph with Object Lock for the ten-year raw record. PostgreSQL with PostGIS for operational state.
• Proof: each report is generated from raw data, hashed, signed with OpenBao, and its digest appended to an immudb ledger.
• Access: APISIX and Keycloak for dispatchers and customers, MapLibre for the live map.

Decisions and trade-offs

• Acknowledge the device only after Kafka confirms. With EMQX Kafka integration, no MQTT offset can be lost, and the device keeps each sample until then. A sequence gap triggers a “resend range” command on the downlink topic. The cost is the EMQX Enterprise licence, a named exception to the open-source rule. Without that budget, I would use Kafka Connect MQTT source: at least once, with the offset in the connector.
• At-least-once plus idempotent writes. The chain is QoS 1 and sequence numbers on the device, acknowledgement after Kafka, Flink checkpoints for state and offsets, and two-phase commit or idempotent upserts in the stores. The result is effectively once, with deduplication at read. It is not a claim of exactly-once. The cost: dashboards must query ClickHouse with FINAL or argMax.
• Two stores by access pattern. A dispatcher needs sub-second queries over 90 days. An auditor needs untouched raw data ten years later. ClickHouse is hot and fast; Iceberg is cheap and locked. The raw layer is the source of truth, and ClickHouse can be rebuilt from it. The cost is running two stores.
• Verify, do not trust. Three layers: Object Lock makes raw objects undeletable, OpenBao signs the report digest, and immudb records it with a Merkle proof. The auditor hashes the PDF again and asks for the proof. The cost: no compaction under Object Lock, so file size is controlled through a 10-minute checkpoint interval.
• Driver privacy versus ten-year evidence. Telemetry belongs to the vehicle. The driver-to-vehicle mapping is the only personal data, tokenised with a per-driver key. Erasure deletes the mapping and destroys the key, so every copy of the token becomes meaningless and the evidence stays intact. The cost: driver monitoring is employee data and needs works council consultation before go-live.
• Late data creates a new report version. A late sample goes through the same rules and is flagged as retrospective: no SMS, but the quality manager is notified and the product is quarantined. Report version 2 is issued, and the digest of version 1 stays in the ledger. The cost is that customers receive report revisions.

What I would revisit

• Real device capabilities. Can every vendor’s device buffer for 72 hours, load certificates and honour a resend command? If not, late data comes only from portal exports, which also decides whether the third vendor stays on the batch API.
• EMQX licence budget. If there is none, the bridge becomes Kafka Connect MQTT source, at least once.
• Retention mode. Is ten years the same in every country, and will legal accept compliance mode, where nobody can delete, or only governance mode?
• Transport system access. If its database is not open to change data capture, reference data comes from API polling, with a freshness cost.

Outcome notes (design assumptions, not measurements)

This is a design exercise, not a deployed platform. The output is a four-page diagram set, a one-page whiteboard version and an 18-page Turkish PDF with capacity arithmetic and open questions. The alert latency estimate (about 8–10 seconds against a 30-second target) is a design estimate.

Sources

• Architecture PDF (Turkish, 18 pages).
• draw.io diagrams (English labels): four-page set and a one-page 30-minute version.