r/dataengineering • u/BookkeeperAutomatic • 22h ago
Blog From 15 year industry veteran - How real-time ETA pipelines work in food delivery/logistics (Online vs. Offline Feature Store architecture + cost tradeoffs)
After 15 years in product engineering and working around logistics/ML systems, one pattern I keep seeing teams underestimate is feature freshness in real-time ETA prediction (like Zomato / Swiggy / DoorDash).
The tutorials you see in YT start at model.fit(). But calculating whether an order takes 22 mins or 38 mins is 80% data engineering and feature pipeline orchestration, and 20% ML.
Here’s how the pipeline actually breaks down in production when you separate ordinary business events from ML serving:
- Business Events First (Zero ML Yet)
Every order emits lifecycle events through API Gateway/Lambda into raw S3 event logs. - Background Feature Aggregation
A Spark job groups these logs. - The Dual Feature Store Split (The Part Teams Get Wrong)
Offline Store (S3/Parquet): Holds historical time-travel snapshots for model training without leakage.
Online Store (Low-Latency KV / SageMaker Online Store): Serves single-digit-millisecond lookups for live features right at checkout.
4. Live Inference Orchestration
This fetches pre-computed features from the Online Feature Store, combines them with live request context (distance, traffic), hits the model endpoint, and returns the ETA.
(Note: If you want to see the full 38-minute whiteboard & sequence diagram walkthrough of this on AWS SageMaker Feature Store + Spark, I broke it down here: https://www.youtube.com/watch?v=v_4vzmv436s - happy to debate feature freshness SLAs or Online/Offline sync tradeoffs in the comments.)
