Book 2 · Lesson 2.4
Chained consumer pattern
- Map PDF5 chained consumer to millipede services
- Explain raw → enriched topic handoff
- Predict backlog when one stage is slow
Prerequisites: 2.3
Instead of one mega-service calling LLM then DB, millipede uses chained consumers — each stage is a small deployable unit.
The chain
Click each link in ChainedConsumerDiagram:
HTTP → ingestion → raw-dev-events
→ llm-worker → enriched-dev-events
→ analyzer → Postgres + Redis
→ radar (poll + SSE)
Benefits
| Benefit | How |
|---|---|
| Scale independently | Add llm-worker replicas |
| Swap enrichment | Change Python without touching Rust ingestion |
| Replay | Rewind consumer offset on enriched topic |
Teach-back prompt
Could analyzer subscribe to raw-dev-events instead? When would you?
raw → enriched topics
Lab complete — nice work.