Book 2 · Lesson 2.4

Chained consumer pattern

  • Map PDF5 chained consumer to millipede services
  • Explain raw → enriched topic handoff
  • Predict backlog when one stage is slow

Prerequisites: 2.3

Instead of one mega-service calling LLM then DB, millipede uses chained consumers — each stage is a small deployable unit.

The chain

Click each link in ChainedConsumerDiagram:

HTTP → ingestion → raw-dev-events
       → llm-worker → enriched-dev-events
       → analyzer → Postgres + Redis
       → radar (poll + SSE)

Benefits

BenefitHow
Scale independentlyAdd llm-worker replicas
Swap enrichmentChange Python without touching Rust ingestion
ReplayRewind consumer offset on enriched topic

Teach-back prompt

Could analyzer subscribe to raw-dev-events instead? When would you?

raw → enriched topics

Lesson 2.4 check

1. millipede LLM output goes to…