Book 2 · Lesson 2.3
Producers and consumers
- Trace produce → broker → consume → commit
- Identify producer and consumer in each service
- Explain consumer groups at a high level
Prerequisites: 2.2
Every pipeline service is both producer and consumer — except ingestion (producer only) and analyzer (final consumer + side effects).
Message lifecycle
Animate KafkaMessageFlow:
- Producer serializes JSON, picks partition
- Broker persists to disk, assigns offset
- Consumer polls batch, processes, commits offset
Rust producer (ingestion)
Uses rdkafka FutureProducer — async send with delivery callback.
Consumer groups
Multiple llm-worker instances share partitions — each partition consumed by one group member at a time.
Teach-back prompt
What happens if a consumer crashes before committing?
Rust producer
Lab complete — nice work.