Real-Time Vector Ingestion Engine

A production-shaped ingestion system for keeping retrieval indexes fresh under streaming load. The goal was to build the part of RAG infrastructure that is easy to underestimate: getting documents from an event stream into a vector database quickly, safely, and observably.

Why

Most RAG demos assume the corpus is already embedded. Real systems are messier. Documents arrive continuously, embedding providers throttle, payloads fail validation, and duplicate events happen. I wanted an ingestion path that treated freshness, idempotency, and failure recovery as first-class design constraints.

Architecture

The pipeline consumes document events from Kafka, applies validation and normalization, batches embedding requests, and writes idempotent upserts into pgvector. Failed payloads move to a DLQ with retry metadata instead of disappearing into logs.

Provider fallback keeps ingestion moving when the primary embedding API slows down. Backpressure is applied at the consumer layer so the system can slow intake without overwhelming downstream storage.

What I verified

• Sustained 500 docs/sec on Confluent Cloud during load testing
• p95 freshness under 10 seconds from event arrival to searchable vector
• Idempotent upserts for duplicate document events
• DLQ retries for malformed payloads and transient provider failures
• Prometheus metrics for lag, embedding latency, upsert latency, and failure rates

What I learned

Vector search quality depends on retrieval design, but retrieval reliability starts earlier. The ingestion system is where stale context, partial failures, and cost surprises usually begin. Treating it like a streaming data platform makes the RAG system easier to trust.