Where Does Kafka Belong in Your Architecture?

Hey there,

Maybe you’ve heard it before: ‘You can use Kafka as your database.’ 

You probably wondered if they're onto something or completely missing the point.

Kafka stores data. It keeps it durable and can serve up queryable views with tools like Kafka Streams. But does that make it a database?

In today's issue, we explore:

• What makes Kafka database-like (and where the similarities end).

• The differences that matter for your architecture decisions.

• When Kafka works as persistent storage, and when it absolutely doesn't.

Let's clear up the confusion.

Kafka wasn't built to be a database. It's an event streaming platform designed to move data between systems in real-time, but it has features that make it look suspiciously database-like. Here are a few:

Long data retention: Unlike most message queues, you can configure Kafka to keep data for weeks, months, or indefinitely. Your events don't disappear after consumption.

Partitioning for scale: Just like databases partition tables across servers, Kafka splits topics into partitions distributed across brokers for massive parallel processing.

Durability through replication: Every partition replicates across multiple brokers. If one server fails, your data remains safe and accessible. 

Log compaction: Kafka can retain only the latest value for each key, functioning like a key-value store where the most recent state is always available.

Despite its storage qualities, Kafka operates on different principles. Understanding these gaps before building on it can help avoid costly architectural mistakes:

No complex queries: Databases excel at joining tables, filtering with WHERE clauses, and running aggregate functions. Kafka doesn't support SQL queries natively; you would need external tools or systems for complex data retrieval.

Retention isn't automatic: Databases assume permanent storage by default. Kafka requires explicit configuration to retain data long-term, and even then, it's optimized for streaming workloads, not historical queries.

Backup mechanisms differ: Databases have built-in backup and recovery features. Kafka uses replication for durability, but true backups require complex cross-cluster setups.

Reach for traditional databases when you need:

• Complex querying: Multi-table joins, subqueries, and ad-hoc analytical queries that require a full query engine.

• Point-in-time lookups: Fast retrieval of specific records by arbitrary attributes using proper indexing.

• Transactional guarantees: ACID transactions across multiple operations that require database transaction managers.

• Data normalization: Structured schemas that rely on constraints, relationships, and normalized tables.

Winning architectures often combine both: Kafka for event streaming and real-time processing, databases for persistent storage, and complex queries.

Read the full blog post here for deeper insights into Kafka's database-like properties and when to use them.

Beyond just moving messages, Kafka’s storage enables durable event-driven architectures. Here are some of the best-fit scenarios:

Event sourcing architectures: Kafka excels in storing every state change as an immutable event and rebuilding application state by replaying the event log. 

Real-time analytics pipelines: It processes streaming data as it arrives, maintains materialized views with Kafka Streams, and serves results immediately.

Audit logs and compliance: Kafka’s immutable, append-only topics with long retention periods are well-suited for tamper-proof audit trails.

Cache warming and data synchronization: Use Kafka topics as the source of truth to keep multiple systems synchronized with consistent, ordered data.

Deepen your understanding of Kafka's capabilities with these focused resources:

Why Python Data Engineers Should Know Kafka and Flink - Why Kafka and Flink matter for modern data pipelines and Python engineers.

Apache Kafka Deep Dive: Core Concepts and Real-World Applications - Comprehensive guide covering Kafka fundamentals and production deployment with practical examples.

Kafka Streams FAQ - Quick answers to common Kafka Streams questions on architecture, state management, and implementation.

Kafka Streams Architecture Explained - How Kafka Streams works under the hood, including topology design and windowing operations.

Read the full blog post here for deeper insights into Kafka's database-like properties and when to use them.

And it’s a wrap!

See you Friday for the week’s news, upcoming events, and opportunities.

If you found this helpful, share this link with a colleague or fellow DevOps engineer.

Divine Odazie
Founder of EverythingDevOps