Event-driven architecture with Apache Kafka: When it makes sense and when it does not

The appeal of event-driven architecture

Event-driven architecture promises loose coupling, independent scalability, and natural audit trails. When done well, it delivers on all three. When done poorly, it creates a distributed system that is harder to debug than the monolith it replaced.

Apache Kafka is the most popular platform for event-driven systems. It is also one of the most frequently misused.

When Kafka makes sense

Kafka excels in scenarios with high-throughput, real-time data flows between multiple consumers:

• Event sourcing: When you need a complete, ordered history of every change to your system state.
• Stream processing: When you need to process, aggregate, or transform data in real time as it arrives.
• Integration hub: When you need to connect multiple systems that should not know about each other.
• Log aggregation: When you need to collect and distribute log data from hundreds of services.

When Kafka is overkill

If your system handles fewer than 10,000 events per day, if you have fewer than three consumers, or if your team has never operated a distributed system, Kafka will add complexity without proportional benefit.

A simple message queue (RabbitMQ, AWS SQS) or even a database-backed job queue might be the right choice. The best architecture is the simplest one that meets your actual requirements.

Operational reality

Kafka clusters require monitoring, tuning, and ongoing maintenance. Partition rebalancing, consumer lag monitoring, retention policy management, and cluster upgrades are all operational concerns that your team needs to handle. Factor this into your decision.