These Apache Kafka interview questions cover core streaming concepts and how to build reliable producers and consumers with KafkaJS in Node.js.
15 Apache Kafka Interview Questions with Answers
Use the short version first, then offer trade-offs if the interviewer wants depth.
1. What problem does Apache Kafka solve compared to a traditional message queue?
Kafka stores events in a durable, ordered log that many consumers can read independently at different speeds. Unlike many queues that delete messages after ack, Kafka retains data for a configurable period, which supports replay, audit trails, and event-driven architectures. It excels at high-throughput stream processing rather than simple task dispatch.
2. Explain the relationship between topics, partitions, and brokers.
Brokers are Kafka servers that host topic data. Each topic is divided into partitions—append-only logs spread across brokers for scale and fault tolerance. Producers write to a specific partition (by key or round-robin), and replicas on other brokers provide durability if a broker fails.
3. Why do consumer groups matter, and how does partition assignment work?
A consumer group lets multiple consumer instances share work: each partition is assigned to exactly one consumer in the group at a time. If you add consumers, Kafka rebalances partitions until each active consumer owns a subset. You cannot scale consumption beyond the partition count without adding partitions first.
4. What is the purpose of a message key in Kafka?
The key determines the partition via hashing (unless a custom partitioner is used). Events with the same key land in the same partition, preserving order for that key. Use keys for entity-scoped ordering—e.g., all events for orderId 42 stay ordered—while still parallelizing across different keys.
5. Compare at-most-once, at-least-once, and exactly-once delivery.
At-most-once may lose messages if you commit offsets before processing completes. At-least-once (common default) commits after processing but can redeliver on failure, so handlers must be idempotent. Exactly-once requires transactional producers and idempotent consumers or stream processing APIs; it adds complexity and is not free in every client setup.
6. How do you create a producer with KafkaJS?
Instantiate Kafka with clientId and brokers, then const producer = kafka.producer(). Call await producer.connect(), then producer.send({ topic, messages: [{ key, value }] }). Always disconnect on shutdown. Set acks and compression in producer config when you need stronger durability or smaller payloads.
7. How do you build a consumer that processes messages safely?
Create kafka.consumer({ groupId }), connect, and await consumer.subscribe({ topic }). In consumer.run, parse eachMessage, perform work, then commit offsets only after success—or disable autoCommit and call commitOffsets manually. Wrap handlers with try/catch, log failures, and consider dead-letter topics for poison messages.
8. What happens during a consumer rebalance?
When consumers join, leave, or crash, the group coordinator reassigns partitions. In-flight messages may be reprocessed after reassignment, which is why idempotency matters. Use rebalance listeners in KafkaJS to pause processing and commit offsets before partitions are revoked to reduce duplicate work.
9. When would you increase the number of partitions on a topic?
Add partitions when consumer lag grows and you need more parallel readers, or when producer throughput exceeds what a single partition can handle. Increasing partitions does not reorder existing data but changes key-to-partition mapping for new records. Plan partition count early; reducing partitions is not supported.
10. What are Kafka retention and compaction policies?
Time- or size-based retention deletes old segments after a limit—good for event streams with a finite window. Log compaction keeps the latest record per key and tombstones deletes, useful for changelog topics like KTable state. Choose policy based on whether consumers need full history or only latest state.
11. How do you handle serialization in a Node.js Kafka service?
Kafka carries bytes; producers typically JSON.stringify or use Avro/Protobuf schemas before send. Consumers parse in eachMessage and validate shape before business logic. Schema Registry enforces compatibility in larger teams. Always version payloads or include a schema id so consumers can evolve safely.
12. What is consumer lag and why is it an important metric?
Lag is the difference between the latest offset on a partition and the consumer group's committed offset. Rising lag means consumers cannot keep up with producers. Alert on sustained lag, then scale consumers (within partition limits), optimize handler code, or add partitions and rebalance load.
13. How do idempotent producers work in Kafka?
Idempotent producers assign a PID and sequence numbers per partition so the broker deduplicates retries caused by network errors. Enable with enable.idempotence=true (KafkaJS maps this in producer config). This prevents duplicate writes from producer retries but does not alone guarantee end-to-end exactly-once consumption.
14. Describe a dead-letter topic (DLT) pattern with Kafka.
When a message fails processing after retries, publish it to a DLT with error metadata instead of blocking the partition. Operators inspect or replay DLT messages after fixes. In KafkaJS, catch handler errors, produce to orders.dlt with the original payload and stack trace, then commit offset on the main topic to move forward.
15. What operational concerns should you mention in a Kafka + Node.js system design answer?
Cover broker replication factor, min in-sync replicas, monitoring lag and under-replicated partitions, graceful shutdown (disconnect clients), and schema evolution. On the Node side, limit concurrent eachMessage work, use backpressure, and run consumers as separate processes or containers so one crash does not take down producers and consumers together.
Pro Tip
In interviews, sketch a flow: producer → topic/partition → consumer group → idempotent handler → offset commit. Mention one failure mode (rebalance or retry) and how your design recovers.