Apache Kafka Cheat Sheet

How to use this Apache Kafka cheat sheet

Topics split into partitions ordered by offset. Producers choose partition by key hash or round-robin. KafkaJS is the common Node.js client for producers and consumers with configurable acks and retry semantics.

Consumer groups assign partitions exclusively per member—rebalance on join/leave. Idempotent producers and exactly-once semantics require enable.idempotence and transactional settings. Monitor lag and use dead-letter topics for poison messages.

Quick Apache Kafka example

import { Kafka } from 'kafkajs';

const kafka = new Kafka({ clientId: 'my-app', brokers: ['localhost:9092'] });
const producer = kafka.producer({ allowAutoTopicCreation: false });
await producer.connect();
await producer.send({
  topic: 'orders',
  messages: [{ key: 'user-42', value: JSON.stringify({ id: 1, total: 99 }) }],
});

Topics, Partitions & Offsets

KafkaJS Producer & Consumer

Acks & Consumer Groups

Idempotence & Reliability

Consumer with manual commit

await consumer.run({
  autoCommit: false,
  eachMessage: async ({ topic, partition, message }) => {
    await processMessage(message);
    await consumer.commitOffsets([
      { topic, partition, offset: (Number(message.offset) + 1).toString() },
    ]);
  },
});

Commit only after successful processing for at-least-once semantics.

Producer with idempotence

const producer = kafka.producer({
  idempotent: true,
  maxInFlightRequests: 5,
  retry: { retries: 5 },
});

Requires acks=all and enables deduplication of producer retries.

Admin create topic

const admin = kafka.admin();
await admin.connect();
await admin.createTopics({
  topics: [{ topic: 'orders', numPartitions: 6, replicationFactor: 3 }],
});
await admin.disconnect();

Disable allowAutoTopicCreation in production; provision topics explicitly.

Common mistakes

• Using auto-commit before processing finishes, losing messages on crash.
• Too few partitions limiting consumer parallelism for high-throughput topics.
• Ignoring consumer lag until processing is hours behind production traffic.
• Sending giant messages near the 1 MB default limit instead of referencing S3.

Key takeaways

• Partition by key for ordering; increase partitions for throughput.
• Use acks=all and idempotent producers for durable writes.
• Consumer groups scale horizontally; one consumer per partition max.
• Manual commits after processing give at-least-once delivery you control.