Apache Kafka — Complete Study Notes

27 June, 2025

How to use these notes: Read top to bottom. Each section builds on the previous one. By the end, you'll be able to explain Kafka in an interview, draw a whiteboard diagram, and know exactly how Spring Boot connects to a Kafka cluster.

Table of Contents

• 1. What is Kafka?
• 2. Core Concepts & Terminology
• 3. Topics
• 4. Partitions
• 5. Brokers & Clusters
• 6. Partition Leader & Follower
• 7. Replication Factor
• 8. Producers
• 9. How Spring Boot Bootstraps & Routes Messages
• 10. Consumers & Consumer Groups
• 11. Kafka Connect
• 12. How to Decide Partition Count
• 13. Creating a Topic — CLI Commands
• 14. Starting Kafka Locally (KRaft Mode)
• 15. Real-World Architecture Example
• 16. Interview Cheat Sheet

1. What is Kafka?

Apache Kafka is an Event Streaming Platform.

Event Streaming means:

• Capturing data in real time from different sources, in the form of streams of events.
• Storing these event streams durably for later retrieval.
• Processing / reacting to event streams in real time and retrospectively.

Kafka's 3 Core Capabilities

All of this is provided in a distributed, fault-tolerant, elastic, and secure manner. Kafka can be deployed on bare metal, VMs, on-premises, or in the cloud.

2. Core Concepts & Terminology

Events

An event is "something that happened" — a record or message.

When you read or write data to Kafka, you do it in the form of events.

Every event conceptually has:

• Key — identifies the entity (e.g., transactionId, userId)
• Value — the actual payload (e.g., JSON data)
• Timestamp — when the event occurred
• Optional metadata headers

Producers

Producers are client applications that publish (write) events to Kafka.

• Producers are fully decoupled from consumers.
• A producer does not wait to know if or when an event is consumed.
• This decoupling is one of Kafka's biggest design strengths.

Consumers

Consumers are client applications that read and process events from Kafka.

• Consumers are independent microservices.
• Events in Kafka are not deleted after consumption — they can be re-read as many times as needed.
• You define how long events are retained using per-topic configuration (retention period).

3. Topics

A Topic is the fundamental way to organize data in Kafka.

Think of a topic like a *folder in a filesystem*, and events are the *files* inside that folder.

Key Properties of a Topic

Naming Convention (Best Practice)

Use descriptive, hyphen-separated names. Examples:

• payments-processed
• user-signup-events
• order-created

4. Partitions

A Partition is the physical subdivision of a Kafka topic. This is the most important concept for scalability.

A Topic is *logical*. A Partition is *physical* (an actual append-only log on disk).

Instead of storing all events of a topic in one place, Kafka splits (shards) the topic into multiple partitions spread across brokers.

Why Partitions Exist

How Events Land in a Partition

Visual: Partition as Append-Only Log

Topic: orders
-----------------------------
P0 → [event1] [event2] [event3]  ← appended in order
P1 → [event4] [event5]
P2 → [event6] [event7]

Events within a partition are ordered and immutable. New events are only ever appended to the end.

5. Brokers & Clusters

Broker

A broker is a single Kafka server (a single running process/node).

• Each broker:
  • Stores partitions (data) on disk
  • Handles read/write requests from producers and consumers
  • Participates in replication

Cluster

A Kafka Cluster is a group of multiple brokers working together.

Kafka Cluster
┌───────────────────────────────────────┐
│  Broker 1    Broker 2    Broker 3     │
│  (Node 1)    (Node 2)    (Node 3)     │
└───────────────────────────────────────┘

• Clusters can span multiple data centers or cloud regions.
• Brokers share partition leadership — each broker is the leader for some partitions and a follower for others.
• Adding more brokers = horizontal scaling.

ZooKeeper vs KRaft

*Use KRaft mode for all new setups.* This is what the local setup commands below use.

6. Partition Leader & Follower

This is the most important concept for understanding how Kafka achieves both performance and fault tolerance.

The Rule

Every partition has:

• Exactly 1 Leader broker — handles ALL reads and writes for that partition
• 0 or more Follower brokers — maintain replicas (copies) of the partition data

Partition P1:
┌──────────────────────────────────────┐
│  Broker-2 (Leader)  ← ALL traffic   │
│  Broker-1 (Follower) ← replica only │
│  Broker-3 (Follower) ← replica only │
└──────────────────────────────────────┘

Why One Leader?

• Simpler consistency model — no risk of stale reads.
• The leader is the single source of truth for a partition.
• Followers only exist for replication and failover.

What Happens When a Leader Fails?

Kafka detects that the leader broker is down.

Kafka automatically elects a new leader from the in-sync followers.

Producers and consumers automatically reconnect to the new leader.

No manual intervention required.

This is how Kafka achieves fault tolerance.

7. Replication Factor

The Replication Factor (RF) defines how many copies of a partition exist across the cluster.

Replication Factor = 3
  → 1 leader copy + 2 follower copies = 3 total replicas

Example: Topic with RF=3 and 2 Partitions, 3 Brokers

          Broker-1        Broker-2        Broker-3
          --------        --------        --------
          P0 (Leader)     P1 (Leader)     P0 (Follower)
          P1 (Follower)   P0 (Follower)   P1 (Follower)

• Broker-1 is leader for P0
• Broker-2 is leader for P1
• Broker-3 only holds follower replicas

What RF to Use in Production?

*Rule of thumb:* RF should never exceed the number of brokers.
RF = 3 requires at least 3 brokers.

How Replication Works (Step by Step)

Producer writes to the partition leader.

Leader writes to its local log.

Followers pull data from the leader and replicate it.

Once followers have copied the data, they are considered in-sync (part of the ISR — In-Sync Replica set).

Data is now durable.

Without vs With Replication

WITHOUT replication:
  Broker crashes → partition data gone forever ❌

WITH replication (RF=3):
  Broker crashes → follower becomes new leader → no data loss ✅

8. Producers

How a Producer Routes a Message

Producer connects to any broker listed in bootstrap-servers (initial contact only).

Downloads cluster metadata — a map of which broker is the leader for each partition.

Determines target partition (via key hash, round-robin, or manual override).

Sends the message directly to the leader broker of that partition.

Leader replicates to followers.

Producer acks Setting

The acks setting controls durability guarantees:

9. How Spring Boot Bootstraps & Routes Messages

This is a very common interview question. Understanding this end-to-end is essential.

Configuration

# application.properties
spring.kafka.bootstrap-servers=broker1:9092,broker2:9092,broker3:9092
spring.kafka.producer.key-serializer=org.apache.kafka.common.serialization.StringSerializer
spring.kafka.producer.value-serializer=org.apache.kafka.common.serialization.StringSerializer

*Common misconception:* "The producer only talks to the bootstrap server."
*Reality:* Bootstrap servers are just the *initial contact point*. After the first handshake, the producer talks directly to whichever broker is the partition leader.

Step-by-Step: What Happens When Spring Boot Sends a Message

Step 1: Bootstrap Connection
      Spring Boot Producer
             │
             │ Initial connection (just for metadata)
             ▼
         Broker-1 (any broker in list)
             │
             │ Responds with cluster metadata:
             │   "Partition 0 → Leader: Broker-1"
             │   "Partition 1 → Leader: Broker-2"
             │   "Partition 2 → Leader: Broker-3"
             ▼

Step 2: Partition Selection
      Producer hashes the message key
      → selects Partition 1

Step 3: Direct Send to Leader
      Producer sends message DIRECTLY to Broker-2
      (leader of Partition 1)

Step 4: Replication
      Broker-2 (Leader)
         │ replicates
         ├──→ Broker-1 (Follower of P1)
         └──→ Broker-3 (Follower of P1)

Whiteboard Diagram

        +----------------------+
        |  Spring Boot Producer|
        +----------+-----------+
                   │
                   │ (1) Bootstrap connection
                   ▼
        +----------------------+
        |   Broker-1           |
        | (Metadata request)   |
        +----------+-----------+
                   │
                   │ (2) Metadata response:
                   │     Partition → Leader mapping
                   ▼

     Topic: payments
     -------------------------
     Partition 0 → Broker-1
     Partition 1 → Broker-2
     Partition 2 → Broker-3

                   │
                   │ (3) Select partition (key hash / round-robin)
                   ▼
        +----------------------+
        |   Broker-2           |  ← Leader of Partition 1
        +----------+-----------+
                   │
                   │ (4) Replication
         ──────────────────────────────
         │                            │
         ▼                            ▼
  +-------------+           +-------------+
  | Broker-1    |           | Broker-3    |
  | (Follower)  |           | (Follower)  |
  +-------------+           +-------------+

Interview Answer (Crisp & Complete)

"Producer connects to any bootstrap broker to fetch cluster metadata — topics, partitions, and their leader brokers. When sending a message, Kafka determines the target partition based on the message key (via hashing) or using round-robin if no key is provided. The producer then sends the message *directly to the leader broker* of that partition, not necessarily the bootstrap broker. After receiving the message, the leader replicates it to follower brokers. If a broker goes down, Kafka elects a new leader and the producer automatically updates its metadata and continues — no manual intervention needed."

10. Consumers & Consumer Groups

Consumer

A consumer reads events from one or more partitions of a topic.

• Consumers track their position using offsets — an incrementing number that marks which events have been read.
• Offsets are committed to Kafka (in the __consumer_offsets internal topic).
• A consumer can replay events by resetting its offset.

Consumer Group

A Consumer Group is a set of consumers that work together to consume a topic.

*Core rule:* Within a consumer group, each partition is consumed by *at most one consumer*. But multiple consumer groups can each independently consume the same topic.

Partition Assignment Within a Group

Topic: orders (3 partitions)
Consumer Group: payment-service

P0 → Consumer-1
P1 → Consumer-2
P2 → Consumer-3

• If you have more consumers than partitions, the extra consumers sit idle.
• If you have fewer consumers than partitions, some consumers handle multiple partitions.

Scaling Consumers

3 partitions, 3 consumers → perfect parallelism ✅

         P0    P1    P2
          │     │     │
          C1    C2    C3


3 partitions, 2 consumers → C1 handles 2 partitions

         P0    P1    P2
          │     │     │
          C1   C1    C2


3 partitions, 4 consumers → C4 is idle

         P0    P1    P2   (nothing)
          │     │     │      │
          C1    C2    C3     C4 ← idle ❌

Maximum useful parallelism = number of partitions

Multiple Consumer Groups (Fan-Out Pattern)

This is where Kafka truly shines. The same data can be consumed independently by completely different services.

                Topic: payments (2 partitions)
               ─────────────────────────────────
                   P0                 P1


Group A (Payment Service):    C1               C2

Group B (Fraud Detection):    C3 ← reads both P0 and P1

Group C (Analytics):          C4 ← reads both P0 and P1

Group D (Audit Logging):      C5 ← reads both P0 and P1

• Each group has its own independent offset — they read at their own pace.
• One group lagging does not affect any other group.
• This is the fan-out pattern — publish once, consume many times for different purposes.

Real-World Example

Same payment event is consumed by:

• Payment Service → processes and updates the database
• Fraud Detection → checks for suspicious patterns
• Analytics Service → aggregates revenue metrics
• Audit Logger → writes to compliance logs

Interview Answer

"If consumers belong to *different consumer groups*, each group independently consumes the same data. The one-partition-per-consumer rule applies only _within_ a consumer group. So multiple services can each receive every event, enabling fan-out patterns. Offsets are tracked per consumer group, so each group reads at its own pace without affecting others."

11. Kafka Connect

Kafka Connect is a framework for moving data into and out of Kafka without writing custom code.

Why It Exists

In enterprise systems, Kafka acts as a central data backbone. But you need to:

Import data INTO Kafka from existing systems (databases, file systems, etc.)

Export data OUT OF Kafka to downstream systems (search engines, data warehouses, etc.)

Writing this integration code manually is error-prone, repetitive, and fragile. Kafka Connect solves this with ready-made connectors.

Import (Source Connectors)

Data flows into Kafka from:

• SQL databases (MySQL, PostgreSQL, Oracle) — via CDC (Change Data Capture)
• File systems (logs, CSV dumps)
• Message queues (RabbitMQ, ActiveMQ)

Export (Sink Connectors)

Data flows out of Kafka to:

• Elasticsearch (search)
• Data warehouses (Snowflake, BigQuery, Redshift)
• Object storage (AWS S3, GCS)
• Other Kafka clusters

Why Not Read Directly from the Database?

Architecture Diagram

Source Systems         Kafka              Sink Systems
─────────────     ──────────────     ──────────────────
  MySQL DB    →   │            │  →    Elasticsearch
  PostgreSQL  →   │   Kafka    │  →    Snowflake DW
  File System →   │  Cluster   │  →    AWS S3
  Oracle      →   │            │  →    Another Kafka
─────────────     ──────────────     ──────────────────
                  ↑ Source          ↑ Sink
                  Connectors        Connectors

12. How to Decide Partition Count

Choosing the right number of partitions is a critical design decision. Too few = bottleneck. Too many = overhead.

Why Partition Count Matters

BAD DESIGN (too few partitions — bottleneck):

        Topic (1 Partition)
                │
                ▼
           Broker-1 (Leader)
        (ALL traffic hits here) ❌


GOOD DESIGN (distributed load):

        Topic (6 Partitions)

   P0 → Broker-1      P3 → Broker-1
   P1 → Broker-2      P4 → Broker-2
   P2 → Broker-3      P5 → Broker-3

   ✔ Load spread across all brokers
   ✔ Parallel producers & consumers

Rules for Deciding Partition Count

Practical Formula

Desired partition count ≈ max(
    target throughput / throughput per partition,
    max consumer instances you'll scale to
)

Real Example

• Target: 600 MB/s throughput
• Throughput per partition: ~100 MB/s
• Max consumers you plan to scale to: 12

→ Use 12 partitions (covers both throughput and consumer parallelism)

Important Caveat

You can *increase* partitions later, but you *cannot decrease* them.
Increasing partitions can also break key-based ordering for existing keys.
So: *over-partition slightly* rather than under-partition.

General Recommendations

13. Creating a Topic — CLI Commands

Start Kafka Locally (KRaft Mode)

# Step 1: Generate Cluster ID
KAFKA_CLUSTER_ID="$(./bin/kafka-storage.sh random-uuid)"

# Step 2: Format storage
./bin/kafka-storage.sh format --standalone -t $KAFKA_CLUSTER_ID -c config/server.properties

# Step 3: Start the broker
./bin/kafka-server-start.sh config/server.properties

Topic Operations

# Create a topic
./bin/kafka-topics.sh \
  --create \
  --topic payments \
  --partitions 3 \
  --replication-factor 1 \
  --bootstrap-server localhost:9092

# Describe a topic (shows partition distribution, leaders, ISR)
./bin/kafka-topics.sh \
  --describe \
  --topic payments \
  --bootstrap-server localhost:9092

# List all topics
./bin/kafka-topics.sh \
  --list \
  --bootstrap-server localhost:9092

# Delete a topic
./bin/kafka-topics.sh \
  --delete \
  --topic payments \
  --bootstrap-server localhost:9092

Producer & Consumer (CLI Testing)

# Start a console producer
./bin/kafka-console-producer.sh \
  --topic payments \
  --bootstrap-server localhost:9092

# Start a console consumer (read from beginning)
./bin/kafka-console-consumer.sh \
  --topic payments \
  --from-beginning \
  --bootstrap-server localhost:9092

# Consumer in a group
./bin/kafka-console-consumer.sh \
  --topic payments \
  --group payment-service \
  --bootstrap-server localhost:9092

14. Starting Kafka Locally (KRaft Mode)

What KRaft Mode Is

Before Kafka 3.x, Kafka required ZooKeeper as a separate service to manage cluster metadata (leader elections, broker registry, etc.).

KRaft (Kafka Raft) removes this dependency. Kafka now manages its own metadata internally using a built-in Raft consensus protocol.

Process Roles in KRaft

In server.properties:

# Combined mode (broker + controller in one process) — good for local dev
process.roles=broker,controller

# Recommended log directory configuration
log.dirs=/path/to/kafka-broker-logs
metadata.log.dir=/path/to/kafka-metadata-logs

For production, these directories should be on separate disks for reliability.

Why Two Log Directories?

Separating them ensures metadata writes (which need low latency) don't compete with data writes.

15. Real-World Architecture Example

Payment Processing Platform

This is a real architecture using Kafka to decouple services:

External        Config      Kafka          Middleware     Targets
Sources         Manager    Cluster         Service        (3rd Party)
─────────      ─────────  ─────────       ─────────      ─────────
                          ┌─────────┐
 Merchant  →  CM Service →│bin data │→ MW consumes →    Cache
 Data                     │merchant │   merchant &       (Redis)
                          │ topic   │   bin data     →  Payment
                          └─────────┘                   Processor A
                                                    →  Payment
                          ┌─────────┐                   Processor B
 Transaction →  App      →│txn      │→ TLM consumes → DB (save txn)
 Events                   │events   │
                          │ topic   │
                          └─────────┘
                           Kafka Cluster

Flow:

CM (Config Manager) publishes merchant and BIN data → Kafka topic.

Middleware (MW) consumes this data → stores in Redis cache.

During a transaction, MW checks Redis first; if cache miss → falls back to CM API.

Transaction data is produced to a Kafka topic → TLM (Transaction Ledger Manager) consumes → saves to DB.

Frontend dashboards (merchants, transactions, monitoring) query the DB and cache.

Why Kafka here?

• CM doesn't get hammered by MW polling it constantly — data is pushed via Kafka.
• Multiple consumers (MW, analytics, audit) all independently read transaction events.
• If MW goes down, events are retained in Kafka and processed when it recovers.

16. Interview Cheat Sheet

One-Line Answers

Key Numbers to Remember

"Pro Tip" Phrases for Interviews

• "Ordering is guaranteed within a partition, not across partitions."
• "The bootstrap server is just an initial contact point — the producer routes directly to the partition leader."
• "Kafka scales throughput not by reading from followers, but by increasing partition count."
• "Consumer groups allow fan-out — the same event is processed independently by multiple services."
• "Replication factor ensures durability — if a leader fails, a follower is automatically elected."
• "The maximum useful consumer parallelism in a group equals the number of partitions."

These notes cover: Event Streaming, Kafka Architecture, Topics, Partitions, Brokers, Leaders & Followers, Replication, Producers, Spring Boot Bootstrap, Consumer Groups, Kafka Connect, Partition Sizing, Topic CLI Commands, and KRaft mode.