Technology deep-dive

Kafka architecture deep-dive for security data.

Apache Kafka shows up in nearly every security pipeline I evaluate, usually inherited from a data engineering team that picked the broker count, partition count, and replication factor without consulting security. Then lag spikes drop alerts, one partition takes 80% of the load, or a broker fails and events disappear. This essay walks through what topics, partitions, consumer groups, and replication actually do (the way I wish someone had explained it to me before I owned a Kafka cluster) and where partition strategy for security workloads tends to break.

Reading time: about 17 minutes. Evidence tier: B overall (Apache Kafka documentation plus published engineering blogs from Netflix, LinkedIn, Confluent), with directional throughput and cost numbers flagged for verification against your own workload.

Topics: append-only event streams

A Kafka topic is a named event stream: a category or feed. In a security pipeline, the topics tend to map cleanly to event sources: cloudtrail-events for AWS CloudTrail logs, edr-telemetry for endpoint detection events, firewall-logs for network firewall events, saas-audit-logs for SaaS application audit trails. Producers write to topics; consumers read from them.

The structural property that matters most: topics are append-only commit logs. Events written to a topic are immutable. They can be appended, never modified, and Kafka retains them for a configured period (commonly seven days, sometimes thirty, occasionally forever). That immutability is what makes Kafka usable as a replay buffer for security investigations months after the original event landed.

Partitions: the parallelism unit

A single topic on a single broker can't absorb millions of events per second. The fix is to split the topic into partitions: independent ordered logs that can live on different brokers and be consumed in parallel. Twelve partitions on a topic means up to twelve consumers can read from it simultaneously, each one owning a slice.

How an event lands in a specific partition depends on the producer's choice of key. When the producer sends an event with a key (say user_id, src_ip, or endpoint_id), Kafka hashes the key and takes the result modulo the partition count. That formula guarantees two useful properties at once. Events with the same key always go to the same partition, which means ordering is preserved within that key (every event for a given user, in the order it happened). And events with different keys are spread across partitions, which provides the parallelism.

The trade-off matters. More partitions means more parallelism and higher throughput, but each partition costs file descriptors, memory, and metadata overhead on every broker. The Apache Kafka documentation has historically suggested a soft ceiling around 4,000 partitions per broker and 200,000 partitions per cluster as a practical limit before metadata operations slow down. These are guidance numbers, not hard caps, and they may move as KRaft (which replaced ZooKeeper for metadata) matures. Verify against your Kafka version before sizing close to them.

Consumer groups: load balancing without duplication

Most security pipelines need the same event stream fed to multiple destinations: the SIEM for real-time alerting, the data lake for historical hunting, and a real-time analytics database like ClickHouse for dashboards. Kafka handles this through consumer groups, which coordinate partition assignment so each partition is consumed by exactly one member of the group.

Three consumers in a single group named data-lake-writer, reading a topic with twelve partitions, will each be assigned four partitions. If one consumer dies, Kafka rebalances and the survivors split the dead consumer's partitions between them. If a new consumer joins, Kafka rebalances again to spread load. Within a group, no event is processed twice.

Multiple consumer groups read independently. A second group named splunk-ingest can read every partition of the same topic without knowing or caring that data-lake-writer exists. That fan-out is the architectural reason Kafka sits at the center of so many security pipelines: one event stream, multiple independent destinations, no duplication of the source data on disk.

Offsets: the consumer's bookmark

Within a partition, every event has a monotonically increasing offset. Consumers commit the highest offset they've successfully processed back to Kafka. On restart, they resume from the next offset rather than re-reading the entire topic from the start.

The commit strategy choice carries weight for security pipelines. Auto-commit, the default, commits every five seconds in the background, which is fast, but if the consumer crashes between commits you lose or duplicate up to five seconds of events. Manual commit, after every successful processing step, gives exactly-once semantics at the cost of throughput, while batch commit every thousand events is the common compromise between the two.

For security workloads I default to manual or batch commit, because auto-commit on a critical-alert pipeline is the kind of decision that produces a stack trace on the second-worst day of your on-call rotation, and the throughput cost you pay to avoid that is rarely the binding constraint when alert reliability is on the line.

Kafka offers three delivery semantics, with steepening trade-offs. At-most-once fires the event and moves on; if the broker crashes before writing, the event is lost. At-least-once waits for broker acknowledgement and retries on failure, which means an acknowledged-but-lost ack will cause the producer to retry and produce a duplicate. Exactly-once, introduced in Kafka 0.11 via KIP-98 and extended in KIP-447, uses idempotent producers and transactional writes to guarantee neither loss nor duplication.

Exactly-once is not free. Confluent's published numbers suggest roughly a 20–30% throughput reduction versus at-least-once delivery, with the overhead varying by batch size and replication factor. Older guidance cited up to 50% overhead; the current number depends on Kafka version and configuration, so I'd treat the 20–30% as directional and verify against your own workload before committing.

For security pipelines, the choice is per-topic, not per-cluster. Compliance-critical events (audit logs that must be defensible in regulatory review, financial transaction logs feeding fraud detection) may justify exactly-once. High-volume telemetry (EDR events, firewall logs, DNS queries) usually doesn't, because the downstream data lake or SIEM can deduplicate on event ID cheaper than Kafka can guarantee no-duplicates upstream.

At-most-once is rarely the right call for security data, because the throughput win is small while the forensic cost of a silently dropped event is unbounded.

This is the section where Kafka guides written for data-engineering audiences and Kafka guides written for security teams diverge. The standard rule of thumb (partitions equal target throughput divided by single-partition throughput) gets you to a reasonable count, but it doesn't tell you how to pick the partition key, and security telemetry has structural properties that punish the wrong choice.

High-cardinality keys are good. Low-cardinality keys cause hotspots.

The textbook partition keys for security data are user ID (for authentication logs), source IP (for network and firewall logs), and endpoint ID (for EDR telemetry). Each preserves ordering within the entity that matters for detection: same user's failed logins in sequence, same IP's port-scan probes in order, same endpoint's process tree intact.

The failure mode shows up when the key distribution is skewed. Authentication logs partitioned by user_id may have one service account or one CI/CD pipeline user generating 60% of all events, so that user's partition gets six times the load of the others and the consumer assigned to it falls behind, which means detection lag against that user (often the most security-relevant user) becomes the longest in the system.

Source IP has the same failure mode in different clothing. A handful of internal load balancers, proxies, or NAT gateways may emit a disproportionate fraction of network events. Partition by src_ip and those few IPs drown out everyone else. Endpoint ID is usually less skewed because endpoints are humans-shaped distributions, but a noisy server endpoint with an unbounded process-spawn loop can still create a hot partition for hours.

Composite keys may help, with caveats

The pattern I've seen used to spread hotspots is a composite key, for example user_id concatenated with a time bucket like floor(timestamp / 60s). That preserves per-minute ordering for a given user (usually enough for detection rules that look for bursts of activity) while spreading a heavy user's events across multiple partitions over time.

The caveat is that you lose strict cross-minute ordering. A detection rule that needs to see all failed logins for a user in order across an hour may have to re-sort downstream, which adds latency and complexity. For burst-detection rules (five failed logins in sixty seconds) the per-minute bucket is fine. For longer windows, the composite key may cause more pain than it saves.

Another option is random round-robin partitioning (no key), which maximizes parallelism and eliminates hotspots entirely at the cost of losing the per-entity ordering guarantees. That's the right choice when downstream processing is order-independent (write-into-Iceberg jobs, archive dumps to S3) and the wrong choice for stream-processing detection, so like the other partition questions it gets settled per-topic rather than once for the whole cluster.

Partition count: how many is enough?

Single-partition throughput on the consumer side is usually limited by consumer processing speed, not Kafka itself. A Python consumer that does ten milliseconds of per-event work tops out around a hundred events per second per partition; a Rust or Go consumer doing simple shape-and-forward work may handle five to ten thousand. So partition count depends as much on what the consumers do as on what the producers emit.

For EDR telemetry at 100,000 events per second with a consumer that processes 2,000 events per second per thread, you need at least fifty partitions, plus a margin on top of that for rebalances and consumer failures. Forty-eight partitions is a common starting number for heavy security topics (it's also a friendly divisor: works for 1, 2, 3, 4, 6, 8, 12, 16, 24, or 48 consumers). Twelve partitions is a common starting number for moderate topics like CloudTrail, and six is the usual starting number for low-volume topics like SaaS audit logs.

Over-partitioning is a softer mistake than under-partitioning. Increasing partition count later requires either creating a new topic (which means rewriting producers and consumers) or accepting that newly created partitions break key-based ordering for existing keys. Start with more partitions than you think you need, within the soft limit guidance above.

The default Kafka retention is seven days, configured via log.retention.hours at the broker or retention.ms at the topic level. For security pipelines, the topic-level override is the one that matters; different event types have wildly different replay needs and wildly different costs.

High-volume topics like EDR telemetry may justify a shorter Kafka retention window (say, 24 to 48 hours) because anything beyond that is going to be served from the data lake anyway. Why keep eight terabytes per day in Kafka for a week when the same data is also being written to Iceberg for ninety-day hot retention and S3 Glacier for longer? Kafka's job in that pipeline is the replay buffer for fixing a broken consumer, not the long-term archive.

Lower-volume but higher-replay-value topics (CloudTrail audit logs, identity provider events, Sigma-rule-driven alerts) may justify longer Kafka retention (thirty days, sometimes longer) because the cost is small and the convenience of replaying directly from Kafka rather than rebuilding from the lake matters when a detection rule has a bug.

Tiered storage changes this calculus in the direction of "keep more, pay less." With KIP-405 in place, the cost difference between seven-day and thirty-day Kafka retention on a high-volume topic shrinks significantly because the older days move to S3 automatically. I'd revisit the retention defaults whenever tiered storage gets enabled.

The two metrics that catch most production problems in security Kafka pipelines are consumer lag and partition skew, in that order.

Consumer lag (the difference between the latest offset on a partition and the consumer group's committed offset) is the canary for downstream processing health. If the SIEM-ingest consumer group's lag on the EDR topic starts climbing, detections are arriving late. The threshold worth alerting on depends on the topic: for real-time alert paths, anything more than a few seconds of lag is worth investigating; for archive-to-lake paths, several minutes may be fine.

Partition skew, the ratio of events on the hottest partition to the average, is the canary for partition-strategy problems. A 2x skew is usually fine. A 6x skew (one partition with 60% of the load that should be 17%) means the partition key is wrong for this workload. Most Kafka monitoring dashboards don't surface skew by default; it's worth building.

Other metrics worth tracking: under-replicated partitions (replication is failing; investigate immediately), broker disk usage (storage exhaustion takes the cluster down hard), ISR shrink rate (replication is degrading), and request handler idle ratio (broker is CPU-saturated). Prometheus plus Grafana with the Kafka JMX exporter is the open-source default; Confluent Control Center, Datadog, and New Relic offer commercial equivalents.

For security teams specifically, I'd add one more: the lag on the dead-letter-queue topic, if you have one. Events that failed downstream processing should not silently accumulate in the DLQ forever. A growing DLQ is a detection-coverage problem disguised as a Kafka problem.