Key Technologies

Apache Flink in Practice

This is the second half of the Flink guide. The first, Apache Flink: Core Concepts, built the model: a job is a dataflow graph of operators running as parallel subtasks; keyBy routes each key to the one subtask that keeps that key's state locally; event time plus watermarks make results correct on late, out-of-order data; and windows slice the infinite stream into finite answers.

This half is about running that machinery in the real world: how a job survives crashes without losing count (checkpoints), how far exactly-once really goes, what happens when a sink slows down (backpressure), how Flink compares to Kafka Streams and Spark Structured Streaming, detecting event sequences with CEP, and the two systems interviewers reach for again and again, card fraud detection and IoT telemetry, worked end to end with code.

Checkpoints: Surviving Failures Without Losing Count

Start with the hard question. Subtasks hold state in local memory or on local disk, machines die, and a stream cannot be recomputed from the beginning of time. How does a job that has been counting for six months survive losing a TaskManager? Flink's answer is the checkpoint: at an interval you choose (say, every minute), the job takes a consistent snapshot of all state across all subtasks, together with the exact source positions (Kafka offsets) that state corresponds to, and writes the snapshot to durable storage such as S3 or HDFS. Consistent is the load-bearing word: the state and the offsets describe the same moment, so "counted through offset 41,230" and "the count is 512" always agree.

The snapshot is taken without pausing the stream, using checkpoint barriers. The JobManager tells the sources to start a checkpoint; each source records its current offset and injects a barrier, a special marker record, into its own stream. The barrier then flows along with the data. When an operator receives the barrier it snapshots its state at exactly that point and passes the barrier on; an operator with several inputs briefly aligns, waiting for the barrier on all of them, so nothing from after the barrier sneaks into the snapshot.

When every sink has seen the barrier and every snapshot is safely in durable storage, the JobManager marks the checkpoint complete. The stream kept flowing the whole time; the barrier just swept a consistent cut through it. One caveat for later: under heavy backpressure barriers travel slowly because they queue behind data, and unaligned checkpoints are the escape hatch that lets a barrier overtake queued records.

The barrier protocol for checkpoint N. Events ahead of the barrier are covered by checkpoint N, events behind it by N+1, and the stream never pauses. On operators with several inputs, brief alignment (waiting for the barrier on all of them) is what keeps the cut consistent.

Recovery is the payoff. When a TaskManager dies, the JobManager restarts the failed tasks on the surviving workers or on replacements, every operator loads its state from the last completed checkpoint, and the sources rewind Kafka to the offsets stored in that same checkpoint. The events processed after the checkpoint are simply read again. Nothing is double-counted, because the state they had previously updated was rolled back to the same cut; the job re-earns those updates as it replays. The result is exactly-once state: every event affects the state exactly once, even across crashes. Notice the quiet dependency: this only works because Kafka can replay from an offset. A source that cannot rewind cannot give you this guarantee, which is one more reason the Kafka plus Flink pairing is standard.

Exactly-once has a boundary

Exactly-once state is not automatically exactly-once everywhere. Between a checkpoint and a crash, the job may already have emitted results that the replay will produce again, so the outside world sees at-least-once unless the sink cooperates.

Flink reaches end-to-end exactly-once with transactional sinks: results are written inside a transaction that commits only when the checkpoint completes (the Kafka sink does this with Kafka transactions, driven by a two-phase commit tied to checkpointing). Where a transaction is not available, make the write idempotent, such as an upsert keyed by card and window, so a replayed result overwrites itself instead of duplicating.

And a side effect Flink cannot undo (an email, a block signal sent to a third party) still needs its own dedupe, exactly like the Kafka rule that exactly-once stops at the system's boundary.

Guarantee	What it means	What it requires
At-least-once	Every event is processed; duplicates possible after a failure	Checkpoints on; any sink; downstream dedupes if it matters
Exactly-once (state)	Job state reflects each event exactly once, even across crashes	Checkpoints plus a replayable source (Kafka offsets rewind)
End-to-end exactly-once	Downstream systems also see each result exactly once	Additionally a transactional sink (two-phase commit) or idempotent writes

Three strengths of guarantee. Each level adds a requirement, and the last one depends on the sink, not on Flink alone.

A savepoint is the same snapshot machinery pointed at operations instead of failure. You trigger one manually to get a named, portable snapshot, then restore a new version of the job from it: upgrade the code, change the parallelism (state redistributes across the new subtasks), migrate clusters, or fork a copy of production state for testing. Checkpoints are automatic and owned by the runtime for crash recovery; savepoints are manual and owned by you for planned change. Two production habits follow: take a savepoint before every deploy, and keep state schema changes compatible so old snapshots still load into new code.

Backpressure: When Downstream Cannot Keep Up

Every streaming system must answer what happens when events arrive faster than some stage can process them (a slow database sink, a hot key, an undersized cluster). Flink neither drops events nor buffers without limit. The network buffers between subtasks are bounded, so a full buffer refuses more data, the stall propagates upstream stage by stage, and eventually the sources slow their Kafka reads. The job sheds no data; it just falls behind, and the lag shows up as Kafka consumer lag, where you should already have an alert. Backpressure also slows checkpoint barriers, since they queue behind data, so "checkpoints started timing out" is often the first visible symptom of a slow sink. The web UI colors busy and backpressured operators, which makes finding the bottleneck straightforward: fix the slow stage, raise its parallelism, or batch its writes.

Flink vs Kafka Streams vs Spark Streaming

Three tools answer "process events in real time," and interviews love the comparison:

Kafka Streams is a Java library, not a cluster: stream processing embedded inside your own service, reading from Kafka and writing to Kafka only. It is operationally the lightest choice and a good default for moderate Kafka-to-Kafka jobs.
Spark Structured Streaming processes micro-batches (latency typically sub-second to seconds rather than milliseconds) and shines when you already live in the Spark and data-lake world and want one engine for batch and streaming.
Flink is a dedicated streaming cluster that processes events one at a time with the strongest state, event-time, and exactly-once machinery, and it runs batch too, treated as a bounded stream.

A fair rule of thumb: Kafka Streams for simpler Kafka-to-Kafka services, Spark when streaming extends an existing batch platform, Flink when the streaming itself is the hard part (very low latency, large state, out-of-order data, complex patterns).

	Flink	Kafka Streams	Spark Structured Streaming
Model	Dedicated cluster, event at a time	Library inside your app	Micro-batches on a Spark cluster
Latency	Milliseconds	Milliseconds	Sub-second to seconds
Sources / sinks	Kafka, files, databases, queues, CDC	Kafka to Kafka only	Kafka, files, lakes, warehouses
State	Local (heap or RocksDB), checkpointed, scales to 100s of GB	Local stores backed by Kafka changelog topics	Checkpointed state store (RocksDB option)
Event time / late data	Strongest: watermarks, allowed lateness, side outputs	Supported, simpler model	Supported, tied to micro-batch cadence
Ops burden	A cluster to run (or a managed service)	None beyond your own app	A Spark cluster, often already present
Best for	Fraud, IoT, pattern detection, large stateful joins, lowest latency	Moderate Kafka-to-Kafka transforms	Streaming added to a batch and lakehouse stack

Choosing between the big three: the embedded library, the micro-batcher, and the dedicated streaming engine.

CEP: Matching Event Sequences

Some fraud is not a threshold but a sequence: several small test charges, then one large charge, all inside ten minutes. You could hand-roll that with a process function and state, but Flink ships a library for it: CEP, complex event processing. You declare a pattern (begin with a small charge, followed by more small charges, then a large one, within ten minutes), apply it to a keyed stream, and Flink compiles it into a state machine per key that emits a match the moment the sequence completes.

Patterns express contiguity (strictly next versus eventually followed by), quantifiers (three or more small charges), and time bounds (within ten minutes). A non-match can matter too: a pattern that times out partway (payment initiated but never completed) comes out of a side output, which is how you catch things that failed to happen. When an interviewer describes behavior over time rather than a number over time, CEP is the name to reach for.

Where Flink Fits: Card Fraud, IoT, and Friends

Card fraud detection is the canonical Flink interview example because it needs everything at once. Transactions stream into a Kafka topic keyed by card id, so each card's events arrive in order, and the Flink job keys by card (keyBy on the card id), which gives every card its own state. Rules then run as each transaction lands:

Velocity: total spend in a sliding five-minute window against the card's normal ceiling.
Impossible travel: the distance between this transaction's location and the previous one against physically possible travel.
The tester pattern: several tiny charges followed by one large one, exactly the sequence shape CEP from the previous section matches directly.

A hit publishes a score to a fraud-alerts topic that the payment service reads while the transaction is still being authorized. Why Flink specifically: per-card state at the scale of millions of cards (local state, no database round trip in the hot path), event time so delayed and retried transactions score correctly, exactly-once so a crash never double-counts spend against a card, and milliseconds of latency because the decision is only useful before authorization completes.

The fraud pipeline end to end. The job keys the stream by card so each card carries its own state (recent spend, last location), applies windowed rules and pattern matching per transaction, and publishes scores fast enough for the authorization flow to block the payment rather than report it afterwards.

IoT and sensor streaming is the other staple, and it stresses different muscles. Picture a logistics fleet: tens of thousands of trucks reporting position, engine temperature, and fuel level every few seconds over cellular connections that drop and recover. Readings arrive late and out of order by design (gateways buffer and flush in bursts), so event time and watermarks are not a nicety here, they are the difference between right and wrong averages. The job keys by device id and typically runs several computations side by side:

Downsampling: tumbling windows roll raw readings up into per-minute aggregates for a time-series store, reducing the flood to what dashboards actually need.
Trend rules: temperature rising across consecutive windows flags failing refrigeration before it fails.
Silence detection: per-device timers (the process-function mechanism from Core Concepts) alert when a truck goes quiet for ten minutes, which windows alone cannot express because silence produces no events to window.
Trip reconstruction: session windows rebuild trips from gaps in movement.

The same shape repeats in smart meters, factory telemetry, and wearables: huge key cardinality, late data as the normal case, and state per device.

The same machinery shows up wherever a number must stay current:

Live dashboards: orders per minute by region or error rate per service, tumbling-window aggregations sunk into a fast store.
Anomaly alerting: a per-key baseline held in state, firing when the current window deviates, such as a payment success rate dipping for one provider.
Real-time ML features: keyed running aggregates (how many rides this user took in the last hour) written to a feature store, so the model reads a fresh number at request time instead of yesterday's batch value.
Streaming ETL and CDC: a database's changelog tailed through Kafka, joined and enriched in Flink, keeping a search index or warehouse continuously in sync instead of reloading it nightly.

When Flink is the wrong answer

Reaching for Flink when a simpler tool fits is a red flag in an interview. A stateless transform (consume, convert, produce) is a plain Kafka consumer or a small Kafka Streams app; a cluster buys nothing. Reports over yesterday's data belong in the warehouse or a scheduled batch job. Latency needs measured in minutes are often served by frequent micro-batches with far less machinery. And running Flink well takes real operational effort (checkpoints, savepoints, state growth, upgrades), which is why teams without that muscle often start on a managed Flink service. The strong move is naming the trigger conditions: choose Flink when you need per-key state at scale, event-time correctness on out-of-order data, exactly-once guarantees, and answers in milliseconds; otherwise say the simpler thing first.

The Fraud Rule in Code

How This Shows Up in Interviews

Expect a familiar set, and practice answering each in two or three tight sentences:

"Kafka vs Flink?" Transport and storage versus computation. They pair rather than compete: Kafka holds and delivers the events, Flink computes over them.
"How does Flink scale?" Operators run as parallel subtasks; keyBy hashes each key to one subtask, spreading state the way Kafka spreads partitions.
"Event time vs processing time?" When it happened versus when it arrived. Event time plus watermarks gives correct results on out-of-order data.
"What is a watermark?" A flowing claim that events up to time T have arrived, usually newest event time minus a slack. Windows fire when it passes their end.
"What happens to late events?" Dropped by default once their window closed; allowed lateness re-fires the window; a side output captures them for reconciliation.
"How does Flink survive failures?" Periodic checkpoints of all state plus source offsets. Recovery restores state, rewinds Kafka, and replays: exactly-once state.
"Is exactly-once real?" For state, yes. End to end it additionally needs a transactional or idempotent sink.
"How would you catch a sequence like small, small, large charges?" A CEP pattern on the keyed stream, with time bounds and a timeout side output.
"Flink vs Spark Streaming?" Event at a time versus micro-batch: stronger event-time and state machinery versus one engine across a batch stack.
"When would you not use Flink?" Stateless transforms, batch reports, minute-level latency needs, or a team without the ops muscle.

And know the design prompts where Flink is the expected answer: fraud scoring during authorization, IoT telemetry with late data, live dashboards, and real-time ML features.

A TaskManager crashes 40 seconds after the last successful checkpoint of a Kafka-fed Flink job. What happens to the events processed during those 40 seconds?

They are reprocessed: the job restores state from the checkpoint and rewinds Kafka to the checkpointed offsets, so state ends up as if each event was applied exactly once

They are lost, because the state they updated lived only in memory on the dead machine

They are double-counted, because the job saw them once before the crash and once again after the replay

Only the failed subtask's events are replayed; the rest of the job continues from where it was

Checkpointing is on, and after a crash and recovery the service consuming your job's fraud-alerts topic reports duplicate alerts. What happened, and what is the fix?

The replay re-emitted results the job had already published between the last checkpoint and the crash; use a transactional sink committed with the checkpoint, or make the consumer idempotent

Checkpoints are too infrequent; shortening the interval prevents the duplicates

The job restored from the wrong snapshot; restoring from a savepoint instead of a checkpoint avoids duplicates

The alerts topic has too few partitions, so Kafka delivered each record twice

Key Takeaways

The mechanics worth being able to reproduce on a whiteboard:

Checkpoints: a barrier sweeps the dataflow and captures every operator's state plus the exact Kafka offsets it corresponds to. Recovery is restore, rewind, replay, and state comes out as if every event was applied exactly once.
End-to-end exactly-once: additionally needs a transactional sink (two-phase commit riding the checkpoint) or idempotent writes, and side effects beyond Flink's reach still need their own dedupe.
Savepoints: the same snapshot machinery for deploys, rescaling, and migrations. Take one before every release.
Backpressure: Flink refusing to lose data when a stage is slow. Sources throttle, Kafka lag grows, checkpoints stretch, and the web UI points at the bottleneck.
Choosing: Kafka Streams for lighter Kafka-to-Kafka services, Spark Structured Streaming when streaming extends a batch platform, Flink when the streaming itself is the hard part.

In interviews, justify choosing Flink with the trigger conditions (per-key state at scale, event-time correctness, exactly-once, millisecond answers) and know the two canonical builds cold: fraud scoring that lands inside the authorization window, and IoT telemetry where late, out-of-order data is the normal case. Then show you know the edges: unbounded state needs a cleanup path, one idle Kafka partition can stall every watermark, a hot key skews a single subtask, and duplicates after recovery mean the sink, not the checkpoint, is the thing to fix.