Key Technologies
Apache Flink: Core Concepts
Apache Flink is a distributed stream processing engine. It runs continuous computations over never-ending streams of events: counting, aggregating, joining, and detecting patterns as each event arrives, with results ready in milliseconds. Where Kafka moves and stores events, Flink is the thing that computes over them. It keeps state (running totals, recent activity per card or per device), understands event time (when something actually happened, not when it arrived), and survives machine failures without losing count. That combination is what makes real-time fraud detection, IoT monitoring, and live analytics possible.
This is the first half of a two-part guide, and it builds the foundation: the core idea, then every unit of the model defined one at a time (events, streams, operators, tasks, subtasks, slots, jobs), the runtime that executes them, keyed state, process functions and timers, event time and watermarks, and windows, all anchored to one running example, a card fraud detector. The second half, Apache Flink in Practice, covers checkpoints and exactly-once, backpressure, CEP, the comparison with Kafka Streams and Spark Streaming, and the fraud and IoT systems as interviewers ask them.
The Core Idea: The Query Never Finishes
Start with how a batch system answers a question. You store the data first (in a database or a data lake), then run a query over what has accumulated, get an answer, and the query ends. Ask again tomorrow and you run it again. The answer is always as stale as the gap between runs.
Flink inverts this. You deploy the query once and it runs forever. Data flows through the computation instead of the computation running over stored data, and every arriving event updates the answer immediately. "How much has each card spent in the last five minutes?" stops being a report you run and becomes a number that is always current. That inversion is the whole point: when the answer must be ready in milliseconds (block this payment, alert on this sensor), you cannot wait for tonight's batch job.
Flink computes, Kafka transports
Keep the two roles separate in your head, because interviewers probe exactly this. Kafka is a durable log: it receives events, stores them, and lets many consumers read them, but it does not aggregate, join, or window anything. Flink is the opposite: it stores nothing long term and exists to run computation over events as they flow. The two pair so often (Kafka in front, Flink behind) that "Kafka for transport, Flink for computation" is a safe one-line summary in a design interview.
The Building Blocks, One by One
Start with the data side. An event (Flink calls it a record) is one immutable fact: "card 4412 spent $90 at store 771 at 10:02:11". It happened, it carries its own timestamp, and it will never change; there is no updating an event, only newer events. A stream is the sequence of such events, and its defining property is that it is unbounded: there is no last transaction, so there is no end you can wait for before answering. Flink also handles bounded streams (a file of last month's transactions has a first and a last record), and that is all a batch job is to Flink: a stream that happens to end. One engine covers both.
Now the compute side. An operator is one step of computation applied to the stream: filter out refunds, parse the payload (map), group by card (keyBy), aggregate a window. A source is the operator that brings events in (most often a Kafka consumer) and a sink is the one that writes results out (Kafka, a database, a search index). Wire them together and you have a job: a dataflow graph with sources at one end, sinks at the other, and transformations in between, which you submit to the cluster once and which then runs until you stop it. You author the graph through two API families that share the same engine underneath: the DataStream API (Java or Python code, full control) and the Table API with SQL (declarative; the companion article shows the same job written both ways).
The graph you write is logical; to run it, Flink stamps out parallel copies. Each operator runs as parallelism-many subtasks, and each subtask handles a slice of the stream: a source with parallelism 3 reads Kafka with three parallel readers splitting the topic's partitions between them (which is why the upstream partition count matters), and a window operator with parallelism 3 splits the key space three ways.
Before deploying, Flink also fuses neighboring operators that can run back to back (the source feeding a filter, a filter feeding a map) into a single task, called operator chaining, so an event passes between them as a plain method call instead of a network hop. The vocabulary, then: an operator is a logical step, a task is one or more chained steps, and a subtask is one parallel copy of a task, the actual unit of running code. Each subtask runs inside a task slot, a fixed slice of a TaskManager's memory; with slot sharing (the default) one slot holds one parallel slice of the entire pipeline, so the number of slots a job needs is simply its highest parallelism.
| Unit | What it is | In the fraud job |
|---|---|---|
| Event (record) | One immutable fact carrying its own timestamp | "card 4412 spent $90 at 10:02:11" |
| Stream | Unbounded sequence of events (bounded = batch) | Every transaction, forever |
| Operator | One logical processing step | filter refunds; window + sum |
| Source / sink | The end operators: read events in / write results out | Kafka consumer in; fraud-alerts topic out |
| Job (dataflow graph) | The whole pipeline, submitted once, runs forever | The fraud detector |
| Task | Neighboring operators chained to run together | source + filter fused into one |
| Subtask | One parallel copy of a task; the running unit of code | window-subtask-2 owns a slice of the cards |
| Parallelism | How many subtasks each operator runs as | 3 source readers, 3 window subtasks |
| Task slot | Fixed share of a TaskManager holding one pipeline slice | slot 1 on TaskManager 2 |
| TaskManager | Worker JVM offering slots; subtasks run here | one per machine or pod |
| JobManager | Coordinator: schedules, checkpoints, recovers | one active per cluster |
Every unit of the model in one place, with its role in the running fraud example.
One question decides whether that parallel picture works at all: when the stream is split across subtasks, how do all the events for one card end up in the same place? That is keyBy's job. It hashes each event's key (the card id) and routes the event to the subtask responsible for that hash range, exactly the way Kafka's producer hashes a record key to pick a partition. All events for card 4412 always land on the same window subtask, which is what makes per-card logic possible, and different cards spread across all subtasks, which is what makes it scale. keyBy is also the network boundary of the graph: operators before and after it can chain into tasks, but crossing it means events move between machines.
Hot keys skew a single subtask
keyBy decides both correctness and balance. Correctness: per-key logic only works because one subtask sees every event for a key. Balance: keys spread by hash, so one enormously busy key (a bot hammering one account, one celebrity user, one chatty device) piles onto a single subtask while its peers sit idle, and raising the parallelism does not help because that key still maps to exactly one subtask. The fix mirrors Kafka's hot-partition fix: choose a finer-grained key when the logic allows it, or aggregate in two stages (split the hot key with a random suffix, then combine the partial results).
The Runtime: JobManager and TaskManagers
A Flink cluster has two kinds of processes. The JobManager is the coordinator: it accepts your job, turns the dataflow graph into tasks, schedules those tasks onto workers, triggers checkpoints on a timer (the state snapshots covered in the companion article), and reacts to failures by restarting what died. The TaskManagers are the workers: JVM processes that offer the task slots where subtasks run.
Two operational facts round this out. Event data never passes through the JobManager; subtasks stream records directly to each other over the network between TaskManagers. And clusters run standalone, on Kubernetes, or on YARN, with production setups keeping a standby JobManager (cluster metadata in ZooKeeper or Kubernetes) so the coordinator itself is not a single point of failure.
State: What the Job Remembers
A filter that drops refunds needs no memory. Almost everything interesting does:
- "Total spend per card in the last five minutes" must remember the amounts seen so far.
- "Alert if a sensor goes silent for ten minutes" must remember when each sensor last spoke.
- "Was this card just used in another city?" must remember the previous transaction's location.
That memory is state, and Flink's core design decision is that state lives locally with the subtask that uses it (in memory or on local disk), not in a remote database. After keyBy, each subtask holds state only for its own keys: a small keyed store per card, per sensor, per user, which the API exposes as typed handles (ValueState, ListState, MapState) automatically scoped to the current key.
Local state is why a subtask can read and update its counters in microseconds and why one job can handle millions of events per second; the same logic calling a remote store on every event would spend its life waiting on the network. It also raises the obvious worry, because the machine holding it can die. Checkpoints answer that, and they open the companion article.
| State backend | Where state lives | Trade-off |
|---|---|---|
| HashMap (heap) | Objects on the JVM heap of each TaskManager | Fastest access; state must fit in memory; garbage-collection pressure when it grows |
| RocksDB (embedded) | An embedded key-value store on each TaskManager's local disk | State larger than RAM and incremental checkpoints; each access pays serialization cost |
Two state backends cover most deployments: heap for small, hot state; RocksDB when state runs to tens or hundreds of gigabytes.
Unbounded state is a slow-motion outage
State that only ever grows will eventually kill the job, and it is the most common Flink production incident. Keyed state for "every card ever seen" expands forever unless something clears it: windows purge themselves when they close, but state you manage yourself in a process function needs a time-to-live (state TTL) or a timer that deletes entries after inactivity. Watch the job's total state size week over week; growth without bound is the early warning that some piece of state has no cleanup path.
Process Functions and Timers: When Windows Are Not Enough
Windows (coming two sections down) cover time-sliced aggregation, but some logic does not fit a window, and the interview favorite is silence: "alert if a sensor sends nothing for ten minutes." Silence produces no events, and with no events a window never fires.
The tool underneath is the KeyedProcessFunction, Flink's low-level building block: for each event it hands you the event, the keyed state for that key, and a timer service. A timer is an alarm you set for a moment in event time or processing time, scoped to the current key; when the watermark or the wall clock passes it, Flink calls you back, event or no event. The silent-sensor alert becomes: on every reading, update last-seen state and reset a timer for last-seen plus ten minutes; if the timer ever fires, no reading arrived in time, so emit the alert.
Windows themselves are built from this same machinery (state per key per window plus a timer at the window's end), which is a useful thing to say in an interview: windows are the packaged pattern, process functions are the raw parts.
Event Time and Watermarks
Every event has two times. Processing time is the clock on the machine when the event happens to arrive. Event time is when the thing actually happened, carried inside the event itself. The two drift apart constantly in real systems: a phone finishes a ride through a tunnel and uploads its GPS points a minute late, an IoT gateway buffers readings and flushes every 30 seconds, a backed-up Kafka partition delivers its records behind the others.
If you compute "spend per card, per minute" by processing time, a burst of late arrivals lands in the wrong minute and the numbers are simply wrong, and not reproducible either: replay the same events tomorrow and the totals come out different. Event time gives the answer the data deserves: each event counts toward the minute it happened in, however late it shows up. Flink is built around event time, and that support is one of the main reasons to pick it over simpler tools.
Event time creates a puzzle: if events can arrive late, when is the 10:00 to 10:01 window safe to close? Wait forever and you never answer; close at 10:01 sharp and stragglers get dropped. Flink's answer is the watermark, a moving marker that flows through the stream and declares "events older than time T have probably all arrived." The usual generator is a bounded-delay heuristic: track the newest event time seen and subtract a slack you choose, say 30 seconds, so after seeing an event stamped 10:01:30 the watermark reads 10:01:00 and the 10:00 to 10:01 window fires. The slack is a knob you own: more slack catches more stragglers but delays every result; less slack answers faster but calls more events late.
| Arrival | Event time | Watermark after (newest - 30s) | Window [10:00, 10:01) |
|---|---|---|---|
| 1st | 10:00:58 | 10:00:28 | open; holds 10:00:58 |
| 2nd | 10:01:05 | 10:00:35 | open |
| 3rd | 10:00:52 (out of order) | 10:00:35 (unchanged) | open; straggler included |
| 4th | 10:01:35 | 10:01:05 | fires with 10:00:58 + 10:00:52 (watermark passed 10:01:00) |
Four arrivals against a 30-second watermark slack. The 10:00:52 reading arrives out of order but is not late: its window has not fired yet when it shows up. Watermarks never move backwards, and the window fires only when the watermark finally passes 10:01:00.
Late events and stalled watermarks
Two traps live here, and both are favorite interview follow-ups.
Late events. Events that arrive after the watermark has passed their window are late, and by default a closed window drops them silently. If late data matters, give the window an allowed lateness (it fires an updated result when stragglers arrive) or route late events to a side output (a second, labeled stream an operator can emit alongside its main output) and reconcile them separately.
Stalled watermarks. An operator's watermark is the minimum across its inputs, so one silent input stalls everything: a Kafka partition with no traffic stops advancing its watermark, downstream windows never close, and the job looks frozen while perfectly healthy. Mark sources idle after a timeout (withIdleness) so a quiet partition stops holding the clock back.
Windows: Finite Answers from an Infinite Stream
You cannot sum an infinite stream; "total spend per card" only means something over a bounded slice of time. Windows are how Flink cuts the endless stream into those slices, usually by event time, and three shapes cover nearly every real use:
- Tumbling windows are back-to-back fixed blocks (every minute, on the minute); each event belongs to exactly one.
- Sliding windows have a size and a smaller slide (a five-minute window that advances every minute), so each event belongs to several windows and you get a smooth rolling view.
- Session windows have no fixed size at all; a session stays open while events keep coming and closes after a gap of inactivity, which matches how people and devices actually behave.
By default an event-time window fires when the watermark passes its end, which is exactly the decision the watermark exists to make.
| Window | Definition | Typical use |
|---|---|---|
| Tumbling | Fixed size, no overlap; every event in exactly one window | Per-minute dashboards, billing counts, rate metrics |
| Sliding | Fixed size with a smaller slide; windows overlap | "Spend in the last 5 minutes, refreshed every minute" fraud checks |
| Session | Grows while events arrive, closes after an inactivity gap | User browsing sessions, device trips, support-chat episodes |
The three window shapes and where each one fits.
You are designing card fraud detection: every transaction must be checked against that card's spend in the last five minutes, across millions of cards. How do you structure the Flink job so each card's total is correct and the job still scales?
An IoT job aggregates sensor readings into one-minute event-time windows with a 30-second watermark slack. A gateway reconnects and delivers a reading stamped 10:00:40 after the watermark has already passed 10:01. What happens to that reading by default?
Key Takeaways
The model in five load-bearing pieces:
- A job is a dataflow graph that runs forever: operators chained into tasks, tasks stamped out as parallel subtasks, subtasks placed into slots on TaskManagers, all coordinated by a JobManager that event data never flows through.
- keyBy is the hinge of the model: it routes every event for a key to the one subtask that owns that key's state, which makes per-card and per-device logic both correct and scalable, with the caveat that a single hot key cannot be split by adding parallelism.
- State lives locally (heap for small and hot, RocksDB for large) and needs a cleanup path or it grows forever.
- Process functions and timers sit underneath windows for logic that events alone cannot trigger, silence being the classic case.
- Event time plus watermarks decide when results are safe to emit despite late, out-of-order data, with allowed lateness and side outputs for the stragglers and idleness marking for quiet sources. Windows then slice the infinite stream into tumbling, sliding, or session-shaped answers.
What this half deliberately postponed: what happens when a machine dies mid-count, how far exactly-once really goes, what happens when the sink cannot keep up, and where Flink beats or loses to Kafka Streams and Spark Streaming. That, plus card fraud detection and IoT telemetry built end to end with code, is the companion article, Apache Flink in Practice.