Real-time systems

Overview

A real-time system is one where the time at which a result arrives is part of whether it is correct. A brake controller that computes the right braking force 200 milliseconds too late has failed. A fraud check that returns after the payment has gone through has lost most of its value. Real-time does not simply mean "fast"; it means "predictably on time", with a known upper bound on how long a response can take.

Engineers usually distinguish hard real-time systems, where missing a deadline counts as a failure and may be dangerous, from soft real-time systems, where late answers lose value but the system keeps working. Industrial control, engine management and medical devices are typical hard real-time systems. Live recommendations, video calls, fraud scoring and streaming analytics are typically soft real-time, judged by how often they meet a latency target.

An everyday picture is a goalkeeper. A save that comes one second after the ball has crossed the line is not a save at all, however good the technique. Real-time systems are designed like goalkeepers: what matters is being in position before the deadline, every time.

What it is

Key building blocks include:

• Deadlines and periods: when each task must finish and how often it repeats.
• Schedulers: rules that decide which task runs next, such as fixed priorities or earliest deadline first.
• Real-time operating systems (RTOS): operating systems designed for predictable timing.
• Latency budgets: the total time allowed for a response, divided between its steps.
• Streaming platforms: systems such as Apache Kafka and Apache Flink that move and process events continuously.

For AI, real time usually means scoring a model inside a request path or event stream within a budget, for example 20 milliseconds for a fraud score or 100 milliseconds for a recommendation.

Where it came from

Real-time computing grew from control and defence systems in the 1950s and 1960s, when computers began to guide aircraft, missiles and industrial processes. Engineers had to guarantee that control loops completed in time, so they studied timing as carefully as correctness.

A foundational paper came in 1973, when C. L. Liu and James Layland published a study of scheduling for hard real-time environments in the Journal of the ACM. They analysed rate-monotonic scheduling, where tasks with shorter periods get higher priority, and showed a utilisation bound below which all deadlines are guaranteed; as the number of tasks grows, that bound approaches about 69 per cent of processor time. They also analysed deadline-driven scheduling, the basis of earliest deadline first. Later work added protocols to prevent priority inversion, where a low-priority task blocks a high-priority one. A widely cited example of priority inversion caused repeated resets on NASA's Mars Pathfinder lander in 1997 and was fixed by enabling priority inheritance.

From the 2000s, "real-time" took on a second meaning in data and web systems: processing streams of events with low latency. Stream processors, in-memory stores such as Redis and low-latency model serving made real-time analytics and AI common. Today, edge devices run models next to machines and cameras, while cloud systems handle streaming decisions at large scale.

How it works

Step 1: define deadlines and budgets

Start with the requirement: for example, "95 per cent of fraud scores within 30 milliseconds, 99.9 per cent within 80". Divide the budget across steps: network, feature lookup, model inference, rules and response. See cost and latency budgets.

Step 2: design for predictable timing

Avoid steps with unbounded or highly variable time on the critical path: large database queries, synchronous calls to slow services, unbounded queues and garbage-collection pauses. Precompute what you can.

Step 3: schedule and prioritise

In embedded and control systems, an RTOS schedules tasks by priority or deadline and analyses worst-case execution times. In distributed systems, separate critical traffic from bulk work, limit queue lengths and shed load when overloaded.

Step 4: keep features fresh and fast

Real-time models need recent features, such as the number of transactions in the last ten minutes. Stream processors compute them continuously and store them in a low-latency store or feature store, so the model only needs a quick lookup at request time.

Step 5: serve models efficiently

Use compact models, optimised runtimes such as ONNX Runtime or TensorRT, batching only where it does not break the budget, and hardware close to the data. For hard deadlines, prefer models with bounded, measurable execution time.

Step 6: plan for failure

Define what happens when a deadline would be missed: a simpler fallback model, a safe default action, or a circuit breaker that stops calling a slow dependency. Monitor tail latency, not just averages, with tools such as Prometheus.

Key concepts and terms

A worked example

This scenario is illustrative. A payments company must approve or decline card transactions within 100 milliseconds end to end, of which 30 milliseconds are allotted to fraud scoring.

The team splits the 30 milliseconds: 3 for network inside the data centre, 8 for feature lookup, 12 for model inference, 4 for business rules and 3 of margin. Spending velocity features are computed continuously by a stream processor from the transaction stream and kept in an in-memory store, so lookups take about 2 milliseconds at p99. The model is a gradient boosting ensemble compiled for fast inference, taking 4 milliseconds at p99 on a standard CPU.

Load testing at twice peak volume shows p99 scoring latency of 18 milliseconds. During a failover drill, the feature store becomes slow; the circuit breaker switches to a fallback model that uses only fields in the transaction itself, and scoring stays within budget with slightly lower accuracy. Dashboards track p50, p99 and p99.9 latency, and an alert fires if p99 exceeds 25 milliseconds for five minutes.

Strengths and limitations

Strengths:

• Enables decisions while they still matter, such as stopping fraud or preventing damage.
• Predictable timing allows safety-critical automation.
• Fresh data improve the accuracy of many decisions.
• Streaming designs avoid large batch delays.
• Clear budgets make performance engineering concrete.

Limitations:

• Designing for worst-case timing is harder and more expensive than for averages.
• Large, accurate models may not fit the latency budget.
• Distributed systems have unpredictable delays from networks and dependencies.
• Testing timing under all conditions is difficult.
• Fallback paths add complexity and must also be tested.

How it compares with alternatives

Practical guidance and best practices

1. Write deadlines and percentiles down as requirements before designing.
2. Break the latency budget into steps and measure each one.
3. Precompute features in streams; keep request-time work to lookups.
4. Measure and alert on tail latency, not averages.
5. Load test beyond expected peak and under dependency failures.
6. Provide a tested fallback for every critical dependency.
7. Choose models that fit the budget, and compare accuracy gains with latency cost.
8. For hard real-time, use an RTOS, bounded algorithms and worst-case timing analysis.

Common mistakes

• Calling a system "real-time" without stating a deadline.
• Reporting average latency while p99 users wait ten times longer.
• Querying a large database or remote service inside the critical path.
• Allowing unbounded queues that hide overload until timeouts cascade.
• Deploying a larger model that silently breaks the latency budget.
• Never testing the fallback path until a real incident.

Frequently asked questions

Is real-time the same as fast?

No. Real-time means meeting a deadline predictably. A system that is very fast on average but occasionally takes several seconds is not real-time for a 100-millisecond deadline. Predictability and bounded worst cases matter more than raw speed.

What is the difference between hard and soft real-time?

In hard real-time systems, missing a deadline is a failure, often with safety consequences, so designs must guarantee timing. In soft real-time systems, occasional misses reduce quality or value but are tolerated, and designs target percentile goals.

Can large AI models run in real time?

Some can, within soft real-time budgets, using optimised runtimes, accelerators, quantisation and smaller variants. Language models usually stream tokens, so the time to the first token and the speed of generation become the latency metrics. For hard real-time control, compact models with bounded execution time are preferred.

Why does tail latency matter so much?

Because a single user request often depends on many internal calls, and the slowest one decides the overall response. If each call has a small chance of being slow, a request touching many of them is likely to hit at least one, so high-percentile latency drives user experience.

What is priority inversion?

It happens when a high-priority task waits for a resource held by a low-priority task, while medium-priority tasks keep the low-priority one from running. Protocols such as priority inheritance temporarily raise the low-priority task's priority to resolve the block.

Glossary

• Real-time system: a system whose correctness depends on timing.
• Deadline: the latest acceptable completion time.
• Latency budget: the total time allowed for a response, divided among steps.
• p99: the 99th percentile of latency.
• Jitter: variability in response time.
• Scheduler: the component that decides which task runs.
• Fallback: a simpler path used when the main path would miss its deadline.
• Edge computing: processing near the data source to reduce delay.

Summary

Real-time systems must respond within bounded, predictable time, from hard real-time control loops where a missed deadline is a failure to soft real-time services such as fraud scoring where late answers lose value. The field rests on scheduling theory, including Liu and Layland's 1973 analysis, and now extends to streaming data and low-latency AI. Success depends on explicit deadlines, step-by-step latency budgets, precomputed features, efficient models, tail-latency monitoring and tested fallbacks.

Related

It belongs to Industries and applications in my stack, under Machine learning. Others in the same group: