How Self-Driving Cars Actually Work—And Why Full Autonomy Is Still Hard

Self-driving cars have been “five years away” for most of the past fifteen years. Some systems have reached genuine limited deployment; others have been quietly wound down after expensive failures. Understanding what’s technically hard about full autonomy—why the last 1% of edge cases is harder than the first 90% of typical driving—requires looking at how the systems actually work, rather than the promotional language around them.

The Sensor Stack: Eyes and Ears of the Vehicle

Autonomous vehicles perceive their environment through a combination of sensor modalities, each with different capabilities and failure modes.

Cameras provide rich visual information—colour, texture, sign reading, lane marking detection—at low cost and high resolution. They produce the most human-readable representation of the driving environment. Their limitations: they fail in direct glare, heavy rain, dense fog, and are affected by lens contamination. Camera-based systems require sophisticated computer vision to extract structured information (distances, object types, trajectories) from raw image data.

LiDAR (Light Detection and Ranging) emits pulses of laser light and measures the time of return, constructing a precise 3D point cloud of the environment. LiDAR provides accurate distance measurements independent of ambient light conditions—it works in darkness—and creates a detailed spatial representation that cameras alone can’t. Its limitations: it’s expensive (though costs have fallen dramatically), performs worse in heavy rain or snow (water droplets scatter the laser returns), and provides no colour or texture information without camera fusion.

Radar uses radio waves rather than light, penetrating rain, fog, and snow effectively. Radar provides velocity information directly through Doppler shift and works in conditions that degrade camera and LiDAR performance. Its resolution is lower than LiDAR—it detects the presence of objects but with less spatial precision. Long-range radar is widely deployed in highway assist systems (adaptive cruise control) for this reliability advantage.

Most serious AV programmes use sensor fusion—combining data from multiple sensor modalities—to build a perception model that’s more robust than any single sensor. Waymo’s fifth-generation sensor suite combines cameras, LiDAR, and radar specifically to have redundant coverage for conditions that stress any individual sensor.

Tesla’s approach is an outlier: the company has deliberately excluded LiDAR, arguing that camera-only vision, if sufficiently sophisticated, should match human capability (since humans navigate using vision alone). This is technically possible in principle—humans manage without LiDAR—but requires the vision system to be robust across conditions where LiDAR would provide an easier path.

Perception: From Raw Sensor Data to a World Model

Sensor data must be converted into a structured model of the vehicle’s environment: what objects are present, where they are, how they’re moving, and what type they are. This perception layer is where the most active AI development is concentrated.

Object detection and classification—recognising pedestrians, cyclists, other vehicles, traffic cones, animals—uses deep learning models trained on enormous labelled datasets. Modern detection systems are highly capable at identifying common object categories in typical conditions. The difficulty is in the long tail: unusual objects, unusual appearances of common objects, partial occlusion, multiple objects interacting in ways the model hasn’t seen. A person in a wheelchair in the road, a large unusual piece of debris, a lane closure with improvised signage—each represents a perceptual challenge that requires the system to correctly classify something its training distribution may not fully represent.

Behaviour prediction is the harder extension of perception: given what objects are present and how they’re moving, what will they do next? A pedestrian facing the street, at a crosswalk, glancing toward traffic is probably about to cross. A cyclist at the edge of a lane making a hand signal is indicating a turn. Reading these behavioural cues accurately—and handling the uncertainty that arises when cues are ambiguous or absent—requires understanding not just the physical state of other road users but their likely intent. This is genuinely hard, not because the AI fundamentals are impossible but because the space of human behaviours in traffic is very large and not fully specifiable.

Planning and Control: Turning Perception Into Action

Given a world model and a destination, the vehicle must plan a path and execute it. This involves several planning layers: route planning (what roads to take), behavioural planning (when to change lanes, how to handle an intersection), and trajectory planning (the specific low-level control outputs to follow the planned path).

The planning challenge is most acute in unstructured environments and at edge cases: a construction zone with unusual lane configurations, a police officer manually directing traffic, a school zone with children unpredictably crossing, a narrow road shared with a slow-moving cyclist. These situations require not just recognising what’s present but reasoning about the correct behaviour in a complex multi-agent social environment.

Traditional autonomous driving programmes (Waymo, Cruise before its pause, Mobileye) use explicit rules-based planning at the behavioural level—a large, hand-crafted set of rules governing how the vehicle should behave in different traffic situations—combined with machine learning at the perception layer. The rules capture known scenarios well but don’t generalise to novel situations.

The emerging approach, most aggressively pursued by Tesla and Waymo’s newer models, uses end-to-end neural network approaches—training models to predict appropriate control outputs directly from sensor data, without explicit rules. This approach potentially generalises better to novel situations but requires enormous training data and is harder to reason about when failures occur.

Why the Last Percent Is Harder Than the First Ninety

The core challenge of full autonomy is the statistical distribution of driving situations. The overwhelming majority of driving—highway travel at normal speeds with clear roads, suburban streets with predictable traffic, parking lots—is handled well by current systems. The small percentage of situations that are unusual, ambiguous, or adversarial is where the difficulty is concentrated.

The engineering challenge is that the consequences of failures in unusual situations are disproportionately severe. A system that handles 99.9% of situations correctly, with failures distributed across the remaining 0.1%, is not safe enough if those failures include high-speed collisions with pedestrians. The safety standard for autonomous vehicles isn’t “better than average drivers”—it’s a level of reliability that makes the tail risk acceptable across hundreds of millions of miles of operation and diverse conditions.

Achieving this requires either solving all edge cases (which is very hard given the combinatorial space of possible traffic situations) or having the system correctly recognise when it’s in a situation outside its competence and transfer control to a human—which is the philosophical basis for Level 2 and Level 3 systems. Full Level 4 autonomy (no human monitoring required, no fallback to human control) removes the human backup, which moves the reliability requirement entirely onto the system.

The Current State in 2026

Waymo operates commercial fully driverless robotaxi service in several US cities—Phoenix, San Francisco (with a more limited operational design domain), Los Angeles—with no human safety driver in the vehicle. This is genuine Level 4 autonomy within defined geographic and weather conditions. The fleet has accumulated millions of driverless miles with a safety record that, by the available data, compares favourably with human drivers for the conditions in which it operates.

Tesla’s “Full Self-Driving” (FSD) remains a Level 2 system requiring driver monitoring despite its marketing. The system handles a broad range of scenarios but produces errors that require driver intervention at a frequency that makes removing the human attention requirement inappropriate under current regulatory frameworks.

Other programmes—Cruise (paused following a serious incident in 2023), Aurora (commercial trucking autonomy), Mobileye (highway assist and urban AV development)—are at various stages of development or limited commercial deployment.

The trajectory suggests that geographically constrained Level 4 autonomy (robotaxis in well-mapped urban areas in favourable weather) will continue to expand. Broad, unconstrained Level 4–5 autonomy that works everywhere a human driver would is genuinely hard and further away than the most optimistic projections have historically suggested. The problem is tractable—the existence of working Level 4 systems proves that—but the remaining difficulties are specifically in the areas that are hardest to enumerate and test comprehensively.