A Research Paper
AI & Traffic Optimization
Abstract
A study comparing five traffic-signal controllers on a 3x3 SUMO grid, introducing a hybrid reinforcement learning controller that outperforms max-pressure post-saturation.
Collabs Scholar
Pressure-Guided Q-Learning: a warm-started hybrid controller for signalised urban grids
by Advik Singh
Leander, TX · Jul 14, 2026
Open Access Library
Keywords: Reinforcement learning, Traffic signal control, Max-pressure control, Q-learning, SUMO simulation.
🚦 1. Introduction
Urban traffic control has traditionally focused on reducing driver waiting time at signalised intersections. Conventional traffic control systems rely on preconfigured signal cycles, which makes them rigid and inefficient when demand fluctuates, flows become asymmetric, or peak and off-peak transitions occur.
Classical adaptive controllers such as max-pressure are decentralised and come with stability guarantees, yet they make decisions that are myopic in time and optimise only on the current step.
Learning-based controllers such as deep Q-networks (DQN) and independent tabular Q-learning are flexible but offer no formal guarantees, and can destabilise under congestion. The PGQL framework proposed in this study is conceptually related to several existing strands of work in reinforcement learning, but makes a specific contribution: the use of max-pressure as the warm-start expert rather than an arbitrary heuristic, providing the learner with a controller of known stability properties at initialisation.
🏙️ 2. Methods
This research evaluated five traffic-signal controllers using high-fidelity microscopic simulation implemented in Python and integrated into the SUMO traffic simulator (version 1.26).
Network & Demand Model
The primary network was a 3×3 bidirectional urban grid, generated with SUMO's netgenerate tool, containing nine signalised intersections and 24 directed single-lane edges, with a block length of 200 m and a free-flow speed of 13.9 m/s. A larger 5×5 grid (25 intersections, 80 edges) was also generated for a second network feasibility test.
Controllers Evaluated
- Fixed-time: Ran SUMO's baked-in 90 s static plan.
- Actuated (gap-out): Advanced the phase early based on empty lanes.
- Max-pressure: Switched to the phase of maximal pressure every 5s.
- Independent tabular Q-learning: Treated each intersection as a separate learner with learning rate α=0.1 and discount γ=0.9.
- Pressure-Guided Q-Learning (PGQL): Augmented Q-learning with three actions and warm-started the Q-table with a positive bias B=1.0.
📊 3. Results
In the moderate-load regime (offered N=200 cars over the 1200 s window on the 3×3 grid), max-pressure achieved an average MWT of 8.34±0.21 s/car. Independent Q-learning achieved 7.92±0.19 s/car, and PGQL achieved 11.67±0.25 s/car. Fixed-time control performed the worst at 26.27±0.84 s/car.
Capacity Model Fitting
Drawing on the M/M/1 single-server queueing result, a closed-form capacity model W(n)=a/(1−n/Ncrit)+b was fitted to each controller.
| Controller | Ncrit (cars in system) | a | b |
|---|---|---|---|
| Fixed-time | 134 [46 to 154] | 35.9 | -14.2 |
| Actuated | 158 [54 to 175] | 40.9 | -20.0 |
| Max-Pressure | 188 [78 to 198] | 27.7 | -20.0 |
| PGQL (Novel) | 193 [51 to 202] | 30.5 | -20.0 |
Wilcoxon Signed-Rank Test: PGQL vs Max-Pressure
A paired comparison revealed three distinct regimes on the 3×3 grid:
| Demand (N) | PGQL median MWT | MP median MWT | Regime |
|---|---|---|---|
| 50 | 11.0 | 6.5 | PGQL worse |
| 600 | 26.8 | 24.9 | Tied |
| 1300 | 94.7 | 107 | PGQL better |
The crossover demand level is approximately N=1300 offered cars. This sits comfortably inside the post-saturation regime, where max-pressure's myopia is expensive.
5×5 Grid Feasibility Validation
On the larger 5×5 grid (25 intersections), PGQL outperformed max-pressure at every tested load level. The absolute improvement was modest at low load (approximately 6.6% at N=200) but persisted throughout the tested demand range.
🧠 4. Discussion
The findings provide several insights into the design and evaluation of hybrid traffic-signal controllers.
The inductive bias provided by max-pressure is close to optimal on small networks where its one-step greedy is rarely incorrect, but its myopia appears more expensive on the larger 5×5 benchmark, leaving more room for the learner to discover beneficial deviations.
The action-usage audit suggests a mechanism for the regime-dependent improvement: the override action is selected ~28% of the time at low and moderate load but increases to 37% under saturation. The learner discovered a state-dependent policy where max-pressure is followed when reliable, and overridden when it is likely to be myopic.
Cold-Start vs Warm-Start Ablation
A cold-started agent produced a catastrophic training episode at N=800 with MWT=111.6 s/car versus 73.7 s/car for the warm-started variant.
These results indicate that the practical value of the warm-start lies in safer early training and a bounded post-saturation improvement, rather than in a large asymptotic performance advantage.🏁 5. Conclusion
This study introduced a warm-started hybrid controller that combines reinforcement learning with a stability-proven classical policy, proposing a carrying-capacity framework based on vehicles concurrently present in the network.
At initialization, PGQL inherits the behavior of max-pressure; once learning modifies the policy, however, performance must be established empirically rather than through the stability guarantee of the underlying controller. The results show that classical adaptive control and RL can be combined so that deployment begins safely while allowing learned deviations to improve performance under heavy congestion.
Finis
Thank you for reading.
About the writer & comments ↓Afterword
Pressure-Guided Q-Learning: a warm-started hybrid controller for signalised urban grids
How to cite
Advik Singh (2026). Pressure-Guided Q-Learning: a warm-started hybrid controller for signalised urban grids. Collabs Scholar Open Access Library.
About the Writer
Advik Singh
Leander, TX
Advik Singh writes for Collabs Scholar.
Comments
Comments are coming soon.