Back to the libraryResearch Paper · AI & Traffic Optimization

A Research Paper

AI & Traffic Optimization

Abstract

A study comparing five traffic-signal controllers on a 3x3 SUMO grid, introducing a hybrid reinforcement learning controller that outperforms max-pressure post-saturation.

Collabs Scholar

Pressure-Guided Q-Learning: a warm-started hybrid controller for signalised urban grids

Leander, TX · Jul 14, 2026

Open Access Library

Keywords: Reinforcement learning, Traffic signal control, Max-pressure control, Q-learning, SUMO simulation.


🚦 1. Introduction

Urban traffic control has traditionally focused on reducing driver waiting time at signalised intersections. Conventional traffic control systems rely on preconfigured signal cycles, which makes them rigid and inefficient when demand fluctuates, flows become asymmetric, or peak and off-peak transitions occur.

Classical adaptive controllers such as max-pressure are decentralised and come with stability guarantees, yet they make decisions that are myopic in time and optimise only on the current step.

— Advik Singh

Learning-based controllers such as deep Q-networks (DQN) and independent tabular Q-learning are flexible but offer no formal guarantees, and can destabilise under congestion. The PGQL framework proposed in this study is conceptually related to several existing strands of work in reinforcement learning, but makes a specific contribution: the use of max-pressure as the warm-start expert rather than an arbitrary heuristic, providing the learner with a controller of known stability properties at initialisation.


🏙️ 2. Methods

This research evaluated five traffic-signal controllers using high-fidelity microscopic simulation implemented in Python and integrated into the SUMO traffic simulator (version 1.26).

Network & Demand Model

The primary network was a 3×33\times3 bidirectional urban grid, generated with SUMO's netgenerate tool, containing nine signalised intersections and 24 directed single-lane edges, with a block length of 200 m and a free-flow speed of 13.9 m/s13.9~m/s. A larger 5×55\times5 grid (25 intersections, 80 edges) was also generated for a second network feasibility test.

Urban Traffic Grid
Figure: Representational image of a dense urban grid where traffic light coordination is critical.

Controllers Evaluated

  1. Fixed-time: Ran SUMO's baked-in 90 s static plan.
  2. Actuated (gap-out): Advanced the phase early based on empty lanes.
  3. Max-pressure: Switched to the phase of maximal pressure every 5s.
  4. Independent tabular Q-learning: Treated each intersection as a separate learner with learning rate α=0.1\alpha=0.1 and discount γ=0.9\gamma=0.9.
  5. Pressure-Guided Q-Learning (PGQL): Augmented Q-learning with three actions and warm-started the Q-table with a positive bias B=1.0B=1.0.

📊 3. Results

In the moderate-load regime (offered N=200N=200 cars over the 1200 s window on the 3×33\times3 grid), max-pressure achieved an average MWT of 8.34±0.21 s/car8.34\pm0.21~s/car. Independent Q-learning achieved 7.92±0.19 s/car7.92\pm0.19~s/car, and PGQL achieved 11.67±0.25 s/car11.67\pm0.25~s/car. Fixed-time control performed the worst at 26.27±0.84 s/car26.27\pm0.84~s/car.

Capacity Model Fitting

Drawing on the M/M/1 single-server queueing result, a closed-form capacity model W(n)=a/(1−n/Ncrit)+bW(n)=a/(1-n/N_{crit})+b was fitted to each controller.

ControllerNcritN_{crit} (cars in system)aabb
Fixed-time134 [46 to 154]35.9-14.2
Actuated158 [54 to 175]40.9-20.0
Max-Pressure188 [78 to 198]27.7-20.0
PGQL (Novel)193 [51 to 202]30.5-20.0

Wilcoxon Signed-Rank Test: PGQL vs Max-Pressure

A paired comparison revealed three distinct regimes on the 3×33\times3 grid:

Demand (NN)PGQL median MWTMP median MWTRegime
5011.06.5PGQL worse
60026.824.9Tied
130094.7107PGQL better

The crossover demand level is approximately N=1300N=1300 offered cars. This sits comfortably inside the post-saturation regime, where max-pressure's myopia is expensive.

5×55\times5 Grid Feasibility Validation

On the larger 5×55\times5 grid (25 intersections), PGQL outperformed max-pressure at every tested load level. The absolute improvement was modest at low load (approximately 6.6% at N=200N=200) but persisted throughout the tested demand range.


🧠 4. Discussion

The findings provide several insights into the design and evaluation of hybrid traffic-signal controllers.

The inductive bias provided by max-pressure is close to optimal on small networks where its one-step greedy is rarely incorrect, but its myopia appears more expensive on the larger 5×55\times5 benchmark, leaving more room for the learner to discover beneficial deviations.

The action-usage audit suggests a mechanism for the regime-dependent improvement: the override action is selected ~28% of the time at low and moderate load but increases to 37% under saturation. The learner discovered a state-dependent policy where max-pressure is followed when reliable, and overridden when it is likely to be myopic.

Cold-Start vs Warm-Start Ablation

A cold-started agent produced a catastrophic training episode at N=800N=800 with MWT=111.6 s/carMWT=111.6~s/car versus 73.7 s/car73.7~s/car for the warm-started variant.

These results indicate that the practical value of the warm-start lies in safer early training and a bounded post-saturation improvement, rather than in a large asymptotic performance advantage.

🏁 5. Conclusion

This study introduced a warm-started hybrid controller that combines reinforcement learning with a stability-proven classical policy, proposing a carrying-capacity framework based on vehicles concurrently present in the network.

At initialization, PGQL inherits the behavior of max-pressure; once learning modifies the policy, however, performance must be established empirically rather than through the stability guarantee of the underlying controller. The results show that classical adaptive control and RL can be combined so that deployment begins safely while allowing learned deviations to improve performance under heavy congestion.

Finis

Thank you for reading.

About the writer & comments ↓

Afterword

Pressure-Guided Q-Learning: a warm-started hybrid controller for signalised urban grids

How to cite

Advik Singh (2026). Pressure-Guided Q-Learning: a warm-started hybrid controller for signalised urban grids. Collabs Scholar Open Access Library.

About the Writer

AS

Advik Singh

Leander, TX

Advik Singh writes for Collabs Scholar.

Comments

Comments are coming soon.