UTS 41118 · AI in Robotics

Pitwall

An F1 inspired autonomous racing simulator. A reinforcement learning agent learns to race a multi lap track while a separate supervised model predicts tire degradation in real time. The objective is to minimise total race time by trading raw speed for tire strategy and pit stops.

Danish Silva · Hari Mahadevan · Ben Costarella · Dennis Asante
University of Technology Sydney
Code Overview Architecture Demo Run It Iteration Log Results
2
AI Components
PPO
RL Algorithm
3
Tire Compounds
96²
px Camera Obs
50
FPS Sim

Abstract

Pitwall is a research grade racing simulator that combines two distinct AI components in a single closed loop. The simulation environment extends Gymnasium's CarRacing-v3 with multi lap race semantics, three tire compounds (soft, medium, hard), a pit lane trigger zone, and a custom reward function shaped for racing strategy. A Proximal Policy Optimisation (PPO) agent learns to drive the car from a 96×96 top down camera feed. A separate gradient boosting regression model trained on stylised F1 tire physics predicts the per step wear rate as a function of speed, cornering load, current wear, and the chosen compound.

The two components close the loop inside the env. Each simulation step, the env reads the car's dynamics, queries the tire model for a wear delta, accumulates it into the agent's state vector, and applies reward shaping for strategy decisions (pit timing, lap completion, off track behaviour). The agent therefore learns not only how to drive (staying on the racing line, hitting apexes) but also when to pit, treating tire degradation as a managed resource.

Beyond the AI itself, Pitwall ships a complete training and evaluation harness: parallel sub process environments, held out evaluation callbacks, racing line visualisations, pit strategy comparison tables, and an A/B mode that pits a tire aware agent against a no strategy baseline on identical track seeds.

Team

Each member owns one subsystem of the project end to end.

Danish

System Architecture · RL Agent

Hari

Environment · Custom Mechanics

Ben

Tire Model · Synthetic Data

Dennis

Evaluation · Plots · Docs

System Architecture

Pitwall is composed of four loosely coupled subsystems. Each is owned by one team member, lives in its own Python package, and communicates with the others through narrow, well typed interfaces.

Hari

Environment

environment/environment.py

A gym.Wrapper around CarRacing-v3 that adds every race specific mechanic.

  • Continuous tire wear (0 to 100 %) updated each step from the supervised tire model
  • Three tire compounds fixed per episode, encoded into the observation
  • Pit lane trigger zone: a circle around the spawn tile (the natural start/finish line)
  • Multi lap support via tile_visited_count: reset tile flags, continue racing
  • Reward shaping: lap completion bonus, soft off track penalty, stagnation termination
  • Optional curriculum: fixed_track_seed forces all envs onto the same track
Danish

RL Agent (PPO)

agents/train.py · agents/policy.py

Proximal Policy Optimisation via Stable Baselines3 with a multi input policy over a Dict observation.

  • Observation: 96×96 RGB camera frame plus 3-d state vector [wear, lap, compound]
  • Action space: continuous (steer, throttle, brake)
  • Frame stacking (n=4) so the policy can perceive motion and angular velocity
  • SubprocVecEnv for 8 way parallelism with near linear throughput scaling
  • EvalCallback writes held out reward to TensorBoard every 10k steps
  • Resume support with CLI hyperparameter overrides (ent_coef, lr, clip_range)
Ben

Tire Model

tire_model/tire_model.py

A supervised regression model that maps driving conditions to a per step wear rate.

  • GradientBoostingRegressor (scikit-learn), 300 trees, depth 4
  • Synthetic dataset: 12 000 samples from stylised F1 tire physics
  • 5 features: speed, cornering load, lap, current wear, compound id
  • Per compound base wear, speed exponent (~v¹·⁴), cornering sensitivity
  • "Cliff" amplifier: degradation accelerates past 50 % wear
  • Per compound R² reported alongside overall metrics
Dennis

Evaluation

evaluation/evaluate.py · watch.py · visualize_track.py

Rollout based offline evaluation plus tools for live inspection and debugging.

  • Markdown summary table: reward, length, laps, pit count, final wear, lap time
  • Pit strategy comparison table that buckets episodes by pit count
  • Plots: tire wear over time, per lap wear, racing line, reward distribution
  • watch.py opens a pygame window to see the agent drive live
  • visualize_track.py previews the random track and pit zone geometry
  • dump_observation.py saves the agent's stacked frames as PNGs

How The Training Loop Works

Training runs a standard on policy PPO loop, but the wrapper environment does most of the interesting work. Every step inside the agent's rollout, Pitwall:

  1. Receives a continuous action (steer, throttle, brake) from the policy.
  2. Steps the underlying CarRacing physics. Box2D integrates the dynamics at 50 Hz.
  3. Reads the resulting car state: speed from car.hull.linearVelocity, lateral acceleration from |ω| · v (normalised into the [0, 10] cornering range the tire model was trained on).
  4. Queries predict_wear_rate(speed, cornering_load, lap, current_wear, compound) and scales the output into a per env step wear delta.
  5. Updates internal state: tire wear, lap counter (via CarRacing's tile_visited_count), stagnation counter, pit zone proximity.
  6. Computes the shaped reward (see below) and returns the new observation, reward, and termination signals to PPO.

Eight env workers run in parallel via SubprocVecEnv. PPO collects 512 steps per worker per rollout (4096 total samples), then runs 10 epochs of mini batch gradient updates with the standard clipped surrogate objective. A separate EvalCallback runs deterministic policy rollouts every 10 000 total steps on a held out single env to track unbiased progress, and saves the best best_model.zip whenever a new high water mark is hit.

Reward Structure (Current Iteration)

Component Value Purpose
Tile visit (base CarRacing) +1000/N per fresh tile Dense forward-progress signal
Time penalty (base CarRacing) −0.1 per step Discourage dawdling, rewards faster laps
Lap completion bonus +200 per lap Terminal goal signal for the value function
Soft off-track penalty −0.1/step beyond 3.0 units "Grass is bad" gradient without killing the episode
Stagnation termination −25 after 300 steps without a new tile Kills the spin-in-place local optimum
Pit-stop penalty −50 on a successful pit (resets tire wear) Models real-world pit-time loss
Tire-wear cost (phase-2, currently 0) −0.001 × wear% per step Enabled once the agent reliably completes laps

Why this shape? Earlier reward structures created a "spin in place on track" local optimum: the agent would visit a few tiles, get stuck, and run out the clock collecting only the small time penalty. Adding the stagnation termination and lap completion bonus broke that equilibrium directly. Sitting still now ends the episode within 6 seconds with a clear penalty, while completing a lap gives a +200 spike that PPO's value function can latch onto.

The Tire Model

The second AI component is a supervised regression model that predicts instantaneous tire wear rate from car state. It runs inside PitwallRacingEnv.step() as a callable: 

predict_wear_rate(
    speed,            # 0 to 100 (m/s ish)
    cornering_load,   # 0 to 10  (lateral g, scaled)
    lap,              # current lap number, 1 indexed
    current_wear,     # 0 to 100 %
    compound,         # "soft" | "medium" | "hard"
) -> float / ndarray

Training data is fully synthetic. 12 000 samples drawn from a stylised F1 physics function with per compound parameters: 

_COMPOUND_PARAMS = {
    "soft":   {"base": 0.20, "speed_gain": 0.0020, "load_gain": 0.50,
               "heat_gain": 0.030, "cliff": 1.6},
    "medium": {"base": 0.12, "speed_gain": 0.0014, "load_gain": 0.40,
               "heat_gain": 0.020, "cliff": 1.4},
    "hard":   {"base": 0.07, "speed_gain": 0.0009, "load_gain": 0.30,
               "heat_gain": 0.012, "cliff": 1.2},
}

Each compound has a different base wear rate, a different sensitivity to speed and cornering load, and a different cliff exponent: a multiplier that ramps degradation in the last half of the tire's life. This captures real F1 tire behaviour where a worn tire degrades much faster than a fresh one, and where soft compounds give more grip but pay for it with shorter lifetime.

A GradientBoostingRegressor with 300 trees and max depth 4 is trained on the synthetic data using an 80/20 split. Performance is reported both overall and per compound, so we can detect if the regressor is failing on one tire type in particular. The model is persisted to models/checkpoints/tire_model.pkl via joblib and lazily loaded into the env on first use.

Demonstration

Watch the trained Pitwall agent in action: lap completion, pit zone detection, tire wear updates in real time, and recovery from corner bobbles. The video below shows the policy driving on the same track it was curriculum trained on.

Recorded with python -m evaluation.watch --model-path models/checkpoints/best/best_model.zip.

Running The Project

Every command below runs from the PitwallAgent/ folder with the project's virtual environment activated. The full flag surface for any script is available with --help.

Setup

Install Dependencies

Python 3.10 or newer. Create a virtual environment and install everything in requirements.txt. 

python -m venv .venv
.venv\Scripts\activate          # Windows
# source .venv/bin/activate     # macOS / Linux

pip install --upgrade pip
pip install -r requirements.txt

On Windows, gymnasium[box2d] builds Box2D from source and needs SWIG. If pip fails on box2d-py, install SWIG first (pip install swig) and rerun.

Step 1

Train The Tire Model

Generates 12 000 synthetic samples, trains the GradientBoostingRegressor, prints overall and per compound R² / MAE, and saves the model and a diagnostic plot. Required before the env will construct (it raises if the .pkl is missing). 

python -m tire_model.tire_model

Outputs: models/checkpoints/tire_model.pkl, data/synthetic/tire_data.npz, evaluation/results/tire_compound_curves.png. Runtime: under one minute.

Step 2

Smoke Test The Pipeline

A short run with 2 parallel envs to confirm everything wires up before committing to a long training session. Should complete in about 5 minutes. 

python -m agents.train --total-timesteps 5000 --n-envs 2 --frame-stack 4
Step 3

Phase A: Curriculum Training

Trains the agent on a single fixed track to learn the basics with low variance. 8 parallel envs, frame stacking, entropy bonus to keep exploration alive. Several hours on CPU. 

python -m agents.train ^
    --total-timesteps 1500000 ^
    --ent-coef 0.01 ^
    --n-envs 8 ^
    --frame-stack 4 ^
    --fixed-track-seed 42

Outputs go to models/checkpoints/ (periodic ppo_pitwall_*_steps.zip, plus best/best_model.zip from EvalCallback) and TensorBoard logs to runs/PPO_*/.

Step 4

Phase B: Generalise To Random Tracks

Once eval/mean_reward is stable on the fixed track, Ctrl+C and resume without --fixed-track-seed. The agent transitions from one memorised track to the full random distribution. 

python -m agents.train ^
    --total-timesteps 3000000 ^
    --ent-coef 0.01 ^
    --n-envs 8 ^
    --frame-stack 4 ^
    --resume models/checkpoints/ppo_pitwall_1500000_steps.zip

Resume honours CLI overrides for ent_coef, learning_rate, and clip_range via custom_objects. Architectural hyperparameters stay baked into the checkpoint.

Monitor

Watch Training In TensorBoard

Open a second terminal and run TensorBoard. The web UI updates roughly every 30 seconds, faster if you change the reload interval in the gear menu. 

tensorboard --logdir runs/

Key curves: eval/mean_reward, rollout/ep_rew_mean, rollout/ep_len_mean, train/explained_variance, train/clip_fraction.

Inspect

Watch The Agent Drive Live

Opens a pygame window and renders the policy driving in real time. Use the random policy first to confirm the env renders correctly, then point at a checkpoint to see the trained agent. 

# Random policy (no model needed)
python -m evaluation.watch --random --frame-stack 4

# Best checkpoint mid training (re-run to refresh)
python -m evaluation.watch ^
    --model-path models/checkpoints/best/best_model.zip ^
    --frame-stack 4

# On the same track the agent was curriculum trained on
python -m evaluation.watch ^
    --model-path models/checkpoints/best/best_model.zip ^
    --frame-stack 4 --fixed-track-seed 42
Evaluate

Full Evaluation Report

Runs N held out episodes on a trained checkpoint, prints summary tables, and saves the full plot set to evaluation/results/. 

python -m evaluation.evaluate --episodes 5 --frame-stack 4

Add --baseline-path to A/B compare two checkpoints on identical seeds; the second model's outputs get a baseline_ prefix. 

python -m evaluation.evaluate ^
    --model-path models/checkpoints/best/best_model.zip ^
    --baseline-path models/baseline/ppo_pitwall_final.zip ^
    --episodes 5 ^
    --frame-stack 4
Debug

Track Preview And Observation Dump

Two utilities for sanity checking the env without needing a trained model. 

# Preview the track shape and pit zone for a given seed
python -m evaluation.visualize_track --seed 42 --n-seeds 4

# Dump the 4 stacked frames the agent actually sees as PNGs
python -m evaluation.dump_observation

The track preview saves evaluation/results/track_preview.png with the pit zone overlay. The observation dump saves evaluation/results/obs_frame_*.png so you can verify the car is centred on the track in every frame of the stack.

Reset

Wipe And Start Fresh

If a training run goes south and you want to restart from scratch (clean checkpoints, clean TensorBoard logs), the safe wipe commands are: 

Remove-Item -Recurse -Force models\checkpoints\best -ErrorAction SilentlyContinue
Remove-Item models\checkpoints\ppo_pitwall_*.zip -ErrorAction SilentlyContinue
Remove-Item -Recurse -Force runs\* -ErrorAction SilentlyContinue

The tire model .pkl is preserved (the env needs it). Source code, the LOGBOOK, and gitignored artefacts are untouched.

Iteration Log

Pitwall took shape over many training cycles. Each lap of the loop revealed a new failure mode, which informed the next change. The honest path here matters more than the final state. Most of what we learned is in why each version stopped working.

Lap 1 · Bootstrap

Repo Skeleton + Tire Model V1

Built the directory structure, dependency list, and a first pass at every subsystem. Tire model trained on 12 k synthetic samples with three compounds and a cliff term; diagnostic plot of wear-rate-vs-lap per compound looked sensible. Env wrapped CarRacing-v3 with placeholder tire wear and a hard-coded pit zone.

scaffolding tire_model.pkl
Lap 2 · Wire The Two Components

Tire Model Consumed Inside env.step

Replaced the placeholder wear formula with a call to predict_wear_rate(...). The two AI components now close the loop: the env reads car dynamics, asks the tire model for a wear delta, applies it to internal state, and surfaces tire wear to the policy through the observation.

closed-loop env.step()
Lap 3 · Lap Detection

Multi Lap Support Via tile_visited_count

CarRacing-v3 terminates after a single circuit. We added a wrapper that watches env.unwrapped.tile_visited_count; when it reaches the full track length, lap count increments, tile flags reset, and the underlying termination is suppressed. The agent can now race up to max_laps circuits in one episode.

Lap 4 · Real Pit Zone

Pit Lane Anchored To The Spawn Tile

The initial pit zone was a hard coded world coord box that never fired because CarRacing builds a random track each reset. Replaced it with a circle centred on track[0] (where the car spawns, the natural start/finish line analogue), cached at reset. Pit now triggers when the car is inside the zone, below a speed threshold, with non trivial tire wear.

Lap 5 · Full Training Loop

EvalCallback, SubprocVecEnv, Resume

Hardened the trainer for real overnight runs: 8-way parallelism via SubprocVecEnv, held-out eval every 10 k total steps with automatic best_model.zip tracking, CarRacing-tuned hyperparameters from the SB3 zoo, full CLI surface with --help, and a --resume path that preserves optimiser state.

Lap 6 · Rich Evaluation

Lap Times, Racing Line, A/B Comparison

Rewrote evaluate.py around an EpisodeResult dataclass that holds per-step telemetry. Added the per-lap wear bar chart, the racing-line plot overlaid on the actual track, a pit-strategy comparison table that buckets episodes by pit count, and a --baseline-path A/B mode that evaluates two checkpoints on identical seeds.

Lap 7 · Aggression Issues

Frame Stacking + Larger CNN

The first long training run plateaued around 30 % lap completion with classic hairpin failures. The agent would initiate a turn but release prematurely. The fix from the literature is frame stacking (give the policy the last 4 frames so it can perceive motion) plus a beefier feature extractor. Added VecFrameStack(n=4) and a custom PitwallFeaturesExtractor with channels 64→128→128.

Lap 8 · Regression

Bigger Model Converged To Worse Equilibrium

To our surprise, the frame-stack + big-CNN run plateaued worse than the baseline. After 2M steps the agent had learned an aggressive driving style that overshot hairpins, spun out, and got terminated by the strict off-track barrier. The extra capacity was learning the wrong thing faster.

red flag over-aggression
Lap 9 · Curriculum + Safety Nets

Fixed Seed Curriculum, Looser Barrier, Waypoints

Added --fixed-track-seed so all envs see the same track during phase 1 (much lower variance, faster learning of one specific corner sequence). Loosened the off track barrier from 10 to 18 units so apex bobbles could recover instead of forcing termination. Added optional waypoint bonuses around the track for a denser reward signal.

Lap 10 · Reward Redesign

Spin In Place Was The Real Bug

After 7M cumulative steps we still hadn't completed a lap. Detailed inspection of long episodes revealed the agent was discovering a "spin in place on track" local optimum: stay on track, accept the small time penalty, never trigger the off-track barrier. The reward function had no signal punishing stagnation.

We redesigned: added a lap completion bonus (+200) for a clean terminal signal, a stagnation termination (−25 after 300 steps without a new tile) that directly kills the spin equilibrium, and a soft per step penalty for being on grass. Reverted the custom CNN back to SB3's default NatureCNN. Smaller models train faster and are less prone to over confidence in bad equilibria.

current version reward redesign CNN revert

Results

Tire Model Accuracy

The regression model achieves high R² on held out synthetic data, with per compound metrics close to the overall figure. This confirms that the regressor discriminates between soft, medium, and hard cleanly rather than averaging them out. The diagnostic plot shows the expected ordering: soft tires degrade fastest, hard tires last longest, and all three exhibit the "cliff" in the second half of life.

(R² and MAE printed by python -m tire_model.tire_model. The per compound wear rate vs lap plot is saved to evaluation/results/tire_compound_curves.png.)

Agent Training

Training metrics live in TensorBoard under runs/PPO_*/. The headline curves to watch are eval/mean_reward (held out reward every 10 k steps) and rollout/ep_len_mean (episode length, which climbs as the agent survives longer on track).

The final reward redesign produced the qualitative shift we were after: stagnation is no longer survivable, and the lap completion bonus gives PPO's value function a clear goal to assign credit toward.

Evaluation Artefacts

Running python -m evaluation.evaluate --episodes 5 against a trained checkpoint produces a complete report inside evaluation/results/:

Add --baseline-path to also evaluate a second checkpoint on the same seeds. The same plot set is saved with a baseline_ prefix for direct A/B comparison.

Reflections

Reward shaping is the dominant lever. Every architectural change we made (frame stacking, the bigger CNN, the camera zoom, the soft barrier) was less impactful than the reward redesign. PPO converges quickly and confidently to whatever the gradient signal points to. If the gradient points somewhere bad, more parameters just get you there faster.

Local optima are sneaky. The "spin in place on track" equilibrium was invisible from aggregate metrics (average reward was just modestly negative), but watching a single episode in watch.py revealed it immediately. The lesson: aggregate metrics tell you the agent is failing, watching tells you why.

Curriculum learning helps but isn't free. Locking to a single seed during phase 1 drops variance dramatically and accelerates early progress, but the transition to random tracks (phase 2) is its own mini training problem with a temporary regression. Worth it overall, not magic.

The two AI components story works. The PPO agent and the tire model are coupled through the reward function and the observation, not through any shared training signal. They were developed independently (different team members, different timelines) and integrated cleanly. That separation is what makes the system testable: you can ablate one without breaking the other.

What we'd do differently. Start with a tighter reward function. Resist the urge to add architecture changes before fixing rewards. Build the evaluator's racing line plot earlier, it's the single most useful debugging artefact and we shipped it late. And run a sanity check on the observation pipeline before training overnight; we eventually wrote dump_observation.py to verify the agent's view matches what we thought it saw.