Problem Statement
Autonomous racing requires more than following a precomputed racing line. In head-to-head racing, the optimal raceline can be blocked by another vehicle, so the ego car must decide when to avoid, when to overtake, and how to return to the raceline safely.
Traditional F1Tenth methods such as Pure Pursuit, Follow-the-Gap, and sampling-based planners can handle path tracking or local obstacle avoidance, but they are often rule-based and may not generalize well to complex overtaking interactions. This final project investigates a reinforcement-learning-based local overtaking controller combined with a conventional raceline tracking controller.
The concept for this solution is a hybrid control framework that switches between:
1) Pure Pursuit for nominal tracking of a precomputed minimum-curvature raceline.
2) Reinforcement Learning overtaking controller when the planned raceline is blocked by another vehicle or obstacle.
This allows the system to retain the stability of classical trajectory tracking while using RL for local, dynamic overtaking behavior that requires more adaptive control.
2) Reinforcement Learning overtaking controller when the planned raceline is blocked by another vehicle or obstacle.
This allows the system to retain the stability of classical trajectory tracking while using RL for local, dynamic overtaking behavior that requires more adaptive control.
Technical Approach
1. Minimum-Curvature Raceline Generation
We generated a reference raceline using a minimum-curvature trajectory optimization framework. This raceline is used by the Pure Pursuit controller when no obstacle blocks the planned path.
We generated a reference raceline using a minimum-curvature trajectory optimization framework. This raceline is used by the Pure Pursuit controller when no obstacle blocks the planned path.
2. Pure Pursuit Controller
During normal driving, the vehicle tracks the optimized raceline using Pure Pursuit. The controller selects a lookahead waypoint, transforms it into the vehicle frame, and computes the steering command.
During normal driving, the vehicle tracks the optimized raceline using Pure Pursuit. The controller selects a lookahead waypoint, transforms it into the vehicle frame, and computes the steering command.
3. LiDAR-Based Obstacle Detection
The vehicle uses LiDAR scans to build a local occupancy grid in the ego frame. LiDAR hit points are converted into grid cells, and occupied cells are inflated to provide a safety margin around walls and obstacles.
The vehicle uses LiDAR scans to build a local occupancy grid in the ego frame. LiDAR hit points are converted into grid cells, and occupied cells are inflated to provide a safety margin around walls and obstacles.
4. Controller Switching Logic
The system checks the reference trajectory segment between the closest waypoint and the Pure Pursuit target waypoint. If any part of this segment intersects with occupied cells in the local occupancy grid, the overtake flag is activated and control switches to the RL policy.
The system checks the reference trajectory segment between the closest waypoint and the Pure Pursuit target waypoint. If any part of this segment intersects with occupied cells in the local occupancy grid, the overtake flag is activated and control switches to the RL policy.
5. RL Overtaking Controller
The RL controller uses processed LiDAR scans, velocity, yaw rate, and previous action information as input. The network outputs normalized steering and speed commands, which are then rescaled to physical vehicle limits.
The RL controller uses processed LiDAR scans, velocity, yaw rate, and previous action information as input. The network outputs normalized steering and speed commands, which are then rescaled to physical vehicle limits.
RL Models
Training:
I built a custom reward reward function to encourage the following:
• Forward progress
• Higher velocity when safe
• Overtaking the opponent
• Smooth steering
• Maintaining distance from walls/obstacles
• Avoiding collisions
Terminal penalties and rewards are added for collision and successful overtaking as well.
• Higher velocity when safe
• Overtaking the opponent
• Smooth steering
• Maintaining distance from walls/obstacles
• Avoiding collisions
Terminal penalties and rewards are added for collision and successful overtaking as well.
The model was a PPO RL model, trained entirely on CPU across multiple race tracks against a opposing agent running Pure Pursuit. It was trained across multiple maps to attempt to build generalizability into the model. Results are shown below.
The models for Houston Hall (where the final race between all teams in the class took place), performed very well in sim. However, on hardware, we saw our car fail in the top right of the track, where there was an alcove along the wall in the top right corner. Thus, we trained the model again with another 2M steps exclusively in that section, tweaking the rewards to prioritize overtaking rather than survival (the survival reward would accumulate when the car would drive into the alcove). See the plots on the right of sim performance. In sim, it learned to make a U-turn and go back down the path it came in a few runs, but when we put this model on hardware, it handled the turn very well.
One condition I had implemented to ensure we could overtake the opponent was the ego vehicle being capable of max speed ~1 m/s greater than the opponent. This makes for some weird and unrealistic speed differences, but it demonstrates how the model learns to overtake only when it's safe (similar to how actual racecar drivers overtake in real life. See below for a demonstration in sim.
Hardware Results
We didn't have access to the Houston Hall track for enough time to collect good videos on the model's overtaking capabilities, so we tested it on a separate racetrack in Levine Hall. The below videos are of this testing.