Summary

<aside> 💡 What did I do? I built a racing game from scratch in Python and trained an RL agent (using Double Q-Learning) to grab coins while driving courses.

Why’d I do it? When I was starting my self-studying ML journey (retrospective detail here), I hadn’t developed a large project in around 2 years. So as a 2-for-1, I taught myself reinforcement learning and refamiliarized myself with larger coding projects.

</aside>

https://github.com/lucasdino/AI-Driving

S/o CodeBullet for giving me the idea for what to build in my first project. This was inspired by his video but improved upon by adding the ability to play multiple courses and have the AI generalize fairly well against courses it had not yet seen before. I think his memorized the map but not shaming…this stuff is FINICKY.

📺Couple GIFs

Track the car was trained on - performs fairly well on this one!

Let’s make this a bit harder…

Found this racetrack on Google Maps. A lot more windy than the others…this AI is lacking on the ‘I’.

Unseen track - slightly more difficult. It’s not bad but a bit of a downgrade.

Took a ton of different prompts to get a closed circuit top-down course from DALL-E 3 but it’s pretty groovy! At least I think it is. The AI doesn’t 🫠

Major Takeaways

1. Coding: Relearned how to code larger projects after my 2-year banking hiatus. I’m happy with how the code turned out though there are obvious improvements I could make.
2. Reinforcement Learning: If I had a dollar for the number of times I wanted to throw my laptop out the window I wouldn’t hesitate to put guac on my burrito bowl at Chipotle. Learned a lot by having to build the physics and data pipeline from the ground up. More on that below.

Coding

Only thing worthy of sharing here - initially I was getting around 40 FPS on training runs.

Note — all these measurements refer to when training my Neural Net (i.e., backprop is running and the model is updating). When my game starts and it hasn’t accumulated enough training data yet, I was clocking in over 200 FPS after all of these 🤠

That’s kind of slow. Like I needed this program to be Level 100 Pacer Test and it was just getting a passing grade.

So I stepped in an made a couple of improvements…

1. Smart Collision / Vision Line Detection: (~40 FPS to ~80FPS)
   • I knew this was going to be the biggest improvement but was definitely surprised by the results. I used a gridding approach mixed with an RTree (spatial data structure) to only check for intersections with the nearest racetrack lines. Previously it was just checking every line on the racetrack.
     • Gridding: When initializing my game, I store the index of each line that exists in each ‘grid’ in the map. When the car is in a grid, I would check the current grid the car is in as well as the neighboring 8 grids - then return a set of the lines that are in those grids and pass that into my intersection function.
     • RTree: I used an RTree to quickly calculate which grid my car was in based on the center point of the sprite.
2. Concept of a Session: (~80 FPS to ~100FPS)
   • I implemented a hierarchy for my game that went as follows:
     1. Session
     2. Race Game
     3. Racecar / Track / Rewards / Etc.
   • The improvement was in migrating as much code as possible into the ‘session’ since every time the car crashes, it resets the ‘RaceGame’. Loading images, importing racetrack lines / rewards from a CSV, calculating the gridding, etc. - these are all things that only need to be done once so I moved them up in hierarchy to reduce unnecessary computation
   • In doing all of this, I also refactored my code a ton which led to finding a few other areas that I could improve performance
3. NumPy / I actually now know linear algebra: (~100 FPS to ~120FPS)
   • It was kind of cute. When I first started this, I didn’t know linear algebra. So I had these long functions that I calculated using raw trigonometry. It worked but it was ugly…
   • Cue me learning linear algebra and NumPy a month or so back. So I was able to convert all my code into NumPy / matrices where possible and got a nice performance boosts here
4. Render Toggle: (~120 FPS to ~140FPS)
   • I also had a theory that I could improve speed by just turning off all my rendering functions. I made a toggle that shows up when I’m training the model and this saw another ~10% improvement in speed

All of this was huge. Faster iteration meant I could take the dog for a walk and cook dinner and have the equivalent of a full overnight training run previously.