Use deep learning autopilot with gps autopilot

[Ezward]

The path follow template currently relies on timely and accurate gps readings. However in the real world gps can fail. During a race this can lead to a crash as we have seen in the Indy Autonomous Challenge. We have seen this at the first race at UCSD where the two finalists, one a Donkeycar and one an F1Tenth-like car running ROS, both failed at the start due to gps issues.

This project would add the ability to run a deep learning autopilot in parallel with the gps autopilot. The deep learning autopilot may be used as a fail safe if the gps fails or it may be used to increase autopilot quality when using a low quality gps.

The development would be in two parts:

1. Update the path follow template to allow it to optionally record data for the deep learning autopilot. So each time through the vehicle loop we would save the image and the throttle and steering inputs. Basically port the related code from the deep learning template. We can assume the linear template, so only the image, steering and throttle need be saved.
2. Update the path follow template to allow it to run deep learning autopilot. Here there is a lot of code to port from the deep learning template. We may want refactor this into a function that can be shared between the deep learning template and the path follow template. Here we want to have two different modes.
   • The first mode would only run the autopilot if the gps 'cuts out'; so if we detect that we are a) the gps name sentence is telling us that the reading is not reliable so we are not getting timely accurate gps readings b) we detect that the serial connection to the gps is lost so that we are not getting any gps readings. So in this mode the deep learning autopilot is a fail safe.
   • The second mode would run the deep learning autopilot between gps readings. So for cheaper gps modes the update rate may only be 5 hz. However we could potentially run faster if we fill in the gaps with a deep learning model.

These two tasks can be accomplished independent of each other; so two different teams could do these tasks in parallel. Indeed they can be used independently. For instance, we can gather the data and train it using the deep learning template as we do now, and if we have the ability to run a deep learning model in the path follow template then just use the model in the path follow (so path follow does not have to save data). If we can gather data in the path follow template then we can use it to train a deep learning autopilot for the deep learning template (so path follow does not have run a deep learning autopilot). But once we have both then we can have a combined template that can use either or both.

[TCIII]

@Ezward,

A very ambitious undertaking to say the least.

TCIII

[TCIII]

@Ezward,

I believe a simpler solution would be to use IMU input in place of image, throttle and steering inputs.
This is basically what the u-Blox ZED-F9R RTK GPS module does which has an integrated IMU on the RTK GPS module for sensor fusion.
If the vehicle momentarily loses GPS input, the IMU can hold the vehicle on the intended course during the loss period.

TCIII

[ShuanSong]

en. I think this ideal is great that achieve the goal: donkey car can drive from A point to B point , then to C point in the counrty.
I think there are another way to achieve this goal.
· creat a new template ,let's GPS data [x,y] and imu data as another channel into the CNN net.