The goals / steps of this project are the following:
- Use the simulator to collect data of good driving behavior
- Build, a convolution neural network in Keras that predicts steering angles from images
- Train and validate the model with a training and validation set
- Test that the model successfully drives around track one without leaving the road
- Summarize the results with a written report
The project includes the following files:
- model.py containing the script to create and train the model
- drive.py for driving the car in autonomous mode
- model.h5 a modelcontaining a trained convolution neural network
- model2.h5 the same model using keras 2.2.1 that uses a GPU
- model.png image of the implemented DNN
- README.md summarizing the results
Using the Udacity provided simulator a drove the track twice, in each direction and repeated the same procedure for both tracks.
The file model.py contains the code for training and saving the convolution neural network. The file shows the pipeline I used for data augmentation, training and validating the model, and it contains comments to explain how the code works.
Initially, I started by implementing the Lenet architecure used in the class, but this was found to be inadequate for producing a model suitable for driving around the track. I then implemented the model published by NVIDIAs autonomous vehicule team. I used the method plot from keras.utils.visualize_util to produce a visualization of the network architecture (line 111 in model.py). The model can be seen in the figure below:
The model consists of a normalization layer followed by a cropping layer, followed by five convolutional layers, followed by four fully connected layers. The data is normalized in the model using a Keras lambda layer (code line 72) and cropped using keras cropping2D layer (code line 61).
The model was trained and validated on different data sets to ensure that the model was not overfitting (code line 106-107). The model was tested by running it through the simulator and ensuring that the vehicle could stay on the track, indeffinatelly.
The model used an adam optimizer, so the learning rate was not tuned manually (model.py line 72). Instead, the default value, as set by keras was used. The model mean squared error loss was plotted (code lines 18-32, 117) to be used as a mean to visuallize the process and detect overfitting. A batch size of 128 was used and the model trainned for 2 epochs, as signs of overfitting were observed. The graph can be seen in the figure below:
As it can be observed, trainning loss descreased, but validation loss remained the same. I tried L2 regularization but this did not improve the situation. I used less data (two runs around the stage in opposite directions, using three cameras), but the model performed in an identical way.
The video demonstrating the performance of the model on the first track can be seen here:
Using the Udacity provided simulator, the initial data collection strategy was to drive around each track twice making sure the wheels never leave the track. At first, the data used for trainning consisted from the images taken only from the central camera only, but this proved insufficient for trainning the model. The next step was to include the data from the other two cameras adjusting the steering angle for each image accordingly. The steering angle was adjusted by subtracting/adding 0.25 from the original steering angle provided. This data also proved to be insufficient for trainning the model. Then, I complemented the dataset by driving each track in the opposite direction. For trainning the first model, I used trainning data only from the first track. This data proved to be sufficient for trainning the model and therefore no other data augmentation method was used, as the trainned model performed extrimely well. At this point of the collection process, I had 24739 of data points. I preprocessed all data by normalizing it and cropped a total of 80 pixes off each image, 25 from the bottom and 55 from the top. For the images collected from the second stage I tried cropping 35px from the top due to the fact that the second track was not flat and the road had a high inclination, extending the drivable region to a larger area within each captured image. Finally, I randomly shuffled the dataset and put 20% of the data into a validation set. I also used a generator to load data into the memory on demand, rather than loading the entire dataset at once.
Despite that the model performed exceptionally well on the first track, it did not do so for the second track. This was due to a number of factors. The driving data quality was not equally good as in the case of the first track, due to the difficulty of the stage. The driving trajectory was not smooth and the model as a result was continously correcting the steering angle in a similar way. The angle used for the side cameras, was constant and despite that it worked well for the first track, it should be replaced by a proper algorithm that predicts the correct angle. The version of keras that comes as default (1.2.1) with python does not use the GPU for trainning, hence the process was particularly lengthy. Despite manually recompiling tensorflow for my system, so that keras 2.0+ can be installed (that utilizes the GPU), speed improvements were marginal. In addition to that the code had to be adjusted for the new syntax. This suggests that a thorough investigation of the model has to be performed in order to detect bottlenecks and improve execution times. But that would require extensive experimentation. Currently, the model needed 12 hours of trainning, using data collected from six runs of the second stage (40.000 images approximatelly), on an 8th gen cpu. This prohibited us from tuning the model's hyper parameters to achieve better performance. The AWS servers used were significantly slower.