The model explains how we can Automate System using Artificial Intelligence. It broadly concerns about:- 1. Lane Detection. 2. Traffic Sign Classification. 3. Behavioural Cloning.

1. Traffic
Automation
System—
”Autonomous
Vehicles”
“Self-driving cars will enable car-
sharing even in spread-out
suburbs. A car will come to you just
when you need it. And when you
are done with it, the car will just
drive away, so you won't even have
to look for parking.”.
~SebastianThrun

2. Computer
Vision
Computer vision is a field of computer science
that works on enabling computers to see, identify
and process images in the same way that human
vision does, and then provide appropriate output.
It is like imparting human intelligence and
instincts to a computer. In reality though, it is a
difficult task to enable computers to recognize
images of different objects.
Computer vision is closely linked with artificial
intelligence, as the computer must interpret what
it sees, and then perform appropriate analysis or
act accordingly.
2
How computer vision works
Computer vision works in three basic steps:
1.Acquiring an image:-
Images, even large sets, can be acquired in real-time through
video, photos or 3D technology for analysis.
2.Processing the image:-
Deep learning models automate much of this process, but the
models are often trained by first being fed thousands of labeled
or pre-identified images.
3.Understanding the image:-
The final step is the interpretative step, where an object is
identified or classified.

3. Finding Lane
Lines on the
Road
In this project, our goal is to:
Find the lanes in the images.
Apply smoothing to draw only
smooth lanes.
.
3
The Pipeline:
The pipeline consists of 09 steps:
1.Get the lane info in the image.
2.Remove noise from the image using Gaussian Blur
Filter.
3.Get the edges in the images using Canny Edge Detector
4.Remove edges outside of the ROI(Region of Interest).
5.Find Lines using Probabilistic Hough Line Transform.
6.Find smooth lanes using Weighted Arithmetic Mean
7.Find the coordinates of the Lanes
8.Draw Lanes
9.Merge results in the original image

4. Project #1
4
Project #1 is about creating a
pipeline that detects lane lines
in images. While the pipeline is
created for a single image, it
can be applied to video footage
by breaking the video down
into frames, passing the frames
through the pipeline, and then
reconstructing the video.

5. Step 1: Get
the Lane info
in the image:
Grayscale
5
HSV channels information is
extracted in order to get the
objects in a frame of specific
color.

6. Step 2:
Remove noise
from the
image using
Gaussian
Blur Filter
6
Gaussian Blur Filter is used to
remove the noise from the
image.

7. Step 3: Find
the edges in
the images
using Canny
Edge-
Detector.
7
Canny Edge Detector is used
to find the edges.The edges
are the points where the
grayscale values change quite
sharply.

8. Step 4:
Remove
edges outside
of the ROI
8
There can be several objects in an
image which are not of interest, e.g.
trees etc. A region of interest (ROI)
needs to be created to remove the
extra details in the image beside the
lanes.

9. Step 5: Find
Lines using
Probabilistic
Hough Line
Transform:
9
Then, Probabilistic Hough
LineTransform is used to get
the lines. It provides the start
and end points of each
detected line.

10. Step 6: Find
smooth
lanes using
Weighted
Arithmetic
Mean
10
Hough LineTransform provides several lines of
different slopes and lengths.The ideal solution
is to have only two lines, one for left lane and
the other for right lane.
Since Hough lines vary in length, the most
important and useful line are those which are
longest in length because the smaller lines
might be due to the noise and random
calculations.
[https://en.wikipedia.org/wiki/Weighted_arithm
etic_mean]
The weighted average can be used where the
line length can be used as a weight.

11. Step 7: Find
the
coordinates
of the lanes:
11
At this point, we have the slopes and the
intercepts of both lanes. Now we need to
calculates the start and end coordinates for
both lanes.
Additionally, we also need to handle the cases
when there is no Hough lines are found.This
problem is solved by keeping the frames and
using the average of the them in case no Hough
lines are found.

12. Step 8: Draw-
Lanes:
12
Now, we can draw the detected
lanes on a blank image for
debugging and testing.This
step is not necessary and the
results can directly be drawn on
the actual image without first
drawing on the blank image.

13. Step 9: Merge
results in the
original-
image:
13
The detected and smooth lanes
can now be drawn on the actual
image using addWeighted
function. It merges two images
with configurable weights
(importance of an individual
image).

14. Classificati
on
Classification is the process of
predicting the class of given data
points. Classes are sometimes called
as targets/ labels or categories.
Classification predictive modeling is
the task of approximating a mapping
function (f) from input variables (X)
to discrete output variables (y).
14
How Classification works
Classification works in three basic steps:
1. Data cleansing and preparation (pre-processing).
2. Design and build the CNN network.
3. Train the model and verify on test / real-life data.

15. Traffic signs-
Classification
project
In this project, our goal is to:
Detecting and ClassifyingTraffic
signs.
15
 Classify traffic signs using a simple convolutional neural
network.
 The CNN learns from the images it is trained on making it
an evolving / learning system.
 Training is done on a subset of the repository keeping
enough samples for validation and testing.
 During training, care is taken to avoid over-fitting.The inner
working of the CNN is analyzed to understand activations
and how the images convolve as it is processed through the
network.

16. Project #2
16
• Detecting and Classifying Traffic signs is a
mandatory problem to solve if we want self
driving cars.
• The dataset we will be using is a German Traffic
sign dataset available online.
• It contains more than 50,000 images in total,
divided into 43 different classes: speed limits,
dangerous curves, slippery road… Here are
some of them.
• This dataset was used in a competition a few
years ago.The best result for the competition
correctly guessed 99.46% of the signs. In
comparison, human performance was
established at 98.84%.Yes, the machine was
more efficient than the human, as it was better
at handling the most difficult cases, such as a
blurry image of a speed limit sign that could be
mistaken for a different speed limit.

17. Initial data exploration shows outliers and
under-representation of a few classes.
These are corrected first. A threshold of
30% is used when making a decision to up-
sample.The threshold is based on general
estimate of the class distribution. The
statistics are shown below before and after.
Next, a sampling of images is selected to
visualize the variations. As seen below,
none of the images are quite the same,
which is good for the learner, but quite a
few are completely illegible. 17
Data
pre-
processing

18. To make the images consistent and to eliminate
the noise, they are taken through a
transformation process— grayscale conversion,
histogram equalization and mean normalization.
Histogram equalization improves the contrast
information ensuring that images are processed
under uniform lighting condition. Mean
normalization helps with the problem of
vanishing gradient as well as help the system
‘converge’ faster.The effect of this processing is
shown below.
The final statistics of the image repository is
shown below.
18
Data pre-
processing

19. A convolution neural network is a
specific type of deep learning for image
recognition. CNNs (and, for that matter,
neural networks) are inspired by how the
human brain works.
CNNs process visual inputs through
small receptive fields, progressively,
across the entire image extracting locally
correlated information.This process is
known as convolution
The starting point chosen for the traffic
sign classifier is the LeNet architecture.
LeNet shown below has two convolution
layers and two fully connected layers. 19
Deep
Learning

20. Each filter in a layer learns specific features
of the image as the model trains.
Visualizing the activations provides an
overview of what features the network
finds important (and hence giving it high
weight) as well as to get an insight into the
workings of the network. A sample set of
activations are shown below for the first
two layers which have 11 and 48 filters .The
images picked were one which the model
almost always predicted correctly (top one)
and other one which it always got wrong.
20
What is the
CNN doing?

21. The first set of activations clearly shows
the unique features of the traffic sign,
including is texture, depth and spatial
information.This a fairly good set of
features that the network has detected.
However, it seems to be having a
tougher time in deciding on the features
for the second image. In subsequent
layers, this becomes more evident as it
fails to separate out any individual
feature.
21
What is the
CNN doing?

22. The features are now better defined
for both images in both the
convolution layers.There are more
more activations seen for the second
image. It should be noted here that
the source images used for this
analysis were downloaded from the
internet and not part of the original
dataset. It is time to evaluate how
the model is doing on those images.
22
What is the
CNN doing?

23. The metrics used to evaluate performance are
accuracy, precision and recall, Precision reflects
the network’s ability to avoid false positives;
recall false negatives. A high number for all three
reflects a well-trained model. Loss is graphed to
see the general trend during training and
validation. A good training session is when loss
decreases and accuracy increases on the training
as well as validation data set.The target accuracy
is 93% on the validation set.
The table below shows the metrics of some of the
training runs.
23
Training
the CNN

24. Behavioral
Cloning
The the goal of behavioral cloning is
to collect data while exhibiting good
behavior and then train a model to
mimic that behavior with the collected
data.
24

25. Behavioral
Cloning for
Self-Driving
Cars
In this project, our goal is to:
Train a CNN to drive a car in a
simulator.
25
 Goal is to teach a Convolutional Neural Network (CNN) to
drive a car in a simulator provided by Udacity.
 The car is equipped with three cameras that provide video
streams and records the values of the steering angle,
speed, throttle and brake.

26. Project #3
26
Goal is to teach a Convolutional Neural
Network (CNN) to drive a car in a simulator
provided by Udacity.The car is equipped with
three cameras that provide video streams and
records the values of the steering angle,
speed, throttle and brake. The steering angle is
the only thing that needs to be predicted, but
more advanced models might also want to
predict throttle and brake.This turns out to be
a regression task, which is very different from
usual applications of CNNs for classification
purposes.

27. I collected the training data by driving
the car on the flat terrain training track.
The performance of theCNN can then be
checked by letting the car drive
autonomously on the same track or
ideally on a second track that is
considerably more windy with steep hills
and that should not be used for training.
27
Step #1
Collecting the
Training Data

28. While driving the car under normal
conditions the steering angle is very close
to zero most of the time.This can clearly be
seen in the raw training data. Below are the
steering angles I recorded while driving the
car around the track while staying as close
as possible to the middle of the lane.
28
Step #2
Balance the
Unbalanced
data

29. I used images from all three cameras.
Each image was randomly flipped
(horizontally) with equal probability in
order to make left and right turns appear as
frequently. Also brightness was randomly
adjusted.
29
Step #3
Data
augmentation

30. So far all transformations had been
performed on the full 320x160 pixel images
coming from the cameras. In the next stage
I chose a window of 280x76 pixels that
eliminated the bonnet from the bottom of
the image and cropped off the part above
the horizon in flat terrain at the top.
30
Step #4
Image
Preprocessing

31. For the actual network architecture,
the Nvidia model is used because of its
simplicity and demonstrated ability to
perform well on self-driving car tasks.
31
Step #5
Use of Nvidia
Network
Architecture

32. .
32
Use of Nvidia
Network
Architecture

33. We need several libraries to build out the
pipeline that will train our model. Here csv is
included to read from, well CSV files; cv2 for
image manipulation; argparse for parsing
command line arguments; utils, a small set of
utilities abstracting some of cv2’s
functionality; shuffle and train_test_split from
sklearn for shuffling data and splitting it into
training and validation sets,
respectively; numpy for awesome numerical
tools; and finally model, where our Keras
model is defined. 33
Step #6 Import
Libraries for
Model

34. 34
Step #7
Running the
vehicle on
Autonomous
mode

35. 35
How the model
works.

36. 36
DATAFL0W-
DIGRAM

37. THANK YOU
Question Session
https://github.com/prabal-chauhan
37