# Convolutional Neural Network

### neuron (node/unit)

input + weight

function f = Activation Function

• Sigmoid σ = [0,1]
• tanh = [-1, 1]
• ReLU f(x) = max(0, x)
• Softmax: Probability (Pass) + Probability (Fail) = 1

### Feedforward Neural Network

neurons, layers, edges, weights

only one direction

#### Backpropagation algorithm

Backward Propagation of Errors (BackProp) = learning from mistakes

1. Forward Propagation
1. Random weight
2. Input + output (training) -> V = f (1*w1 + 35*w2 + 67*w3)
2. Back Propagation and Weight Updation
1. calculate total error at the output nodes
2. calculate the gradients (propagate these errors back through)

https://www.cs.ryerson.ca/~aharley/vis/fc/

#### MNIST Database of handwritten digits

• Inputs: 28 x 28 image = 784 numeric pixel values
• first hidden layer: 300 nodes
• second hidden layer: 100 nodes
• output layer: 10 nodes (10 digits)

### ConvNet

Operations:

1. Convolution
2. Non Linearity (ReLU)
3. Pooling or Sub Sampling
4. Classification (Fully Connected Layer)

### Convolution

Image: three channels (red, green and blue) range 0 to 255 (0=black, 255=white)
grayscale image: one channel

3 x 3 matrix = filter / kernel / feature detector

output matrix = Convolved Feature / Feature Map / Activation Map

size of the Feature Map controlled by three parameters:

• Depth: number of filters
• Stride: number of pixels. larger stride = smaller feature maps

### Non Linearity (ReLU)

want our ConvNet to learn would be non-linear (Convolution is a linear operation)

### The Pooling Step

Spatial Pooling / subsampling / downsampling

• reduces dimensionality but retains important information
• Max, Average, Sum etc

### Fully Connected Layer

softmax activation: sum of output probabilities = 1

### training process:

1. randomly initialize filters and parameters / weights
2. forward propagation
1. goes through the step
2. finds the output probabilities, i.e. [0.2, 0.4, 0.1, 0.3]
3. Calculate the total error at the output layer
4. Backpropagation
1. calculate the gradients of the error
2. minimize error: gradient descent – update all filter values / weights and parameter
2. output [0.1, 0.1, 0.7, 0.1], closer to the target vector [0, 0, 1, 0]
3. only the values of the filter matrix and connection weights get updated
5. Repeat steps 2-4

new image: forward propagation step and output a probability for each class (calculated using the weights of previous training data)

http://scs.ryerson.ca/~aharley/vis/conv/flat.html

https://www.cs.ryerson.ca/~aharley/vis/conv/

1024 pixels (32 x 32 image)

• Convolution Layer 1: six unique 5 × 5 (stride 1) filters
• feature map of depth six
• Pooling Layer 1: 2 × 2 max pooling (with stride 2)
• sixteen 5 × 5 (stride 1) convolutional filters
• 2 × 2 max pooling (with stride 2)
• three fully-connected (FC) layers
• 120 neurons in the first FC layer
• 100 neurons in the second FC layer
• 10 neurons in the third FC layer corresponding to the 10 digits