Deep learning 101. Chapter 1.

What is neural networks?

What is Deep learning?

Deep learning were invented in 1960s, it caught wind around 1980s

Everybody thought that deep learning or neural networks were this new things that is going to impact the world, is going to solve all the world problems, and then it kind of slowly died off over the next decades.

Why did neural network not survive for a long time?

So what happened?

1. Was it a not a good invention?

The reason, it the fact that the technology back then was not up to the right standard in order to facilitate neural networks.

In order to Deep learning and Neural networks to work properly you need:

1. Lots of data
2. Processing power to process that data and facilitate neural networks.
3. How did the storage looked back in time?
4. 5 mb hard drive in 1956:
5. Hard drive in 1980
6. right now: 20$ for 256 gb drive
7. 

Still what is Deep learning?

Idea behind deep learning is to mimic how the human brain operates and recreate it

Human brains is one of the most creative, powerful tool on this planet for learning and adapting. If computer could copy that , we can just leverage that.

Example of brain neurons:

These neurons are connected to 1000s of other neighbour neurons.

These neurons are responsible for any activities we perform.

How do we recreate this in computer?

1. We create a artificial structure, called artificial neural net
2. where we have nodes i.e neurons
3. we have neurons for input values in input layers
4. these are the values you know about a certain situation.
5. for instance if you're modeling something, you want to predict something to start your predictions off.
6. Output layer: The value we want to predict.
7. Fraudulent transaction vs real transactions
8. Spam vs Ham or whatever
9. In Between these two layers, there is hidden layers

For example,

1. if you heard a roar of a lion or a whistle from someone
2. it's the input, which can be represented as input layer
3. Information are collected through senses and passed to the input layers of the brain
4. These information are going through billions & billions of neurons before a command is send from the brain.
5. Brain process this information of sound along with other noises
6. this is hidden layers
7. There are billions and billions of neurons to process the information.
8. There are lots of hidden layers in the brain.
9. Finally the information passes through the hidden layers to output layer
10. which may be make you turn around and see what it is or maybe make you runaway

Next, we will discuss:

1. The Neuron
2. The Activation Function
3. How do Neural Networks work?
4. How do Neural Networks learn?
5. Gradient Descent
6. Stochastic Gradient Descent
7. Backpropagation

The Neuron:

The very first step to creating artificial neural networks is to recreate a neuron.

So how do we do that?

Let's see how neuron looks first

This is how the neuron looks like. It has

1. Body
2. Dendrites
3. Axon

Neuron by themselves are like an ant,

1. They can't build ant hill by themselves.
2. They can't establish colony
3. They barely can carry food for survivals.

But the game changes when there are lots of ants, millions of ants they can do anything.

Similarly, How do neurons work together then?

That's what the dendrites and axon are for.

1. Dendrites are receiver of signal
2. Axon are transmitters of the signal

Axon of A Neuron ==passes spikes to ==> Dendrites of neighbouring neurons.

Now let's see how we can represent this on a computer?

In the right side image:

Just like how a neuron is composed:

1. Dendrites (input layer)
2. Nucleus , a process of information
3. Axon (output layer)

Similarly, in a artificial neuron on the left side of the image:

1. We have input layers X, these are independent variables
2. Linear function & Activation function to process information
3. Output layer y.

Whatever inputs you are putting in, that's for 1 row, and the output you get back is for the exact same row.

What is independent variables?

1. Age of a person
2. Race
3. Salary of a person
4. location

1. We need to standardize them:
2. Has mean of zero and variance of One
3. Or normalize them
4. you subtract the minimum value and then you / (max - min) i.e range to get the value between 0 & 1

What can be our output value?

1. Binary value i.e 0 or 1
2. Male or female
3. Categorical Value
4. Continuous value i,e price

Synapses:

Tiny gaps between neurons that allow them to communicate with each other.

In artificial neural networks, synapses between nodes, they are assigned weights.

Comparing to artificial neuron node:

Here, w1, w2, w3, wn are weights assigned to synapeses.

Weights are crucial for neural networks:

1. Weights are how neural networks learn
2. By adjusting weight, in every single case, neural network decides what signal is important vs not to a certain neuron,
3. What signal gets passed along vs what signals are dropped
4. Or to what strength or extent signals gets passed along.
5. They are the parameter that gets tuned/adjusted during the process of learning.

When you are training your artificial neural network, you're basically adjusting all of the weights in all of the synapses across the whole neural network.

Thus the birth of Gradient Descent and Backpropagation.

So signals go to neuron and what happens to the neuron?

What processing happens inside the neuron?

1. All of the values passed X1, X2, X3, Xn & W1, W2, W3, Wn gets added up.
2. We calculate the weighted sum .
3. Then it applies activation function
4. It is applied to weighted sum

Depending on the function , the neuron will either pass the signal or it won't pass the signal on to the next neuron.

What is Activation Function?

1. Threshold function (yes or no type)
2. if value < 0, return 0
3. 0 <= value, return 1
4. Sigmoid function
5. It's a function which is used in logistic regression.
6. It is smooth and useful in the output layer, especially when trying to predict probability.
7. Rectifier Function.
8. Hyperbolic Tangent Function.

Assuming your independent variable is binary i.e 0 or 1. Which activation function would you use?

1. There is two option
2. Threshold activation (either 0 or 1)
3. Sigmoid activation (between 0 and 1)

How many layers of hidden layer it has to be to considered deep learning neural network?

1. Shallow vs. Deep:
2. A neural network with 1-2 hidden layers is typically considered shallow.
3. Networks with 3 or more hidden layers are generally called deep.
4. Typical ranges:
5. Many practical deep learning models have anywhere from 3 to 150 layers.
6. Some very large models can have hundreds or even thousands of layers.
7. It depends on the task:
8. Simpler problems might only need a few layers.
9. More complex tasks often benefit from more layers.
10. Diminishing returns:
11. Adding more layers doesn't always improve performance.
12. At some point, extra layers can lead to overfitting or training difficulties.
13. The number of neurons per layer, type of layers, and other architectural choices are also important.

This is the note i took from udemy lecture & CSCE 598 Deep learning class @ ULL.

https://www.udemy.com/course/deeplearning/learn/lecture/6743222#overview

#deeplearning #neuralnetworks #machinelearning #chapter1