Simple Perceptron Activation Function:

We've seen how single perceptron behaves, now let's expand on this concept to understand the idea of a neural network.

Let's see how to connect many perceptrons together and then see how to represent this mathematically.

How does multiple perceptron network look like?

Here we can see various layers of single perceptron connected to each other through inputs and outputs.

image is copied from investopedia:

Here we have input layers, hidden layers and output layer.

Essentially hidden layers are layers that in between input and output layer, that don't get to see the outside.

1. Input layers:
2. Real values from the data
3. Hidden Layers:
4. Layers in between input and output
5. 3 or more layers is deep network
6. Output layer
7. final estimate of the output.

As you go forwards through more layers, the level of abstraction increases.

Let's discuss the activation function in a little more detail.

In previous post, our activation function was just a simple function that output 0 or 1. We were just taking the sum of the input, and had activation function output either 0 or 1. based off weather the number was positive or negative.

our function , weight times the input + bias

z = wx + b,

we will now refer z as function.

Simple Perceptron Activation Function:

• Activation Function: In a simple perceptron, the activation function checks if Z is positive or negative.
• If Z is positive, the output is 1.
• If Z is negative, the output is 0.

Example:

Imagine Z is calculated as follows: Z=WX+bZ = WX + bZ=WX+b

Scenario 1:

• Z=0.5Z = 0.5Z=0.5
• The output is 1 (because 0.5 is positive).

Scenario 2:

• Z=0.6Z = 0.6Z=0.6
• The output is still 1 (because 0.6 is positive).

Scenario 3:

• Z=−1Z = -1Z=−1
• The output is 0 (because -1 is negative).

Scenario 4:

• Z=−1000Z = -1000Z=−1000
• The output is still 0 (because -1000 is negative).

Key Point:

• Small Changes in Z: When Z changes from 0.5 to 0.6, the output remains the same (1) because both are positive.
• Dramatic Changes in Z: When Z changes from -1 to -1000, the output remains the same (0) because both are negative.

Why is this a Problem?

The simple perceptron activation function is not sensitive to the magnitude of ZZZ; it only cares about the sign (positive or negative). This means:

• Small changes in Z (like from 0.5 to 0.6) don't affect the output.
• Large changes in Z (like from -1 to -1000) also don't affect the output.

Solution: Using More Sophisticated Activation Functions

To make the model more sensitive to changes in ZZZ, we use more complex activation functions like the sigmoid function.

image is copied from researchgate.

This red line here, our output is not 0 and 1. Sigmoid function is

Formula: σ(x)=1 /[1 + e^{x}]

|Output Range: The sigmoid function outputs a value between 0 and 1.

So changing the activation function can be really beneficial depending on the particular task.

Few more activations functions:

1. Hyperbolic Tangent; tanh(Z)
2. Rectified Linear Unit ReLU
3. max(0,z)
4. Based off your z value and 0 just returns maximum value
5. if your value of z if negative, max between (0, -negative) is 0
6. between (0, positive), max is positive.
7. ReLu tends to have best performance in many situations.
8. Often we'll use something like softmax at the very end of a layer. In order to get some sort of classification output.

image from medium:

In future post, we will understand how to implement and build our own neural networks models with Keras.