What is GPT ? Transformer | Machine Learning Model

What GPT-3 Means:

GPT means generative pre-trained transformer, 3 means 3rd generation.

Generative: Means bot that can generate new text.
Pre-trained: Pre-trained refers to how the model went through a process of learning from a massive amount of data
Transformer: A transformer is a specific kind of neural network, a machine learning model. It's the core invention underlying the current boom in AI.

GPT 3 is just one example of transformer. Something that transforms from one sequence into another. Language translation is just a great example.

Transformers are powerful deep learning model.

Machine Learning Basics:

A machine learning is very flexible structure with tun-able parameters like bunch of knob & dials, then somehow we use many examples of what the output should look like give an input, tweaking and tuning the values of these parameters to mimic this behaviour.

A simplest form of machine learning is linear regression, where input and output are a single number. For example sq.ft of a house and a price, we want a line of best fit throughout the data, that can predict future house prices.

Deep Learning:

Deep learning models get much more complicated than this simple linear regression. GPT3 has 175 Billion parameters.

When you create a giant model with huge number of parameters without it either grossly overfitting the training data or being too complex & unmanageable to train.

Deep learning describes a class of models that in the last decade have proven to scale remarkably well. What unifies them is they all use the same training algorithm, called back-propagation.

Transformers as a Model Class:

How transformer processes language or data?

Input Data Formatting:

Input has to be formatted in the array of numbers :

or formatted in higher dimensional of arrays i.e tensors.

Transformers' Operation:

These input data are progressively transformed into many distinct layers.

Until you get into the final layer i.e output. Final layer in our text processing model is the numbers representing the probability distribution for all possible next tokens.

In deep learning these parameters are referred as weights.

The only way parameters/weights interact with the data being processed is through weighted sums. We also sprinkle some simple non linear function throughout.These weighted sums are packaged together as various component in a matrix vector product.

Parameters in GPT-3:

Those 175 billion parameters/weights in GPT-3 are organised into 27,938 matrices.

These 27,983 matrices falls in 8 different categories.

These weights are the actual brain. They are the things learned during training. And they determine how a model behaves.

In the next thread we will look into details of these categories.

#AI #MachineLearning #DeepLearning #GPT3 #ArtificialIntelligence #Transformers #NeuralNetworks #DataScience #NaturalLanguageProcessing #NLP #DeepLearningModels #TechExplained #AIModels #MLBasics #GenerativeAI #TechEducation #AIInnovation #DataScienceCommunity #AIResearch #LearnAI #AIExplained #TechKnowledge #MLTechniques

this images & notes are taken from the following video, all credit goes to 3Blue1Brown