Constructing GPT from Scratch

-Inspired heavily from Andrej Karpathy’s video and Illustrated Transformer blog.

Below is a diagram showing the Transformer -decoder architecture I implemented in PyTorch (Full Code in Github).

To envision if my implementation is correct, I initially trained the model on dataset. I built a small network, and trained the model for just 1 epoch on a small subset of the dataset. Below are some sample results:

Here I used to be just attempting to see if the model is capable of generate some semblance of natural language — and it does.

Dataset: piano adl midi (link) — comprising 2000+ midi files.

Data Preprocessing: tokenized using miditok remiPlus tokenizer — a fast approach to tokenize and de-tokenize the midi files. Sequence Length: 256

There’s some loss in tokenizing and detokenizing the files. But it surely’s alright

• Attention Heads: 8
• Transformer Layers : 1
• Embedding Dimension: 769
• Token Vocabulary: 633
• Epochs: 5

Ran for about 12 hours/epoch on my small GPU — 1080Ti