AIdventure.

Speeding up Attention Layers

Multi-head, Multi-Query & Grouped-Query Attention layers clearly explained. How cache works in the Attention layers

DINO - Emerging Properties in Self-Supervised Vision Transformers

Unsupervised visual feature learning using knowledge distillation and transformers

Vision Transformer - An Image is Worth 16x16 Words

Google shows how treating image patches as tokens can revolutionize computer vision

DistillBERT - A distilled version of BERT

Knowledge distillation compresses BERT: smaller, faster, with almost all performance

RoBERTa - Optimized BERT Pretraining Approach

Unlocking the true potential of BERT through rigorous optimization and strategic training choices

BERT - Bidirectional Encoder Representations from Transformers

Pre-training bidirectional by jointly conditioning on both left and right context

GPT1 - Improving Language Understanding by Pre-Training

Semi-supervised learning through generative pre-training on unlabeled text and task-specific fine-tuning

Transformer - Attention Is All You Need

Demystifying the Transformer architecture, explaining the Encoder, Decoder, and Attention mechanisms block by block with PyTorch implementation