AIdventure.

DistillBERT - A distilled version of BERT

Knowledge distillation compresses BERT: smaller, faster, with almost all performance

RoBERTa - Optimized BERT Pretraining Approach

Unlocking the true potential of BERT through rigorous optimization and strategic training choices

BERT - Bidirectional Encoder Representations from Transformers

Pre-training bidirectional by jointly conditioning on both left and right context