Multi-Head AttentionMulti-Head Attention
Tutorial
- AttentionScaled dot-product attention and its training step
- Attention with Weights and a Residual ConnectionCausal self-attention with its projections, and its training step
- Multi-Head AttentionCausal, each head with a query, a key and a value of its own
- Grouped-Query AttentionQuery heads sharing key and value heads
Classic models
- Attention Is All You NeedThe original transformer · Jun 2017
- Mixtral-8x7BSparse mixture of experts · Dec 2023
- DeepSeek-V3Latent attention, mixture of experts · Dec 2024
Modern models
FP
Form