---
issue: 2
status: working
tags: model
---
# qk-norm-relu2

QK-norm, ReLU² and untied embeddings.

## What happened

- 2026-09-11: 15.2 minutes to 3.28 (−29.8 on the baseline)

## Next

Add value embeddings and skip connections.