亚洲模型实验室 · arXiv 论文
Attention-head Scaling Facets
复现 Kimi K2 Figure 6：在不同 compute 档位比较 heads=layers 与 doubled heads 的验证 loss。
attentionHeadFacetsInformation grammar
它如何组织信息
四 compute 档折线小多图 + 方/圆 marker 协议 + loss 改善直标
- Visual system
- VS-62 Kimi head-count facets
- Component ID
kimi-attention-head-scaling
Use when
适用判断
适合架构消融跨多个 compute scale 的比较。
Figure-level evidence
Figure 6 · PDF p.7
不同训练 compute 下比较 heads=layers 与 doubled heads 的 validation loss。
- Verified
- 2026-07-12
- Paper source
- Kimi K2 Technical Report
Structured data
可替换的数据模型
示意数据，仅复现信息结构，不代表最新榜单结果。
{
"facets": [
{
"label": "1.2e20 FLOPs",
"min": 1.62,
"max": 1.76,
"delta": 0.5,
"base": [
1.74,
1.7,
1.67,
1.66,
1.67
],
"double": [
1.73,
1.69,
1.66,
1.64,
1.65
]
},
{
"label": "2.2e20 FLOPs",
"min": 1.53,
"max": 1.65,
"delta": 0.7,
"base": [
1.63,
1.59,
1.57,
1.56,
1.58
],
"double": [
1.62,
1.58,
1.55,
1.54,
1.56
]
},
{
"label": "4.5e20 FLOPs",
"min": 1.43,
"max": 1.54,
"delta": 0.9,
"base": [
1.52,
1.49,
1.47,
1.46,
1.48
],
"double": [
1.51,
1.48,
1.45,
1.44,
1.46
]
},
{
"label": "9.0e20 FLOPs",
"min": 1.34,
"max": 1.43,
"delta": 1.2,
"base": [
1.41,
1.39,
1.38,
1.37,
1.39
],
"double": [
1.4,
1.38,
1.36,
1.35,
1.37
]
}
]
}Related components