For a targeted weight \(W\in\mathbb{R}^{m\times n}\), LoRA trains two factors and
leaves \(W\) frozen:
\[
\Delta W=AB,\qquad
A\in\mathbb{R}^{m\times r},\quad
B\in\mathbb{R}^{r\times n},\qquad
N_W(r)=r(m+n).
\]
We count the two factors only: no trainable bias, embedding, normalization, or
output-head parameters. A fused projection is one weight matrix, so its inputs share
one LoRA factor. One cube is \(10^6\) parameters, rounded to the nearest cube.
Method: LoRA paper.
Qwen3 4B
There are \(L=36\) layers, model width \(d=2560\), MLP width \(f=9728\),
\(32\) query heads, \(8\) key/value heads, and head width \(128\). The seven
separate targets are Q, K, V, O, gate, up, and down.
\[
\begin{aligned}
N_{\mathrm{attn/layer}}/r
&= (2560+4096)+2(2560+1024)+(4096+2560) \\
&= 20{,}480,\\
N_{\mathrm{MLP/layer}}/r
&=3(2560+9728)=36{,}864,\\
N_{\mathrm{Qwen3\ 4B}}(r)
&=36r(20{,}480+36{,}864)\\
&=\mathbf{2{,}064{,}384r}.
\end{aligned}
\]
\[
N(1)=2{,}064{,}384,\quad
N(4)=8{,}257{,}536,\quad
N(32)=66{,}060{,}288.
\]
Dimensions: official Qwen3 4B config.
Qwen3.6 27B
The main model has \(64\) layers with width \(d=5120\) and MLP width
\(f=17{,}408\). Sixteen layers use full attention. This visual follows the
checkpoint's fused Megatron targets: linear_qkv,
linear_proj, linear_fc1, and linear_fc2.
In a full-attention layer, fused QKV has output width
\(12{,}288+1{,}024+1{,}024=14{,}336\). Fused FC1 has output width
\(2f=34{,}816\).
\[
\begin{aligned}
N_{\mathrm{full\ attn/layer}}/r
&=(5120+14{,}336)+(6144+5120)\\
&=30{,}720,\\
N_{\mathrm{MLP/layer}}/r
&=(5120+34{,}816)+(17{,}408+5120)\\
&=62{,}464,\\
N_{\mathrm{Qwen3.6\ 27B}}(r)
&=16r(30{,}720)+64r(62{,}464)\\
&=491{,}520r+3{,}997{,}696r\\
&=\mathbf{4{,}489{,}216r}.
\end{aligned}
\]
\[
N(1)=4{,}489{,}216,\quad
N(4)=17{,}956{,}864,\quad
N(32)=143{,}654{,}912.
\]
The \(48\) GDN linear-attention layers and the MTP prediction layer are not
targeted here. Counting Q, K, and V as three independent LoRA modules would give a
different answer because it would train three input-side factors instead of the one
shared by fused QKV.
Dimensions and layer pattern: official Qwen3.6 27B config and official Transformers implementation.
GLM 5.2 (744B total, 40B active)
This calculation targets the five MLA projections plus the dense,
shared-expert, and routed-expert MLP projections. The 256 routed experts in each
MoE layer use one tied LoRA adapter.
There are \(78\) main layers: \(3\) dense and \(75\) MoE. The model width is
\(d=6144\); dense MLP width is \(12{,}288\); expert width is \(2048\); and each
MoE layer has \(256\) routed experts. MLA uses query rank \(2048\), KV rank \(512\),
\(64\) heads, Q/K head width \(256=192+64\), and value head width \(256\).
Gate and up are treated as one fused projection.
\[
\begin{aligned}
N_{\mathrm{MLA}}/r
&=78[(6144+2048)+(2048+64\cdot256)\\
&\quad +(6144+576)+(512+64\cdot448)\\
&\quad +(64\cdot256+6144)]\\
&=6{,}634{,}368,\\
N_{\mathrm{dense}}/r
&=3[(6144+2\cdot12{,}288)+(12{,}288+6144)]\\
&=147{,}456,\\
N_{\mathrm{shared}}/r
&=75[(6144+2\cdot2048)+(2048+6144)]\\
&=1{,}382{,}400,\\
N_{\mathrm{routed}}/r
&=75[(6144+2\cdot2048)+(2048+6144)]\\
&=1{,}382{,}400,\\
N_{\mathrm{GLM\ 5.2}}(r)
&=r(N_{\mathrm{MLA}}/r+N_{\mathrm{dense}}/r
+N_{\mathrm{shared}}/r+N_{\mathrm{routed}}/r)\\
&=\mathbf{9{,}546{,}624r}.
\end{aligned}
\]
\[
N(1)=9{,}546{,}624,\quad
N(4)=38{,}186{,}496,\quad
N(32)=305{,}491{,}968.
\]
The architectural shared expert has its own adapter; the routed experts share a
second adapter. The DSA indexer, router, embeddings, output head, and MTP prediction
layer are not targeted.
Architecture: official GLM 5.2 config and Transformers implementation.