## tldr: - went from 16~ t/s to 27~ t/s generatio…
went from 16~ t/s to 27~ t/s generation.
got my usable context up from 220k to the full 262k without sacrificing anything.
prefill also increased from 376 to 573
llama-server -m Qwen3.8-27B-UD-Q6_K_XL.gguf -c 262144 -ngl 999 -fa on \ -ctk q8_0 -ctv q8_0 --spec-type draft-mtp,ngram-map-k4v --spec-draft-n-max 3 \ --spec-draft-device CUDA0 --split-mode layer -dev CUDA0,Vulkan2 -ts 40,60 \ --jinja -fitt 256
P1 Gen 6 (an RTX 4090 laptop, effectively a 4080 chip running with a 95~ w cap and 16gb vram at 536ish gb/s vram speeds, something like that.) + XTX 7900, over TB4, on an AG02 egpu dock.
I had the laptop lying around, so this was the cheapest way to get 40gb vram and run qwen with q6 k xl and full context. Overall very happy and impressed with the setup.
I originally thought I'd likely see a big benefit if I just reduced context and gave the XTX a bigger part of the split. Turns out they balance each other super well, actually.
Prefill: the 4090 wins by 34% on a 22k cold prompt.
Prefill, but 7B @ depth 0: the 4090 by 82%? damn.
Generation with no speculative decode: XTX wins by 27%.
Generation, 7B @ depth 0: XTX by 23%.
Generation adding MTP (speculative decode) back in: XTX by 72%.
Generation at depth, 7B @ 98k depth: XTX by 2.5x.
So, the XTX is the more powerful decode machine, even over TB4. But it mostly isn't the crazy difference you'd make, and the 4090 wins a lot on prefill to earn back quite a lot.
Also, that huge MTP gap on generation? pretty sure that's a bug, ended up filing it. Found that MTP is still a net 25% gain for me in spite of this, but my prefill drops from the 900s to the 500s with MTP. It seems to be a multi-gpu thing only, tested and found that the penalty wasn't there on either card by itself, including the 4090 on vulkan or on nvidia. Issue filed, hopefully I didn't embarrass myself: https://github.com/ggml-org/llama.cpp/issues/27428
Anyways, like I said: turns out I have like 3% or less performance gain by reducing context and trying to use the xtx more than the 40/60 split that they organically have. The context is worth more than 3%.
Not nearly as bad as you'd think at first blush. If it's what you have, use it. When doing layer split, it's theoretically possible for me to get almost 1000 pp t/s on this setup (when not using MTP because of the bug I found) on qwen 27b q6 k xl. The bandwidth is 40gbs and the latency is .025ms (that's 1/4th of a millisecond) when I was testing the lag between two computers of tb4, and that was with the latency running 'ping' adds I guess. You aren't going to do tensor parallelism I imagine (I haven't been able to test that yet, need an nvidia card), but this setup is a lot easier than selling my laptop and buying a server, etc. (which I may eventually do).
I actually at first had two laptops like this, one with a 3080 with 16gb vram and the 4090 I have now. Sold the 3080 to buy the AG02 and the XTX. But while I had them both, I actually experimented with running (at the time 3.6) 27B q5 with 262k context, by making the 3080 machine an RPC node and connecting the two laptops over the TB4 connection (TB4 networking is pretty great, but hard to find two machines with TB4 compatible motherboards; that said, gb and 10gb ethernet works, and I had surprisingly usable results even over LAN wifi). I got t/s that were very usable, which blew me away.
Didn't have the same detailed benching setup, so I don't have a lot of numbers to give from that, but I just wanted to throw that out there.
Back to my current setup.
(that, fwiw, I took a very active role in; some of this Claude would not have found on its own without my actual understanding what was going on and suggesting my own thoughts)
"knob" as claude like to call it gain notes
--spec-type draft-mtp,ngram-map-k4v (all defaults) +50–70% generation (17→26-28; copy-heavy requests spike to 85 t/s at 100% acceptance) but... cuts my prefill in half on my setup, big surprise. Worth it because my workload is 77% generation time
NVIDIA card on CUDA backend, AMD on Vulkan ( -dev CUDA0,Vulkan2 ) +28% prefill no-spec, +41% prefill with the spec stack (376→532) Originally was running both on vulkan, seemed close, deeper testing showed benefits for nvidia for the 4090. Had to have AI help me through getting this running, but it was worth it. (side note: CUDA's q8_0-KV decode collapses −50% at depth on dense models, fine on hybrid-SSM)
--spec-draft-device CUDA0 +8% prefill (532→573) my very last test trying to guess at source of the MTP prefill penalty yielded this little finding. Nice.
rebuild llama.cpp to a 2 weeks newer build... +13% generation free (25.0→28.3, identical config) at first I did all kinds of little tuning stuff... then on rebuild, those all washed out to worthless and the benefits were now built in for free. lol.
UD-Q6_K_XL over Q8_0 +9% gen, +3GB context, equal prefill ONLY on CUDA — on Vulkan, Q8 prefills 12% faster (K-quant dequant is expensive in RADV, free in CUDA). Huh. But anyways, I get 262k context on q6, would be sacrificing that on q8, but was worth testing anyways.
q8_0 KV cache 262k context vs ~190k Note that this runs slower. I didn't know about this before. Apparently it costs −23% prefill/−9% gen at depth in llama-bench, but only −7.5% prefill in the real server; took the context
-ts 40,60 (VRAM-proportional) baseline ±5 points moves ~2%; not worth losing context, which I'd have to do if I play with these numbers. 262 context means a just about 100% full 4090 and nearly 100% full xtx.
attempt result
-ub 1024 / 2048 (the classic "raise ubatch for prefill") −13% / −31% prefill. This can help apparently if you have a single card, but I found that it inverts on my multi-GPU layer split. smaller chunks pipeline better across cards. So I tried tuning smaller, but once I updated to latest, that benefit disapeared.
-b 4096 +9% in llama-bench, exactly 0% in the real server. Bench your production path before adopting anything... Thought I found something there.
ngram tuning (min-hits, draft length M, p-min) won +17% on the old build; after rebuilding, ALL deltas collapsed into noise. upstream refactors absorbed what the knobs were buying. I felt fancy, but we all benefit when this stuff is just better out of the box, so yay.
ngram-mod (adaptive ngram) −17% generation
a trained DSpark draft head for my exact model 1–5% acceptance (GGUF mislabeled dflash), 33% forced as dspark — still 3× slower than MTP+ngram. Trained drafters aren't plug-and-play yet I guess. wamp wamp.
-sm row not possible for me on this setup. will test for the hell of it when/if I get a 3090 to play with.
mixed KV types ( -ctk q8_0 -ctv f16 or mirror) 4–13× prefill collapse — silent kernel fallback; KV types must match. total fail. was an attempt to maybe win back some of the loss from q8 k/v.
-fa 0 can't even launch: quantized KV requires flash attention. AI had told me this was nvidia only for the longest time, this is actually also vulkan supported apparently. neat. til.
raising the 80W power cap on windows, btw, I can see this card hold a sustained 135w~ at best during gaming. In linux it seems capped at 95w. But tbh I watched many telemetric graphs for a long time. This card isn't watt bound in this setup--compute power isn't the bottleneck.
all-Vulkan pairing (revisited after the MTP findings) 376 vs 532 prefill with the spec stack; gen equal. CUDA still has an edge on this card.
--fitt 256 I would get 'not enough vram' when trying to do 262k context until I used this, bu…
本条由桃子采集流水线（启发式模式）自动整理，原文见文末信源。
