https://orkun.soylu.me/Notes from my homelab: open-weight LLMs tested on my own GPUs, plus writing on AI, technology and running IT. What actually works, with the numbers.enhttps://orkun.soylu.me/posts/2026-10-10-strata-dual-gpu/https://orkun.soylu.me/posts/2026-10-10-strata-dual-gpu/Sat, 10 Oct 2026 00:00:00 GMTI split Qwen3.8-Flash-Next by layer across 72 GB of VRAM and system RAM dropped out of the picture entirely. Decode barely moved (72 → 74 tok/s); long-prompt prefill gained 24% (961 → 1,190 tok/s) — less than I expected, because each card needs its own prompt buffers.stratamulti-gpumoeamdhttps://orkun.soylu.me/posts/2026-10-09-strata/https://orkun.soylu.me/posts/2026-10-09-strata/Fri, 09 Oct 2026 00:00:00 GMTA purpose-built engine runs a 125B mixture-of-experts on my single W7800 by keeping the hot experts in VRAM and the rest in RAM. Tuned, the 3-bit build matches or beats my 27B baseline on quality and is 40–60% faster per token — but tool-heavy answers arrive 38% later.stratamoerocmhttps://orkun.soylu.me/posts/2026-10-09-coder390/https://orkun.soylu.me/posts/2026-10-09-coder390/Fri, 09 Oct 2026 00:00:00 GMTA community fine-tune that promises to cure over-long thinking tied the official Qwen3.8-27B on quality in all three of my tests, and was 4–5% slower per token. The long tail it's built to fix never showed up in my workloads.codingfine-tunesqwenhttps://orkun.soylu.me/posts/2026-09-23-thinking-token-claims/https://orkun.soylu.me/posts/2026-09-23-thinking-token-claims/Wed, 23 Sep 2026 00:00:00 GMTTwo fine-tunes that promise far fewer thinking tokens didn't deliver: TURBO thinks the most of all three (58% of its output), and Swift's claimed 58% reduction came out at 15%. One run per prompt would have told me the opposite story.reasoningfine-tuneshttps://orkun.soylu.me/posts/2026-09-23-llamacpp-rocm-vs-vulkan/https://orkun.soylu.me/posts/2026-09-23-llamacpp-rocm-vs-vulkan/Wed, 23 Sep 2026 00:00:00 GMTCurrent llama.cpp is only 3% faster than the runtime bundled with Ollama. Vulkan (RADV) decodes 4.6% faster than ROCm but prefills 16% slower. And the card is capped at 230 W, with no way to raise it.llama.cpprocmvulkanamdhttps://orkun.soylu.me/posts/2026-09-23-bare-metal-vs-vm/https://orkun.soylu.me/posts/2026-09-23-bare-metal-vs-vm/Wed, 23 Sep 2026 00:00:00 GMTMoving my W7800 from a bare-metal box into a Proxmox VM on another host changed generation by 2% (noise) and made prompt processing 8% faster. Passthrough itself costs nothing measurable — but I had to get past four setup traps first.virtualizationamdperformancehttps://orkun.soylu.me/posts/2026-09-22-own-quantization-q6k/https://orkun.soylu.me/posts/2026-09-22-own-quantization-q6k/Tue, 22 Sep 2026 00:00:00 GMTI made a Q6_K with a 145K-token calibration set (40% Turkish), and it stays 10× closer to Q8_0 than ISTA's 3-bit does (KLD 0.0056 vs 0.056). My targeted imatrix beat a bigger generic one by 4–7%. But with MTP on, Q6_K is slower than Q8_0.quantizationllama.cpphttps://orkun.soylu.me/posts/2026-09-22-mtp-speculative-decoding/https://orkun.soylu.me/posts/2026-09-22-mtp-speculative-decoding/Tue, 22 Sep 2026 00:00:00 GMTTurning on the model's built-in multi-token-prediction head more than doubled generation speed without changing quality. Four draft tokens is the sweet spot, and the gain depends a lot on content — code accepts drafts far more often than Turkish prose.speculative-decodingollamaperformancehttps://orkun.soylu.me/posts/2026-09-22-cve-empero-q8-ista/https://orkun.soylu.me/posts/2026-09-22-cve-empero-q8-ista/Tue, 22 Sep 2026 00:00:00 GMTA distilled 35B-A3B MoE answers 2.4× faster but drops to 0.84, with unsafe upgrade advice and a hallucinated CVE. On my W7800, Q8_0+MTP ties ISTA IQ3_S at 0.96 while answering 1.7× faster.securitymoefine-tuneshttps://orkun.soylu.me/posts/2026-09-21-vllm-rocm-w7800/https://orkun.soylu.me/posts/2026-09-21-vllm-rocm-w7800/Mon, 21 Sep 2026 00:00:00 GMTOf the seven quantization formats I tried, two ran on RDNA3 under vLLM 0.27. int8 W8A16 + MTP reached 42 tok/s; int4 W4A16 reached 57 tok/s but locked up the GPU at ~30K tokens. I removed vLLM — and Ollama with MTP matched it the next day.vllmrocmamdhttps://orkun.soylu.me/posts/2026-09-18-ornith-cve/https://orkun.soylu.me/posts/2026-09-18-ornith-cve/Fri, 18 Sep 2026 00:00:00 GMTA 35B mixture-of-experts with only 1.9B active parameters decodes at 84 tok/s and answers in 39 s — but its score drops to 0.83, with consistently unsafe upgrade advice. My CVE prompt actually made it slightly worse.securitymoefine-tuneshttps://orkun.soylu.me/posts/2026-09-17-ista-general-capability/https://orkun.soylu.me/posts/2026-09-17-ista-general-capability/Thu, 17 Sep 2026 00:00:00 GMTOn my 16 GB card, ISTA's 3-bit Qwen3.8-27B nearly maxes out everyday benchmarks: 40/41 HumanEval, 49/50 GSM8K, 9/9 long-context recall, 15/15 tool calls. The one weakness I could see is Turkish reading comprehension (36/40 vs 40/40 in English).quantizationqwennvidiahttps://orkun.soylu.me/posts/2026-09-17-hauhau-q8-cve/https://orkun.soylu.me/posts/2026-09-17-hauhau-q8-cve/Thu, 17 Sep 2026 00:00:00 GMTRunning the same fine-tune at Q8 on my 48 GB W7800 lifts its CVE score from 0.92 to 0.97 — most of its weakness was the 3-bit compression. But it still doesn't beat ISTA's 3-bit, and it costs 2.4× the time and 2.5× the energy per token.securityquantizationfine-tuneshttps://orkun.soylu.me/posts/2026-09-17-cve-prompt-v2/https://orkun.soylu.me/posts/2026-09-17-cve-prompt-v2/Thu, 17 Sep 2026 00:00:00 GMTA stricter grounding rule plus two tool fixes took ISTA to 0.98, with zero CVEs from memory and zero language drift. HauhauCS slipped to 0.92: it gives up researching upgrades too early.securitypromptingtool-usehttps://orkun.soylu.me/posts/2026-09-17-cve-prompt-v1/https://orkun.soylu.me/posts/2026-09-17-cve-prompt-v1/Wed, 16 Sep 2026 00:00:00 GMTOne shared system prompt took HauhauCS's English answers from 15 down to 1 and improved fix-version accuracy for both models. The facts from memory stayed — now labelled 'unverified', but still wrong.securitypromptingtool-usehttps://orkun.soylu.me/posts/2026-09-16-ista-vs-unsloth-iq3/https://orkun.soylu.me/posts/2026-09-16-ista-vs-unsloth-iq3/Wed, 16 Sep 2026 00:00:00 GMTOn my RTX 4070 Ti SUPER under vLLM, ISTA's IQ3_S gets 1.2% lower perplexity from a smaller file and prefills 8% faster; Unsloth's decodes 5% faster and leaves a bit more room for KV.quantizationvllmnvidiahttps://orkun.soylu.me/posts/2026-09-16-ista-vs-hauhau-features/https://orkun.soylu.me/posts/2026-09-16-ista-vs-hauhau-features/Wed, 16 Sep 2026 00:00:00 GMTBoth 3-bit Qwen3.8-27B builds pass vision, tool calling (with and without thinking), using tool results and thinking control on my RTX 4070 Ti SUPER, at ~43–44 tok/s with 64K fully on the GPU.tool-usevisionfine-tuneshttps://orkun.soylu.me/posts/2026-09-16-cve-baseline/https://orkun.soylu.me/posts/2026-09-16-cve-baseline/Wed, 16 Sep 2026 00:00:00 GMTThe first run of my tool-using CVE evaluation: both 3-bit 27B models scored 0.96. What separated them wasn't accuracy but language discipline — HauhauCS answered a third of my Turkish questions in English.securitytool-usenvidiahttps://orkun.soylu.me/posts/2026-09-16-vllm-vs-ollama-power/https://orkun.soylu.me/posts/2026-09-16-vllm-vs-ollama-power/Tue, 15 Sep 2026 00:00:00 GMTFor a single user, Ollama beat vLLM on every axis I measured: 28% faster decode, 1.9× faster prefill, 2.5× the context in VRAM, a 3-second start and less energy per token.vllmollamapowernvidiahttps://orkun.soylu.me/posts/2026-08-19-thinking-on-off/https://orkun.soylu.me/posts/2026-08-19-thinking-on-off/Wed, 19 Aug 2026 00:00:00 GMTWith think=false the model just moves its reasoning into the visible answer, which gets longer. It only wins on plain fact lookups; on questions that need reasoning, thinking mode was 22–40% faster.reasoningqwenhttps://orkun.soylu.me/posts/2026-08-19-pi-agent-27b/https://orkun.soylu.me/posts/2026-08-19-pi-agent-27b/Wed, 19 Aug 2026 00:00:00 GMTAll 13 of my agent runs fixed the bug or built the feature, with zero failed exact-match edits. Q8_0 was up to 78% slower and no more accurate. Almost all the waiting went into re-reading the prompt.agentscodingqwenhttps://orkun.soylu.me/posts/2026-08-19-168k-context/https://orkun.soylu.me/posts/2026-08-19-168k-context/Wed, 19 Aug 2026 00:00:00 GMTI fed a 168,084-token prompt to Qwen3.8-27B Q8_0 on my W7800: prefill took 11.7 minutes, and generation fell from 20.2 to 9.6 tok/s. It still recalled a marker from the very start perfectly.long-contextvramollamahttps://orkun.soylu.me/posts/2026-08-18-w7800-vs-7900xtx/https://orkun.soylu.me/posts/2026-08-18-w7800-vs-7900xtx/Tue, 18 Aug 2026 00:00:00 GMTI ran the identical Q4_K_M blob and Ollama version on both cards: the consumer 7900 XTX is 29% faster at generation and 44% faster at prefill. The gap follows compute, not bandwidth — what the W7800 really offers is its 48 GB.gpuamdrocmhttps://orkun.soylu.me/posts/2026-08-18-qwen38-27b-q8-vs-q4/https://orkun.soylu.me/posts/2026-08-18-qwen38-27b-q8-vs-q4/Tue, 18 Aug 2026 00:00:00 GMTDropping from Q8_0 to Q4_K_M on my W7800 bought 29% faster generation and 12 GB of VRAM — far less than the bandwidth maths promised — and made long-prompt prefill 7% slower.quantizationqwenollamahttps://orkun.soylu.me/posts/2026-08-05-ai-memory-crisis-sap-hana/https://orkun.soylu.me/posts/2026-08-05-ai-memory-crisis-sap-hana/Wed, 05 Aug 2026 00:00:00 GMTMemory prices keep breaking records. A thought experiment on what that means for refreshing in-memory SAP HANA servers and for HANA capacity in the cloud.saphanadramcloudhttps://orkun.soylu.me/posts/2026-07-06-vram-fit-48gb/https://orkun.soylu.me/posts/2026-07-06-vram-fit-48gb/Mon, 06 Jul 2026 00:00:00 GMTWith one model loaded at a time, both Qwen3.6 models get their full 262K context entirely in VRAM on my W7800. Gemma 4 31B doesn't — above 64K it spills to the CPU.vramollamagemmaqwenhttps://orkun.soylu.me/posts/2026-07-06-moe-vs-dense-long-context/https://orkun.soylu.me/posts/2026-07-06-moe-vs-dense-long-context/Mon, 06 Jul 2026 00:00:00 GMTOn my 48 GB W7800 the mixture-of-experts model generates 3.4× faster than the dense 27B. Filling a quarter of the context barely dents generation speed — almost all of the wait is prefill.moelong-contextvramollama