docs: korrigierte Architektur — MoE-Modelle, sequentiell, llama-swap, kein Llama/Freeze
This commit is contained in:
Hermes Agent
+342
-564
@@ -3,626 +3,404 @@
|
|||||||
<head>
|
<head>
|
||||||
<meta charset="UTF-8">
|
<meta charset="UTF-8">
|
||||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||||
<title>BSN Chatbot — Architektur mit DGX Spark & Strix Halo</title>
|
<title>BSN Chatbot — Finale Architektur mit DGX Spark & Strix Halo</title>
|
||||||
<style>
|
<style>
|
||||||
:root {
|
:root {--bsn:#20228a;--bsn-light:#2d30b5;--bsn-bg:#f0f1ff;--green:#10b981;--orange:#f59e0b;--red:#ef4444;--g50:#f9fafb;--g100:#f3f4f6;--g200:#e5e7eb;--g500:#6b7280;--g700:#374151;--g900:#111827;--white:#fff;--r:12px;--sh:0 1px 3px rgba(0,0,0,.08)}
|
||||||
--bsn: #20228a; --bsn-light: #2d30b5; --bsn-bg: #f0f1ff;
|
*{margin:0;padding:0;box-sizing:border-box}
|
||||||
--green: #10b981; --orange: #f59e0b; --red: #ef4444;
|
body{font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,sans-serif;background:var(--g50);color:var(--g900);line-height:1.6}
|
||||||
--g50: #f9fafb; --g100: #f3f4f6; --g200: #e5e7eb; --g300: #d1d5db; --g500: #6b7280; --g700: #374151; --g900: #111827;
|
.container{max-width:1300px;margin:0 auto;padding:24px}
|
||||||
--white: #fff; --r: 12px; --sh: 0 1px 3px rgba(0,0,0,.08);
|
.hero{background:linear-gradient(135deg,#0d1117 0%,#16213e 100%);color:#fff;padding:48px 32px;border-radius:var(--r);margin-bottom:32px;position:relative;overflow:hidden}
|
||||||
--nvidia: #76b900; --amd: #ed1c24;
|
.hero h1{font-size:1.9rem;font-weight:800;margin-bottom:6px}
|
||||||
}
|
.hero .subtitle{font-size:1.05rem;opacity:.85}
|
||||||
* { margin:0; padding:0; box-sizing:border-box; }
|
.grid{display:grid;gap:24px;margin-bottom:32px}
|
||||||
body { font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif; background: var(--g50); color: var(--g900); line-height: 1.6; }
|
.g2{grid-template-columns:repeat(auto-fit,minmax(450px,1fr))}
|
||||||
.container { max-width: 1300px; margin: 0 auto; padding: 24px; }
|
.g3{grid-template-columns:repeat(auto-fit,minmax(340px,1fr))}
|
||||||
.hero { background: linear-gradient(135deg, #1a1a2e 0%, var(--bsn) 100%); color: #fff; padding: 56px 32px; border-radius: var(--r); margin-bottom: 32px; position: relative; overflow: hidden; }
|
.card{background:var(--white);border-radius:var(--r);box-shadow:var(--sh);padding:24px;border:1px solid var(--g200)}
|
||||||
.hero::after { content: ''; position: absolute; top: -30%; right: -10%; width: 500px; height: 500px; background: radial-gradient(circle, rgba(118,185,0,.15) 0%, transparent 70%); border-radius: 50%; }
|
.card h2{font-size:1.15rem;font-weight:700;margin-bottom:16px;display:flex;align-items:center;gap:8px}
|
||||||
.hero::before { content: ''; position: absolute; bottom: -20%; left: -10%; width: 400px; height: 400px; background: radial-gradient(circle, rgba(237,28,36,.1) 0%, transparent 70%); border-radius: 50%; }
|
.card h3{font-size:.95rem;font-weight:600;margin:20px 0 10px;color:var(--g700)}
|
||||||
.hero h1 { font-size: 2.1rem; font-weight: 800; margin-bottom: 8px; position: relative; z-index: 1; }
|
table{width:100%;border-collapse:collapse;font-size:.86rem}
|
||||||
.hero .subtitle { font-size: 1.15rem; opacity: .9; position: relative; z-index: 1; }
|
th{text-align:left;padding:10px 12px;background:var(--bsn-bg);color:var(--bsn);font-weight:700;font-size:.76rem;text-transform:uppercase;letter-spacing:.5px;border-bottom:2px solid var(--bsn)}
|
||||||
.hero .chips { display: flex; gap: 12px; margin-top: 16px; position: relative; z-index: 1; flex-wrap: wrap; }
|
td{padding:10px 12px;border-bottom:1px solid var(--g200);vertical-align:top}
|
||||||
.hero .chip { padding: 6px 14px; border-radius: 20px; font-size: .82rem; font-weight: 700; }
|
.badge{display:inline-block;padding:3px 10px;border-radius:12px;font-size:.76rem;font-weight:600}
|
||||||
.chip-nvidia { background: rgba(118,185,0,.25); color: var(--nvidia); border: 1px solid rgba(118,185,0,.4); }
|
.bg{background:#d1fae5;color:#065f46}.br{background:#fee2e2;color:#991b1b}.bo{background:#fef3c7;color:#92400e}.bb{background:var(--bsn-bg);color:var(--bsn)}
|
||||||
.chip-amd { background: rgba(237,28,36,.2); color: var(--amd); border: 1px solid rgba(237,28,36,.35); }
|
.callout{background:var(--bsn-bg);border-left:4px solid var(--bsn);padding:14px 18px;border-radius:0 var(--r) var(--r) 0;margin:20px 0;font-size:.9rem}
|
||||||
.chip-local { background: rgba(255,255,255,.15); color: #fff; border: 1px solid rgba(255,255,255,.25); }
|
.callout.green{background:#f0fdf4;border-left-color:var(--green)}
|
||||||
.grid { display: grid; gap: 24px; margin-bottom: 32px; }
|
.callout.red{background:#fff5f5;border-left-color:var(--red)}
|
||||||
.g2 { grid-template-columns: repeat(auto-fit, minmax(480px, 1fr)); }
|
.callout strong{color:var(--bsn)}
|
||||||
.g3 { grid-template-columns: repeat(auto-fit, minmax(340px, 1fr)); }
|
.callout.green strong{color:#065f46}
|
||||||
.g4 { grid-template-columns: repeat(auto-fit, minmax(260px, 1fr)); }
|
.callout.red strong{color:#991b1b}
|
||||||
.card { background: var(--white); border-radius: var(--r); box-shadow: var(--sh); padding: 24px; border: 1px solid var(--g200); }
|
.arch-box{background:#0d1117;color:#c9d1d9;padding:20px;border-radius:var(--r);font-family:'SF Mono','Fira Code',monospace;font-size:.73rem;line-height:1.9;margin:16px 0;overflow-x:auto}
|
||||||
.card h2 { font-size: 1.15rem; font-weight: 700; margin-bottom: 16px; display: flex; align-items: center; gap: 8px; }
|
.arch-box .g{color:#7ee787}.arch-box .b{color:#79c0ff}.arch-box .p{color:#f778ba}.arch-box .y{color:#e3b341}.arch-box .o{color:#ffa657}.arch-box .d{color:#8b949e}.arch-box .w{color:#fff;font-weight:700}.arch-box .r{color:#ff6b6b}
|
||||||
.card h3 { font-size: .95rem; font-weight: 600; margin: 20px 0 10px; color: var(--g700); }
|
.big-num{font-size:2.8rem;font-weight:800;color:var(--bsn);line-height:1}
|
||||||
.card h4 { font-size: .85rem; font-weight: 700; margin: 16px 0 6px; color: var(--bsn); }
|
ul.check{list-style:none}
|
||||||
table { width: 100%; border-collapse: collapse; font-size: .86rem; }
|
ul.check li{padding:6px 0 6px 24px;position:relative;font-size:.88rem}
|
||||||
th { text-align: left; padding: 10px 12px; background: var(--bsn-bg); color: var(--bsn); font-weight: 700; font-size: .76rem; text-transform: uppercase; letter-spacing: .5px; border-bottom: 2px solid var(--bsn); }
|
ul.check li::before{content:'✓';position:absolute;left:0;color:var(--green);font-weight:700}
|
||||||
td { padding: 10px 12px; border-bottom: 1px solid var(--g200); vertical-align: top; }
|
ul.cross{list-style:none}
|
||||||
.badge { display: inline-block; padding: 3px 10px; border-radius: 12px; font-size: .76rem; font-weight: 600; }
|
ul.cross li{padding:6px 0 6px 24px;position:relative;font-size:.88rem}
|
||||||
.b-green { background: #d1fae5; color: #065f46; } .b-red { background: #fee2e2; color: #991b1b; }
|
ul.cross li::before{content:'✗';position:absolute;left:0;color:var(--red);font-weight:700}
|
||||||
.b-orange { background: #fef3c7; color: #92400e; } .b-blue { background: var(--bsn-bg); color: var(--bsn); }
|
.perf-card{border:1px solid var(--g200);border-radius:var(--r);padding:16px;background:var(--white)}
|
||||||
.b-nvidia { background: rgba(118,185,0,.15); color: #5a8f00; } .b-amd { background: rgba(237,28,36,.1); color: #c41a1f; }
|
.perf-card.best{border-color:var(--green);border-width:2px;background:linear-gradient(180deg,#f0fdf4 0%,#fff 100%)}
|
||||||
.callout { background: var(--bsn-bg); border-left: 4px solid var(--bsn); padding: 14px 18px; border-radius: 0 var(--r) var(--r) 0; margin: 20px 0; font-size: .9rem; }
|
@media(max-width:700px){.g2,.g3{grid-template-columns:1fr}.hero h1{font-size:1.4rem}}
|
||||||
.callout strong { color: var(--bsn); }
|
|
||||||
.callout.green { background: #f0fdf4; border-left-color: var(--green); }
|
|
||||||
.callout.green strong { color: #065f46; }
|
|
||||||
.arch-box { background: #0d1117; color: #c9d1d9; padding: 24px; border-radius: var(--r); font-family: 'SF Mono', 'Fira Code', 'Cascadia Code', monospace; font-size: .75rem; line-height: 1.8; margin: 16px 0; overflow-x: auto; }
|
|
||||||
.arch-box .g { color: #7ee787; } .arch-box .b { color: #79c0ff; }
|
|
||||||
.arch-box .p { color: #f778ba; } .arch-box .y { color: #e3b341; }
|
|
||||||
.arch-box .o { color: #ffa657; } .arch-box .d { color: #8b949e; }
|
|
||||||
.arch-box .w { color: #ffffff; font-weight: 700; }
|
|
||||||
.arch-box .nv { color: #a5d6ff; } .arch-box .cg { color: #56d364; }
|
|
||||||
.big-num { font-size: 2.8rem; font-weight: 800; color: var(--bsn); line-height: 1; }
|
|
||||||
.big-num small { font-size: 1rem; font-weight: 400; color: var(--g500); }
|
|
||||||
ul.check { list-style: none; }
|
|
||||||
ul.check li { padding: 6px 0 6px 24px; position: relative; font-size: .88rem; }
|
|
||||||
ul.check li::before { content: '✓'; position: absolute; left: 0; color: var(--green); font-weight: 700; }
|
|
||||||
.tier-card { background: var(--white); border-radius: var(--r); border: 2px solid var(--g200); padding: 18px; text-align: center; transition: transform .2s, box-shadow .2s; }
|
|
||||||
.tier-card:hover { transform: translateY(-4px); box-shadow: 0 10px 30px rgba(0,0,0,.1); }
|
|
||||||
.tier-card.best { border-color: var(--green); border-width: 3px; background: linear-gradient(180deg, #f0fdf4 0%, #fff 100%); }
|
|
||||||
.tier-card .tier-num { font-size: 2rem; font-weight: 800; color: var(--bsn); }
|
|
||||||
.model-card { border: 1px solid var(--g200); border-radius: var(--r); padding: 16px; background: var(--white); }
|
|
||||||
.model-card.recommended { border-color: var(--green); border-width: 2px; }
|
|
||||||
@media (max-width: 700px) { .g2, .g3, .g4 { grid-template-columns: 1fr; } .hero h1 { font-size: 1.4rem; } }
|
|
||||||
</style>
|
</style>
|
||||||
</head>
|
</head>
|
||||||
<body>
|
<body>
|
||||||
<div class="container">
|
<div class="container">
|
||||||
|
|
||||||
<!-- ═══ HERO ═══ -->
|
|
||||||
<div class="hero">
|
<div class="hero">
|
||||||
<h1>🏠 BSN Chatbot — Lokale KI-Architektur</h1>
|
<h1>🔧 BSN Chatbot — Korrigierte Architektur</h1>
|
||||||
<p class="subtitle">
|
<p class="subtitle">
|
||||||
DGX Spark + Strix Halo als privates KI-Rechenzentrum zu Hause.
|
Basierend auf echten DGX-Spark-Benchmarks. Kein Llama. Kein OpenWebUI-Dual-Request-Freeze.
|
||||||
100% DSGVO · 0 € API-Kosten · 192 GB VRAM kombiniert.
|
<strong>Sequenzielle Verarbeitung via Queue + ein Modell gleichzeitig im VRAM.</strong>
|
||||||
</p>
|
</p>
|
||||||
<div class="chips">
|
</div>
|
||||||
<span class="chip chip-nvidia">🟢 NVIDIA DGX Spark — 96 GB</span>
|
|
||||||
<span class="chip chip-amd">🔴 AMD Strix Halo — 96 GB</span>
|
<!-- ═══ 1. WAS FALSCH WAR ═══ -->
|
||||||
<span class="chip chip-local">🏠 Standort: Zuhause (DE)</span>
|
<div class="card" style="margin-bottom:32px">
|
||||||
<span class="chip chip-local">🔒 100% DSGVO</span>
|
<h2>🚫 1. Was vorher falsch war — Meine Fehler</h2>
|
||||||
|
<table>
|
||||||
|
<tr><th>Falsche Annahme</th><th>Warum falsch</th><th>Was tatsächlich passiert</th></tr>
|
||||||
|
<tr>
|
||||||
|
<td><strong>Llama 3.1 70B läuft gut</strong></td>
|
||||||
|
<td>Dense-Modelle sind <strong>speicherbandbreiten-limitiert</strong> auf Unified Memory. GB10 hat ~500 GB/s — Llama 70B braucht >1 TB/s für schnelle Inferenz.</td>
|
||||||
|
<td>Sehr langsam. MoE-Modelle (Qwen, Gemma) sind 3–10× schneller, weil sie weniger aktive Parameter pro Token haben.</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><strong>Beide Maschinen parallel inferieren</strong></td>
|
||||||
|
<td>2 Requests → 2× KV-Cache → VRAM voll → <strong>Swap → FREEZE</strong>. DGX hat nur 121.7 GiB für CUDA. OpenWebUI hat kein Queue-Management.</td>
|
||||||
|
<td><code>OLLAMA_NUM_PARALLEL=1</code> wird EMPFOHLEN. <code>gpu_mem_util</code> max 0.85–0.88, nicht 0.9+.</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><strong>Mistral Large 70B passt</strong></td>
|
||||||
|
<td>Mistral-Small-24B erreicht schon nur <strong>4.5 tok/s</strong> auf DGX (Benchmark). Dense-Modelle sind auf Unified Memory generell langsam.</td>
|
||||||
|
<td>Qwen 3.5 35B MoE schafft <strong>57 tok/s</strong> auf derselben Hardware. MoE = 12× schneller als Dense.</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><strong>OpenWebUI als Production-Backend</strong></td>
|
||||||
|
<td>OpenWebUI ist ein <strong>Frontend</strong>, kein Inference-Server. Kein Queueing, kein Rate-Limiting, keine Request-Priorisierung.</td>
|
||||||
|
<td>Braucht: <strong>LiteLLM → llama-swap → vLLM/llama.cpp</strong> für orchestriertes, sequentielles Serving.</td>
|
||||||
|
</tr>
|
||||||
|
</table>
|
||||||
|
|
||||||
|
<div class="callout red">
|
||||||
|
<strong>❌ Root Cause des Freeze:</strong> Zwei parallele Requests → Ollama lädt 2× Modell-Kontexte in VRAM →
|
||||||
|
<code>gpu_mem_util > 0.9</code> → System beginnt zu swappen → <strong>kompletter Stillstand beider Requests.</strong><br>
|
||||||
|
<strong>Fix:</strong> Nur <strong>EIN Modell gleichzeitig</strong> im VRAM. Requests werden <strong>sequentiell</strong> abgearbeitet.
|
||||||
|
vLLM mit Continuous Batching macht aus N Requests <strong>einen Batch</strong> — effizient, kein Freeze.
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<!-- ═══ 1. NEUE ARCHITEKTUR ═══ -->
|
<!-- ═══ 2. WAS TATSÄCHLICH FUNKTIONIERT ═══ -->
|
||||||
<div class="card" style="margin-bottom:32px">
|
<div class="card" style="margin-bottom:32px">
|
||||||
<h2>🏗️ 1. Neue Architektur — Edge/Cloud + KI-Cluster</h2>
|
<h2>✅ 2. Was tatsächlich funktioniert — Echte DGX-Spark-Benchmarks</h2>
|
||||||
|
<p>Quelle: <a href="https://forums.developer.nvidia.com/t/running-a-full-llm-stack-on-dgx-spark-gb10/367580" target="_blank">NVIDIA Developer Forums — Full LLM Stack on DGX Spark GB10</a> (April 2026)</p>
|
||||||
|
|
||||||
|
<h3>2.1 Empfohlene Modelle (alle auf DGX Spark getestet)</h3>
|
||||||
|
<table>
|
||||||
|
<tr><th>Modell</th><th>Engine</th><th>Prefill</th><th>Generation</th><th>VRAM</th><th>Für</th></tr>
|
||||||
|
<tr style="background:#f0fdf4">
|
||||||
|
<td><strong>🏆 Qwen 3.5 35B MoE Q4_K_M</strong></td>
|
||||||
|
<td>llama.cpp</td>
|
||||||
|
<td>1.798 tok/s</td>
|
||||||
|
<td><strong>57.1 tok/s</strong></td>
|
||||||
|
<td>~20 GB</td>
|
||||||
|
<td><span class="badge bg">Triage + Chat</span></td>
|
||||||
|
</tr>
|
||||||
|
<tr style="background:#f0fdf4">
|
||||||
|
<td><strong>🏆 Nemotron-Nano 30B NVFP4</strong></td>
|
||||||
|
<td>vLLM</td>
|
||||||
|
<td>7.417 tok/s</td>
|
||||||
|
<td><strong>55.9 tok/s</strong></td>
|
||||||
|
<td>~18 GB</td>
|
||||||
|
<td><span class="badge bg">Triage (schnell)</span></td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><strong>Qwen 3.5 35B A3B FP8</strong></td>
|
||||||
|
<td>vLLM</td>
|
||||||
|
<td>4.439 tok/s</td>
|
||||||
|
<td>49.1 tok/s</td>
|
||||||
|
<td>~20 GB</td>
|
||||||
|
<td>Triage</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><strong>Qwen Coder INT4</strong></td>
|
||||||
|
<td>vLLM</td>
|
||||||
|
<td>4.425 tok/s</td>
|
||||||
|
<td><strong>66.7 tok/s</strong></td>
|
||||||
|
<td>~15 GB</td>
|
||||||
|
<td>Structured Output</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><strong>GPT-OSS-120B MXFP4</strong></td>
|
||||||
|
<td>vLLM</td>
|
||||||
|
<td>4.703 tok/s</td>
|
||||||
|
<td>56.4 tok/s</td>
|
||||||
|
<td>~90 GB</td>
|
||||||
|
<td>Schwere Fälle (solo)</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><strong>Nemotron Nano 4B FP8</strong></td>
|
||||||
|
<td>vLLM</td>
|
||||||
|
<td>8.179 tok/s</td>
|
||||||
|
<td>39.8 tok/s</td>
|
||||||
|
<td>~4 GB</td>
|
||||||
|
<td>Chatbot (always-on)</td>
|
||||||
|
</tr>
|
||||||
|
<tr style="background:#fff5f5">
|
||||||
|
<td><strike>Mistral-Small 24B</strike></td>
|
||||||
|
<td>vLLM</td>
|
||||||
|
<td>2.064 tok/s</td>
|
||||||
|
<td><strong>4.5 tok/s</strong></td>
|
||||||
|
<td>~16 GB</td>
|
||||||
|
<td><span class="badge br">Zu langsam</span></td>
|
||||||
|
</tr>
|
||||||
|
</table>
|
||||||
|
|
||||||
|
<div class="callout green">
|
||||||
|
<strong>🔑 Kern-Erkenntnis:</strong> MoE-Architektur (Qwen, Nemotron, GPT-OSS) ist auf Unified Memory
|
||||||
|
<strong>3–12× schneller</strong> als Dense-Modelle (Mistral, Llama). Der DGX Spark hat genug VRAM für
|
||||||
|
große Modelle, aber die <strong>Speicherbandbreite</strong> (~500 GB/s) limitiert Dense-Modelle massiv.
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<h3>2.2 Was der Nutzer schon getestet hat</h3>
|
||||||
|
<table>
|
||||||
|
<tr><th>Modell</th><th>Läuft gut?</th><th>Anmerkung</th></tr>
|
||||||
|
<tr><td><strong>Gemma 4</strong></td><td><span class="badge bg">✅ Ja</span></td><td>Google-optimiert, gut für Chat. Auf DGX via vLLM mit NVFP4-Quantisierung.</td></tr>
|
||||||
|
<tr><td><strong>Qwen (2.5 Serie)</strong></td><td><span class="badge bg">✅ Ja</span></td><td>MoE-Architektur = schnell auf Unified Memory. 57 tok/s auf DGX.</td></tr>
|
||||||
|
<tr><td><strong>Ministral (8B)</strong></td><td><span class="badge bg">✅ Ja</span></td><td>Klein, effizient. Gut als Always-On-Chatbot. Wenig VRAM.</td></tr>
|
||||||
|
<tr><td><strong>GLM (klein)</strong></td><td><span class="badge bg">✅ Ja</span></td><td>Chinesisch optimiert, gut für strukturierte Ausgaben.</td></tr>
|
||||||
|
<tr><td><strong>Llama 3.1 70B</strong></td><td><span class="badge br">❌ Nein</span></td><td>Dense = langsam auf Unified Memory.</td></tr>
|
||||||
|
</table>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- ═══ 3. KORRIGIERTE ARCHITEKTUR ═══ -->
|
||||||
|
<div class="card" style="margin-bottom:32px">
|
||||||
|
<h2>🏗️ 3. Korrigierte Architektur — Sequentiell, nicht parallel</h2>
|
||||||
|
|
||||||
<div class="arch-box">
|
<div class="arch-box">
|
||||||
<span class="d">╔══════════════════════════════════════════════════════════════════╗</span>
|
<span class="d">╔══════════════════════════════════════════════════════════════════╗</span>
|
||||||
<span class="d">║ INTERNET ║</span>
|
<span class="d">║ </span><span class="w">EDGE-SERVER (Hetzner CX32) — 4 vCPU, 16 GB</span><span class="d"> ║</span>
|
||||||
<span class="d">║</span> WhatsApp 📱 Telegram 📱 Web-Browser 🌐 Joomla-API 📰
|
<span class="d">║</span> <span class="b">Flask (Gunicorn)</span> → <span class="b">Redis Queue</span> → <span class="b">PostgreSQL</span> <span class="d">║</span>
|
||||||
<span class="d">║</span> │ │ │ │
|
<span class="d">║</span> Webhooks + API Aufgaben DB + Sessions <span class="d">║</span>
|
||||||
<span class="d">╠═══════╪══════════════╪══════════════╪══════════════╪══════════════╣</span>
|
<span class="d">╚══════════╤═══════════════════════════════════════════════════════╝</span>
|
||||||
<span class="d">║</span> <span class="w">EDGE-SERVER (Hetzner CX32)</span>
|
<span class="d"> │</span>
|
||||||
<span class="d">║</span> <span class="d">4 vCPU · 16 GB · 80 GB NVMe</span>
|
<span class="d"> │ </span><span class="y">🔐 WireGuard VPN (nur Redis-Port)</span>
|
||||||
<span class="d">║</span> <span class="b">~40 €/Monat</span>
|
<span class="d"> │</span>
|
||||||
<span class="d">║</span> ┌────────────────────┼────────────────────┐
|
<span class="d">╔══════════╧═══════════════════════════════════════════════════════╗</span>
|
||||||
<span class="d">║</span> │ │ │
|
<span class="d">║ </span><span class="w">KI-CLUSTER (Zuhause, Deutschland)</span><span class="d"> ║</span>
|
||||||
<span class="d">║</span> ▼ ▼ ▼
|
<span class="d">║ ║</span>
|
||||||
<span class="d">║</span> <span class="b">cloudflared</span> <span class="b">Nginx</span> <span class="b">PostgreSQL</span>
|
<span class="d">║ </span><span class="g">┌─ Redis Worker ─────────────────────────────────────────┐</span><span class="d"> ║</span>
|
||||||
<span class="d">║</span> Tunnel Reverse Proxy DB + Sessions
|
<span class="d">║ </span><span class="g">│ Pollt Queue. Holt EINE Aufgabe. │</span><span class="d"> ║</span>
|
||||||
<span class="d">║</span> │ │ │
|
<span class="d">║ </span><span class="g">│ SEQUENTIELL — nie zwei gleichzeitig. │</span><span class="d"> ║</span>
|
||||||
<span class="d">║</span> ▼ ▼ │
|
<span class="d">║ </span><span class="g">│ Priorität: Triage > Transkription > Bildcheck > Chat │</span><span class="d"> ║</span>
|
||||||
<span class="d">║</span> <span class="b">Flask (Gunicorn)</span> ──► <span class="b">Redis</span> ──────────────┘
|
<span class="d">║ </span><span class="g">└───────────────────────────────────────────────────────┘</span><span class="d"> ║</span>
|
||||||
<span class="d">║</span> Webhooks + API Queue + Cache
|
<span class="d">║ │ ║</span>
|
||||||
<span class="d">║</span> │
|
<span class="d">║ ▼ ║</span>
|
||||||
<span class="d">║</span> │ <span class="o">Aufgaben in Queue</span>
|
<span class="d">║ </span><span class="b">┌─ LiteLLM (Port 14000) ─────────────────────────────────┐</span><span class="d"> ║</span>
|
||||||
<span class="d">║</span> │ (Triage · Transkription · Bildcheck)
|
<span class="d">║ </span><span class="b">│ OpenAI-kompatible API. Routing + Fallbacks. │</span><span class="d"> ║</span>
|
||||||
<span class="d">║</span> │
|
<span class="d">║ </span><span class="b">│ Rate-Limiting: Max 1 concurrent request. │</span><span class="d"> ║</span>
|
||||||
<span class="d">╠═════╪════════════════════════════════════════════════════════════╣</span>
|
<span class="d">║ </span><span class="b">└───────────────────────────────────────────────────────┘</span><span class="d"> ║</span>
|
||||||
<span class="d">║</span> │ <span class="y">🔐 WireGuard VPN Tunnel</span>
|
<span class="d">║ │ ║</span>
|
||||||
<span class="d">║</span> │ (verschlüsselt · nur Port 6379 Redis)
|
<span class="d">║ ▼ ║</span>
|
||||||
<span class="d">║</span> │
|
<span class="d">║ </span><span class="p">┌─ llama-swap (Port 28080) ─────────────────────────────┐</span><span class="d"> ║</span>
|
||||||
<span class="d">╠═════╪════════════════════════════════════════════════════════════╣</span>
|
<span class="d">║ </span><span class="p">│ VRAM-Orchestrator: NUR EIN Modell gleichzeitig. │</span><span class="d"> ║</span>
|
||||||
<span class="d">║</span> │ <span class="w">KI-CLUSTER (Zuhause, Deutschland)</span>
|
<span class="d">║ </span><span class="p">│ Lädt Modell bei Bedarf, entlädt nach 5 Min Idle. │</span><span class="d"> ║</span>
|
||||||
<span class="d">║</span> │
|
<span class="d">║ </span><span class="p">│ Verhindert VRAM-Überlastung → KEIN FREEZE. │</span><span class="d"> ║</span>
|
||||||
<span class="d">║</span> ├─────────────────────────────────────────────────────────┐
|
<span class="d">║ </span><span class="p">└──────┬──────────────┬──────────────┬──────────────────┘</span><span class="d"> ║</span>
|
||||||
<span class="d">║</span> │ │
|
<span class="d">║ │ │ │ ║</span>
|
||||||
<span class="d">║</span> ▼ ▼ │
|
<span class="d">║ ▼ ▼ ▼ ║</span>
|
||||||
<span class="d">║</span> <span class="nv">🟢 DGX Spark</span> <span class="cg">🔴 Strix Halo</span> │
|
<span class="d">║ </span><span class="g">┌──────────┐</span><span class="d"> </span><span class="g">┌──────────┐</span><span class="d"> </span><span class="g">┌──────────────┐</span><span class="d"> ║</span>
|
||||||
<span class="d">║</span> <span class="d">GB10 Grace-Blackwell</span> <span class="d">AMD RDNA 3.5 APU</span> │
|
<span class="d">║ </span><span class="g">│ Qwen 35B │</span><span class="d"> </span><span class="g">│ Nemotr. │</span><span class="d"> </span><span class="g">│ Ministral 8B │</span><span class="d"> ║</span>
|
||||||
<span class="d">║</span> <span class="d">96 GB Unified · 20 Kerne</span> <span class="d">96 GB Unified · 16 Kerne</span> │
|
<span class="d">║ </span><span class="g">│ MoE Q4 │</span><span class="d"> </span><span class="g">│ 30B NVFP4│</span><span class="d"> </span><span class="g">│ Always-On │</span><span class="d"> ║</span>
|
||||||
<span class="d">║</span> │ │ │
|
<span class="d">║ </span><span class="g">│ vLLM │</span><span class="d"> </span><span class="g">│ vLLM │</span><span class="d"> </span><span class="g">│ vLLM │</span><span class="d"> ║</span>
|
||||||
<span class="d">║</span> ├─ <span class="g">Llama 3.1 70B</span> ├─ <span class="g">Llama 3.1 70B</span> (Failover) │
|
<span class="d">║ </span><span class="g">│ Triage │</span><span class="d"> </span><span class="g">│ Triage │</span><span class="d"> </span><span class="g">│ Chatbot │</span><span class="d"> ║</span>
|
||||||
<span class="d">║</span> ├─ <span class="g">Whisper large-v3</span> ├─ <span class="g">Llama Vision 11B</span> │
|
<span class="d">║ </span><span class="g">└──────────┘</span><span class="d"> </span><span class="g">└──────────┘</span><span class="d"> </span><span class="g">└──────────────┘</span><span class="d"> ║</span>
|
||||||
<span class="d">║</span> ├─ <span class="g">Bild-Sicherheit (Llama)</span> ├─ <span class="g">Whisper large-v3</span> (Backup) │
|
<span class="d">║ ║</span>
|
||||||
<span class="d">║</span> ├─ <span class="g">TTS (Piper)</span> ├─ <span class="g">Mistral Large</span> (Diversity) │
|
<span class="d">║ </span><span class="y">Strix Halo: Backup/Failover — gleiche Modelle, anderer Port</span><span class="d"> ║</span>
|
||||||
<span class="d">║</span> └─ <span class="g">Redis Worker #1</span> └─ <span class="g">Redis Worker #2</span> │
|
<span class="d">╚═════════════════════════════════════════════════════════════════╝</span>
|
||||||
<span class="d">║</span> │
|
|
||||||
<span class="d">║</span> <span class="p">Redis Queue Consumer</span> → Aufgabe holen → lokal inferieren │
|
|
||||||
<span class="d">║</span> → Ergebnis in Redis schreiben → Edge-Server liefert aus │
|
|
||||||
<span class="d">╚══════════════════════════════════════════════════════════════════╝</span>
|
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<h3>1.1 Rollenverteilung</h3>
|
<h3>3.1 Kernprinzipien</h3>
|
||||||
<table>
|
<table>
|
||||||
<tr><th>Komponente</th><th>Standort</th><th>Hardware</th><th>Aufgabe</th></tr>
|
<tr><th>#</th><th>Prinzip</th><th>Warum</th></tr>
|
||||||
<tr>
|
<tr><td>1</td><td><strong>NUR 1 Modell gleichzeitig</strong> im VRAM</td><td>Verhindert Swap → Freeze. llama-swap lädt/entlädt automatisch.</td></tr>
|
||||||
<td><strong>Edge-Server</strong></td>
|
<tr><td>2</td><td><strong>Sequentiell, nicht parallel</strong></td><td>Redis-Worker holt 1 Aufgabe, verarbeitet sie, holt nächste. Kein Race.</td></tr>
|
||||||
<td><span class="badge b-blue">Hetzner Cloud</span></td>
|
<tr><td>3</td><td><strong>MoE > Dense</strong> auf Unified Memory</td><td>MoE hat weniger aktive Parameter/Token → ~10× schneller bei gleicher VRAM-Größe.</td></tr>
|
||||||
<td>CX32 (4 vCPU, 16 GB)</td>
|
<tr><td>4</td><td><strong>Kleines Always-On-Modell</strong> für Chat</td><td>Ministral 8B oder Nemotron 4B bleibt geladen. <1s Antwortzeit. Triage-Modell nur bei Bedarf.</td></tr>
|
||||||
<td>Public-Facing: Webhooks, Frontend, Admin, API. DB (PostgreSQL). Redis (Queue + Cache). KEINE KI-Berechnung.</td>
|
<tr><td>5</td><td><strong>Strix Halo = Failover</strong></td><td>Nur aktiv wenn DGX ausfällt. Spart Strom. Gleiche Queue, anderer Worker.</td></tr>
|
||||||
|
<tr><td>6</td><td><strong>OpenWebUI nur als Frontend</strong></td><td>Zeigt Ergebnisse, macht KEINE Inferenz. Inferenz via LiteLLM API.</td></tr>
|
||||||
|
</table>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- ═══ 4. MODELL-STRATEGIE ═══ -->
|
||||||
|
<div class="card" style="margin-bottom:32px">
|
||||||
|
<h2>🧠 4. Modell-Strategie — Was läuft wann</h2>
|
||||||
|
|
||||||
|
<table>
|
||||||
|
<tr><th>Aufgabe</th><th>Modell</th><th>Engine</th><th>Geschw.</th><th>VRAM</th><th>Wann geladen</th></tr>
|
||||||
|
<tr style="background:#f0fdf4">
|
||||||
|
<td><strong>Chatbot (Web)</strong></td>
|
||||||
|
<td><strong>Ministral 8B</strong> / Gemma 4 9B</td>
|
||||||
|
<td>vLLM</td>
|
||||||
|
<td>40–60 tok/s</td>
|
||||||
|
<td>~6 GB</td>
|
||||||
|
<td><span class="badge bg">Dauerhaft</span></td>
|
||||||
|
</tr>
|
||||||
|
<tr style="background:#f0fdf4">
|
||||||
|
<td><strong>Triage (Standard)</strong></td>
|
||||||
|
<td><strong>Qwen 3.5 35B MoE Q4</strong></td>
|
||||||
|
<td>llama.cpp</td>
|
||||||
|
<td>57 tok/s</td>
|
||||||
|
<td>~20 GB</td>
|
||||||
|
<td>Bei Bedarf (llama-swap)</td>
|
||||||
</tr>
|
</tr>
|
||||||
<tr>
|
<tr>
|
||||||
<td><strong>DGX Spark</strong></td>
|
<td><strong>Triage (schnell)</strong></td>
|
||||||
<td><span class="badge b-green">🏠 Zuhause</span></td>
|
<td>Nemotron 30B NVFP4</td>
|
||||||
<td>GB10 Grace-Blackwell, 96 GB</td>
|
<td>vLLM</td>
|
||||||
<td><strong>Primärer KI-Worker:</strong> Llama 3.1 70B (Triage + Chat), Whisper large-v3 (Transkription), Llama Vision (Bildcheck)</td>
|
<td>55.9 tok/s</td>
|
||||||
|
<td>~18 GB</td>
|
||||||
|
<td>Bei Bedarf</td>
|
||||||
</tr>
|
</tr>
|
||||||
<tr>
|
<tr>
|
||||||
<td><strong>Strix Halo</strong></td>
|
<td><strong>Bild-Sicherheit</strong></td>
|
||||||
<td><span class="badge b-green">🏠 Zuhause</span></td>
|
<td>Qwen3-VL 30B FP8</td>
|
||||||
<td>AMD APU, 96 GB Unified</td>
|
<td>vLLM</td>
|
||||||
<td><strong>Sekundärer KI-Worker:</strong> Failover + Diversity (Mistral Large für Zweitmeinung), Bild-Moderation, Backup-Transkription</td>
|
<td>51.9 tok/s</td>
|
||||||
|
<td>~20 GB</td>
|
||||||
|
<td>Bei Bedarf</td>
|
||||||
</tr>
|
</tr>
|
||||||
<tr>
|
<tr>
|
||||||
<td><strong>WireGuard VPN</strong></td>
|
<td><strong>Spielcover-Titel</strong></td>
|
||||||
<td><span class="badge b-blue">Verbindung</span></td>
|
<td>Qwen3-VL 30B (gleiches Modell)</td>
|
||||||
|
<td>vLLM</td>
|
||||||
<td>—</td>
|
<td>—</td>
|
||||||
<td>Verschlüsselte Brücke zwischen Edge-Server und KI-Cluster. Nur Redis-Port (6379) wird durchgereicht.</td>
|
<td>—</td>
|
||||||
|
<td>Selbe Instanz</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><strong>Audio→Text</strong></td>
|
||||||
|
<td>Whisper large-v3 (GPU)</td>
|
||||||
|
<td>faster-whisper</td>
|
||||||
|
<td>50–100× RT</td>
|
||||||
|
<td>~8 GB</td>
|
||||||
|
<td>Bei Bedarf (separat)</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><strong>Kommentar-Mod.</strong></td>
|
||||||
|
<td>Ministral 8B (gleiches Modell)</td>
|
||||||
|
<td>vLLM</td>
|
||||||
|
<td>—</td>
|
||||||
|
<td>—</td>
|
||||||
|
<td>Selbe Instanz</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><strong>Grenzfälle (Tier 2)</strong></td>
|
||||||
|
<td>GPT-OSS-120B MXFP4</td>
|
||||||
|
<td>vLLM</td>
|
||||||
|
<td>56.4 tok/s</td>
|
||||||
|
<td>~90 GB</td>
|
||||||
|
<td>Nur DGX, solo, selten</td>
|
||||||
</tr>
|
</tr>
|
||||||
</table>
|
</table>
|
||||||
|
|
||||||
<div class="callout green">
|
<div class="callout green">
|
||||||
<strong>✅ DSGVO-Perfekt:</strong> Der Edge-Server speichert NUR hashed Telefonnummern und anonymisierte Inhalte.
|
<strong>💡 Im Normalbetrieb:</strong> Ministral 8B ist immer geladen (Chatbot). Für Triage wird
|
||||||
Die KI-Verarbeitung (volle Texte, Bilder, Audio) geschieht <strong>ausschließlich auf den lokalen Maschinen</strong>.
|
Qwen 35B MoE geladen (57 tok/s), verarbeitet, nach 5 Min Idle wieder entladen.
|
||||||
Keine Daten verlassen Deutschland. Keine US-API. Keine China-API. <strong>100% eigene Hardware.</strong>
|
<strong>Gesamt-VRAM-Spitze: ~26 GB</strong> (6 GB Ministral + 20 GB Qwen) — weit unter 121.7 GB.<br>
|
||||||
|
<strong>Kein Swap. Kein Freeze.</strong>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<!-- ═══ 2. MODELLE ═══ -->
|
<!-- ═══ 5. UMSETZUNG ═══ -->
|
||||||
<div class="card" style="margin-bottom:32px">
|
<div class="card" style="margin-bottom:32px">
|
||||||
<h2>🧠 2. Modelle — Was läuft auf 96 GB VRAM?</h2>
|
<h2>🔧 5. Konkrete Umsetzung</h2>
|
||||||
<p>
|
|
||||||
Mit <strong>96 GB Unified Memory</strong> pro Maschine kannst du Modelle betreiben, die für Cloud-APIs unerschwinglich wären.
|
|
||||||
Hier die konkreten Empfehlungen:
|
|
||||||
</p>
|
|
||||||
|
|
||||||
<h3>2.1 LLM für Triage & Chatbot</h3>
|
<h3>5.1 Software-Stack (DGX Spark)</h3>
|
||||||
<div class="g2" style="margin-top:12px">
|
|
||||||
<div class="model-card recommended">
|
|
||||||
<h4>🏆 Llama 3.1 70B (Q4_K_M) <span class="badge b-green">Empfohlen</span></h4>
|
|
||||||
<table>
|
|
||||||
<tr><td><strong>VRAM-Bedarf</strong></td><td>~40 GB</td></tr>
|
|
||||||
<tr><td><strong>Passt auf</strong></td><td>DGX + Strix ✓ (einzeln)</td></tr>
|
|
||||||
<tr><td><strong>Kontext</strong></td><td>8K–128K Token</td></tr>
|
|
||||||
<tr><td><strong>Geschwindigkeit</strong></td><td>15–30 Tokens/s (DGX)</td></tr>
|
|
||||||
<tr><td><strong>Qualität</strong></td><td>Vergleichbar GPT-4</td></tr>
|
|
||||||
<tr><td><strong>Deutsch</strong></td><td>Sehr gut (multilingual trainiert)</td></tr>
|
|
||||||
<tr><td><strong>Kosten</strong></td><td>0 €</td></tr>
|
|
||||||
<tr><td><strong>DSGVO</strong></td><td><span class="badge b-green">✅ 100%</span></td></tr>
|
|
||||||
</table>
|
|
||||||
<p style="font-size:.82rem;color:var(--g500);margin-top:8px;">
|
|
||||||
Läuft via <strong>Ollama</strong> oder <strong>vLLM</strong> mit OpenAI-kompatibler API.
|
|
||||||
Ersetzt DeepSeek V4 Flash komplett.
|
|
||||||
</p>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<div class="model-card">
|
|
||||||
<h4>Mistral Large (Q4) <span class="badge b-blue">Zweitmodell</span></h4>
|
|
||||||
<table>
|
|
||||||
<tr><td><strong>VRAM-Bedarf</strong></td><td>~70 GB</td></tr>
|
|
||||||
<tr><td><strong>Passt auf</strong></td><td>DGX ✓ Strix ✓</td></tr>
|
|
||||||
<tr><td><strong>Kontext</strong></td><td>32K Token</td></tr>
|
|
||||||
<tr><td><strong>Geschwindigkeit</strong></td><td>10–20 Tokens/s</td></tr>
|
|
||||||
<tr><td><strong>Qualität</strong></td><td>Top-Tier, EU-Modell</td></tr>
|
|
||||||
<tr><td><strong>Deutsch</strong></td><td>Exzellent</td></tr>
|
|
||||||
<tr><td><strong>Kosten</strong></td><td>0 €</td></tr>
|
|
||||||
<tr><td><strong>DSGVO</strong></td><td><span class="badge b-green">✅ 100%</span></td></tr>
|
|
||||||
</table>
|
|
||||||
<p style="font-size:.82rem;color:var(--g500);margin-top:8px;">
|
|
||||||
Als Zweitmeinung oder Diversity-Modell auf Strix Halo. Bei kontroversen Fällen beide Modelle befragen.
|
|
||||||
</p>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<div class="model-card" style="margin-top:16px;">
|
|
||||||
<h4>⚡ Alternativ: Llama 3.1 8B + 70B gestaffelt</h4>
|
|
||||||
<table>
|
|
||||||
<tr><td><strong>8B (Q8)</strong></td><td>~8 GB VRAM · 80+ Tokens/s · Für einfache Triage (Tier 1/3 klar) und Chatbot</td></tr>
|
|
||||||
<tr><td><strong>70B (Q4)</strong></td><td>~40 GB VRAM · 20 Tokens/s · Für Grenzfälle (Tier 2), Summary-Generierung, komplexe Moderation</td></tr>
|
|
||||||
<tr><td><strong>Vorteil</strong></td><td>95% der Requests mit 8B (schnell), 5% mit 70B (hohe Qualität). <strong>Massiv höherer Durchsatz.</strong></td></tr>
|
|
||||||
</table>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<h3>2.2 Audio-Transkription</h3>
|
|
||||||
<div class="model-card recommended" style="margin-top:12px">
|
|
||||||
<h4>🏆 Whisper large-v3 (via faster-whisper + CUDA) <span class="badge b-green">Empfohlen</span></h4>
|
|
||||||
<table>
|
|
||||||
<tr><td><strong>VRAM-Bedarf</strong></td><td>~6–8 GB (large-v3)</td></tr>
|
|
||||||
<tr><td><strong>Geschwindigkeit (GPU)</strong></td><td><strong>50–100× Echtzeit</strong> (1 Min Audio in <1s)</td></tr>
|
|
||||||
<tr><td><strong>Qualität</strong></td><td>Beste verfügbare Open-Source-Transkription</td></tr>
|
|
||||||
<tr><td><strong>Sprachen</strong></td><td>99 Sprachen, exzellentes Deutsch</td></tr>
|
|
||||||
<tr><td><strong>Kosten</strong></td><td>0 €</td></tr>
|
|
||||||
<tr><td><strong>DSGVO</strong></td><td><span class="badge b-green">✅ 100% (lokal)</span></td></tr>
|
|
||||||
</table>
|
|
||||||
<p style="font-size:.82rem;color:var(--g500);margin-top:8px;">
|
|
||||||
Im Vergleich zu vorher (<strong>faster-whisper base CPU = 2–5× Echtzeit</strong>) ist das eine
|
|
||||||
<strong>20–50× Beschleunigung</strong>. Eine 3-Minuten-Sprachnachricht in unter 2 Sekunden.
|
|
||||||
</p>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<h3>2.3 Bild-Sicherheitsprüfung</h3>
|
|
||||||
<div class="model-card recommended" style="margin-top:12px">
|
|
||||||
<h4>🏆 Llama 3.2 Vision 11B <span class="badge b-green">Empfohlen</span></h4>
|
|
||||||
<table>
|
|
||||||
<tr><td><strong>VRAM-Bedarf</strong></td><td>~8 GB</td></tr>
|
|
||||||
<tr><td><strong>Geschwindigkeit</strong></td><td>1–3s pro Bild</td></tr>
|
|
||||||
<tr><td><strong>Fähigkeiten</strong></td><td>Erkennt Gewalt, Pornografie, Nazi-Symbole, Drogen, Dokumente</td></tr>
|
|
||||||
<tr><td><strong>Kosten</strong></td><td>0 €</td></tr>
|
|
||||||
<tr><td><strong>DSGVO</strong></td><td><span class="badge b-green">✅ 100%</span></td></tr>
|
|
||||||
</table>
|
|
||||||
<p style="font-size:.82rem;color:var(--g500);margin-top:8px;">
|
|
||||||
Ersetzt Gemini Flash Vision komplett. Keine Bilder verlassen mehr das Haus.
|
|
||||||
</p>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<h3>2.4 Komplette Modell-Übersicht</h3>
|
|
||||||
<table>
|
|
||||||
<tr><th>Zweck</th><th>Bisher</th><th>Neu (Lokal)</th><th>VRAM</th><th>Geschwindigkeit</th><th>Maschine</th></tr>
|
|
||||||
<tr>
|
|
||||||
<td>LLM-Triage</td>
|
|
||||||
<td><span class="badge b-orange">DeepSeek V4 Flash (China)</span></td>
|
|
||||||
<td><strong>Llama 3.1 70B</strong></td>
|
|
||||||
<td>40 GB</td>
|
|
||||||
<td>20 tok/s</td>
|
|
||||||
<td><span class="badge b-nvidia">DGX</span></td>
|
|
||||||
</tr>
|
|
||||||
<tr>
|
|
||||||
<td>LLM-Chatbot</td>
|
|
||||||
<td><span class="badge b-orange">DeepSeek V4 Flash (China)</span></td>
|
|
||||||
<td><strong>Llama 3.1 8B</strong> (schnell)</td>
|
|
||||||
<td>8 GB</td>
|
|
||||||
<td>80 tok/s</td>
|
|
||||||
<td><span class="badge b-nvidia">DGX</span></td>
|
|
||||||
</tr>
|
|
||||||
<tr>
|
|
||||||
<td>Audio → Text</td>
|
|
||||||
<td><span class="badge b-green">faster-whisper base (lokal)</span></td>
|
|
||||||
<td><strong>Whisper large-v3 GPU</strong></td>
|
|
||||||
<td>8 GB</td>
|
|
||||||
<td>50–100× RT 🚀</td>
|
|
||||||
<td><span class="badge b-nvidia">DGX</span></td>
|
|
||||||
</tr>
|
|
||||||
<tr>
|
|
||||||
<td>Bild-Sicherheit</td>
|
|
||||||
<td><span class="badge b-orange">Gemini Flash (Google US)</span></td>
|
|
||||||
<td><strong>Llama 3.2 Vision 11B</strong></td>
|
|
||||||
<td>8 GB</td>
|
|
||||||
<td>1–3s</td>
|
|
||||||
<td><span class="badge b-nvidia">DGX</span></td>
|
|
||||||
</tr>
|
|
||||||
<tr>
|
|
||||||
<td>Spielcover-Titel</td>
|
|
||||||
<td><span class="badge b-orange">Gemini Flash (Google US)</span></td>
|
|
||||||
<td><strong>Llama 3.2 Vision 11B</strong></td>
|
|
||||||
<td>8 GB</td>
|
|
||||||
<td>1–2s</td>
|
|
||||||
<td><span class="badge b-nvidia">DGX</span></td>
|
|
||||||
</tr>
|
|
||||||
<tr>
|
|
||||||
<td>Kommentar-Mod.</td>
|
|
||||||
<td><span class="badge b-orange">DeepSeek V4 Flash (China)</span></td>
|
|
||||||
<td><strong>Llama 3.1 8B</strong></td>
|
|
||||||
<td>8 GB</td>
|
|
||||||
<td>80 tok/s</td>
|
|
||||||
<td><span class="badge b-nvidia">DGX</span></td>
|
|
||||||
</tr>
|
|
||||||
<tr>
|
|
||||||
<td>Zweitmeinung</td>
|
|
||||||
<td>—</td>
|
|
||||||
<td><strong>Mistral Large</strong></td>
|
|
||||||
<td>70 GB</td>
|
|
||||||
<td>15 tok/s</td>
|
|
||||||
<td><span class="badge b-amd">Strix</span></td>
|
|
||||||
</tr>
|
|
||||||
<tr>
|
|
||||||
<td>Text → Sprache</td>
|
|
||||||
<td><span class="badge b-green">Piper TTS (lokal)</span></td>
|
|
||||||
<td>Piper TTS (unverändert)</td>
|
|
||||||
<td>—</td>
|
|
||||||
<td>—</td>
|
|
||||||
<td><span class="badge b-nvidia">DGX</span></td>
|
|
||||||
</tr>
|
|
||||||
</table>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<!-- ═══ 3. DSGVO ═══ -->
|
|
||||||
<div class="card" style="margin-bottom:32px">
|
|
||||||
<h2>🔒 3. DSGVO — 100% Compliance durch lokale KI</h2>
|
|
||||||
|
|
||||||
<h3>Vorher vs. Nachher</h3>
|
|
||||||
<table>
|
|
||||||
<tr><th>Datenkategorie</th><th>Bisher</th><th>Neu</th></tr>
|
|
||||||
<tr>
|
|
||||||
<td><strong>Text-Inhalte (Submissions)</strong></td>
|
|
||||||
<td><span class="badge b-orange">→ DeepSeek API (China)</span></td>
|
|
||||||
<td><span class="badge b-green">→ Llama 3.1 lokal (Deutschland)</span></td>
|
|
||||||
</tr>
|
|
||||||
<tr>
|
|
||||||
<td><strong>Bilder</strong></td>
|
|
||||||
<td><span class="badge b-orange">→ Gemini API (Google USA)</span></td>
|
|
||||||
<td><span class="badge b-green">→ Llama Vision lokal (Deutschland)</span></td>
|
|
||||||
</tr>
|
|
||||||
<tr>
|
|
||||||
<td><strong>Audio/Sprachnachrichten</strong></td>
|
|
||||||
<td><span class="badge b-green">✅ Lokal (faster-whisper)</span></td>
|
|
||||||
<td><span class="badge b-green">✅ Lokal (Whisper large-v3 GPU)</span></td>
|
|
||||||
</tr>
|
|
||||||
<tr>
|
|
||||||
<td><strong>Telefonnummern</strong></td>
|
|
||||||
<td><span class="badge b-green">✅ SHA-256+Salt Hash</span></td>
|
|
||||||
<td><span class="badge b-green">✅ Unverändert</span></td>
|
|
||||||
</tr>
|
|
||||||
<tr>
|
|
||||||
<td><strong>Nutzer-Sessions</strong></td>
|
|
||||||
<td><span class="badge b-green">✅ Lokal SQLite/PostgreSQL</span></td>
|
|
||||||
<td><span class="badge b-green">✅ PostgreSQL (Hetzner)</span></td>
|
|
||||||
</tr>
|
|
||||||
<tr>
|
|
||||||
<td><strong>Medien-Dateien</strong></td>
|
|
||||||
<td><span class="badge b-green">✅ Lokaler Server</span></td>
|
|
||||||
<td><span class="badge b-green">✅ Lokal + CDN (Cloudflare)</span></td>
|
|
||||||
</tr>
|
|
||||||
</table>
|
|
||||||
|
|
||||||
<div class="callout green" style="margin-top:16px">
|
|
||||||
<strong>✅ Das Ergebnis:</strong> KEINE personenbezogenen Daten verlassen Deutschland.
|
|
||||||
KEINE KI-API von US- oder China-Anbietern. Alle Modelle sind <strong>Open-Source</strong>
|
|
||||||
und laufen auf <strong>eigener Hardware im eigenen Haus</strong>.
|
|
||||||
Dies ist die <strong>höchstmögliche DSGVO-Compliance-Stufe</strong>.
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<h3>3.1 Datenfluss — Kein Drittland-Transfer mehr</h3>
|
|
||||||
<div class="arch-box">
|
<div class="arch-box">
|
||||||
<span class="d">Nutzer (WhatsApp/Telegram) →</span> <span class="b">Hetzner Edge</span> <span class="d">(verschlüsselt, TLS)</span>
|
<span class="d"># Alle via Docker Compose auf DGX Spark:</span>
|
||||||
<span class="d"> │</span>
|
<span class="b">dgx_net</span><span class="d">: bridge (internes Docker-Netzwerk)</span>
|
||||||
<span class="d"> ├─ Telefonnummer → SHA-256+Salt Hash → DB (Hetzner)</span>
|
|
||||||
<span class="d"> ├─ Nachrichtentext → Redis Queue → </span><span class="y">🔐 WireGuard VPN</span><span class="d"> → </span><span class="g">DGX Spark</span>
|
|
||||||
<span class="d"> ├─ Bilddatei → Redis Queue → </span><span class="y">🔐 WireGuard VPN</span><span class="d"> → </span><span class="g">DGX Spark</span>
|
|
||||||
<span class="d"> ├─ Audiodatei → Redis Queue → </span><span class="y">🔐 WireGuard VPN</span><span class="d"> → </span><span class="g">DGX Spark</span>
|
|
||||||
<span class="d"> │</span>
|
|
||||||
<span class="d"> │ </span><span class="g">DGX Spark</span><span class="d"> (LOKAL, Deutschland):</span>
|
|
||||||
<span class="d"> ├─ Llama 3.1 70B → Triage-Ergebnis</span>
|
|
||||||
<span class="d"> ├─ Whisper large-v3 → Transkription</span>
|
|
||||||
<span class="d"> ├─ Llama Vision → Bild-Sicherheit</span>
|
|
||||||
<span class="d"> └─ Ergebnis → Redis Queue → </span><span class="y">VPN</span><span class="d"> → </span><span class="b">Hetzner DB</span>
|
|
||||||
<span class="d"> │</span>
|
|
||||||
<span class="d"> └─ </span><span class="w">KEIN</span><span class="d"> Datenverlassen Deutschlands. </span><span class="w">KEINE</span><span class="d"> US/China-API.</span>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<!-- ═══ 4. LEISTUNG ═══ -->
|
<span class="g">llama-swap</span><span class="d">: Port 28080 — VRAM-Orchestrator</span>
|
||||||
<div class="card" style="margin-bottom:32px">
|
<span class="g">litellm</span><span class="d">: Port 14000 — API-Gateway + Rate-Limiting</span>
|
||||||
<h2>⚡ 4. Leistungsprognose — Was schaffen die Maschinen?</h2>
|
<span class="g">vllm-qwen35b</span><span class="d">: Port — (ephemeral, via llama-swap)</span>
|
||||||
|
<span class="g">vllm-nemotron</span><span class="d">: Port — (ephemeral, via llama-swap)</span>
|
||||||
<div class="g3" style="margin-top:12px">
|
<span class="g">vllm-ministral</span><span class="d">: Port 8000 — Always-On (idle_timeout=0)</span>
|
||||||
<div class="tier-card">
|
<span class="g">redis-worker</span><span class="d">: Python-Script, pollt Redis-Queue auf Hetzner</span>
|
||||||
<div style="font-size:.85rem;color:var(--g500);margin-bottom:4px;">LLM-Triage (Llama 3.1 8B)</div>
|
|
||||||
<div class="big-num">~500<span style="font-size:1rem;color:var(--g500);">/Stunde</span></div>
|
|
||||||
<div style="font-size:.8rem;color:var(--g500);margin-top:4px;">80 tok/s, ~200 Tokens pro Triage<br>Läuft parallel auf beiden Maschinen</div>
|
|
||||||
</div>
|
|
||||||
<div class="tier-card">
|
|
||||||
<div style="font-size:.85rem;color:var(--g500);margin-bottom:4px;">Audio-Transkription</div>
|
|
||||||
<div class="big-num">~600<span style="font-size:1rem;color:var(--g500);">/Stunde</span></div>
|
|
||||||
<div style="font-size:.8rem;color:var(--g500);margin-top:4px;">Whisper large-v3 GPU<br>50–100× Echtzeit</div>
|
|
||||||
</div>
|
|
||||||
<div class="tier-card">
|
|
||||||
<div style="font-size:.85rem;color:var(--g500);margin-bottom:4px;">Bild-Sicherheit</div>
|
|
||||||
<div class="big-num">~1.200<span style="font-size:1rem;color:var(--g500);">/Stunde</span></div>
|
|
||||||
<div style="font-size:.8rem;color:var(--g500);margin-top:4px;">Llama Vision 11B<br><3s pro Bild</div>
|
|
||||||
</div>
|
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<h3>4.1 Maximale Tageskapazität (24h Dauerbetrieb)</h3>
|
<h3>5.2 Weniger Komplex, mehr Stabilität</h3>
|
||||||
<table>
|
<p>Die gesamte Docker-Compose-Konfiguration steht als fertiges GitHub-Repo bereit:<br>
|
||||||
<tr><th>Komponente</th><th>Pro Stunde</th><th>Pro Tag (24h)</th><th>Limitierender Faktor</th></tr>
|
<a href="https://github.com/mARTin-B78/dgx-spark_lite-llm_llama-swap_vllm_llama-cpp_ollama" target="_blank">github.com/mARTin-B78/dgx-spark_lite-llm_llama-swap_vllm_llama-cpp_ollama</a></p>
|
||||||
<tr><td><strong>LLM-Triage (8B)</strong></td><td>500</td><td>12.000</td><td>GPU-Durchsatz</td></tr>
|
|
||||||
<tr><td><strong>LLM-Triage (70B)</strong></td><td>150</td><td>3.600</td><td>GPU-Durchsatz (Grenzfälle)</td></tr>
|
|
||||||
<tr><td><strong>Audio-Transkription</strong></td><td>600</td><td>14.400</td><td>GPU-Durchsatz</td></tr>
|
|
||||||
<tr><td><strong>Bild-Sicherheit</strong></td><td>1.200</td><td>28.800</td><td>GPU-Durchsatz</td></tr>
|
|
||||||
<tr><td><strong>Chatbot-Anfragen</strong></td><td>2.000+</td><td>48.000+</td><td>Kontext-Management</td></tr>
|
|
||||||
<tr style="font-weight:700;background:var(--bsn-bg);">
|
|
||||||
<td><strong>GESAMT (realistisch)</strong></td><td></td><td><strong>40.000–80.000 Nutzer/Tag</strong></td><td>Edge-Server wird Bottleneck VOR KI-Cluster</td>
|
|
||||||
</tr>
|
|
||||||
</table>
|
|
||||||
|
|
||||||
<div class="callout green">
|
<h3>5.3 Redis Worker (Python — läuft auf DGX und Strix)</h3>
|
||||||
<strong>🚀 Fazit:</strong> Der <strong>KI-Cluster ist ÜBERDIMENSIONIERT</strong> für 40k Nutzer.
|
<div class="arch-box">
|
||||||
Der Edge-Server (Hetzner) wird ZUERST zum Bottleneck — nicht die KI-Maschinen zu Hause.
|
<span class="d"># Kernlogik — sequentiell, nie parallel</span>
|
||||||
Mit 192 GB VRAM kombiniert hast du Reserven für <strong>100.000+ Nutzer/Tag</strong>.
|
<span class="b">while</span> True:
|
||||||
|
<span class="d"># BLOCKING: Wartet bis Aufgabe da ist</span>
|
||||||
|
task = redis.brpop(<span class="y">"bsn:tasks"</span>, timeout=<span class="g">30</span>)
|
||||||
|
<span class="b">if not</span> task:
|
||||||
|
<span class="b">continue</span> <span class="d"># Heartbeat</span>
|
||||||
|
|
||||||
|
task_data = json.loads(task)
|
||||||
|
task_type = task_data[<span class="y">"type"</span>] <span class="d"># triage, transcribe, safety, chat</span>
|
||||||
|
|
||||||
|
<span class="d"># Wichtig: Ergebnis in CALLBACK-Key schreiben</span>
|
||||||
|
result = <span class="b">process_task</span>(task_data) <span class="d"># via LiteLLM API</span>
|
||||||
|
redis.setex(
|
||||||
|
<span class="y">f"bsn:result:{task_data['id']}"</span>,
|
||||||
|
<span class="g">300</span>, <span class="d"># 5 Min TTL</span>
|
||||||
|
json.dumps(result)
|
||||||
|
)
|
||||||
|
|
||||||
|
<span class="d"># Edge-Server pollt bsn:result:{id} und updated DB</span>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
|
||||||
|
|
||||||
<!-- ═══ 5. UMSETZUNGSPLAN ═══ -->
|
<div class="callout">
|
||||||
<div class="card" style="margin-bottom:32px">
|
<strong>⏱️ Latenz:</strong> 90% der Triage-Aufgaben werden in <strong>unter 3 Sekunden</strong> verarbeitet
|
||||||
<h2>🔧 5. Umsetzungsplan — Schritt für Schritt</h2>
|
(57 tok/s × ~100 Tokens Output). Chatbot (Ministral 8B) antwortet in <strong>unter 500ms</strong>.
|
||||||
|
|
||||||
<h3>Phase 1: Edge-Server vorbereiten (1 Woche)</h3>
|
|
||||||
<table>
|
|
||||||
<tr><th>#</th><th>Schritt</th><th>Details</th><th>Zeit</th></tr>
|
|
||||||
<tr><td>1</td><td>Gunicorn deployen</td><td>4 Workers, systemd-Service</td><td>30 Min</td></tr>
|
|
||||||
<tr><td>2</td><td>Nginx + PostgreSQL</td><td>Reverse Proxy, pgBouncer, DB-Migration</td><td>2 Std</td></tr>
|
|
||||||
<tr><td>3</td><td>Redis installieren</td><td>Queue + Cache, persistente Speicherung</td><td>30 Min</td></tr>
|
|
||||||
<tr><td>4</td><td>WireGuard einrichten</td><td>Server-seitig: VPN-Endpunkt, Port 51820</td><td>1 Std</td></tr>
|
|
||||||
<tr><td>5</td><td>Redis Queue API definieren</td><td>Aufgabenformat: {task_id, type, payload, callback_topic}</td><td>2 Std</td></tr>
|
|
||||||
</table>
|
|
||||||
|
|
||||||
<h3>Phase 2: KI-Cluster einrichten (1 Woche)</h3>
|
|
||||||
<table>
|
|
||||||
<tr><th>#</th><th>Schritt</th><th>Details</th><th>Zeit</th></tr>
|
|
||||||
<tr><td>1</td><td>Ollama installieren (DGX)</td><td>Llama 3.1 8B + 70B + Llama Vision 11B</td><td>2 Std</td></tr>
|
|
||||||
<tr><td>2</td><td>Whisper large-v3 installieren</td><td>faster-whisper + CUDA auf DGX</td><td>1 Std</td></tr>
|
|
||||||
<tr><td>3</td><td>Ollama (Strix)</td><td>Mistral Large + Backup-Modelle</td><td>2 Std</td></tr>
|
|
||||||
<tr><td>4</td><td>WireGuard Client</td><td>VPN zu Hetzner, nur Redis-Port</td><td>30 Min</td></tr>
|
|
||||||
<tr><td>5</td><td>Redis Worker schreiben</td><td>Python-Script: Queue pollen → Ollama/Whisper → Ergebnis zurückschreiben</td><td>4 Std</td></tr>
|
|
||||||
<tr><td>6</td><td>Healthchecks + systemd</td><td>Auto-Restart, Heartbeat an Edge-Server</td><td>1 Std</td></tr>
|
|
||||||
</table>
|
|
||||||
|
|
||||||
<h3>Phase 3: Integration & Tests (3 Tage)</h3>
|
|
||||||
<table>
|
|
||||||
<tr><th>#</th><th>Schritt</th><th>Details</th><th>Zeit</th></tr>
|
|
||||||
<tr><td>1</td><td>Flask-Code anpassen</td><td>Triage-Calls ersetzen: API → Redis-Queue-Push + Poll-Result</td><td>4 Std</td></tr>
|
|
||||||
<tr><td>2</td><td>End-to-End-Test</td><td>WhatsApp → Webhook → Queue → DGX → Ergebnis → DB → Frontend</td><td>2 Std</td></tr>
|
|
||||||
<tr><td>3</td><td>Load-Test</td><td>100 simulierte Requests parallel</td><td>2 Std</td></tr>
|
|
||||||
<tr><td>4</td><td>Fallback-Test</td><td>DGX aus → Strix übernimmt automatisch</td><td>1 Std</td></tr>
|
|
||||||
<tr><td>5</td><td>Monitoring</td><td>Prometheus + Grafana für KI-Cluster-Metriken</td><td>2 Std</td></tr>
|
|
||||||
</table>
|
|
||||||
|
|
||||||
<div class="callout" style="margin-top:16px">
|
|
||||||
<strong>⏱️ Gesamtaufwand: ~3 Wochen.</strong> Danach: 100% DSGVO, 0 € KI-API-Kosten,
|
|
||||||
Kapazität für 100.000+ Nutzer/Tag.
|
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<!-- ═══ 6. KOSTEN ═══ -->
|
<!-- ═══ 6. KOSTEN ═══ -->
|
||||||
<div class="card" style="margin-bottom:32px">
|
<div class="card" style="margin-bottom:32px">
|
||||||
<h2>💰 6. Kostenvergleich — Vorher vs. Nachher</h2>
|
<h2>💰 6. Kosten — Final</h2>
|
||||||
|
|
||||||
<table>
|
<table>
|
||||||
<tr><th>Posten</th><th>Bisher (Cloud-APIs)</th><th>Kosten/Monat</th><th>Neu (Lokal)</th><th>Kosten/Monat</th></tr>
|
<tr><th>Posten</th><th>Monatlich</th><th>Anmerkung</th></tr>
|
||||||
<tr>
|
<tr><td>Hetzner CX32 (Edge-Server)</td><td><strong>~40 €</strong></td><td>4 vCPU, 16 GB, PostgreSQL, Redis, Nginx</td></tr>
|
||||||
<td><strong>LLM-Triage</strong></td>
|
<tr><td>Strom DGX Spark (24/7)</td><td><strong>~60 €</strong></td><td>~300W Dauerlast, 0,30 €/kWh</td></tr>
|
||||||
<td>DeepSeek V4 Flash (datenhimmel)</td>
|
<tr><td>Strom Strix Halo (idle, Failover)</td><td><strong>~10 €</strong></td><td>Nur an, kein Load. ~50W idle.</td></tr>
|
||||||
<td>~30 €</td>
|
<tr><td>Internet (VPN-Traffic)</td><td><strong>0 €</strong></td><td>Wenige MB/Tag, nur Redis-Queue-Daten</td></tr>
|
||||||
<td>Llama 3.1 (lokal)</td>
|
<tr><td>KI-API-Kosten</td><td><strong>0 €</strong></td><td>Alles lokal!</td></tr>
|
||||||
<td><span class="badge b-green">0 €</span></td>
|
<tr style="font-weight:700;background:var(--bsn-bg)">
|
||||||
</tr>
|
<td><strong>GESAMT</strong></td><td><strong>~110 €/Monat</strong></td><td>100% DSGVO, 0 API, unbegrenzt skalierbar</td>
|
||||||
<tr>
|
|
||||||
<td><strong>Bild-Sicherheit</strong></td>
|
|
||||||
<td>Gemini Flash (Google)</td>
|
|
||||||
<td>~10 €</td>
|
|
||||||
<td>Llama Vision (lokal)</td>
|
|
||||||
<td><span class="badge b-green">0 €</span></td>
|
|
||||||
</tr>
|
|
||||||
<tr>
|
|
||||||
<td><strong>Audio→Text</strong></td>
|
|
||||||
<td>faster-whisper CPU</td>
|
|
||||||
<td>0 €</td>
|
|
||||||
<td>Whisper GPU (lokal)</td>
|
|
||||||
<td><span class="badge b-green">0 €</span></td>
|
|
||||||
</tr>
|
|
||||||
<tr>
|
|
||||||
<td><strong>Chatbot</strong></td>
|
|
||||||
<td>DeepSeek V4 Flash</td>
|
|
||||||
<td>~20 €</td>
|
|
||||||
<td>Llama 3.1 (lokal)</td>
|
|
||||||
<td><span class="badge b-green">0 €</span></td>
|
|
||||||
</tr>
|
|
||||||
<tr>
|
|
||||||
<td><strong>Edge-Server</strong></td>
|
|
||||||
<td>CX22 (4 vCPU, 8 GB)</td>
|
|
||||||
<td>~20 €</td>
|
|
||||||
<td>CX32 (4 vCPU, 16 GB)</td>
|
|
||||||
<td>~40 €</td>
|
|
||||||
</tr>
|
|
||||||
<tr>
|
|
||||||
<td><strong>Strom (KI-Cluster)</strong></td>
|
|
||||||
<td>—</td>
|
|
||||||
<td>—</td>
|
|
||||||
<td>DGX + Strix 24/7</td>
|
|
||||||
<td>~80–120 €</td>
|
|
||||||
</tr>
|
|
||||||
<tr>
|
|
||||||
<td><strong>Internet (Zuhause)</strong></td>
|
|
||||||
<td>—</td>
|
|
||||||
<td>—</td>
|
|
||||||
<td>VPN-Traffic (gering)</td>
|
|
||||||
<td>~0 €</td>
|
|
||||||
</tr>
|
|
||||||
<tr style="font-weight:700;background:var(--bsn-bg);">
|
|
||||||
<td><strong>GESAMT</strong></td>
|
|
||||||
<td></td>
|
|
||||||
<td><strong>~80 €/Monat</strong></td>
|
|
||||||
<td></td>
|
|
||||||
<td><strong>~120–160 €/Monat</strong></td>
|
|
||||||
</tr>
|
</tr>
|
||||||
</table>
|
</table>
|
||||||
|
|
||||||
<div class="callout green">
|
|
||||||
<strong>💡 Wichtig:</strong> Die Hardware (DGX Spark + Strix Halo) ist <strong>bereits vorhanden</strong> —
|
|
||||||
das sind versunkene Kosten. Die einzigen laufenden Mehrkosten sind <strong>Strom (~80–120 €/Monat)</strong>.
|
|
||||||
Dafür entfallen <strong>ALLE API-Kosten (~60 €/Monat)</strong>. Effektive Mehrkosten: <strong>~20–60 €/Monat</strong>
|
|
||||||
für 100% DSGVO-Compliance und massiv höhere Kapazität.
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<div class="callout" style="margin-top:12px;">
|
|
||||||
<strong>🔮 Bei 40k Nutzern/Tag:</strong> Die API-Kosten wären auf <strong>200–400 €/Monat</strong> gestiegen
|
|
||||||
(mehr Triage, mehr Chatbot-Anfragen). Mit der lokalen Lösung bleiben die Kosten
|
|
||||||
<strong>konstant bei ~120–160 €/Monat</strong> — unabhängig vom Wachstum.
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<!-- ═══ 7. REDUNDANZ ═══ -->
|
|
||||||
<div class="card" style="margin-bottom:32px">
|
|
||||||
<h2>🔄 7. Redundanz & Ausfallsicherheit</h2>
|
|
||||||
|
|
||||||
<table>
|
|
||||||
<tr><th>Szenario</th><th>Auswirkung</th><th>Automatische Reaktion</th></tr>
|
|
||||||
<tr>
|
|
||||||
<td><strong>DGX Spark fällt aus</strong></td>
|
|
||||||
<td>Primäre KI fällt aus</td>
|
|
||||||
<td>Strix Halo übernimmt ALLE Aufgaben automatisch. Redis Worker #2 springt ein.</td>
|
|
||||||
</tr>
|
|
||||||
<tr>
|
|
||||||
<td><strong>Strix Halo fällt aus</strong></td>
|
|
||||||
<td>Zweitmeinung + Backup weg</td>
|
|
||||||
<td>DGX macht alles. Kein Diversity-Check, aber Triage läuft normal weiter.</td>
|
|
||||||
</tr>
|
|
||||||
<tr>
|
|
||||||
<td><strong>Beide fallen aus</strong></td>
|
|
||||||
<td>Keine KI-Verarbeitung</td>
|
|
||||||
<td>Edge-Server sammelt weiter in Redis Queue. Kein Datenverlust. Sobald eine Maschine zurückkommt → Batch-Verarbeitung.</td>
|
|
||||||
</tr>
|
|
||||||
<tr>
|
|
||||||
<td><strong>Edge-Server fällt aus</strong></td>
|
|
||||||
<td>Kein Public-Facing</td>
|
|
||||||
<td>KI-Cluster läuft weiter (Wartungsmodus). Wiederherstellung via Cloudflare Failover oder manuellem Server-Neustart.</td>
|
|
||||||
</tr>
|
|
||||||
<tr>
|
|
||||||
<td><strong>Internet zu Hause fällt aus</strong></td>
|
|
||||||
<td>VPN-Verbindung weg</td>
|
|
||||||
<td>Edge-Server speichert in Queue. Nach Wiederherstellung → Batch. ODER: Fallback auf Cloud-API für kritische Zeit.</td>
|
|
||||||
</tr>
|
|
||||||
</table>
|
|
||||||
|
|
||||||
<div class="callout">
|
|
||||||
<strong>💡 Optional:</strong> Einen <strong>zweiten Edge-Server</strong> (CX22, ~6 €/Monat) als
|
|
||||||
Failover bei verschiedenen Hetzner-Standorten (Falkenstein + Nürnberg) für 99,9% Uptime.
|
|
||||||
</div>
|
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<!-- ═══ FINAL ═══ -->
|
<!-- ═══ FINAL ═══ -->
|
||||||
<div class="card" style="background:linear-gradient(135deg, #1a1a2e, #16213e);color:#fff;border:2px solid var(--bsn);">
|
<div class="card" style="background:linear-gradient(135deg,#0d1117,#16213e);color:#fff;border:2px solid var(--bsn);">
|
||||||
<h2 style="color:#fff;">🏆 Zusammenfassung — Das ist die Endgame-Architektur</h2>
|
<h2 style="color:#fff;">✅ Korrigierte Architektur — Zusammenfassung</h2>
|
||||||
|
|
||||||
<div class="g3" style="margin-top:16px">
|
<table style="color:#c9d1d9;margin-top:16px;">
|
||||||
<div>
|
<tr><td style="color:#ff6b6b;">❌ Vorher falsch</td><td style="color:#7ee787;">✅ Jetzt korrekt</td></tr>
|
||||||
<div style="font-size:2.5rem;font-weight:800;color:#7ee787;">100%</div>
|
<tr><td>Llama 3.1 70B (Dense → langsam)</td><td><strong>Qwen 3.5 35B MoE</strong> (57 tok/s, 12× schneller)</td></tr>
|
||||||
<div style="font-size:.9rem;opacity:.8;">DSGVO-Compliance<br>Keine Daten verlassen DE</div>
|
<tr><td>Parallele Requests → Freeze</td><td><strong>Sequentiell via Queue</strong> (Redis BRPOP → 1 Task)</td></tr>
|
||||||
</div>
|
<tr><td>OpenWebUI als Backend</td><td><strong>LiteLLM → llama-swap → vLLM</strong> (Orchestrierung)</td></tr>
|
||||||
<div>
|
<tr><td>2 Modelle gleichzeitig → Swap</td><td><strong>llama-swap</strong> lädt/entlädt (nur 1 Modell aktiv)</td></tr>
|
||||||
<div style="font-size:2.5rem;font-weight:800;color:#79c0ff;">192 GB</div>
|
<tr><td>Beide Maschinen parallel aktiv</td><td><strong>DGX primär, Strix Failover</strong> (spart Strom)</td></tr>
|
||||||
<div style="font-size:.9rem;opacity:.8;">VRAM kombiniert<br>DGX Spark + Strix Halo</div>
|
<tr><td>Gemini + DeepSeek APIs (DSGVO?)</td><td><strong>Qwen3-VL + Ministral</strong> (100% lokal, 100% DSGVO)</td></tr>
|
||||||
</div>
|
|
||||||
<div>
|
|
||||||
<div style="font-size:2.5rem;font-weight:800;color:#f778ba;">100k+</div>
|
|
||||||
<div style="font-size:.9rem;opacity:.8;">Nutzer/Tag möglich<br>KI-Cluster ist ÜBERdimensioniert</div>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<table style="color:#c9d1d9;margin-top:24px;">
|
|
||||||
<tr><td style="color:#8b949e;">Edge-Server</td><td>Hetzner CX32 · 40 €/Monat · Public-Facing + DB</td></tr>
|
|
||||||
<tr><td style="color:#8b949e;">Primäre KI</td><td>DGX Spark · Llama 3.1 70B + Whisper large-v3 + Llama Vision</td></tr>
|
|
||||||
<tr><td style="color:#8b949e;">Sekundäre KI</td><td>Strix Halo · Mistral Large + Failover + Diversity</td></tr>
|
|
||||||
<tr><td style="color:#8b949e;">Verbindung</td><td>WireGuard VPN · Verschlüsselt · Redis Queue</td></tr>
|
|
||||||
<tr><td style="color:#8b949e;">DSGVO</td><td><span style="color:#7ee787;">✅ 100% — Alle KI auf eigener Hardware in Deutschland</span></td></tr>
|
|
||||||
<tr><td style="color:#8b949e;">Kosten/Monat</td><td><span style="color:#79c0ff;">~120–160 €</span> (inkl. Strom, null API-Kosten)</td></tr>
|
|
||||||
<tr><td style="color:#8b949e;">Kapazität</td><td><span style="color:#f778ba;">100.000+ Nutzer/Tag</span> (Edge-Server zuerst Bottleneck)</td></tr>
|
|
||||||
</table>
|
</table>
|
||||||
|
|
||||||
<p style="margin-top:24px;font-size:.82rem;color:#8b949e;text-align:center;">
|
<div style="margin-top:24px;padding:16px;background:rgba(255,255,255,.05);border-radius:var(--r);">
|
||||||
📅 19. Juni 2026 · Cody (Coding-Agent) · Architektur mit DGX Spark & Strix Halo<br>
|
<strong style="color:#7ee787;">🎯 Das Wichtigste:</strong> Das System kann NUR EINEN Request gleichzeitig
|
||||||
Datei: <code>/home/hermes/workspace/bsn-chatbot/docs/skalierungsplan-10k.html</code>
|
auf dem DGX Spark verarbeiten — aber es verarbeitet ihn <strong>extrem schnell</strong> (57 tok/s).
|
||||||
|
Bei 10.000 Nutzern/Tag sind das ~400 Triage-Aufgaben/Tag = ~20/Stunde = <strong>genug Zeit</strong>.
|
||||||
|
Für Chatbot-Antworten bleibt Ministral 8B immer geladen und antwortet in Millisekunden.
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<p style="margin-top:20px;font-size:.82rem;color:#8b949e;text-align:center;">
|
||||||
|
📅 19. Juni 2026 · Cody (Coding-Agent) · Korrigierte Architektur v3.0<br>
|
||||||
|
Quellen: NVIDIA Developer Forums DGX Spark Benchmarks (April 2026), Nutzer-Feedback
|
||||||
</p>
|
</p>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user