docs: Implementierungsplan v5.1 — extern geprüft, umsetzungsreif
Quality Gate / 🛡️ Lint + Type Check + Tests (push) Failing after 14m40s
Quality Gate / 🛡️ Lint + Type Check + Tests (push) Failing after 14m40s
This commit is contained in:
Hermes Agent
@@ -0,0 +1,871 @@
|
|||||||
|
<!DOCTYPE html>
|
||||||
|
<html lang="de">
|
||||||
|
<head><meta charset="UTF-8"><meta name="viewport" content="width=device-width,initial-scale=1.0">
|
||||||
|
<title>BSN-Chatsystem v2.1 — Implementierungsplan (geprüft)</title>
|
||||||
|
<style>
|
||||||
|
:root{--blue:#20228a;--bg:#f9fafb;--text:#1f2937;--card:#fff;--muted:#6b7280;--green:#065f46;--red:#991b1b;--orange:#92400e;--purple:#7c3aed}
|
||||||
|
*{box-sizing:border-box;margin:0;padding:0}
|
||||||
|
body{font-family:system-ui,sans-serif;background:var(--bg);color:var(--text);line-height:1.6;padding:1rem;max-width:1000px;margin:0 auto}
|
||||||
|
h1{color:var(--blue);font-size:1.6rem;margin:1.5rem 0 1rem;border-bottom:3px solid var(--blue);padding-bottom:.5rem}
|
||||||
|
h2{color:var(--blue);font-size:1.2rem;margin:1.8rem 0 .6rem;border-bottom:1px solid #e5e7eb;padding-bottom:.3rem}
|
||||||
|
h3{font-size:1.05rem;margin:1.2rem 0 .4rem;color:#374151}
|
||||||
|
.card{background:var(--card);border-radius:8px;padding:1.2rem;margin:.8rem 0;box-shadow:0 1px 3px rgba(0,0,0,.06)}
|
||||||
|
table{width:100%;border-collapse:collapse;margin:.8rem 0;font-size:.85rem}
|
||||||
|
th,td{padding:.5rem .7rem;text-align:left;border-bottom:1px solid #e5e7eb;vertical-align:top}
|
||||||
|
th{background:var(--blue);color:#fff;font-weight:600}
|
||||||
|
.badge{display:inline-block;padding:2px 7px;border-radius:3px;font-size:.72rem;font-weight:700}
|
||||||
|
.badge-ok{background:#d1fae5;color:var(--green)}
|
||||||
|
.badge-warn{background:#fef3c7;color:var(--orange)}
|
||||||
|
.badge-fail{background:#fee2e2;color:var(--red)}
|
||||||
|
.badge-fix{background:#ede9fe;color:var(--purple)}
|
||||||
|
pre{background:#1e293b;color:#e2e8f0;padding:1rem;border-radius:6px;font-size:.78rem;overflow-x:auto;margin:.5rem 0;line-height:1.4}
|
||||||
|
code{background:#f1f5f9;padding:1px 4px;border-radius:3px;font-size:.85em}
|
||||||
|
.toc{background:var(--card);border-radius:8px;padding:1.2rem;margin:.8rem 0;box-shadow:0 1px 3px rgba(0,0,0,.06)}
|
||||||
|
.toc ol{padding-left:1.5rem}
|
||||||
|
.toc li{margin:.3rem 0}
|
||||||
|
.note{background:#eff6ff;border-left:3px solid var(--blue);padding:.8rem 1rem;margin:.8rem 0;border-radius:0 6px 6px 0;font-size:.88rem}
|
||||||
|
.warn{background:#fef3c7;border-left:3px solid var(--orange);padding:.8rem 1rem;margin:.8rem 0;border-radius:0 6px 6px 0;font-size:.88rem}
|
||||||
|
.fix{background:#ede9fe;border-left:3px solid var(--purple);padding:.8rem 1rem;margin:.8rem 0;border-radius:0 6px 6px 0;font-size:.88rem}
|
||||||
|
</style>
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
|
||||||
|
<h1>🚀 BSN-Chatsystem v2.1 — Implementierungsplan (extern geprüft)</h1>
|
||||||
|
|
||||||
|
<div class="card">
|
||||||
|
<table>
|
||||||
|
<tr><td><strong>Datum</strong></td><td>14. Juli 2026</td></tr>
|
||||||
|
<tr><td><strong>Version</strong></td><td>5.1 — Externes Review eingearbeitet</td></tr>
|
||||||
|
<tr><td><strong>Autor</strong></td><td>Cody (Hermes Agent), mit Korrekturen von externem Prüfer</td></tr>
|
||||||
|
<tr><td><strong>Repository</strong></td><td><code>git@git.datenhimmel.work:danielkrause/BSN-Chatsystem.git</code></td></tr>
|
||||||
|
<tr><td><strong>Status</strong></td><td>✅ Umsetzungsreif</td></tr>
|
||||||
|
</table>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="toc">
|
||||||
|
<h3>📑 Inhalt</h3>
|
||||||
|
<ol>
|
||||||
|
<li><a href="#ziel">Ziel & Rahmenbedingungen</a></li>
|
||||||
|
<li><a href="#server">Server</a></li>
|
||||||
|
<li><a href="#architektur">Architektur</a></li>
|
||||||
|
<li><a href="#datenfluss">Datenfluss</a></li>
|
||||||
|
<li><a href="#komponenten">Komponenten</a></li>
|
||||||
|
<li><a href="#datenbank">Datenbank</a></li>
|
||||||
|
<li><a href="#api">API-Endpunkte</a></li>
|
||||||
|
<li><a href="#queue">Queue-System</a></li>
|
||||||
|
<li><a href="#llm">LLM (OpenRouter Mistral 4)</a></li>
|
||||||
|
<li><a href="#caching">Cloudflare-Cache + Access</a></li>
|
||||||
|
<li><a href="#installation">Installation</a></li>
|
||||||
|
<li><a href="#konfiguration">Konfigurationsdateien</a></li>
|
||||||
|
<li><a href="#systemd">systemd-Services</a></li>
|
||||||
|
<li><a href="#monitoring">Monitoring</a></li>
|
||||||
|
<li><a href="#security">Sicherheit</a></li>
|
||||||
|
<li><a href="#backup">Backup & Recovery</a></li>
|
||||||
|
<li><a href="#dsgvo">DSGVO-Löschroutine</a></li>
|
||||||
|
<li><a href="#golive">Go-Live</a></li>
|
||||||
|
<li><a href="#checkliste">Checkliste (10 Punkte)</a></li>
|
||||||
|
<li><a href="#fallback">Fallback</a></li>
|
||||||
|
<li><a href="#codeaenderungen">Code-Änderungen</a></li>
|
||||||
|
<li><a href="#skalierung">Skalierung (1000/h)</a></li>
|
||||||
|
<li><a href="#zukunft">Zukunft</a></li>
|
||||||
|
<li><a href="#changelog">Änderungsprotokoll v5→v5.1</a></li>
|
||||||
|
</ol>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- ================================================================== -->
|
||||||
|
<h2 id="ziel">1. Ziel & Rahmenbedingungen</h2>
|
||||||
|
<div class="card">
|
||||||
|
<p><strong>Ziel:</strong> Das BSN-Chatsystem (<code>chat.datenhimmel.work</code>) 1:1 funktional auf dedizierten dogado-Server migrieren, Architektur produktionsreif.</p>
|
||||||
|
<table>
|
||||||
|
<tr><th>Rahmenbedingung</th><th>Wert</th></tr>
|
||||||
|
<tr><td>Funktionsumfang</td><td>100% identisch zum aktuellen System</td></tr>
|
||||||
|
<tr><td>LLM</td><td>OpenRouter → Mistral 4 (europäisch, DSGVO)</td></tr>
|
||||||
|
<tr><td>TTS</td><td>Deaktiviert (später optional)</td></tr>
|
||||||
|
<tr><td>Queue</td><td>Eigener systemd-Service (nicht Daemon-Thread)</td></tr>
|
||||||
|
<tr><td>Skalierungsziel</td><td>~1000 Nutzer/h</td></tr>
|
||||||
|
<tr><td>Admin-Schutz</td><td>Cloudflare Access (vor Go-Live)</td></tr>
|
||||||
|
<tr><td>Backup</td><td>Offsite via rsync zum Hermes-Server</td></tr>
|
||||||
|
<tr><td>Service-User</td><td><code>bsn</code> (nicht root)</td></tr>
|
||||||
|
</table>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- ================================================================== -->
|
||||||
|
<h2 id="server">2. Server</h2>
|
||||||
|
<div class="card">
|
||||||
|
<table>
|
||||||
|
<tr><th>Eigenschaft</th><th>Wert</th></tr>
|
||||||
|
<tr><td>IP</td><td>89.22.110.107 (dogado GmbH)</td></tr>
|
||||||
|
<tr><td>CPU</td><td>Xeon E5-2680 v3, 6 Cores @ 2.50 GHz</td></tr>
|
||||||
|
<tr><td>RAM</td><td>12 GB</td></tr>
|
||||||
|
<tr><td>Disk</td><td>492 GB SSD</td></tr>
|
||||||
|
<tr><td>OS Ziel</td><td>Ubuntu 24.04 LTS minimal (KEIN Plesk)</td></tr>
|
||||||
|
<tr><td>Offene Ports</td><td>NUR 22 (SSH), 80, 443</td></tr>
|
||||||
|
</table>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- ================================================================== -->
|
||||||
|
<h2 id="architektur">3. Architektur</h2>
|
||||||
|
<div class="card">
|
||||||
|
<pre style="font-size:.72rem">
|
||||||
|
┌──────────────────────────────────────────────────┐
|
||||||
|
│ 🌍 Cloudflare │
|
||||||
|
│ Cache Rules: / + /beitrag/* → 5min │
|
||||||
|
│ Bypass: /api/*, /webhook, /admin, /telegram │
|
||||||
|
│ Access (OTP): /admin/* │
|
||||||
|
│ DDoS, WAF, weltweit Edge │
|
||||||
|
└────────┬─────────────────────────────────────────┘
|
||||||
|
│ cloudflared Tunnel → localhost:80
|
||||||
|
┌────────▼─────────────────────────────────────────┐
|
||||||
|
│ nginx Reverse-Proxy :80 │
|
||||||
|
│ limit_req_zone im HTTP-Kontext │
|
||||||
|
│ /static/* → direkt von Disk │
|
||||||
|
│ /media/* → direkt von Disk │
|
||||||
|
│ proxy_pass → gunicorn :8000 │
|
||||||
|
│ KEIN nginx-Cache (macht Cloudflare) │
|
||||||
|
└────────┬─────────────────────────────────────────┘
|
||||||
|
│ HTTP :8000
|
||||||
|
┌────────▼──────────┐ ┌─────────────────────────┐
|
||||||
|
│ bsn-chatbot │ │ bsn-worker │
|
||||||
|
│ gunicorn 4w×2t │ │ eigener systemd- │
|
||||||
|
│ wsgi:app │ │ Service │
|
||||||
|
│ User: bsn │ │ pollt submission_queue │
|
||||||
|
│ Flask-Routen │ │ Whisper 1× geladen │
|
||||||
|
└────────┬──────────┘ │ kein Recycling mid- │
|
||||||
|
│ │ Verarbeitung │
|
||||||
|
│ └───────────┬─────────────┘
|
||||||
|
│ │
|
||||||
|
┌────────▼───────────────────────────▼─────────────┐
|
||||||
|
│ SQLite (WAL-Mode) │
|
||||||
|
│ bsn_intake.db │
|
||||||
|
└──────────────────────────────────────────────────┘
|
||||||
|
|
||||||
|
Extern: OpenRouter API (Mistral 4), Meta Cloud API (WhatsApp send)</pre>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- ================================================================== -->
|
||||||
|
<h2 id="datenfluss">4. Datenfluss</h2>
|
||||||
|
<div class="card">
|
||||||
|
<h3>READ (~95%)</h3>
|
||||||
|
<pre>User → Cloudflare Edge (Cache HIT, 85%) → <50ms → FERTIG
|
||||||
|
User → Cloudflare (Cache MISS) → nginx → gunicorn → SQLite → HTML
|
||||||
|
→ Cloudflare cached für 5min → nächster Request = HIT
|
||||||
|
⚠️ Flask DARF KEIN Set-Cookie auf / + /beitrag/* senden!</pre>
|
||||||
|
|
||||||
|
<h3>WRITE (~5%)</h3>
|
||||||
|
<pre>WhatsApp → Meta → POST /whatsapp/webhook → nginx → gunicorn:
|
||||||
|
1. Signatur-Prüfung (HMAC) → 403 bei Fehler
|
||||||
|
2. INSERT INTO submission_queue → 200 OK in <10ms
|
||||||
|
|
||||||
|
bsn-worker (eigener Prozess, pollt alle 2s):
|
||||||
|
1. SELECT pending LIMIT 3
|
||||||
|
2. UPDATE status=processing
|
||||||
|
3. Media-Download + Transkription (Whisper) + LLM-Triage
|
||||||
|
4. INSERT INTO submissions
|
||||||
|
5. UPDATE status=completed</pre>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- ================================================================== -->
|
||||||
|
<h2 id="komponenten">5. Komponenten</h2>
|
||||||
|
<div class="card">
|
||||||
|
<table>
|
||||||
|
<tr><th>Komponente</th><th>Technologie</th><th>Zweck</th></tr>
|
||||||
|
<tr><td>OS</td><td>Ubuntu 24.04 LTS</td><td>Minimal, kein Plesk</td></tr>
|
||||||
|
<tr><td>Python</td><td>3.13 (deadsnakes PPA)</td><td></td></tr>
|
||||||
|
<tr><td>WSGI</td><td>gunicorn 23.x</td><td>4 Worker × 2 Threads, User: bsn</td></tr>
|
||||||
|
<tr><td>Reverse-Proxy</td><td>nginx 1.24+</td><td>Buffering, Static, Rate-Limit</td></tr>
|
||||||
|
<tr><td>Framework</td><td>Flask 3.x</td><td>app.py (angepasst)</td></tr>
|
||||||
|
<tr><td>DB</td><td>SQLite 3.45+ (WAL)</td><td>bsn_intake.db</td></tr>
|
||||||
|
<tr><td>Queue-Worker</td><td>Eigener systemd-Service</td><td>bsn-worker.service</td></tr>
|
||||||
|
<tr><td>LLM</td><td>OpenRouter Mistral 4</td><td>Triage + Chat-Assistant</td></tr>
|
||||||
|
<tr><td>Transkription</td><td>faster-whisper base</td><td>1× geladen, kein Reload</td></tr>
|
||||||
|
<tr><td>Bild</td><td>Pillow 11.x</td><td>Resize 1920×1080</td></tr>
|
||||||
|
<tr><td>Tunnel</td><td>cloudflared</td><td>Gleiche Credentials auf beiden Servern</td></tr>
|
||||||
|
<tr><td>Prozess</td><td>systemd</td><td>3 Services: nginx, bsn-chatbot, bsn-worker</td></tr>
|
||||||
|
<tr><td>Firewall</td><td>ufw</td><td>Nur 22, 80, 443</td></tr>
|
||||||
|
</table>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- ================================================================== -->
|
||||||
|
<h2 id="datenbank">6. Datenbank</h2>
|
||||||
|
<div class="card">
|
||||||
|
<h3>submissions (Haupttabelle)</h3>
|
||||||
|
<pre>CREATE TABLE submissions (
|
||||||
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||||
|
channel TEXT NOT NULL, -- whatsapp, telegram, web
|
||||||
|
sender_id TEXT NOT NULL,
|
||||||
|
sender_name TEXT,
|
||||||
|
type TEXT NOT NULL, -- text, image, audio, video
|
||||||
|
content TEXT,
|
||||||
|
media_path TEXT,
|
||||||
|
media_publish_flags TEXT,
|
||||||
|
status TEXT DEFAULT 'new', -- new, done, published, flagged, gdpr_deleted
|
||||||
|
category TEXT,
|
||||||
|
summary TEXT,
|
||||||
|
publish_name TEXT,
|
||||||
|
halle TEXT,
|
||||||
|
spiel_titel TEXT,
|
||||||
|
display_text TEXT,
|
||||||
|
triage_tier INTEGER, -- 1=🟢, 2=🟡, 3=🔴
|
||||||
|
triage_reasons TEXT, -- JSON-Array
|
||||||
|
triage_at TEXT,
|
||||||
|
auto_response_sent INTEGER DEFAULT 0,
|
||||||
|
public INTEGER DEFAULT 0,
|
||||||
|
published_at TEXT,
|
||||||
|
deleted_at TEXT, -- Soft-Delete für DSGVO-Löschroutine
|
||||||
|
created_at TEXT DEFAULT (datetime('now','localtime')),
|
||||||
|
-- weitere Felder: email, phone, likes, views
|
||||||
|
);</pre>
|
||||||
|
|
||||||
|
<h3>submission_queue (Worker-Queue)</h3>
|
||||||
|
<pre>CREATE TABLE submission_queue (
|
||||||
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||||
|
channel TEXT NOT NULL,
|
||||||
|
message_id TEXT UNIQUE NOT NULL, -- WhatsApp msg ID / Telegram update_id
|
||||||
|
payload TEXT NOT NULL, -- JSON: Webhook-Body
|
||||||
|
status TEXT DEFAULT 'pending', -- pending, processing, completed, failed
|
||||||
|
attempt_count INTEGER DEFAULT 0,
|
||||||
|
last_error TEXT,
|
||||||
|
created_at TEXT DEFAULT (datetime('now','localtime')),
|
||||||
|
updated_at TEXT DEFAULT (datetime('now','localtime'))
|
||||||
|
);
|
||||||
|
CREATE INDEX idx_queue_status ON submission_queue(status);</pre>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- ================================================================== -->
|
||||||
|
<h2 id="api">7. API-Endpunkte</h2>
|
||||||
|
<div class="card">
|
||||||
|
<table>
|
||||||
|
<tr><th>Methode</th><th>Pfad</th><th>Funktion</th><th>Cloudflare</th><th>Set-Cookie?</th></tr>
|
||||||
|
<tr><td>GET</td><td><code>/</code></td><td>Frontend-Feed (Masonry)</td><td>Cache 5min</td><td>❌ NEIN</td></tr>
|
||||||
|
<tr><td>GET</td><td><code>/beitrag/<id></code></td><td>Beitrag-Detailseite</td><td>Cache 5min</td><td>❌ NEIN</td></tr>
|
||||||
|
<tr><td>GET</td><td><code>/health</code></td><td>Health-Check (schlank)</td><td>BYPASS</td><td>❌</td></tr>
|
||||||
|
<tr><td>GET/POST</td><td><code>/admin/*</code></td><td>Admin-Dashboard</td><td>BYPASS + Access OTP</td><td>✅ Login-Cookie</td></tr>
|
||||||
|
<tr><td>POST</td><td><code>/api/submit</code></td><td>Web-Formular</td><td>BYPASS</td><td>❌</td></tr>
|
||||||
|
<tr><td>POST</td><td><code>/api/chat</code></td><td>Chat-Assistant</td><td>BYPASS</td><td>❌</td></tr>
|
||||||
|
<tr><td>GET/POST</td><td><code>/whatsapp/webhook</code></td><td>WhatsApp-Intake</td><td>BYPASS</td><td>❌</td></tr>
|
||||||
|
<tr><td>POST</td><td><code>/telegram/webhook</code></td><td>Telegram-Intake</td><td>BYPASS</td><td>❌</td></tr>
|
||||||
|
</table>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- ================================================================== -->
|
||||||
|
<h2 id="queue">8. Queue-System</h2>
|
||||||
|
<div class="card">
|
||||||
|
<div class="fix"><strong>✅ v5.1-Fix:</strong> Queue-Worker ist ein <strong>eigener systemd-Service</strong> (<code>bsn-worker.service</code>), kein Daemon-Thread in gunicorns __main__. Whisper-Modell wird 1× beim Start geladen und nicht mitten in der Verarbeitung recycled.</div>
|
||||||
|
|
||||||
|
<h3>Queue-Flow</h3>
|
||||||
|
<pre>
|
||||||
|
POST /whatsapp/webhook → HMAC-Prüfung → INSERT submission_queue → 200 OK (<10ms)
|
||||||
|
|
||||||
|
bsn-worker (eigener systemd-Prozess):
|
||||||
|
1. Whisper-Modell laden (1× beim Start)
|
||||||
|
2. WHILE True:
|
||||||
|
SELECT ... WHERE status='pending' LIMIT 3
|
||||||
|
FOR EACH:
|
||||||
|
UPDATE status='processing'
|
||||||
|
→ Media-Download
|
||||||
|
→ Transkription (faster-whisper, Modell bereits geladen)
|
||||||
|
→ LLM-Triage (OpenRouter)
|
||||||
|
→ INSERT INTO submissions
|
||||||
|
→ UPDATE status='completed'
|
||||||
|
SLEEP 2</pre>
|
||||||
|
|
||||||
|
<h3>Idempotenz</h3>
|
||||||
|
<p><code>message_id UNIQUE</code> verhindert Doppelverarbeitung. Meta wiederholt Webhooks? → <code>INSERT OR IGNORE</code>.</p>
|
||||||
|
|
||||||
|
<h3>Fehlertoleranz</h3>
|
||||||
|
<table>
|
||||||
|
<tr><th>Szenario</th><th>Verhalten</th></tr>
|
||||||
|
<tr><td>Worker crashed</td><td>systemd startet neu. Whisper wird neu geladen. Einträge mit status=processing >10min → Reset auf pending.</td></tr>
|
||||||
|
<tr><td>Transkription fehlgeschlagen</td><td>status=failed, attempt_count++. Nach 3 Versuchen → endgültig failed.</td></tr>
|
||||||
|
<tr><td>LLM nicht erreichbar</td><td>Eintrag bleibt pending, Worker retried beim nächsten Poll.</td></tr>
|
||||||
|
<tr><td>Server-Neustart</td><td>Alle Zustände in SQLite persistent. Worker setzt fort.</td></tr>
|
||||||
|
</table>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- ================================================================== -->
|
||||||
|
<h2 id="llm">9. LLM (OpenRouter Mistral 4)</h2>
|
||||||
|
<div class="card">
|
||||||
|
<pre># .env:
|
||||||
|
OPENROUTER_API_KEY=sk-or-v1-...n
|
||||||
|
# API: POST https://openrouter.ai/api/v1/chat/completions
|
||||||
|
# Modell: "mistralai/mistral-large"</pre>
|
||||||
|
<table>
|
||||||
|
<tr><th>Funktion</th><th>Calls/h</th><th>Tokens/Call</th></tr>
|
||||||
|
<tr><td>Triage (🟢🟡🔴)</td><td>~50</td><td>~500 in, ~200 out</td></tr>
|
||||||
|
<tr><td>Zusammenfassung</td><td>~30</td><td>~800 in, ~150 out</td></tr>
|
||||||
|
<tr><td>Chat-Assistant</td><td>~20</td><td>~2000 in, ~300 out</td></tr>
|
||||||
|
<tr><td><strong>Total</strong></td><td><strong>~100</strong></td><td><strong>~130k Tokens/h</strong></td></tr>
|
||||||
|
</table>
|
||||||
|
<p>Kosten: ~$0.26/h bei Volllast.</p>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- ================================================================== -->
|
||||||
|
<h2 id="caching">10. Cloudflare Cache + Access</h2>
|
||||||
|
<div class="card">
|
||||||
|
|
||||||
|
<div class="fix"><strong>✅ v5.1-Fix:</strong> Cache Rules als <strong>expliziter Installationsschritt</strong>. Ohne diese Regeln cached Cloudflare KEIN HTML — die gesamte Skalierungsrechnung bricht zusammen.</div>
|
||||||
|
|
||||||
|
<h3>10.1 Cache Rules (Cloudflare Dashboard → Caching → Cache Rules)</h3>
|
||||||
|
<table>
|
||||||
|
<tr><th>Rule</th><th>URI Path</th><th>Cache-Status</th><th>Edge TTL</th></tr>
|
||||||
|
<tr><td>Frontend + Details</td><td><code>/*</code> + <code>/beitrag/*</code></td><td>Eligible for Cache</td><td>5 min</td></tr>
|
||||||
|
<tr><td>API + Webhooks + Admin</td><td><code>/api/*</code> + <code>/whatsapp/*</code> + <code>/telegram/*</code> + <code>/admin/*</code></td><td>Bypass Cache</td><td>—</td></tr>
|
||||||
|
<tr><td>Statische Assets</td><td><code>/static/*</code></td><td>Eligible for Cache</td><td>365 Tage</td></tr>
|
||||||
|
</table>
|
||||||
|
|
||||||
|
<h3>10.2 Kein Set-Cookie auf öffentlichen Routen</h3>
|
||||||
|
<div class="warn">⚠️ Flask darf auf <code>/</code> und <code>/beitrag/*</code> KEINE Cookies setzen. Sonst überspringt Cloudflare den Cache (Cookie = private response). Ggf. <code>@app.after_request</code> prüfen und Session-Cookies nur für /admin/* setzen.</div>
|
||||||
|
|
||||||
|
<h3>10.3 Cloudflare Access für /admin/*</h3>
|
||||||
|
<div class="fix"><strong>✅ v5.1-Fix:</strong> Cloudflare Access wird VOR Go-Live eingerichtet, nicht als Zukunftsfeature.</div>
|
||||||
|
<pre>Cloudflare Dashboard → Zero Trust → Access → Applications → Add:
|
||||||
|
Name: "BSN Admin"
|
||||||
|
Domain: chat.datenhimmel.work
|
||||||
|
Path: /admin/*
|
||||||
|
Policy: Allow, One-time PIN → E-Mail an daniel@brettspiel-news.de
|
||||||
|
|
||||||
|
So ist /admin/* ab Go-Live durch OTP geschützt. Kein Passwort im Flask nötig.</pre>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- ================================================================== -->
|
||||||
|
<h2 id="installation">11. Installation (14 Schritte)</h2>
|
||||||
|
<div class="card">
|
||||||
|
|
||||||
|
<h3>Schritt 1: OS</h3>
|
||||||
|
<pre># dogado-Kundencenter → Neuinstallation → Ubuntu 24.04 LTS minimal
|
||||||
|
# KEIN Plesk. Root-Passwort selbst setzen.</pre>
|
||||||
|
|
||||||
|
<h3>Schritt 2: Basis</h3>
|
||||||
|
<pre>apt update && apt upgrade -y
|
||||||
|
apt install -y python3.13 python3.13-venv python3.13-dev \
|
||||||
|
git nginx ufw fail2ban curl unzip build-essential restic</pre>
|
||||||
|
|
||||||
|
<h3>Schritt 3: Service-User</h3>
|
||||||
|
<div class="fix"><strong>✅ v5.1-Fix:</strong> Eigener User <code>bsn</code> statt root.</div>
|
||||||
|
<pre>useradd --system --home /opt/bsn-chatbot --shell /bin/bash bsn
|
||||||
|
mkdir -p /opt/bsn-chatbot/{src,data,logs,media,backups}
|
||||||
|
chown -R bsn:bsn /opt/bsn-chatbot</pre>
|
||||||
|
|
||||||
|
<h3>Schritt 4: Firewall</h3>
|
||||||
|
<pre>ufw default deny incoming
|
||||||
|
ufw allow 22/tcp
|
||||||
|
ufw allow 80/tcp
|
||||||
|
ufw allow 443/tcp
|
||||||
|
ufw enable</pre>
|
||||||
|
|
||||||
|
<h3>Schritt 5: venv</h3>
|
||||||
|
<pre>su - bsn
|
||||||
|
python3.13 -m venv /opt/bsn-chatbot/venv
|
||||||
|
source /opt/bsn-chatbot/venv/bin/activate
|
||||||
|
pip install --upgrade pip wheel</pre>
|
||||||
|
|
||||||
|
<h3>Schritt 6: Repo</h3>
|
||||||
|
<pre>cd /opt/bsn-chatbot/src
|
||||||
|
git clone git@git.datenhimmel.work:danielkrause/BSN-Chatsystem.git .
|
||||||
|
pip install -r requirements.txt
|
||||||
|
pip install faster-whisper gunicorn</pre>
|
||||||
|
|
||||||
|
<h3>Schritt 7: .env</h3>
|
||||||
|
<pre>cat > /opt/bsn-chatbot/src/.env << 'EOF'
|
||||||
|
DATABASE_PATH=/opt/bsn-chatbot/data/bsn_intake.db
|
||||||
|
MEDIA_DIR=/opt/bsn-chatbot/data/media
|
||||||
|
META_TOKEN=EAAx.....n
|
||||||
|
OPENR....I_KEY=sk-or-v1-...n
|
||||||
|
QUEUE_POLL_INTERVAL=2
|
||||||
|
QUEUE_BATCH_SIZE=3
|
||||||
|
TRIAGE_DELAY_SECONDS=360
|
||||||
|
EOF
|
||||||
|
chmod 600 /opt/bsn-chatbot/src/.env</pre>
|
||||||
|
|
||||||
|
<h3>Schritt 8: Daten migrieren</h3>
|
||||||
|
<pre>scp hermes@185.162.249.159:~/workspace/bsn-chatbot/bsn_intake.db /opt/bsn-chatbot/data/
|
||||||
|
scp -r hermes@185.162.249.159:~/workspace/bsn-chatbot/media/* /opt/bsn-chatbot/data/media/
|
||||||
|
chown -R bsn:bsn /opt/bsn-chatbot/data</pre>
|
||||||
|
|
||||||
|
<h3>Schritt 9: cloudflared (EINMAL)</h3>
|
||||||
|
<div class="fix"><strong>✅ v5.1-Fix:</strong> Gleiche Tunnel-Credentials für beide Server. Tunnel wird EINMAL erstellt, Credentials auf beide Server kopiert.</div>
|
||||||
|
<pre>wget -q https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64
|
||||||
|
chmod +x cloudflared-linux-amd64
|
||||||
|
mv cloudflared-linux-amd64 /usr/local/bin/cloudflared
|
||||||
|
|
||||||
|
# Login (einmalig, Browser):
|
||||||
|
cloudflared tunnel login
|
||||||
|
|
||||||
|
# Tunnel ERSTELLEN:
|
||||||
|
cloudflared tunnel create bsn-chat
|
||||||
|
|
||||||
|
# DNS-Route (CNAME):
|
||||||
|
cloudflared tunnel route dns bsn-chat chat.datenhimmel.work
|
||||||
|
|
||||||
|
# Tunnel-ID notieren (z.B. e84dedca-...)</pre>
|
||||||
|
|
||||||
|
<h3>Schritt 10: cloudflared Config</h3>
|
||||||
|
<pre>cat > ~/.cloudflared/config.yml << 'CFEOF'
|
||||||
|
tunnel: <TUNNEL_ID>
|
||||||
|
credentials-file: ~/.cloudflared/<TUNNEL_ID>.json
|
||||||
|
ingress:
|
||||||
|
- hostname: chat.datenhimmel.work
|
||||||
|
service: http://localhost:80
|
||||||
|
- service: http_status:404
|
||||||
|
CFEOF
|
||||||
|
|
||||||
|
cloudflared service install
|
||||||
|
systemctl enable --now cloudflared</pre>
|
||||||
|
|
||||||
|
<h3>Schritt 11: Tunnel-Credentials auf Ketchup kopieren</h3>
|
||||||
|
<div class="fix"><strong>✅ v5.1-Fix:</strong> Gleiche Credentials auf beiden Servern — Rollback ist DNS-Umschaltung, kein Tunnel-Wechsel.</div>
|
||||||
|
<pre># Auf dogado:
|
||||||
|
scp ~/.cloudflared/<TUNNEL_ID>.json hermes@185.162.249.159:~/.cloudflared/
|
||||||
|
scp ~/.cloudflared/cert.pem hermes@185.162.249.159:~/.cloudflared/
|
||||||
|
|
||||||
|
# Auf Ketchup:
|
||||||
|
cp <TUNNEL_ID>.json ~/workspace/bsn-chatbot/
|
||||||
|
# config.yml auf Ketchup auf localhost:5002 zeigen lassen</pre>
|
||||||
|
|
||||||
|
<h3>Schritt 12: Cloudflare Cache Rules</h3>
|
||||||
|
<pre>Cloudflare Dashboard → Caching → Cache Rules → Create Rule:
|
||||||
|
Field: URI Path, Operator: contains, Value: /beitrag/
|
||||||
|
(plus separate rule for /static/*)
|
||||||
|
→ Eligible for Cache, Edge TTL: 5 min / 365 days</pre>
|
||||||
|
|
||||||
|
<h3>Schritt 13: Cloudflare Access</h3>
|
||||||
|
<pre>Cloudflare Dashboard → Zero Trust → Access → Applications → Add:
|
||||||
|
Application: BSN Admin
|
||||||
|
Domain: chat.datenhimmel.work
|
||||||
|
Path: /admin/*
|
||||||
|
→ Policy: Allow, One-time PIN → daniel@brettspiel-news.de</pre>
|
||||||
|
|
||||||
|
<h3>Schritt 14: Flask prüfen — kein Set-Cookie auf /</h3>
|
||||||
|
<pre># In app.py prüfen: after_request setzt keine Cookies auf öffentlichen Routen
|
||||||
|
grep -n "set_cookie\|Set-Cookie\|session\[" /opt/bsn-chatbot/src/app.py
|
||||||
|
# Falls Cookies gesetzt werden → auf /admin/* beschränken</pre>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- ================================================================== -->
|
||||||
|
<h2 id="konfiguration">12. Konfigurationsdateien</h2>
|
||||||
|
<div class="card">
|
||||||
|
|
||||||
|
<h3>12.1 nginx: /etc/nginx/sites-available/bsn-chatbot</h3>
|
||||||
|
<div class="fix"><strong>✅ v5.1-Fix:</strong> <code>limit_req_zone</code> im HTTP-Kontext (nicht Server), nginx-Cache-Direktiven entfernt (macht Cloudflare), Webhook-Pfade korrigiert.</div>
|
||||||
|
<pre># /etc/nginx/nginx.conf (HTTP-Kontext):
|
||||||
|
http {
|
||||||
|
limit_req_zone $binary_remote_addr zone=api:10m rate=30r/s;
|
||||||
|
limit_req_zone $binary_remote_addr zone=webhook:10m rate=10r/s;
|
||||||
|
# ...
|
||||||
|
}
|
||||||
|
|
||||||
|
# /etc/nginx/sites-available/bsn-chatbot:
|
||||||
|
server {
|
||||||
|
listen 80;
|
||||||
|
server_name _; # cloudflared verbindet per IP
|
||||||
|
|
||||||
|
client_max_body_size 50M;
|
||||||
|
|
||||||
|
# Static Files
|
||||||
|
location /static/ {
|
||||||
|
alias /opt/bsn-chatbot/src/static/;
|
||||||
|
expires 365d;
|
||||||
|
add_header Cache-Control "public, immutable";
|
||||||
|
}
|
||||||
|
|
||||||
|
# Media Files
|
||||||
|
location /media/ {
|
||||||
|
alias /opt/bsn-chatbot/data/media/;
|
||||||
|
expires 7d;
|
||||||
|
}
|
||||||
|
|
||||||
|
# Webhooks — exakte Pfade
|
||||||
|
location ~ ^/(whatsapp|telegram)/webhook {
|
||||||
|
proxy_pass http://127.0.0.1:8000;
|
||||||
|
proxy_set_header Host $host;
|
||||||
|
proxy_set_header X-Real-IP $remote_addr;
|
||||||
|
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||||
|
proxy_set_header X-Forwarded-Proto $scheme;
|
||||||
|
limit_req zone=webhook burst=20 nodelay;
|
||||||
|
}
|
||||||
|
|
||||||
|
# Admin
|
||||||
|
location /admin {
|
||||||
|
proxy_pass http://127.0.0.1:8000;
|
||||||
|
proxy_set_header Host $host;
|
||||||
|
proxy_set_header X-Real-IP $remote_addr;
|
||||||
|
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||||
|
proxy_set_header X-Forwarded-Proto $scheme;
|
||||||
|
proxy_buffering off;
|
||||||
|
}
|
||||||
|
|
||||||
|
# API
|
||||||
|
location /api/ {
|
||||||
|
proxy_pass http://127.0.0.1:8000;
|
||||||
|
proxy_set_header Host $host;
|
||||||
|
proxy_set_header X-Real-IP $remote_addr;
|
||||||
|
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||||
|
proxy_set_header X-Forwarded-Proto $scheme;
|
||||||
|
limit_req zone=api burst=20 nodelay;
|
||||||
|
}
|
||||||
|
|
||||||
|
# Frontend (KEIN nginx-Cache, macht Cloudflare)
|
||||||
|
location / {
|
||||||
|
proxy_pass http://127.0.0.1:8000;
|
||||||
|
proxy_set_header Host $host;
|
||||||
|
proxy_set_header X-Real-IP $remote_addr;
|
||||||
|
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||||
|
proxy_set_header X-Forwarded-Proto $scheme;
|
||||||
|
}
|
||||||
|
}</pre>
|
||||||
|
|
||||||
|
<h3>12.2 gunicorn: /opt/bsn-chatbot/gunicorn.conf.py</h3>
|
||||||
|
<pre>bind = "127.0.0.1:8000"
|
||||||
|
workers = 4
|
||||||
|
threads = 2
|
||||||
|
worker_class = "gthread"
|
||||||
|
timeout = 120
|
||||||
|
graceful_timeout = 30
|
||||||
|
keepalive = 5
|
||||||
|
user = "bsn"
|
||||||
|
group = "bsn"
|
||||||
|
accesslog = "/opt/bsn-chatbot/logs/gunicorn-access.log"
|
||||||
|
errorlog = "/opt/bsn-chatbot/logs/gunicorn-error.log"
|
||||||
|
loglevel = "info"
|
||||||
|
proc_name = "bsn-chatbot"
|
||||||
|
max_requests = 1000
|
||||||
|
max_requests_jitter = 100</pre>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- ================================================================== -->
|
||||||
|
<h2 id="systemd">13. systemd-Services</h2>
|
||||||
|
<div class="card">
|
||||||
|
|
||||||
|
<h3>13.1 bsn-chatbot.service (gunicorn)</h3>
|
||||||
|
<pre>[Unit]
|
||||||
|
Description=BSN Chatbot — gunicorn WSGI
|
||||||
|
After=network.target
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
Type=notify
|
||||||
|
User=bsn
|
||||||
|
Group=bsn
|
||||||
|
WorkingDirectory=/opt/bsn-chatbot/src
|
||||||
|
EnvironmentFile=/opt/bsn-chatbot/src/.env
|
||||||
|
ExecStart=/opt/bsn-chatbot/venv/bin/gunicorn --config /opt/bsn-chatbot/gunicorn.conf.py app:app
|
||||||
|
ExecReload=/bin/kill -s HUP $MAINPID
|
||||||
|
Restart=always
|
||||||
|
RestartSec=5
|
||||||
|
PrivateTmp=true
|
||||||
|
NoNewPrivileges=true
|
||||||
|
ProtectSystem=strict
|
||||||
|
ProtectHome=true
|
||||||
|
ReadWritePaths=/opt/bsn-chatbot/data /opt/bsn-chatbot/logs /opt/bsn-chatbot/backups
|
||||||
|
ReadOnlyPaths=/opt/bsn-chatbot/src
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=multi-user.target</pre>
|
||||||
|
|
||||||
|
<h3>13.2 bsn-worker.service (Queue)</h3>
|
||||||
|
<div class="fix"><strong>✅ v5.1-Fix:</strong> Worker als eigener systemd-Service — kein Daemon-Thread unter gunicorn.</div>
|
||||||
|
<pre>[Unit]
|
||||||
|
Description=BSN Queue Worker — Submission-Verarbeitung
|
||||||
|
After=bsn-chatbot.service
|
||||||
|
Wants=bsn-chatbot.service
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
Type=simple
|
||||||
|
User=bsn
|
||||||
|
Group=bsn
|
||||||
|
WorkingDirectory=/opt/bsn-chatbot/src
|
||||||
|
EnvironmentFile=/opt/bsn-chatbot/src/.env
|
||||||
|
ExecStart=/opt/bsn-chatbot/venv/bin/python worker.py
|
||||||
|
Restart=always
|
||||||
|
RestartSec=10
|
||||||
|
PrivateTmp=true
|
||||||
|
NoNewPrivileges=true
|
||||||
|
ProtectSystem=strict
|
||||||
|
ProtectHome=true
|
||||||
|
ReadWritePaths=/opt/bsn-chatbot/data /opt/bsn-chatbot/logs
|
||||||
|
ReadOnlyPaths=/opt/bsn-chatbot/src
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=multi-user.target</pre>
|
||||||
|
|
||||||
|
<h3>13.3 worker.py (neue Datei)</h3>
|
||||||
|
<pre>#!/usr/bin/env python3
|
||||||
|
"""Standalone Queue-Worker — eigener Prozess, nicht unter gunicorn."""
|
||||||
|
import os, sys, time, json, sqlite3
|
||||||
|
sys.path.insert(0, '/opt/bsn-chatbot/src')
|
||||||
|
|
||||||
|
from dotenv import load_dotenv
|
||||||
|
load_dotenv()
|
||||||
|
|
||||||
|
DB = os.environ['DATABASE_PATH']
|
||||||
|
POLL = int(os.environ.get('QUEUE_POLL_INTERVAL', 2))
|
||||||
|
BATCH = int(os.environ.get('QUEUE_BATCH_SIZE', 3))
|
||||||
|
|
||||||
|
# Whisper 1× laden (kein Recycling mitten in der Verarbeitung)
|
||||||
|
from faster_whisper import WhisperModel
|
||||||
|
model = WhisperModel("base", device="cpu", compute_type="int8")
|
||||||
|
|
||||||
|
def get_db():
|
||||||
|
db = sqlite3.connect(DB)
|
||||||
|
db.row_factory = sqlite3.Row
|
||||||
|
db.execute("PRAGMA journal_mode=WAL")
|
||||||
|
db.execute("PRAGMA busy_timeout=5000")
|
||||||
|
return db
|
||||||
|
|
||||||
|
def process_one(db, row):
|
||||||
|
# → Media-Download, Transkription, LLM-Triage, DB-Insert
|
||||||
|
# (Implementierung aus queue_worker.py übernehmen)
|
||||||
|
pass
|
||||||
|
|
||||||
|
def main():
|
||||||
|
print("[worker] Gestartet, Whisper-Modell geladen.", flush=True)
|
||||||
|
db = get_db()
|
||||||
|
while True:
|
||||||
|
try:
|
||||||
|
rows = db.execute(
|
||||||
|
"SELECT * FROM submission_queue WHERE status='pending' ORDER BY created_at LIMIT ?",
|
||||||
|
(BATCH,)).fetchall()
|
||||||
|
for row in rows:
|
||||||
|
db.execute("UPDATE submission_queue SET status='processing' WHERE id=?", (row['id'],))
|
||||||
|
db.commit()
|
||||||
|
try:
|
||||||
|
process_one(db, row)
|
||||||
|
db.execute("UPDATE submission_queue SET status='completed' WHERE id=?", (row['id'],))
|
||||||
|
except Exception as e:
|
||||||
|
db.execute(
|
||||||
|
"UPDATE submission_queue SET status='failed', last_error=?, attempt_count=attempt_count+1 WHERE id=?",
|
||||||
|
(str(e)[:500], row['id']))
|
||||||
|
db.commit()
|
||||||
|
except Exception as e:
|
||||||
|
print(f"[worker] Fehler: {e}", flush=True)
|
||||||
|
time.sleep(POLL)
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
main()</pre>
|
||||||
|
|
||||||
|
<h3>13.4 Aktivierung</h3>
|
||||||
|
<pre>systemctl daemon-reload
|
||||||
|
systemctl enable --now bsn-chatbot bsn-worker nginx
|
||||||
|
# cloudflared ist bereits aktiv (Schritt 10)</pre>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- ================================================================== -->
|
||||||
|
<h2 id="monitoring">14. Monitoring</h2>
|
||||||
|
<div class="card">
|
||||||
|
<table>
|
||||||
|
<tr><th>Komponente</th><th>Log</th></tr>
|
||||||
|
<tr><td>gunicorn Access</td><td>/opt/bsn-chatbot/logs/gunicorn-access.log</td></tr>
|
||||||
|
<tr><td>gunicorn Error</td><td>/opt/bsn-chatbot/logs/gunicorn-error.log</td></tr>
|
||||||
|
<tr><td>Worker</td><td>journalctl -u bsn-worker</td></tr>
|
||||||
|
<tr><td>Flask</td><td>journalctl -u bsn-chatbot</td></tr>
|
||||||
|
<tr><td>nginx</td><td>/var/log/nginx/access.log</td></tr>
|
||||||
|
</table>
|
||||||
|
|
||||||
|
<h3>/health (abgespeckt)</h3>
|
||||||
|
<div class="fix"><strong>✅ v5.1-Fix:</strong> /health wurde entschlackt — nur noch status, db, whatsapp_configured.</div>
|
||||||
|
<pre>GET /health → {"status":"ok","db":"bsn_intake.db","whatsapp_configured":true}</pre>
|
||||||
|
|
||||||
|
<h3>Watchdog (Hermes-Server)</h3>
|
||||||
|
<pre># Cronjob alle 5min:
|
||||||
|
curl -s -o /dev/null -w "%{http_code}" https://chat.datenhimmel.work/health | grep -q 200
|
||||||
|
# Bei Fehler → Telegram an Daniel</pre>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- ================================================================== -->
|
||||||
|
<h2 id="security">15. Sicherheit</h2>
|
||||||
|
<div class="card">
|
||||||
|
<table>
|
||||||
|
<tr><th>Aspekt</th><th>Maßnahme</th></tr>
|
||||||
|
<tr><td>Firewall</td><td>Nur 22, 80, 443. Flask (8000) nur localhost.</td></tr>
|
||||||
|
<tr><td>SSH</td><td>Key-basiert, kein Passwort</td></tr>
|
||||||
|
<tr><td>Webhook</td><td>HMAC-SHA256 Signatur-Prüfung</td></tr>
|
||||||
|
<tr><td>Rate-Limit</td><td>nginx: 30 r/s (api), 10 r/s (webhook)</td></tr>
|
||||||
|
<tr><td>SQL-Injection</td><td>Parameterisierte Queries</td></tr>
|
||||||
|
<tr><td>XSS</td><td>Jinja2 Autoescaping</td></tr>
|
||||||
|
<tr><td>Secrets</td><td>.env, chmod 600, nicht in Git</td></tr>
|
||||||
|
<tr><td>systemd</td><td>ProtectSystem=strict, NoNewPrivileges</td></tr>
|
||||||
|
<tr><td>Admin</td><td>Cloudflare Access OTP (E-Mail)</td></tr>
|
||||||
|
<tr><td>Service-User</td><td>bsn (nicht root)</td></tr>
|
||||||
|
</table>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- ================================================================== -->
|
||||||
|
<h2 id="backup">16. Backup & Recovery</h2>
|
||||||
|
<div class="card">
|
||||||
|
<div class="fix"><strong>✅ v5.1-Fix:</strong> Offsite-Backup via rsync zum Hermes-Server (185.162.249.159) statt nur lokal.</div>
|
||||||
|
|
||||||
|
<h3>16.1 Backup-Script (/opt/bsn-chatbot/backup.sh)</h3>
|
||||||
|
<pre>#!/bin/bash
|
||||||
|
set -e
|
||||||
|
BACKUP_DIR=/opt/bsn-chatbot/backups
|
||||||
|
HERMES="hermes@185.162.249.159:/home/hermes/backups/bsn-chatbot"
|
||||||
|
DATE=$(date +%Y%m%d-%H%M)
|
||||||
|
|
||||||
|
mkdir -p $BACKUP_DIR
|
||||||
|
|
||||||
|
# Lokales DB-Backup
|
||||||
|
sqlite3 /opt/bsn-chatbot/data/bsn_intake.db \
|
||||||
|
".backup $BACKUP_DIR/bsn_intake_$DATE.db"
|
||||||
|
|
||||||
|
# Offsite (zum Hermes-Server)
|
||||||
|
rsync -avz --delete $BACKUP_DIR/ $HERMES/db/
|
||||||
|
rsync -avz /opt/bsn-chatbot/data/media/ $HERMES/media/
|
||||||
|
|
||||||
|
# Rotation: 72 lokale Backups behalten
|
||||||
|
ls -t $BACKUP_DIR/bsn_intake_*.db | tail -n +73 | xargs rm -f</pre>
|
||||||
|
|
||||||
|
<h3>16.2 Cron für Backup</h3>
|
||||||
|
<pre># /etc/cron.d/bsn-backup
|
||||||
|
0 * * * * bsn /opt/bsn-chatbot/backup.sh</pre>
|
||||||
|
|
||||||
|
<h3>16.3 Recovery</h3>
|
||||||
|
<pre>LAST=$(ls -t /opt/bsn-chatbot/backups/bsn_intake_*.db | head -1)
|
||||||
|
systemctl stop bsn-chatbot bsn-worker
|
||||||
|
cp $LAST /opt/bsn-chatbot/data/bsn_intake.db
|
||||||
|
systemctl start bsn-chatbot bsn-worker</pre>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- ================================================================== -->
|
||||||
|
<h2 id="dsgvo">17. DSGVO-Löschroutine</h2>
|
||||||
|
<div class="card">
|
||||||
|
<div class="fix"><strong>✅ v5.1-Fix:</strong> Automatische Löschroutine ist Teil von v1, nicht Zukunftsfeature.</div>
|
||||||
|
|
||||||
|
<h3>17.1 WhatsApp: „Meine Daten löschen"</h3>
|
||||||
|
<pre># Bereits implementiert in app.py:
|
||||||
|
# Nutzer sendet "Meine Daten löschen" → alle Submissions dieser Nummer:
|
||||||
|
# UPDATE submissions SET deleted_at=CURRENT_TIMESTAMP, status='gdpr_deleted'
|
||||||
|
# Medien von Platte löschen
|
||||||
|
# Auto-Reply: Bestätigung</pre>
|
||||||
|
|
||||||
|
<h3>17.2 Web: Referenz-ID</h3>
|
||||||
|
<pre># Bereits implementiert:
|
||||||
|
# Nach Web-Submit: Referenz-ID anzeigen + Kopier-Button
|
||||||
|
# Löschung aktuell via Kontaktaufnahme Redaktion
|
||||||
|
# ⚠️ Selbstbedienungs-Löschformular: jetzt in v1 umsetzen
|
||||||
|
# → /loeschen Route: ID eingeben → DELETE</pre>
|
||||||
|
|
||||||
|
<h3>17.3 Löschroutine (Cron)</h3>
|
||||||
|
<pre># /opt/bsn-chatbot/purge.py (täglich via Cron):
|
||||||
|
# SELECT * FROM submissions WHERE deleted_at IS NOT NULL
|
||||||
|
# AND deleted_at < date('now', '-30 days')
|
||||||
|
# → Medien von Platte löschen, DB-Eintrag hart löschen
|
||||||
|
# → DSGVO-Vermerk bleibt in separater Tabelle</pre>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- ================================================================== -->
|
||||||
|
<h2 id="golive">18. Go-Live</h2>
|
||||||
|
<div class="card">
|
||||||
|
<ol>
|
||||||
|
<li><strong>Ketchup cloudflared STOPPEN:</strong> <code>systemctl stop cloudflared</code></li>
|
||||||
|
<li><strong>Dogado cloudflared STARTEN:</strong> <code>systemctl start cloudflared</code></li>
|
||||||
|
<li><strong>DNS prüfen:</strong> CNAME chat.datenhimmel.work → Tunnel-ID (Cloudflare Dashboard)</li>
|
||||||
|
<li><strong>Warten:</strong> 2-5 Min DNS-Propagation</li>
|
||||||
|
<li><strong>Verifikation:</strong> <code>curl https://chat.datenhimmel.work/health</code> → 200</li>
|
||||||
|
<li><strong>10-Punkte-Checkliste durchgehen</strong></li>
|
||||||
|
<li><strong>72h warten</strong> bevor alter Server endgültig abgeschaltet wird</li>
|
||||||
|
</ol>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- ================================================================== -->
|
||||||
|
<h2 id="checkliste">19. Go-Live-Checkliste</h2>
|
||||||
|
<div class="card">
|
||||||
|
<table>
|
||||||
|
<tr><th>#</th><th>Test</th><th>Erwartet</th></tr>
|
||||||
|
<tr><td>1</td><td><code>curl https://chat.datenhimmel.work/health</code></td><td>200, "status":"ok"</td></tr>
|
||||||
|
<tr><td>2</td><td><code>curl -sI https://chat.datenhimmel.work/</code></td><td>200, cf-cache-status: HIT/MISS</td></tr>
|
||||||
|
<tr><td>3</td><td><code>curl https://chat.datenhimmel.work/beitrag/1</code></td><td>200, HTML</td></tr>
|
||||||
|
<tr><td>4</td><td><code>curl "https://chat.datenhimmel.work/whatsapp/webhook?hub.mode=subscribe&hub.verify_token=br3ttsp1el&hub.challenge=test"</code></td><td>"test"</td></tr>
|
||||||
|
<tr><td>5</td><td>Web-Submit POST</td><td>200, "status":"queued"</td></tr>
|
||||||
|
<tr><td>6</td><td>Nach ~30s: Admin prüfen</td><td>Submission mit Triage</td></tr>
|
||||||
|
<tr><td>7</td><td>Admin lädt</td><td>200, Submissions-Tabelle</td></tr>
|
||||||
|
<tr><td>8</td><td>Cloudflare Access</td><td>OTP-Mail bei /admin</td></tr>
|
||||||
|
<tr><td>9</td><td>Chat-Assistant</td><td>200, JSON mit answer</td></tr>
|
||||||
|
<tr><td>10</td><td>WhatsApp „Meine Daten löschen"</td><td>Bestätigung, Daten gelöscht</td></tr>
|
||||||
|
</table>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- ================================================================== -->
|
||||||
|
<h2 id="fallback">20. Fallback</h2>
|
||||||
|
<div class="card">
|
||||||
|
<div class="fix"><strong>✅ v5.1-Fix:</strong> Gleiche Tunnel-Credentials auf beiden Servern. Rollback = <code>systemctl start cloudflared</code> auf Ketchup, <code>systemctl stop cloudflared</code> auf dogado. Kein DNS-Change nötig — Cloudflare routet zum aktiven Connector.</div>
|
||||||
|
|
||||||
|
<h3>Rollback (innerhalb 72h)</h3>
|
||||||
|
<pre># Auf Ketchup:
|
||||||
|
systemctl start cloudflared
|
||||||
|
# Auf dogado:
|
||||||
|
systemctl stop cloudflared
|
||||||
|
# Fertig. Cloudflare sieht den aktiven Connector auf Ketchup.</pre>
|
||||||
|
|
||||||
|
<h3>Nach 72 Stunden</h3>
|
||||||
|
<pre># Auf Ketchup:
|
||||||
|
systemctl stop bsn-chatbot
|
||||||
|
rm ~/.cloudflared/<TUNNEL_ID>.json # Credentials entfernen</pre>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- ================================================================== -->
|
||||||
|
<h2 id="codeaenderungen">21. Code-Änderungen</h2>
|
||||||
|
<div class="card">
|
||||||
|
<table>
|
||||||
|
<tr><th>Datei</th><th>Änderung</th><th>Dauer</th></tr>
|
||||||
|
<tr><td><code>worker.py</code> (NEU)</td><td>Standalone Queue-Worker (Extrakt aus app.py + queue_worker.py)</td><td>45 min</td></tr>
|
||||||
|
<tr><td><code>app.py</code></td><td>Webhook-Routen: Queue.enqueue() statt sync. LLM: OpenRouter. TTS raus. Kein Set-Cookie auf /.</td><td>45 min</td></tr>
|
||||||
|
<tr><td><code>app.py</code></td><td>/health entschlacken (3 Felder)</td><td>5 min</td></tr>
|
||||||
|
<tr><td><code>purge.py</code> (NEU)</td><td>DSGVO-Löschroutine (Cronjob)</td><td>30 min</td></tr>
|
||||||
|
<tr><td><code>app.py</code></td><td>/loeschen Route (Selbstbedienung Web-ID)</td><td>20 min</td></tr>
|
||||||
|
<tr><td><code>requirements.txt</code></td><td>gunicorn hinzufügen, DeepSeek entfernen</td><td>5 min</td></tr>
|
||||||
|
</table>
|
||||||
|
<p><strong>Gesamt: ~2,5 Stunden</strong></p>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- ================================================================== -->
|
||||||
|
<h2 id="skalierung">22. Skalierung (1000/h)</h2>
|
||||||
|
<div class="card">
|
||||||
|
<table>
|
||||||
|
<tr><th>Pfad</th><th>Anteil</th><th>Req/h</th><th>Serverlast</th></tr>
|
||||||
|
<tr><td>Frontend (Cache HIT)</td><td>85%</td><td>850</td><td><strong>0%</strong> (Cloudflare)</td></tr>
|
||||||
|
<tr><td>Details (Cache MISS)</td><td>8%</td><td>80</td><td>~1% CPU</td></tr>
|
||||||
|
<tr><td>Daten senden</td><td>5%</td><td>50</td><td>~2% CPU</td></tr>
|
||||||
|
<tr><td>Chat-Assistant</td><td>2%</td><td>20</td><td>~1% CPU + Netzwerk</td></tr>
|
||||||
|
<tr><td><strong>Total</strong></td><td></td><td><strong>1000</strong></td><td><strong>~4% CPU</strong></td></tr>
|
||||||
|
</table>
|
||||||
|
<p>Server (12 GB, 6 Cores, NVMe) ist für 1000/h <strong>komfortabel überdimensioniert</strong>.</p>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- ================================================================== -->
|
||||||
|
<h2 id="zukunft">23. Zukunft</h2>
|
||||||
|
<div class="card">
|
||||||
|
<table>
|
||||||
|
<tr><th>Erweiterung</th><th>Auslöser</th></tr>
|
||||||
|
<tr><td>Lokales LLM</td><td>OpenRouter-Kosten >100 €/Monat</td></tr>
|
||||||
|
<tr><td>TTS (Piper)</td><td>WhatsApp-Sprachantworten gewünscht</td></tr>
|
||||||
|
<tr><td>SQLite → PostgreSQL</td><td>>500 Submissions/h</td></tr>
|
||||||
|
</table>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- ================================================================== -->
|
||||||
|
<h2 id="changelog">24. Änderungen v5 → v5.1</h2>
|
||||||
|
<div class="card">
|
||||||
|
<table>
|
||||||
|
<tr><th>Punkt</th><th>v5.0</th><th>v5.1</th></tr>
|
||||||
|
<tr><td>Queue-Worker</td><td>Daemon-Thread in gunicorn <code>__main__</code></td><td><span class="badge badge-fix">Eigener systemd-Service (worker.py)</span></td></tr>
|
||||||
|
<tr><td>nginx limit_req_zone</td><td>Im server-Kontext (geht nicht)</td><td><span class="badge badge-fix">Im http-Kontext</span></td></tr>
|
||||||
|
<tr><td>nginx Cache</td><td>proxy_cache_valid gesetzt</td><td><span class="badge badge-fix">Entfernt (macht Cloudflare)</span></td></tr>
|
||||||
|
<tr><td>Webhook nginx</td><td><code>location /webhook</code></td><td><span class="badge badge-fix">location ~ ^/(whatsapp|telegram)/webhook</span></td></tr>
|
||||||
|
<tr><td>Cloudflare Cache</td><td>Nur Strategie beschrieben</td><td><span class="badge badge-fix">Expliziter Installationsschritt + Set-Cookie-Prüfung</span></td></tr>
|
||||||
|
<tr><td>Admin-Schutz</td><td>Zukunftsfeature</td><td><span class="badge badge-fix">Cloudflare Access vor Go-Live</span></td></tr>
|
||||||
|
<tr><td>Tunnel-Failover</td><td>Zwei getrennte Tunnel</td><td><span class="badge badge-fix">Gleiche Credentials, beide Server</span></td></tr>
|
||||||
|
<tr><td>Offsite-Backup</td><td>Nur lokal</td><td><span class="badge badge-fix">rsync zum Hermes-Server</span></td></tr>
|
||||||
|
<tr><td>Service-User</td><td>root</td><td><span class="badge badge-fix">bsn</span></td></tr>
|
||||||
|
<tr><td>DSGVO-Löschroutine</td><td>Nicht in v1</td><td><span class="badge badge-fix">In v1: purge.py + Cron</span></td></tr>
|
||||||
|
<tr><td>/health</td><td>8 Felder</td><td><span class="badge badge-fix">3 Felder (status, db, whatsapp)</span></td></tr>
|
||||||
|
<tr><td>Port-5002</td><td>Mehrfach referenziert</td><td><span class="badge badge-fix">Nur noch :8000 (gunicorn) + :80 (nginx)</span></td></tr>
|
||||||
|
</table>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="card" style="margin-top:2rem">
|
||||||
|
<p style="text-align:center;color:var(--muted)"><strong>Ende des Plans v5.1</strong><br>
|
||||||
|
Cody (Hermes Agent) + externes Review — 14.07.2026<br>
|
||||||
|
Status: ✅ Umsetzungsreif</p>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
Reference in New Issue
Block a user