BSN-Chatsystem/docs/plans/2026-06-20-kickstarter-fetcher.md

# Kickstarter-Zuverlässigkeits-Plan

> **Für Hermes/Cody:** Schrittweise umsetzen, task-by-task. Zuerst Test-Aufbau, dann Integration.

**Goal:** Kickstarter-Kampagnenseiten **immer** zuverlässig abrufen können — Funding-Daten, Backer-Zahlen, Kampagnenstatus — trotz Cloudflare-Blockade.

**Architecture:** Mehrstufige Fallback-Strategie: (1) Headless-Browser mit Residential Proxy → (2) Tavily Extract API → (3) Wayback Machine/Google Cache → (4) Indirekte Pressequellen. Integration als Python-Modul `kickstarter_fetcher.py` mit einheitlichem Interface.

**Tech Stack:** Python 3.13, Playwright (headless Chromium), Proxy (vorhanden), Tavily API, requests, BeautifulSoup

**Status 2026-06-20:**
- Kickstarter.com: Cloudflare-blockiert auf ALLEN Endpunkten (Projektseiten, Widget-API, Discover)
- Tavily Extract: Getestet mit Earthborne Trailblazer → `"Failed to fetch url"` (auch blockiert)
- Wayback Machine: ✅ HTTP 200 (aber historische Snapshots, nicht live)
- Google Cache: ✅ HTTP 200 (aber stark verzögert)

---

## Task 1: Playwright + Headless Chromium installieren

**Objective:** Browser-Automation auf dem Server verfügbar machen.

**Step 1: Playwright installieren**

```bash
cd /home/hermes/workspace
/home/hermes/venv/bin/pip install playwright
/home/hermes/venv/bin/playwright install chromium
```

**Step 2: Abhängigkeiten prüfen**

```bash
/home/hermes/venv/bin/playwright install-deps chromium 2>&1 | tail -5
```

**Step 3: Smoke-Test**

```python
# test_playwright.py
from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://example.com", timeout=10000)
    print(f"Title: {page.title()}")
    browser.close()
```

Run: `/home/hermes/venv/bin/python test_playwright.py`
Expected: `Title: Example Domain`

---

## Task 2: Proxy-Konfiguration ermitteln

**Objective:** Den vorhandenen Proxy identifizieren und für Playwright nutzbar machen.

**Step 1: Proxy-Details finden**

```bash
# Prüfen, ob ein Proxy konfiguriert ist
grep -r 'proxy\|PROXY\|socks\|SOCKS' /home/hermes/.hermes/ 2>/dev/null | head -10
env | grep -i proxy
cat ~/.bashrc 2>/dev/null | grep -i proxy
```

**Step 2: Proxy in Playwright einbinden**

```python
browser = p.chromium.launch(
    headless=True,
    proxy={
        "server": "http://proxy-ip:port",  # Aus Schritt 1
    }
)
```

**Fallback falls kein Proxy:** Residential Proxy-Dienst evaluieren (Bright Data, Oxylabs, o.ä. — ca. 5-10€/Monat für leichtes Scraping)

---

## Task 3: Kickstarter-Fetcher-Modul bauen (`kickstarter_fetcher.py`)

**Objective:** Einheitliches Modul, das Kickstarter-Seiten abruft — mit mehrstufigem Fallback.

**Files:**
- Create: `/home/hermes/workspace/bsn-chatbot/kickstarter_fetcher.py`

**Step 1: Modulgerüst mit Typsignatur**

```python
#!/usr/bin/env python3
"""
Kickstarter Campaign Fetcher — Multi-Stage Fallback Strategy
===========================================================
Stages:
  1. Playwright Headless Browser (via Proxy)
  2. Tavily Extract API
  3. Wayback Machine
  4. Google Cache
  5. Indirect Press Sources (BoardGameWire, Dicebreaker)

Returns: dict with campaign data or None
"""

from dataclasses import dataclass, field
from typing import Optional
import re, json, time

@dataclass
class KickstarterCampaign:
    url: str
    title: str = ""
    pledged: str = ""        # "€123,456"
    goal: str = ""           # "€10,000"
    backers: int = 0
    days_left: int = -1
    status: str = "unknown"  # "live", "funded", "cancelled"
    description: str = ""
    image_url: str = ""
    source: str = ""         # "playwright", "tavily", "wayback", "google_cache", "press"
    raw_html: str = ""
    fetched_at: str = ""

def fetch_campaign(url: str, timeout: int = 30) -> Optional[KickstarterCampaign]:
    """
    Fetch a Kickstarter campaign page with multi-stage fallback.
    Returns KickstarterCampaign or None if all stages fail.
    """
    # Stage 1: Playwright (best quality)
    result = _stage_playwright(url, timeout)
    if result:
        return result

    # Stage 2: Tavily Extract
    result = _stage_tavily(url, timeout)
    if result:
        return result

    # Stage 3: Wayback Machine
    result = _stage_wayback(url, timeout)
    if result:
        return result

    # Stage 4: Google Cache
    result = _stage_google_cache(url, timeout)
    if result:
        return result

    # Stage 5: Press sources (title only, no funding data)
    result = _stage_press_sources(url, timeout)
    if result:
        return result

    return None
```

**Step 2: Stage-1 Playwright-Implementierung**

```python
def _stage_playwright(url: str, timeout: int) -> Optional[KickstarterCampaign]:
    """Fetch via headless Chromium with proxy."""
    try:
        from playwright.sync_api import sync_playwright
    except ImportError:
        return None

    try:
        with sync_playwright() as p:
            browser = p.chromium.launch(headless=True)
            context = browser.new_context(
                user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36",
                viewport={"width": 1920, "height": 1080},
            )
            page = context.new_page()
            page.goto(url, timeout=timeout * 1000, wait_until="domcontentloaded")
            # Wait for Cloudflare challenge to resolve
            page.wait_for_timeout(5000)  # 5s for JS to execute
            html = page.content()
            browser.close()

            return _parse_campaign_html(html, url, "playwright")
    except Exception as e:
        print(f"[kickstarter_fetcher] Playwright failed: {e}")
        return None
```

**Step 3: HTML-Parser für Kampagnendaten**

```python
def _parse_campaign_html(html: str, url: str, source: str) -> Optional[KickstarterCampaign]:
    """Parse Kickstarter campaign HTML to extract structured data."""
    from bs4 import BeautifulSoup

    soup = BeautifulSoup(html, "html.parser")

    # Title
    title_tag = soup.find("title")
    title = title_tag.text.strip() if title_tag else ""
    # Strip " by ... — Kickstarter" suffix
    title = re.sub(r"\s*by\s+.*?—\s*Kickstarter\s*$", "", title, flags=re.I)

    # Pledged amount (multiple patterns)
    pledged = ""
    goal = ""
    backers = 0

    # Pattern 1: JSON-LD structured data
    for script in soup.find_all("script", type="application/ld+json"):
        try:
            data = json.loads(script.string)
            if isinstance(data, dict):
                # ...
                pass
        except:
            pass

    # Pattern 2: Meta tags
    # Pattern 3: Specific CSS classes (varies by Kickstarter template version)

    # Regex fallback on full text
    text = soup.get_text(" ", strip=True)

    # Pledged: "€123,456 pledged" or "pledged of €10,000 goal"
    pledged_match = re.search(r'(?:€|EUR|USD|\$|£)\s*([\d,.]+)\s*(?:pledged|von)', text, re.I)
    if pledged_match:
        pledged = pledged_match.group(0)

    # Goal
    goal_match = re.search(r'(?:goal|Ziel)\s*(?:of|von)?\s*(?:€|EUR|USD|\$|£)\s*([\d,.]+)', text, re.I)
    if goal_match:
        goal = goal_match.group(0)

    # Backers
    backers_match = re.search(r'([\d,]+)\s*backers?', text, re.I)
    if backers_match:
        backers = int(backers_match.group(1).replace(",", ""))

    # Days left
    days_match = re.search(r'(\d+)\s*(?:days?|Tage)\s*(?:to go|left|verbleibend)', text, re.I)
    days_left = int(days_match.group(1)) if days_match else -1

    # Status
    status = "unknown"
    if "successfully funded" in text.lower() or "erfolgreich finanziert" in text.lower():
        status = "funded"
    elif "cancel" in text.lower():
        status = "cancelled"
    elif days_left >= 0:
        status = "live"

    if not title:
        return None

    return KickstarterCampaign(
        url=url,
        title=title,
        pledged=pledged,
        goal=goal,
        backers=backers,
        days_left=days_left,
        status=status,
        description=(text[:500] if text else ""),
        source=source,
        raw_html=html[:50000],
        fetched_at=time.strftime("%Y-%m-%d %H:%M:%S"),
    )
```

**Step 4: Stages 2-5 Implementierung**

```python
def _stage_tavily(url: str, timeout: int) -> Optional[KickstarterCampaign]:
    """Fetch via Tavily Extract API."""
    import requests
    tavily_key = os.environ.get("TAVILY_API_KEY", "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")
    try:
        r = requests.post("https://api.tavily.com/extract",
            json={"api_key": tavily_key, "urls": [url], "extract_depth": "advanced"},
            timeout=timeout,
        )
        data = r.json()
        results = data.get("results", [])
        if results and results[0].get("raw_content"):
            html = results[0]["raw_content"]
            return _parse_campaign_html(html, url, "tavily")
    except Exception as e:
        print(f"[kickstarter_fetcher] Tavily failed: {e}")
    return None


def _stage_wayback(url: str, timeout: int) -> Optional[KickstarterCampaign]:
    """Fetch via Wayback Machine (most recent snapshot)."""
    import requests
    try:
        # First get the most recent snapshot URL
        cdx_url = f"https://web.archive.org/web/timemap/json?url={url}&matchType=prefix&limit=1"
        r = requests.get(cdx_url, timeout=10)
        snapshots = r.json()
        if snapshots:
            snapshot_url = f"https://web.archive.org/web/{snapshots[0][1]}/{url}"
            r = requests.get(snapshot_url,
                headers={"User-Agent": "Mozilla/5.0 (X11; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0"},
                timeout=timeout,
            )
            if r.status_code == 200:
                return _parse_campaign_html(r.text, url, "wayback")
    except Exception as e:
        print(f"[kickstarter_fetcher] Wayback failed: {e}")
    return None


def _stage_google_cache(url: str, timeout: int) -> Optional[KickstarterCampaign]:
    """Fetch via Google Cache."""
    import requests
    try:
        cache_url = f"https://webcache.googleusercontent.com/search?q=cache:{url}"
        r = requests.get(cache_url,
            headers={"User-Agent": "Mozilla/5.0 (X11; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0"},
            timeout=timeout,
        )
        if r.status_code == 200 and len(r.text) > 1000:
            return _parse_campaign_html(r.text, url, "google_cache")
    except Exception as e:
        print(f"[kickstarter_fetcher] Google Cache failed: {e}")
    return None


def _stage_press_sources(url: str, timeout: int) -> Optional[KickstarterCampaign]:
    """Last resort: extract title from press coverage."""
    # Extract slug from URL and search for mentions
    slug_match = re.search(r'projects/([^/]+/[^/?#]+)', url)
    if not slug_match:
        return None
    slug = slug_match.group(1)
    project_name = slug.split("/")[-1].replace("-", " ")

    # Return minimal campaign object
    return KickstarterCampaign(
        url=url,
        title=project_name.title(),
        source="press",
        status="unknown",
        description=f"[Indirekte Quelle — keine Live-Daten verfügbar. Projekt: {project_name}]",
    )
```

---

## Task 4: Integration in bestehenden Workflow

**Objective:** `kickstarter_fetcher.py` in den Recherche-Workflow einbinden.

**Step 1: Als Cronjob-Script registrieren**

```bash
cp kickstarter_fetcher.py /home/hermes/.hermes/profiles/cody/scripts/
```

**Schritt 2: CLI-Wrapper für manuellen Aufruf**

```bash
# /home/hermes/.hermes/profiles/cody/scripts/kickstarter_fetch.sh
#!/bin/bash
URL="$1"
/home/hermes/venv/bin/python -c "
import sys
sys.path.insert(0, '/home/hermes/workspace/bsn-chatbot')
from kickstarter_fetcher import fetch_campaign
c = fetch_campaign('$URL')
if c:
    print(f'Title: {c.title}')
    print(f'Pledged: {c.pledged}')
    print(f'Goal: {c.goal}')
    print(f'Backers: {c.backers}')
    print(f'Status: {c.status}')
    print(f'Source: {c.source}')
else:
    print('ALL STAGES FAILED')
"
```

**Step 3: In den Recherche-Skill integrieren**

Die `crowdfunding_platforms.md`-Referenz aktualisieren:
- Status Kickstarter: "BLOCKED → Multi-Stage-Fallback via `kickstarter_fetcher.py`"
- Verweis auf das Fetcher-Modul

---

## Task 5: Test & Verifikation

**Objective:** Alle Stages an einer echten Kampagne testen.

**Step 1: Playwright-Stage testen (mit Proxy falls vorhanden)**

```bash
/home/hermes/venv/bin/python -c "
from kickstarter_fetcher import fetch_campaign
c = fetch_campaign('https://www.kickstarter.com/projects/earthbornegames/earthborne-trailblazer')
if c:
    print(f'✅ Source: {c.source}')
    print(f'   Title: {c.title}')
    print(f'   Pledged: {c.pledged or \"N/A\"}')
    print(f'   Goal: {c.goal or \"N/A\"}')
    print(f'   Backers: {c.backers}')
    print(f'   Status: {c.status}')
else:
    print('❌ All stages failed')
"
```

Expected: Mindestens Stage 3 (Wayback) liefert Daten. Ideal: Stage 1 (Playwright) liefert Live-Daten.

**Step 2: Edge Cases testen**

- Live-Kampagne (‚earthborne-trailblazer')
- Abgeschlossene Kampagne (‚earthborne-rangers')
- Deutsche Kampagne (‚palace-quest')
- Kampagne mit Video-Hintergrund (JS-lastig)

**Step 3: Zeit messen**

```bash
time /home/hermes/venv/bin/python -c "from kickstarter_fetcher import fetch_campaign; fetch_campaign('...')"
```

Expected: < 10 Sekunden pro erfolgreichem Fetch, < 30 Sekunden für kompletten Fallback-Durchlauf.

---

## Task 6: Commit & Dokumentation

```bash
cd /home/hermes/workspace/bsn-chatbot
git add kickstarter_fetcher.py docs/plans/
git commit -m "feat: Kickstarter-Fetcher mit 5-Stufen-Fallback (Playwright→Tavily→Wayback→Cache→Presse)"
git push
```

---

## Zusammenfassung: Fallback-Kette

| Stufe | Methode | Datenqualität | Latenz | Status |
|---|---|---|---|---|
| 1 | **Playwright Headless Browser** (via Proxy) | Vollständig (Live-Daten) | 5-10s | Zu implementieren |
| 2 | **Tavily Extract API** | Hoch (HTML) | 3-5s | ⚠️ Aktuell blockiert |
| 3 | **Wayback Machine** | Mittel (Snapshot, 1-30 Tage alt) | 3-8s | ✅ Funktioniert |
| 4 | **Google Cache** | Mittel (Cache, 1-7 Tage alt) | 3-5s | ✅ Funktioniert |
| 5 | **Pressequellen** | Niedrig (nur Titel) | 2-5s | ✅ Immer |

**Kritischer Pfad:** Stage 1 (Playwright mit Proxy) ist die einzige Option für **Live-Funding-Daten**. Ohne Proxy wird Cloudflare den Headless-Browser erkennen und blockieren.