BSN-Chatsystem/docs/plans/2026-06-20-kickstarter-fetcher.md

Kickstarter-Zuverlässigkeits-Plan

Für Hermes/Cody: Schrittweise umsetzen, task-by-task. Zuerst Test-Aufbau, dann Integration.

Goal: Kickstarter-Kampagnenseiten immer zuverlässig abrufen können — Funding-Daten, Backer-Zahlen, Kampagnenstatus — trotz Cloudflare-Blockade.

Architecture: Mehrstufige Fallback-Strategie: (1) Headless-Browser mit Residential Proxy → (2) Tavily Extract API → (3) Wayback Machine/Google Cache → (4) Indirekte Pressequellen. Integration als Python-Modul kickstarter_fetcher.py mit einheitlichem Interface.

Tech Stack: Python 3.13, Playwright (headless Chromium), Proxy (vorhanden), Tavily API, requests, BeautifulSoup

Status 2026-06-20:

• Kickstarter.com: Cloudflare-blockiert auf ALLEN Endpunkten (Projektseiten, Widget-API, Discover)
• Tavily Extract: Getestet mit Earthborne Trailblazer → "Failed to fetch url" (auch blockiert)
• Wayback Machine: ✅ HTTP 200 (aber historische Snapshots, nicht live)
• Google Cache: ✅ HTTP 200 (aber stark verzögert)

---

Task 1: Playwright + Headless Chromium installieren

Objective: Browser-Automation auf dem Server verfügbar machen.

Step 1: Playwright installieren

cd /home/hermes/workspace
/home/hermes/venv/bin/pip install playwright
/home/hermes/venv/bin/playwright install chromium

Step 2: Abhängigkeiten prüfen

/home/hermes/venv/bin/playwright install-deps chromium 2>&1 | tail -5

Step 3: Smoke-Test

# test_playwright.py
from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://example.com", timeout=10000)
    print(f"Title: {page.title()}")
    browser.close()

Run: /home/hermes/venv/bin/python test_playwright.py Expected: Title: Example Domain

---

Task 2: Proxy-Konfiguration ermitteln

Objective: Den vorhandenen Proxy identifizieren und für Playwright nutzbar machen.

Step 1: Proxy-Details finden

# Prüfen, ob ein Proxy konfiguriert ist
grep -r 'proxy\|PROXY\|socks\|SOCKS' /home/hermes/.hermes/ 2>/dev/null | head -10
env | grep -i proxy
cat ~/.bashrc 2>/dev/null | grep -i proxy

Step 2: Proxy in Playwright einbinden

browser = p.chromium.launch(
    headless=True,
    proxy={
        "server": "http://proxy-ip:port",  # Aus Schritt 1
    }
)

Fallback falls kein Proxy: Residential Proxy-Dienst evaluieren (Bright Data, Oxylabs, o.ä. — ca. 5-10€/Monat für leichtes Scraping)

---

Task 3: Kickstarter-Fetcher-Modul bauen (kickstarter_fetcher.py)

Objective: Einheitliches Modul, das Kickstarter-Seiten abruft — mit mehrstufigem Fallback.

Files:

• Create: /home/hermes/workspace/bsn-chatbot/kickstarter_fetcher.py

Step 1: Modulgerüst mit Typsignatur

#!/usr/bin/env python3
"""
Kickstarter Campaign Fetcher — Multi-Stage Fallback Strategy
===========================================================
Stages:
  1. Playwright Headless Browser (via Proxy)
  2. Tavily Extract API
  3. Wayback Machine
  4. Google Cache
  5. Indirect Press Sources (BoardGameWire, Dicebreaker)

Returns: dict with campaign data or None
"""

from dataclasses import dataclass, field
from typing import Optional
import re, json, time

@dataclass
class KickstarterCampaign:
    url: str
    title: str = ""
    pledged: str = ""        # "€123,456"
    goal: str = ""           # "€10,000"
    backers: int = 0
    days_left: int = -1
    status: str = "unknown"  # "live", "funded", "cancelled"
    description: str = ""
    image_url: str = ""
    source: str = ""         # "playwright", "tavily", "wayback", "google_cache", "press"
    raw_html: str = ""
    fetched_at: str = ""

def fetch_campaign(url: str, timeout: int = 30) -> Optional[KickstarterCampaign]:
    """
    Fetch a Kickstarter campaign page with multi-stage fallback.
    Returns KickstarterCampaign or None if all stages fail.
    """
    # Stage 1: Playwright (best quality)
    result = _stage_playwright(url, timeout)
    if result:
        return result

    # Stage 2: Tavily Extract
    result = _stage_tavily(url, timeout)
    if result:
        return result

    # Stage 3: Wayback Machine
    result = _stage_wayback(url, timeout)
    if result:
        return result

    # Stage 4: Google Cache
    result = _stage_google_cache(url, timeout)
    if result:
        return result

    # Stage 5: Press sources (title only, no funding data)
    result = _stage_press_sources(url, timeout)
    if result:
        return result

    return None

Step 2: Stage-1 Playwright-Implementierung

def _stage_playwright(url: str, timeout: int) -> Optional[KickstarterCampaign]:
    """Fetch via headless Chromium with proxy."""
    try:
        from playwright.sync_api import sync_playwright
    except ImportError:
        return None

    try:
        with sync_playwright() as p:
            browser = p.chromium.launch(headless=True)
            context = browser.new_context(
                user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36",
                viewport={"width": 1920, "height": 1080},
            )
            page = context.new_page()
            page.goto(url, timeout=timeout * 1000, wait_until="domcontentloaded")
            # Wait for Cloudflare challenge to resolve
            page.wait_for_timeout(5000)  # 5s for JS to execute
            html = page.content()
            browser.close()

            return _parse_campaign_html(html, url, "playwright")
    except Exception as e:
        print(f"[kickstarter_fetcher] Playwright failed: {e}")
        return None

Step 3: HTML-Parser für Kampagnendaten

def _parse_campaign_html(html: str, url: str, source: str) -> Optional[KickstarterCampaign]:
    """Parse Kickstarter campaign HTML to extract structured data."""
    from bs4 import BeautifulSoup

    soup = BeautifulSoup(html, "html.parser")

    # Title
    title_tag = soup.find("title")
    title = title_tag.text.strip() if title_tag else ""
    # Strip " by ... — Kickstarter" suffix
    title = re.sub(r"\s*by\s+.*?—\s*Kickstarter\s*$", "", title, flags=re.I)

    # Pledged amount (multiple patterns)
    pledged = ""
    goal = ""
    backers = 0

    # Pattern 1: JSON-LD structured data
    for script in soup.find_all("script", type="application/ld+json"):
        try:
            data = json.loads(script.string)
            if isinstance(data, dict):
                # ...
                pass
        except:
            pass

    # Pattern 2: Meta tags
    # Pattern 3: Specific CSS classes (varies by Kickstarter template version)

    # Regex fallback on full text
    text = soup.get_text(" ", strip=True)

    # Pledged: "€123,456 pledged" or "pledged of €10,000 goal"
    pledged_match = re.search(r'(?:€|EUR|USD|\$|£)\s*([\d,.]+)\s*(?:pledged|von)', text, re.I)
    if pledged_match:
        pledged = pledged_match.group(0)

    # Goal
    goal_match = re.search(r'(?:goal|Ziel)\s*(?:of|von)?\s*(?:€|EUR|USD|\$|£)\s*([\d,.]+)', text, re.I)
    if goal_match:
        goal = goal_match.group(0)

    # Backers
    backers_match = re.search(r'([\d,]+)\s*backers?', text, re.I)
    if backers_match:
        backers = int(backers_match.group(1).replace(",", ""))

    # Days left
    days_match = re.search(r'(\d+)\s*(?:days?|Tage)\s*(?:to go|left|verbleibend)', text, re.I)
    days_left = int(days_match.group(1)) if days_match else -1

    # Status
    status = "unknown"
    if "successfully funded" in text.lower() or "erfolgreich finanziert" in text.lower():
        status = "funded"
    elif "cancel" in text.lower():
        status = "cancelled"
    elif days_left >= 0:
        status = "live"

    if not title:
        return None

    return KickstarterCampaign(
        url=url,
        title=title,
        pledged=pledged,
        goal=goal,
        backers=backers,
        days_left=days_left,
        status=status,
        description=(text[:500] if text else ""),
        source=source,
        raw_html=html[:50000],
        fetched_at=time.strftime("%Y-%m-%d %H:%M:%S"),
    )

Step 4: Stages 2-5 Implementierung

def _stage_tavily(url: str, timeout: int) -> Optional[KickstarterCampaign]:
    """Fetch via Tavily Extract API."""
    import requests
    tavily_key = os.environ.get("TAVILY_API_KEY", "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")
    try:
        r = requests.post("https://api.tavily.com/extract",
            json={"api_key": tavily_key, "urls": [url], "extract_depth": "advanced"},
            timeout=timeout,
        )
        data = r.json()
        results = data.get("results", [])
        if results and results[0].get("raw_content"):
            html = results[0]["raw_content"]
            return _parse_campaign_html(html, url, "tavily")
    except Exception as e:
        print(f"[kickstarter_fetcher] Tavily failed: {e}")
    return None


def _stage_wayback(url: str, timeout: int) -> Optional[KickstarterCampaign]:
    """Fetch via Wayback Machine (most recent snapshot)."""
    import requests
    try:
        # First get the most recent snapshot URL
        cdx_url = f"https://web.archive.org/web/timemap/json?url={url}&matchType=prefix&limit=1"
        r = requests.get(cdx_url, timeout=10)
        snapshots = r.json()
        if snapshots:
            snapshot_url = f"https://web.archive.org/web/{snapshots[0][1]}/{url}"
            r = requests.get(snapshot_url,
                headers={"User-Agent": "Mozilla/5.0 (X11; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0"},
                timeout=timeout,
            )
            if r.status_code == 200:
                return _parse_campaign_html(r.text, url, "wayback")
    except Exception as e:
        print(f"[kickstarter_fetcher] Wayback failed: {e}")
    return None


def _stage_google_cache(url: str, timeout: int) -> Optional[KickstarterCampaign]:
    """Fetch via Google Cache."""
    import requests
    try:
        cache_url = f"https://webcache.googleusercontent.com/search?q=cache:{url}"
        r = requests.get(cache_url,
            headers={"User-Agent": "Mozilla/5.0 (X11; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0"},
            timeout=timeout,
        )
        if r.status_code == 200 and len(r.text) > 1000:
            return _parse_campaign_html(r.text, url, "google_cache")
    except Exception as e:
        print(f"[kickstarter_fetcher] Google Cache failed: {e}")
    return None


def _stage_press_sources(url: str, timeout: int) -> Optional[KickstarterCampaign]:
    """Last resort: extract title from press coverage."""
    # Extract slug from URL and search for mentions
    slug_match = re.search(r'projects/([^/]+/[^/?#]+)', url)
    if not slug_match:
        return None
    slug = slug_match.group(1)
    project_name = slug.split("/")[-1].replace("-", " ")

    # Return minimal campaign object
    return KickstarterCampaign(
        url=url,
        title=project_name.title(),
        source="press",
        status="unknown",
        description=f"[Indirekte Quelle — keine Live-Daten verfügbar. Projekt: {project_name}]",
    )

---

Task 4: Integration in bestehenden Workflow

Objective: kickstarter_fetcher.py in den Recherche-Workflow einbinden.

Step 1: Als Cronjob-Script registrieren

cp kickstarter_fetcher.py /home/hermes/.hermes/profiles/cody/scripts/

Schritt 2: CLI-Wrapper für manuellen Aufruf

# /home/hermes/.hermes/profiles/cody/scripts/kickstarter_fetch.sh
#!/bin/bash
URL="$1"
/home/hermes/venv/bin/python -c "
import sys
sys.path.insert(0, '/home/hermes/workspace/bsn-chatbot')
from kickstarter_fetcher import fetch_campaign
c = fetch_campaign('$URL')
if c:
    print(f'Title: {c.title}')
    print(f'Pledged: {c.pledged}')
    print(f'Goal: {c.goal}')
    print(f'Backers: {c.backers}')
    print(f'Status: {c.status}')
    print(f'Source: {c.source}')
else:
    print('ALL STAGES FAILED')
"

Step 3: In den Recherche-Skill integrieren

Die crowdfunding_platforms.md-Referenz aktualisieren:

• Status Kickstarter: "BLOCKED → Multi-Stage-Fallback via kickstarter_fetcher.py"
• Verweis auf das Fetcher-Modul

---

Task 5: Test & Verifikation

Objective: Alle Stages an einer echten Kampagne testen.

Step 1: Playwright-Stage testen (mit Proxy falls vorhanden)

/home/hermes/venv/bin/python -c "
from kickstarter_fetcher import fetch_campaign
c = fetch_campaign('https://www.kickstarter.com/projects/earthbornegames/earthborne-trailblazer')
if c:
    print(f'✅ Source: {c.source}')
    print(f'   Title: {c.title}')
    print(f'   Pledged: {c.pledged or \"N/A\"}')
    print(f'   Goal: {c.goal or \"N/A\"}')
    print(f'   Backers: {c.backers}')
    print(f'   Status: {c.status}')
else:
    print('❌ All stages failed')
"

Expected: Mindestens Stage 3 (Wayback) liefert Daten. Ideal: Stage 1 (Playwright) liefert Live-Daten.

Step 2: Edge Cases testen

• Live-Kampagne (‚earthborne-trailblazer')
• Abgeschlossene Kampagne (‚earthborne-rangers')
• Deutsche Kampagne (‚palace-quest')
• Kampagne mit Video-Hintergrund (JS-lastig)

Step 3: Zeit messen

time /home/hermes/venv/bin/python -c "from kickstarter_fetcher import fetch_campaign; fetch_campaign('...')"

Expected: < 10 Sekunden pro erfolgreichem Fetch, < 30 Sekunden für kompletten Fallback-Durchlauf.

---

Task 6: Commit & Dokumentation

cd /home/hermes/workspace/bsn-chatbot
git add kickstarter_fetcher.py docs/plans/
git commit -m "feat: Kickstarter-Fetcher mit 5-Stufen-Fallback (Playwright→Tavily→Wayback→Cache→Presse)"
git push

---

Zusammenfassung: Fallback-Kette

Stufe	Methode	Datenqualität	Latenz	Status
1	Playwright Headless Browser (via Proxy)	Vollständig (Live-Daten)	5-10s	Zu implementieren
2	Tavily Extract API	Hoch (HTML)	3-5s	⚠️ Aktuell blockiert
3	Wayback Machine	Mittel (Snapshot, 1-30 Tage alt)	3-8s	✅ Funktioniert
4	Google Cache	Mittel (Cache, 1-7 Tage alt)	3-5s	✅ Funktioniert
5	Pressequellen	Niedrig (nur Titel)	2-5s	✅ Immer

Kritischer Pfad: Stage 1 (Playwright mit Proxy) ist die einzige Option für Live-Funding-Daten. Ohne Proxy wird Cloudflare den Headless-Browser erkennen und blockieren.