@clawhub-billjamno58-737e0d743c
Upload CSV/Excel files and describe your visualization needs in natural language to get AI-recommended professional charts with PNG export.
# Smart Dashboard Generator
**One sentence, one chart** — Upload a CSV/Excel file, describe what you want in natural language, and AI generates professional charts instantly.
---
## Overview
Smart Dashboard Generator is an AI-powered data visualization tool that recommends and renders the best chart types based on your data and natural language requests.
---
## Features
### Core Capabilities
- **File Upload** — Parse CSV and Excel (.xlsx/.xls) automatically
- **AI Chart Recommendation** — Automatically suggest optimal chart types based on data structure
- **Multi-Chart Generation** — Generate multiple related charts in one request
- **PNG Export** — Download high-resolution chart images
- **Data Overview** — Display row/column count, column names, data types
### Supported Chart Types
| Chart Type | Best For |
|------------|----------|
| Bar | Category comparison |
| Line | Trends over time |
| Pie | Proportion/composition |
| Scatter | Relationship between variables |
| HeatMap | Density distribution |
| Radar | Multi-dimensional comparison |
| Gauge | KPI display |
| Funnel | Conversion funnel |
---
## Usage
### Step 1: Upload Data File
Upload a CSV or Excel file. The system automatically parses field types.
### Step 2: Describe Your Request
Use natural language to describe the chart you want:
- "Show monthly sales trends"
- "Compare product category sales"
- "Display user age distribution"
### Step 3: Get AI Recommendation
AI recommends the best chart types based on your data and request.
### Step 4: Download Chart
Export charts as PNG format, ready for reports and presentations.
---
## Pricing
| Tier | Price | Data Rows | Features |
|------|-------|-----------|----------|
| **FREE** | Free | 500 rows | 10 uses total, basic charts |
| **PRO** | $0.01 USDT/use | Full | All chart types, unlimited |
**FREE tier: 10 total uses (not per month), 500 row limit per file.**
---
## Billing
This skill uses **SkillPay** for billing.
- Each PRO use costs **$0.01 USDT**
- FREE tier: 10 total uses (not monthly)
- Purchase credits at: https://skillpay.me/smart-dashboard
---
## Env Variables
| Variable | Description |
|----------|-------------|
| `AI_API_KEY` | Your API key for AI recommendations |
| `AI_PROVIDER` | AI provider: `openai`, `claude`, `zhipu`, `minimax` |
| `AI_MODEL` | Specific model (optional) |
### Supported AI Providers
- **OpenAI** (GPT-4o) — `export AI_PROVIDER=openai`
- **Claude** (Claude 3.5 Sonnet) — `export AI_PROVIDER=claude`
- **Zhipu GLM** — `export AI_PROVIDER=zhipu`
- **MiniMax** — `export AI_PROVIDER=minimax`
---
## Technical Details
- **Data Parsing** — pandas for CSV/Excel processing
- **Chart Rendering** — Apache ECharts (pyecharts)
- **AI Recommendation** — Bring your own API key (OpenAI/Claude/GLM/MiniMax)
- **Data Security** — All processing is local, no server upload
---
## Limitations
- FREE tier: 10 total uses (not monthly), 500 row limit
- Recommended file size under 10MB
- AI features require your own API key
FILE:billing.py
# billing.py - ClawHub SkillPay Per-Use Billing (Python)
# Smart Dashboard Generator - $0.01 USDT per use
# slug: smart-dashboard
import os
import requests
BILLING_URL = "https://skillpay.me/api/v1/billing"
BUILDER_API_KEY = os.environ.get("SKILLPAY_API_KEY", "")
SKILL_ID = "smart-dashboard"
DEV_MODE = not BUILDER_API_KEY
def charge_user(user_id: str) -> dict:
"""
Charge a user for one API call (balance check, no actual charge).
Returns dict with ok=True/False and balance.
Dev mode: returns balance=999.0 without network call.
"""
if DEV_MODE:
return {"ok": True, "balance": 999.0, "reason": "dev_mode"}
if not BUILDER_API_KEY:
return {"ok": False, "balance": 0.0, "reason": "no_builder_key"}
try:
resp = requests.post(
f"{BILLING_URL}/charge",
headers={
"Content-Type": "application/json",
"X-API-Key": BUILDER_API_KEY,
},
json={
"user_id": user_id,
"skill_id": SKILL_ID,
"amount": 0,
},
timeout=10,
)
data = resp.json()
if resp.ok and data.get("success"):
return {"ok": True, "balance": data.get("balance", 0.0)}
return {
"ok": False,
"balance": data.get("balance", 0.0),
"payment_url": data.get("payment_url", f"https://skillpay.me/{SKILL_ID}"),
}
except Exception as e:
# Network error → allow usage, do not block
return {"ok": True, "balance": 0.0, "reason": f"network_error: {e}"}
def validate_token(api_key: str) -> dict:
"""
Validate user API key and return tier/balance.
"""
if DEV_MODE or not api_key:
return {"valid": True, "plan": "PRO", "balance": 999.0, "reason": "dev_mode"}
result = charge_user(api_key)
return {
"valid": result["ok"],
"plan": "PRO" if result["ok"] else "FREE",
"balance": result.get("balance", 0),
}
FILE:requirements.txt
pandas>=2.0.0
pyecharts>=2.0.0
requests>=2.28.0
openpyxl>=3.1.0
FILE:scripts/chart_recommender.py
# chart_recommender.py - AI Chart Type Recommender
"""Use AI to recommend best chart types based on data structure."""
import json
import requests
from typing import Dict, Any, List, Optional
from .config import AI_PROVIDERS, CHART_TYPES
class ChartRecommender:
"""AI-powered chart type recommendation."""
def __init__(self, api_key: str, provider: str = "openai", model: Optional[str] = None):
self.api_key = api_key
self.provider = provider.lower()
self.model = model or self._default_model()
self.base_url = AI_PROVIDERS.get(self.provider, AI_PROVIDERS["openai"])
def _default_model(self) -> str:
"""Get default model for provider."""
defaults = {
"openai": "gpt-4o",
"claude": "claude-3-5-sonnet-20241022",
"zhipu": "glm-4-flash",
"minimax": "MiniMax-Text-01",
}
return defaults.get(self.provider, "gpt-4o")
def _build_prompt(self, data_overview: Dict[str, Any], user_request: str) -> str:
"""Build prompt for chart recommendation."""
columns = data_overview.get("columns", [])
preview = data_overview.get("preview", [])
col_desc = "\n".join([
f"- {c['name']}: {c['semantic_type']} ({c['dtype']})"
for c in columns
])
preview_sample = json.dumps(preview[:3], ensure_ascii=False, indent=2)
return (
"You are a data visualization expert. Given a dataset and a user's request, recommend the best chart types.\n\n"
f"Dataset Overview:\n"
f"- Total rows: {data_overview['total_rows']}\n"
f"- Columns ({len(columns)}):\n"
f"{col_desc}\n\n"
"Preview data (first 3 rows):\n"
f"{preview_sample}\n\n"
"User request: \"" + user_request + "\"\n\n"
"Available chart types: " + ", ".join(CHART_TYPES) + "\n\n"
"Respond with a JSON object:\n"
"{{\n"
' "recommended_charts": [\n'
' {{\n'
' "chart_type": "bar|line|pie|scatter|heatmap|radar|gauge|funnel",\n'
' "title": "Chart title in English",\n'
' "x_axis": "column name for x-axis",\n'
' "y_axis": ["list of column names for y-axis"],\n'
' "reason": "why this chart type is recommended",\n'
' "style": {{"color": "#5470c6", ...}}\n'
' }}\n'
' ],\n'
' "data_mapping": {{\n'
' "x_column": "column name",\n'
' "y_columns": ["list of columns"]\n'
' }}\n'
"}}\n\n"
"Rules:\n"
"- Return 1-3 chart recommendations\n"
"- For trend/time data, prefer line chart\n"
"- For category comparisons, prefer bar chart\n"
"- For composition/proportion, prefer pie chart\n"
"- For relationships between two numeric variables, prefer scatter\n"
"- Output valid JSON only, no markdown code blocks\n"
)
def _call_ai(self, prompt: str) -> str:
"""Call AI API and return response text."""
if self.provider == "openai":
return self._call_openai(prompt)
elif self.provider == "claude":
return self._call_claude(prompt)
elif self.provider == "zhipu":
return self._call_zhipu(prompt)
elif self.provider == "minimax":
return self._call_minimax(prompt)
else:
raise ValueError(f"Unsupported provider: {self.provider}")
def _call_openai(self, prompt: str) -> str:
"""Call OpenAI API."""
headers = {
"Authorization": f"Bearer {self.api_key}",
"Content-Type": "application/json",
}
payload = {
"model": self.model,
"messages": [{"role": "user", "content": prompt}],
"temperature": 0.3,
}
resp = requests.post(
f"{self.base_url}",
headers=headers,
json=payload,
timeout=30,
)
resp.raise_for_status()
return resp.json()["choices"][0]["message"]["content"]
def _call_claude(self, prompt: str) -> str:
"""Call Claude API."""
headers = {
"x-api-key": self.api_key,
"anthropic-version": "2023-06-01",
"Content-Type": "application/json",
}
payload = {
"model": self.model,
"max_tokens": 1024,
"messages": [{"role": "user", "content": prompt}],
}
resp = requests.post(
f"{self.base_url}",
headers=headers,
json=payload,
timeout=30,
)
resp.raise_for_status()
return resp.json()["content"][0]["text"]
def _call_zhipu(self, prompt: str) -> str:
"""Call Zhipu (GLM) API."""
headers = {
"Authorization": f"Bearer {self.api_key}",
"Content-Type": "application/json",
}
payload = {
"model": self.model,
"messages": [{"role": "user", "content": prompt}],
"temperature": 0.3,
}
resp = requests.post(
f"{self.base_url}",
headers=headers,
json=payload,
timeout=30,
)
resp.raise_for_status()
return resp.json()["choices"][0]["message"]["content"]
def _call_minimax(self, prompt: str) -> str:
"""Call MiniMax API."""
headers = {
"Authorization": f"Bearer {self.api_key}",
"Content-Type": "application/json",
}
payload = {
"model": self.model,
"messages": [{"role": "user", "content": prompt}],
"temperature": 0.3,
}
resp = requests.post(
f"{self.base_url}",
headers=headers,
json=payload,
timeout=30,
)
resp.raise_for_status()
data = resp.json()
return data["choices"][0]["text"] if "text" in data["choices"][0] else data["choices"][0]["message"]["content"]
def recommend(
self,
data_overview: Dict[str, Any],
user_request: str,
) -> Dict[str, Any]:
"""Get chart recommendation from AI."""
# If no API key, use fallback
if not self.api_key:
return self._fallback_recommendation(data_overview)
prompt = self._build_prompt(data_overview, user_request)
response = self._call_ai(prompt)
# Parse JSON from response
try:
# Try to extract JSON from response
json_str = response.strip()
if json_str.startswith("```"):
json_str = json_str.split("```")[1]
if json_str.startswith("json"):
json_str = json_str[4:]
return json.loads(json_str)
except json.JSONDecodeError:
# Return fallback
return self._fallback_recommendation(data_overview)
def _fallback_recommendation(self, data_overview: Dict[str, Any]) -> Dict[str, Any]:
"""Fallback recommendation when AI parsing fails."""
columns = data_overview.get("columns", [])
numeric_cols = [c["name"] for c in columns if c["semantic_type"] == "numeric"]
categorical_cols = [c["name"] for c in columns if c["semantic_type"] == "categorical"]
datetime_cols = [c["name"] for c in columns if c["semantic_type"] == "datetime"]
x_col = datetime_cols[0] if datetime_cols else (categorical_cols[0] if categorical_cols else columns[0]["name"] if columns else "")
y_col = numeric_cols[:3] if numeric_cols else []
chart_type = "line" if datetime_cols else "bar"
return {
"recommended_charts": [{
"chart_type": chart_type,
"title": f"{y_col[0] if y_col else 'Data'} by {x_col}",
"x_axis": x_col,
"y_axis": y_col,
"reason": "Auto-selected based on data structure",
}],
"data_mapping": {
"x_column": x_col,
"y_columns": y_col,
},
}
def recommend_chart(
data_overview: Dict[str, Any],
user_request: str,
api_key: str,
provider: str = "openai",
model: Optional[str] = None,
) -> Dict[str, Any]:
"""Convenience function for chart recommendation."""
recommender = ChartRecommender(api_key, provider, model)
return recommender.recommend(data_overview, user_request)
FILE:scripts/chart_renderer.py
# chart_renderer.py - Chart Renderer using pyecharts
"""Render charts to PNG using pyecharts + screenshot."""
import os
import subprocess
from typing import Dict, Any, List, Optional
from pyecharts import options as opts
from pyecharts.charts import Bar, Line, Pie, Scatter, HeatMap, Radar, Gauge, Funnel
from pyecharts.globals import ThemeType
from .config import OUTPUT_DIR, DEFAULT_COLORS
class ChartRenderer:
"""Render chart configurations to PNG images."""
def __init__(self, output_dir: str = OUTPUT_DIR):
self.output_dir = output_dir
os.makedirs(output_dir, exist_ok=True)
def _load_data(self, data_mapping: Dict[str, Any], file_path: str) -> Dict[str, Any]:
"""Load actual data for chart rendering."""
from .file_parser import FileParser
parser = FileParser()
parser.parse(file_path)
x_col = data_mapping.get("x_column", "")
y_cols = data_mapping.get("y_columns", [])
return parser.get_data_for_chart(x_col, y_cols)
def render(
self,
chart_config: Dict[str, Any],
data_overview: Dict[str, Any],
file_path: str,
output_name: str,
) -> str:
"""Render a single chart to PNG."""
chart_type = chart_config.get("chart_type", "bar")
title = chart_config.get("title", "Chart")
style = chart_config.get("style", {})
# Build data_mapping from chart_config (handles both formats)
data_mapping = chart_config.get("data_mapping", {})
if not data_mapping:
# Fallback: use x_axis and y_axis directly
x_col = chart_config.get("x_axis", "")
y_cols = chart_config.get("y_axis", [])
data_mapping = {"x_column": x_col, "y_columns": y_cols}
# Load actual data
data = self._load_data(data_mapping, file_path)
# Render based on chart type
if chart_type == "bar":
chart = self._render_bar(title, data, data_mapping, style)
elif chart_type == "line":
chart = self._render_line(title, data, data_mapping, style)
elif chart_type == "pie":
chart = self._render_pie(title, data, data_mapping, style)
elif chart_type == "scatter":
chart = self._render_scatter(title, data, data_mapping, style)
elif chart_type == "heatmap":
chart = self._render_heatmap(title, data, data_mapping, style)
elif chart_type == "radar":
chart = self._render_radar(title, data, data_mapping, style)
elif chart_type == "gauge":
chart = self._render_gauge(title, data, data_mapping, style)
elif chart_type == "funnel":
chart = self._render_funnel(title, data, data_mapping, style)
else:
chart = self._render_bar(title, data, data_mapping, style)
# Save HTML
html_path = os.path.join(self.output_dir, f"{output_name}.html")
chart.render(html_path)
# Convert to PNG using screenshot
png_path = os.path.join(self.output_dir, f"{output_name}.png")
png_created = self._html_to_png(html_path, png_path)
if png_created:
return png_path
else:
return html_path
def _html_to_png(self, html_path: str, png_path: str):
"""Convert HTML to PNG using puppeteer.
Security: html_path and png_path must be inside OUTPUT_DIR.
Paths are sanitized before use to prevent command injection.
"""
try:
# Security: resolve and validate paths are inside output_dir
abs_html = os.path.abspath(html_path)
abs_png = os.path.abspath(png_path)
abs_out = os.path.abspath(self.output_dir)
if not (abs_html.startswith(abs_out) and abs_png.startswith(abs_out)):
return False # Path traversal attempt, reject
# Write fixed script (no user input in script content)
script_content = """const { chromium } = require('puppeteer');
(async () => {
const browser = await chromium.launch();
const page = await browser.newPage();
await page.setViewport({ width: 1200, height: 800 });
const args = process.argv.slice(2);
const htmlFile = args[0];
const pngFile = args[1];
await page.goto('file://' + htmlFile, { waitUntil: 'networkidle0' });
await page.screenshot({ path: pngFile, fullPage: true });
await browser.close();
})();
"""
script_path = os.path.join(self.output_dir, "_screenshot.js")
with open(script_path, "w") as f:
f.write(script_content)
# Use list form: node script.js <arg1> <arg2> — no shell injection possible
subprocess.run(
["node", script_path, abs_html, abs_png],
check=True,
capture_output=True,
timeout=60,
)
os.remove(script_path)
except (subprocess.CalledProcessError, FileNotFoundError, OSError):
# PNG conversion failed — return False, HTML still available
return False
return True
def _render_bar(
self,
title: str,
data: Dict[str, Any],
data_mapping: Dict[str, Any],
style: Dict[str, Any],
) -> Bar:
"""Render bar chart."""
x_data = data.get("x", [])
y_keys = [k for k in data.keys() if k != "x"]
chart = Bar(init_opts=opts.InitOpts(theme=ThemeType.LIGHT, width="1200px", height="800px"))
chart.add_xaxis(x_data)
colors = style.get("color", DEFAULT_COLORS[0])
for i, y_key in enumerate(y_keys):
color = DEFAULT_COLORS[i % len(DEFAULT_COLORS)] if isinstance(colors, list) else colors
chart.add_yaxis(
y_key,
data.get(y_key, []),
itemstyle_opts=opts.ItemStyleOpts(color=color),
)
chart.set_global_opts(
title_opts=opts.TitleOpts(title=title),
legend_opts=opts.LegendOpts(is_show=True),
tooltip_opts=opts.TooltipOpts(trigger="axis"),
xaxis_opts=opts.AxisOpts(name=data_mapping.get("x_column", "")),
yaxis_opts=opts.AxisOpts(name=""),
)
return chart
def _render_line(
self,
title: str,
data: Dict[str, Any],
data_mapping: Dict[str, Any],
style: Dict[str, Any],
) -> Line:
"""Render line chart."""
x_data = data.get("x", [])
y_keys = [k for k in data.keys() if k != "x"]
chart = Line(init_opts=opts.InitOpts(theme=ThemeType.LIGHT, width="1200px", height="800px"))
chart.add_xaxis(x_data)
for i, y_key in enumerate(y_keys):
color = DEFAULT_COLORS[i % len(DEFAULT_COLORS)]
chart.add_yaxis(
y_key,
data.get(y_key, []),
linestyle_opts=opts.LineStyleOpts(color=color, width=3),
itemstyle_opts=opts.ItemStyleOpts(color=color),
)
chart.set_global_opts(
title_opts=opts.TitleOpts(title=title),
legend_opts=opts.LegendOpts(is_show=True),
tooltip_opts=opts.TooltipOpts(trigger="axis"),
xaxis_opts=opts.AxisOpts(name=data_mapping.get("x_column", "")),
yaxis_opts=opts.AxisOpts(name=""),
datazoom_opts=opts.DataZoomOpts(),
)
return chart
def _render_pie(
self,
title: str,
data: Dict[str, Any],
data_mapping: Dict[str, Any],
style: Dict[str, Any],
) -> Pie:
"""Render pie chart."""
# For pie, use first numeric column
y_keys = [k for k in data.keys() if k != "x"]
if not y_keys:
y_keys = list(data.keys())
values = data.get(y_keys[0], []) if y_keys else []
x_data = data.get("x", [])
pairs = list(zip(x_data, values))
pairs = [(str(k), v) for k, v in pairs if v is not None and str(v) != "nan"]
chart = Pie(init_opts=opts.InitOpts(theme=ThemeType.LIGHT, width="1200px", height="800px"))
chart.add(
series_name="",
data_pair=pairs,
radius=["30%", "70%"],
label_opts=opts.LabelOpts(formatter="{b}: {d}%"),
)
chart.set_colors(DEFAULT_COLORS)
chart.set_global_opts(
title_opts=opts.TitleOpts(title=title),
legend_opts=opts.LegendOpts(is_show=True, orient="vertical", pos_left="left"),
)
return chart
def _render_scatter(
self,
title: str,
data: Dict[str, Any],
data_mapping: Dict[str, Any],
style: Dict[str, Any],
) -> Scatter:
"""Render scatter chart."""
y_keys = [k for k in data.keys() if k != "x"]
x_data = data.get("x", [])
y_data = data.get(y_keys[0], []) if y_keys else []
# Pair x and y
scatter_data = [[x_data[i], y_data[i]] for i in range(len(x_data)) if i < len(y_data)]
chart = Scatter(init_opts=opts.InitOpts(theme=ThemeType.LIGHT, width="1200px", height="800px"))
chart.add_xaxis(x_data)
chart.add_yaxis(
y_keys[0] if y_keys else "value",
scatter_data,
itemstyle_opts=opts.ItemStyleOpts(color=DEFAULT_COLORS[0]),
)
chart.set_global_opts(
title_opts=opts.TitleOpts(title=title),
legend_opts=opts.LegendOpts(is_show=True),
tooltip_opts=opts.TooltipOpts(formatter="{c}"),
xaxis_opts=opts.AxisOpts(name=data_mapping.get("x_column", "")),
yaxis_opts=opts.AxisOpts(name=y_keys[0] if y_keys else ""),
)
return chart
def _render_heatmap(
self,
title: str,
data: Dict[str, Any],
data_mapping: Dict[str, Any],
style: Dict[str, Any],
) -> HeatMap:
"""Render heatmap chart."""
# Simplified heatmap: use numeric columns as dimensions
x_data = data.get("x", [])
y_keys = [k for k in data.keys() if k != "x"]
heatmap_data = []
for i, y_key in enumerate(y_keys):
y_values = data.get(y_key, [])
for j, val in enumerate(y_values):
if j < len(x_data):
heatmap_data.append([j, i, val])
chart = HeatMap(init_opts=opts.InitOpts(theme=ThemeType.LIGHT, width="1200px", height="800px"))
chart.add_xaxis(x_data)
chart.add_yaxis("value", y_keys, heatmap_data)
chart.set_global_opts(
title_opts=opts.TitleOpts(title=title),
visualmap_opts=opts.VisualMapOpts(),
xaxis_opts=opts.AxisOpts(name=data_mapping.get("x_column", ""), type="category"),
yaxis_opts=opts.AxisOpts(type="category"),
)
return chart
def _render_radar(
self,
title: str,
data: Dict[str, Any],
data_mapping: Dict[str, Any],
style: Dict[str, Any],
) -> Radar:
"""Render radar chart."""
y_keys = [k for k in data.keys() if k != "x"]
if not y_keys:
return self._render_bar(title, data, data_mapping, style)
# Average values for each dimension
x_data = data.get("x", [])
values = []
for y_key in y_keys:
y_values = data.get(y_key, [])
valid = [v for v in y_values if v is not None and str(v) != "nan"]
values.append(sum(valid) / len(valid) if valid else 0)
chart = Radar(init_opts=opts.InitOpts(theme=ThemeType.LIGHT, width="1200px", height="800px"))
chart.add_schema(schema=[
opts.RadarIndicatorItem(name=n, max_=max(values) * 1.2 if max(values) > 0 else 100)
for n in x_data
])
chart.add("value", [values], areastyle_opts=opts.AreaStyleOpts(opacity=0.3))
chart.set_global_opts(
title_opts=opts.TitleOpts(title=title),
legend_opts=opts.LegendOpts(is_show=True),
)
return chart
def _render_gauge(
self,
title: str,
data: Dict[str, Any],
data_mapping: Dict[str, Any],
style: Dict[str, Any],
) -> Gauge:
"""Render gauge chart."""
y_keys = [k for k in data.keys() if k != "x"]
y_values = data.get(y_keys[0], []) if y_keys else []
value = y_values[0] if y_values else 0
chart = Gauge(init_opts=opts.InitOpts(theme=ThemeType.LIGHT, width="1200px", height="800px"))
chart.add(
series_name=title,
data_pair=[["value", value]],
detail_label_opts=opts.GaugeDetailOpts(formatter="{value}"),
)
chart.set_global_opts(
title_opts=opts.TitleOpts(title=title),
)
return chart
def _render_funnel(
self,
title: str,
data: Dict[str, Any],
data_mapping: Dict[str, Any],
style: Dict[str, Any],
) -> Funnel:
"""Render funnel chart."""
x_data = data.get("x", [])
y_keys = [k for k in data.keys() if k != "x"]
values = data.get(y_keys[0], []) if y_keys else []
pairs = list(zip([str(x) for x in x_data], values))
pairs = [(k, v) for k, v in pairs if v is not None and str(v) != "nan"]
chart = Funnel(init_opts=opts.InitOpts(theme=ThemeType.LIGHT, width="1200px", height="800px"))
chart.add(
series_name=title,
data_pair=pairs,
label_opts=opts.LabelOpts(formatter="{b}: {c}"),
)
chart.set_colors(DEFAULT_COLORS)
chart.set_global_opts(
title_opts=opts.TitleOpts(title=title),
legend_opts=opts.LegendOpts(is_show=True),
)
return chart
def render_chart(
chart_config: Dict[str, Any],
data_overview: Dict[str, Any],
file_path: str,
output_name: str,
) -> str:
"""Convenience function to render a chart to PNG."""
renderer = ChartRenderer()
return renderer.render(chart_config, data_overview, file_path, output_name)
FILE:scripts/web_app.py
#!/usr/bin/env python3
# web_app.py - Web interface for Smart Dashboard Generator
"""Simple web interface for Smart Dashboard Generator.
This provides a web UI that handles:
1. File upload (CSV/Excel)
2. AI chart recommendation
3. Chart rendering to PNG
4. Download
Usage:
python -m smart_dashboard.src.web_app [--port PORT]
"""
import argparse
import base64
import io
import json
import os
import sys
import uuid
import webbrowser
from http.server import HTTPServer, SimpleHTTPRequestHandler
from pathlib import Path
from threading import Thread
from typing import Optional
# Add src to path for imports
sys.path.insert(0, str(Path(__file__).parent.parent))
from src.file_parser import FileParser, parse_file
from src.chart_recommender import recommend_chart
from src.chart_renderer import render_chart, ChartRenderer
from src.config import BASE_DIR, OUTPUT_DIR, FREE_USES_LIMIT, ROW_LIMITS
# Import billing (ClawHub: clawhub.billing Python module)
try:
import sys
import os
_clawhub_root = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) # scripts/
sys.path.insert(0, _clawhub_root)
from clawhub.billing import charge_user, DEV_MODE
BILLING_AVAILABLE = True
except Exception as e:
print(f"[Billing] Import failed: {e}")
BILLING_AVAILABLE = False
HTML_TEMPLATE = """<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Smart Dashboard Generator</title>
<script src="https://cdn.jsdelivr.net/npm/[email protected]/dist/echarts.min.js"></script>
<style>
* { box-sizing: border-box; margin: 0; padding: 0; }
body { font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif; background: #f5f7fa; color: #333; min-height: 100vh; }
.container { max-width: 1200px; margin: 0 auto; padding: 20px; }
header { background: linear-gradient(135deg, #667eea 0%, #764ba2 100%); color: white; padding: 30px 20px; border-radius: 0 0 20px 20px; text-align: center; margin-bottom: 30px; }
header h1 { font-size: 2em; margin-bottom: 10px; }
header p { opacity: 0.9; font-size: 1.1em; }
.card { background: white; border-radius: 12px; padding: 24px; margin-bottom: 20px; box-shadow: 0 2px 8px rgba(0,0,0,0.08); }
.card h2 { font-size: 1.3em; margin-bottom: 16px; color: #444; border-bottom: 2px solid #667eea; padding-bottom: 8px; }
.upload-zone { border: 2px dashed #ddd; border-radius: 12px; padding: 40px; text-align: center; transition: all 0.3s; cursor: pointer; }
.upload-zone:hover { border-color: #667eea; background: #f8f9ff; }
.upload-zone.dragover { border-color: #667eea; background: #f0f2ff; }
.upload-zone input[type="file"] { display: none; }
.upload-icon { font-size: 48px; margin-bottom: 16px; }
.btn { background: #667eea; color: white; border: none; padding: 12px 24px; border-radius: 8px; cursor: pointer; font-size: 1em; transition: background 0.2s; }
.btn:hover { background: #5568d3; }
.btn:disabled { background: #ccc; cursor: not-allowed; }
.btn-secondary { background: #6c757d; }
.btn-secondary:hover { background: #5a6268; }
.form-group { margin-bottom: 16px; }
.form-group label { display: block; margin-bottom: 6px; font-weight: 500; }
.form-group input, .form-group select { width: 100%; padding: 10px; border: 1px solid #ddd; border-radius: 6px; font-size: 1em; }
.form-row { display: grid; grid-template-columns: 1fr 1fr; gap: 16px; }
@media (max-width: 768px) { .form-row { grid-template-columns: 1fr; } }
.preview-table { width: 100%; border-collapse: collapse; margin-top: 16px; overflow-x: auto; display: block; }
.preview-table th, .preview-table td { padding: 10px; text-align: left; border-bottom: 1px solid #eee; white-space: nowrap; }
.preview-table th { background: #f8f9fa; font-weight: 600; }
.preview-table tr:hover { background: #f8f9ff; }
.chart-container { background: white; border-radius: 12px; padding: 20px; margin: 20px 0; box-shadow: 0 2px 8px rgba(0,0,0,0.08); }
.chart-wrapper { width: 100%; height: 400px; }
.charts-grid { display: grid; grid-template-columns: repeat(auto-fit, minmax(500px, 1fr)); gap: 20px; }
@media (max-width: 768px) { .charts-grid { grid-template-columns: 1fr; } }
.status { padding: 12px 16px; border-radius: 8px; margin-bottom: 16px; }
.status.info { background: #e7f3ff; color: #0066cc; border: 1px solid #b3d9ff; }
.status.error { background: #ffe7e7; color: #cc0000; border: 1px solid #ffb3b3; }
.status.success { background: #e7ffe7; color: #006600; border: 1px solid #b3ffb3; }
.usage-info { background: #f8f9fa; padding: 12px; border-radius: 8px; margin-top: 16px; font-size: 0.9em; color: #666; }
.loading { display: inline-block; width: 20px; height: 20px; border: 3px solid #f3f3f3; border-top: 3px solid #667eea; border-radius: 50%; animation: spin 1s linear infinite; }
@keyframes spin { 0% { transform: rotate(0deg); } 100% { transform: rotate(360deg); } }
.hidden { display: none; }
.footer { text-align: center; padding: 20px; color: #888; font-size: 0.9em; }
</style>
</head>
<body>
<header>
<h1>Smart Dashboard Generator</h1>
<p>Upload data, describe what you want, get professional charts instantly</p>
</header>
<div class="container">
<div id="status-area"></div>
<!-- Upload Section -->
<div class="card" id="upload-section">
<h2>Step 1: Upload Data File</h2>
<div class="upload-zone" id="drop-zone">
<div class="upload-icon">📁</div>
<p><strong>Drop CSV or Excel file here</strong></p>
<p style="color: #888; margin-top: 8px;">or click to browse</p>
<input type="file" id="file-input" accept=".csv,.xlsx,.xls">
</div>
<div id="file-info" class="hidden">
<p><strong>File:</strong> <span id="file-name"></span></p>
<p><strong>Size:</strong> <span id="file-size"></span></p>
</div>
</div>
<!-- Data Preview Section -->
<div class="card hidden" id="preview-section">
<h2>Step 2: Data Overview</h2>
<div id="data-overview"></div>
<h3 style="margin: 16px 0 8px; font-size: 1.1em;">Preview (first 10 rows)</h3>
<div style="overflow-x: auto;">
<table class="preview-table" id="preview-table"></table>
</div>
</div>
<!-- AI Request Section -->
<div class="card hidden" id="request-section">
<h2>Step 3: Describe Your Chart</h2>
<div class="form-row">
<div class="form-group">
<label>AI Provider</label>
<select id="ai-provider">
<option value="openai">OpenAI (GPT-4o)</option>
<option value="claude">Claude</option>
<option value="zhipu">Zhipu GLM</option>
<option value="minimax">MiniMax</option>
</select>
</div>
<div class="form-group">
<label>Chart Title (optional)</label>
<input type="text" id="chart-title" placeholder="e.g., Monthly Sales Report">
</div>
</div>
<div class="form-group">
<label>Your Request (natural language)</label>
<input type="text" id="user-request" placeholder="e.g., Show sales trends over time, compare categories">
</div>
<button class="btn" id="generate-btn" onclick="generateCharts()">Generate Charts</button>
</div>
<!-- Charts Section -->
<div class="card hidden" id="charts-section">
<h2>Generated Charts</h2>
<div class="charts-grid" id="charts-container"></div>
<div style="margin-top: 20px;">
<button class="btn btn-secondary" onclick="downloadAllCharts()">Download All as PNG</button>
</div>
</div>
<!-- Usage Info -->
<div class="usage-info" id="usage-info"></div>
</div>
<div class="footer">
<p>Smart Dashboard Generator • All data processed locally</p>
</div>
<script>
let currentData = null;
let currentCharts = [];
// File upload handling
const dropZone = document.getElementById('drop-zone');
const fileInput = document.getElementById('file-input');
dropZone.addEventListener('click', () => fileInput.click());
dropZone.addEventListener('dragover', (e) => {
e.preventDefault();
dropZone.classList.add('dragover');
});
dropZone.addEventListener('dragleave', () => {
dropZone.classList.remove('dragover');
});
dropZone.addEventListener('drop', (e) => {
e.preventDefault();
dropZone.classList.remove('dragover');
const file = e.dataTransfer.files[0];
if (file) handleFile(file);
});
fileInput.addEventListener('change', (e) => {
const file = e.target.files[0];
if (file) handleFile(file);
});
async function handleFile(file) {
const validTypes = ['.csv', '.xlsx', '.xls'];
const ext = '.' + file.name.split('.').pop().toLowerCase();
if (!validTypes.includes(ext)) {
showStatus('Please upload a CSV or Excel file', 'error');
return;
}
showStatus('Parsing file...', 'info');
const formData = new FormData();
formData.append('file', file);
formData.append('command', 'parse');
try {
const resp = await fetch('/api', {
method: 'POST',
body: formData
});
const data = await resp.json();
if (data.error) {
showStatus(data.error, 'error');
return;
}
currentData = data;
document.getElementById('file-name').textContent = data.file_name;
document.getElementById('file-size').textContent = formatBytes(file.size);
document.getElementById('file-info').classList.remove('hidden');
document.getElementById('preview-section').classList.remove('hidden');
document.getElementById('request-section').classList.remove('hidden');
// Show data overview
const overview = document.getElementById('data-overview');
overview.innerHTML = `
<p><strong>Rows:</strong> data.total_rowsdata.truncated ? ` (of ${data.original_rows)` : ''}</p>
<p><strong>Columns:</strong> data.total_columns</p>
<p><strong>Column Types:</strong></p>
<ul style="margin-left: 20px; margin-top: 8px;">
data.columns.map(c => `<li>${c.name: c.semantic_type (c.dtype)</li>`).join('')}
</ul>
`;
// Show preview table
const previewTable = document.getElementById('preview-table');
const preview = data.preview.slice(0, 10);
const cols = data.columns.map(c => c.name);
previewTable.innerHTML = `
<thead><tr>cols.map(c => `<th>${c</th>`).join('')}</tr></thead>
<tbody>
preview.map(row => `<tr>${cols.map(c => `<td>${row[c] ?? ''</td>`).join('')}</tr>`).join('')}
</tbody>
`;
// Update usage info
updateUsageInfo(data.remaining_uses);
showStatus('File parsed successfully', 'success');
} catch (err) {
showStatus('Error parsing file: ' + err.message, 'error');
}
}
async function generateCharts() {
const request = document.getElementById('user-request').value.trim();
if (!request) {
showStatus('Please enter your chart request', 'error');
return;
}
if (!currentData) {
showStatus('Please upload a file first', 'error');
return;
}
const btn = document.getElementById('generate-btn');
btn.disabled = true;
btn.innerHTML = '<span class="loading"></span> Generating...';
showStatus('Generating charts with AI...', 'info');
try {
const resp = await fetch('/api', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
command: 'generate',
file_name: currentData.file_name,
data_overview: currentData,
request: request,
provider: document.getElementById('ai-provider').value,
chart_title: document.getElementById('chart-title').value
})
});
const data = await resp.json();
if (data.error) {
showStatus(data.error, 'error');
btn.disabled = false;
btn.textContent = 'Generate Charts';
return;
}
// Render charts
currentCharts = data.charts || [];
renderCharts(data.charts);
document.getElementById('charts-section').classList.remove('hidden');
updateUsageInfo(data.remaining_uses);
showStatus(`Generated currentCharts.length chart(s)`, 'success');
} catch (err) {
showStatus('Error generating charts: ' + err.message, 'error');
} finally {
btn.disabled = false;
btn.textContent = 'Generate Charts';
}
}
function renderCharts(charts) {
const container = document.getElementById('charts-container');
container.innerHTML = '';
charts.forEach((chart, i) => {
if (!chart.success) return;
const div = document.createElement('div');
div.className = 'chart-container';
div.innerHTML = `
<h3 style="margin-bottom: 12px;">chart.title || 'Chart ' + (i+1)</h3>
<div class="chart-wrapper" id="chart-i"></div>
<button class="btn btn-secondary" style="margin-top: 12px;" onclick="downloadChart(i)">Download PNG</button>
`;
container.appendChild(div);
// Initialize ECharts
const chartDom = document.getElementById(`chart-i`);
const myChart = echarts.init(chartDom);
try {
const chartData = typeof chart.chart_data === 'string' ? JSON.parse(chart.chart_data) : chart.chart_data;
myChart.setOption(chartData);
chart._echarts = myChart;
} catch (err) {
chartDom.innerHTML = `<p style="color: red;">Error rendering chart: err.message</p>`;
}
});
}
function downloadChart(index) {
const chart = currentCharts[index];
if (!chart || !chart.success) return;
const a = document.createElement('a');
a.href = chart.png_data;
a.download = `chart_index + 1.png`;
a.click();
}
function downloadAllCharts() {
currentCharts.forEach((chart, i) => {
if (chart.success) {
setTimeout(() => downloadChart(i), i * 200);
}
});
}
function showStatus(message, type) {
const statusArea = document.getElementById('status-area');
statusArea.innerHTML = `<div class="status type">message</div>`;
setTimeout(() => { if (statusArea) statusArea.innerHTML = ''; }, 5000);
}
function updateUsageInfo(remaining) {
const info = document.getElementById('usage-info');
if (info && remaining !== undefined) {
info.innerHTML = `<strong>Remaining uses:</strong> remaining / 0 FREE uses`;
}
}
function formatBytes(bytes) {
if (bytes === 0) return '0 Bytes';
const k = 1024;
const sizes = ['Bytes', 'KB', 'MB', 'GB'];
const i = Math.floor(Math.log(bytes) / Math.log(k));
return parseFloat((bytes / Math.pow(k, i)).toFixed(2)) + ' ' + sizes[i];
}
// Initialize
updateUsageInfo(10);
</script>
</body>
</html>
"""
class DashboardHandler(SimpleHTTPRequestHandler):
"""HTTP handler for dashboard web app."""
def do_GET(self):
"""Serve the web app."""
if self.path == '/' or self.path == '/index.html':
self.send_response(200)
self.send_header('Content-type', 'text/html')
self.end_headers()
self.wfile.write(HTML_TEMPLATE.encode())
else:
super().do_GET()
def do_POST(self):
"""Handle API requests."""
if self.path == '/api':
content_length = int(self.headers.get('Content-Length', 0))
content_type = self.headers.get('Content-Type', '')
if 'multipart/form-data' in content_type:
# File upload - parse
body = self.rfile.read(content_length)
import cgi
fields = cgi.parse_multipart(io.BytesIO(body), self.headers)
file_data = fields.get('file')[0] if fields.get('file') else None
command = fields.get('command', [''])[0]
if file_data and command == 'parse':
# Save temp file
file_name = file_data.filename if hasattr(file_data, 'filename') else 'uploaded_file'
import tempfile
with tempfile.NamedTemporaryFile(mode='wb', suffix=os.path.splitext(file_name)[1], delete=False) as f:
f.write(file_data)
temp_path = f.name
try:
tracker = UsageTracker()
if not tracker.check_and_increment():
self.send_json({"error": "FREE tier exhausted", "remaining": 0})
return
parser = FileParser(max_rows=ROW_LIMITS["FREE"])
overview = parser.parse(temp_path)
overview["remaining_uses"] = tracker.get_remaining()
self.send_json(overview)
except Exception as e:
self.send_json({"error": str(e)})
finally:
os.unlink(temp_path)
else:
self.send_json({"error": "Invalid request"})
else:
# JSON request
body = self.rfile.read(content_length)
data = json.loads(body)
command = data.get('command', '')
if command == 'generate':
self.handle_generate(data)
else:
self.send_json({"error": "Unknown command"})
else:
self.send_json({"error": "Not found"})
def handle_generate(self, data):
"""Handle generate command."""
try:
# Get user billing key and check via SkillPay
billing_api_key = data.get('billing_api_key', '')
user_id = data.get('user_id', 'anon')
is_free_user = not billing_api_key
if is_free_user:
# FREE tier: use local UsageTracker (10 uses total)
tracker = UsageTracker()
if not tracker.check_and_increment():
self.send_json({"error": "FREE tier exhausted (10 uses). Please upgrade.", "remaining": 0})
return
else:
# PRO tier: call SkillPay billing
if BILLING_AVAILABLE:
billing_result = charge_user(billing_api_key)
if not billing_result.get('ok', False):
self.send_json({
"error": "Insufficient balance or billing failed",
"payment_url": billing_result.get('payment_url', f'https://skillpay.me/smart-dashboard'),
"remaining": -1
})
return
data_overview = data.get('data_overview', {})
user_request = data.get('request', '')
provider = data.get('provider', 'openai')
chart_title = data.get('chart_title', '')
# Get AI recommendation
api_key = os.environ.get('AI_API_KEY', '')
if not api_key:
# Return demo recommendation
charts = self._demo_charts(data_overview, chart_title)
result = {
"charts": charts,
"remaining_uses": tracker.get_remaining() if is_free_user else -1,
}
self.send_json(result)
return
# Get AI recommendation
recommendation = recommend_chart(
data_overview=data_overview,
user_request=user_request,
api_key=api_key,
provider=provider,
)
charts = []
recommended = recommendation.get('recommended_charts', [])
for i, chart_config in enumerate(recommended):
try:
chart_data = self._generate_chart_data(chart_config, data_overview)
charts.append({
"chart_type": chart_config.get('chart_type', 'bar'),
"title": chart_config.get('title', f'Chart {i+1}'),
"chart_data": chart_data,
"png_data": None,
"success": True,
})
except Exception as e:
charts.append({
"chart_type": chart_config.get('chart_type', 'unknown'),
"title": chart_config.get('title', f'Chart {i+1}'),
"success": False,
"error": str(e),
})
result = {
"charts": charts,
"remaining_uses": tracker.get_remaining() if is_free_user else -1,
}
self.send_json(result)
except Exception as e:
self.send_json({"error": str(e)})
def _demo_charts(self, data_overview, chart_title):
"""Generate demo charts without AI."""
cols = data_overview.get('columns', [])
numeric_cols = [c['name'] for c in cols if c['semantic_type'] == 'numeric']
cat_cols = [c['name'] for c in cols if c['semantic_type'] == 'categorical']
x_col = cat_cols[0] if cat_cols else (cols[0]['name'] if cols else 'x')
y_col = numeric_cols[0] if numeric_cols else 'value'
# Demo bar chart
bar_data = {
"xAxis": {"type": "category", "data": ["Jan", "Feb", "Mar", "Apr", "May"]},
"yAxis": {"type": "value"},
"series": [{
"data": [120, 200, 150, 80, 70],
"type": "bar",
"itemStyle": {"color": "#5470c6"}
}],
"title": {"text": chart_title or f'{y_col} by {x_col}'},
"tooltip": {},
}
return [{
"chart_type": "bar",
"title": chart_title or f'{y_col} by {x_col}',
"chart_data": bar_data,
"png_data": None,
"success": True,
}]
def _generate_chart_data(self, chart_config, data_overview):
"""Generate ECharts config from chart recommendation."""
chart_type = chart_config.get('chart_type', 'bar')
title_text = chart_config.get('title', 'Chart')
x_col = chart_config.get('x_axis', '')
y_cols = chart_config.get('y_axis', [])
cols = data_overview.get('columns', [])
preview = data_overview.get('preview', [])
x_data = [row.get(x_col, '') for row in preview[:10]]
y_data = [[i, row.get(y_cols[0], 0) if y_cols else 0] for i, row in enumerate(preview[:10])]
if chart_type == 'bar':
return {
"xAxis": {"type": "category", "data": x_data, "name": x_col},
"yAxis": {"type": "value"},
"series": [{
"data": y_data,
"type": "bar",
"itemStyle": {"color": "#5470c6"}
}],
"title": {"text": title_text},
"tooltip": {},
}
elif chart_type == 'line':
return {
"xAxis": {"type": "category", "data": x_data, "name": x_col},
"yAxis": {"type": "value"},
"series": [{
"data": y_data,
"type": "line",
"lineStyle": {"color": "#5470c6", "width": 3},
"itemStyle": {"color": "#5470c6"},
}],
"title": {"text": title_text},
"tooltip": {},
}
elif chart_type == 'pie':
pie_data = [[str(row.get(x_col, '')), row.get(y_cols[0], 0) if y_cols else 0] for row in preview[:10]]
return {
"series": [{
"type": "pie",
"radius": ["30%", "70%"],
"data": pie_data,
"label": {"formatter": "{b}: {d}%"},
}],
"title": {"text": title_text},
"tooltip": {},
}
else:
return {
"xAxis": {"type": "category", "data": x_data},
"yAxis": {"type": "value"},
"series": [{"data": y_data, "type": chart_type}],
"title": {"text": title_text},
}
def send_json(self, data):
"""Send JSON response."""
body = json.dumps(data, ensure_ascii=False).encode()
self.send_response(200)
self.send_header('Content-Type', 'application/json')
self.send_header('Content-Length', len(body))
self.end_headers()
self.wfile.write(body)
class UsageTracker:
"""Track usage for FREE tier."""
def __init__(self, storage_path: str = os.path.join(BASE_DIR, "usage.json")):
self.storage_path = storage_path
os.makedirs(os.path.dirname(storage_path), exist_ok=True)
self._load()
def _load(self):
if os.path.exists(self.storage_path):
with open(self.storage_path, "r") as f:
self.data = json.load(f)
else:
self.data = {"used": 0, "total": FREE_USES_LIMIT}
def _save(self):
with open(self.storage_path, "w") as f:
json.dump(self.data, f)
def check_and_increment(self) -> bool:
if self.data["used"] >= self.data["total"]:
return False
self.data["used"] += 1
self._save()
return True
def get_remaining(self) -> int:
return max(0, self.data["total"] - self.data["used"])
def run_server(port: int = 8080):
"""Run the web server."""
os.makedirs(BASE_DIR, exist_ok=True)
handler = DashboardHandler
server = HTTPServer(('0.0.0.0', port), handler)
print(f"Smart Dashboard Generator running at http://localhost:{port}")
print("Press Ctrl+C to stop")
server.serve_forever()
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument("--port", type=int, default=8080)
args = parser.parse_args()
run_server(args.port)
FILE:scripts/config.py
# config.py - Smart Dashboard Generator Configuration
"""Configuration for Smart Dashboard Generator."""
import os
# Base paths - all file operations use /tmp/ only
BASE_DIR = "/tmp/smart-dashboard"
DATA_DIR = os.path.join(BASE_DIR, "data")
OUTPUT_DIR = os.path.join(BASE_DIR, "output")
# Ensure directories exist
for d in [BASE_DIR, DATA_DIR, OUTPUT_DIR]:
os.makedirs(d, exist_ok=True)
# Row limits per tier
ROW_LIMITS = {
"FREE": 500,
"STANDARD": 10_000,
"PRO": 100_000,
"ENTERPRISE": float("inf"),
}
# Chart types supported
CHART_TYPES = [
"bar",
"line",
"pie",
"scatter",
"heatmap",
"radar",
"gauge",
"funnel",
]
# AI API endpoint mappings
AI_PROVIDERS = {
"openai": "https://api.openai.com/v1/chat/completions",
"claude": "https://api.anthropic.com/v1/messages",
"zhipu": "https://open.bigmodel.cn/api/paas/v4/chat/completions",
"minimax": "https://api.minimax.chat/v1/text/chatcompletion_v2",
}
# Default chart colors
DEFAULT_COLORS = [
"#5470c6", "#91cc75", "#fac858", "#ee6666", "#73c0de",
"#3ba272", "#fc8452", "#9a60b4", "#ea7ccc",
]
# Preview rows for data
PREVIEW_ROWS = 20
# Usage limits
FREE_USES_LIMIT = 10
FILE:scripts/file_parser.py
# file_parser.py - CSV/Excel File Parser
"""Parse CSV and Excel files with pandas, generate data overview."""
import pandas as pd
import os
from typing import Dict, Any, Tuple, Optional
from .config import PREVIEW_ROWS, ROW_LIMITS
class FileParser:
"""Parse CSV/Excel files and generate data overview."""
def __init__(self, max_rows: int = ROW_LIMITS["FREE"]):
self.max_rows = max_rows
self.df: Optional[pd.DataFrame] = None
self.file_path: Optional[str] = None
self.file_name: Optional[str] = None
def parse(self, file_path: str) -> Dict[str, Any]:
"""Parse file and return data overview.
Security: file_path is resolved to absolute path and validated
to be inside BASE_DIR to prevent LFI/path traversal attacks.
"""
# Security: resolve absolute path and validate it's within allowed dir
from .config import BASE_DIR
abs_path = os.path.abspath(file_path)
abs_base = os.path.abspath(BASE_DIR)
if not abs_path.startswith(abs_base + os.sep):
raise ValueError(f"Access denied: file path outside allowed directory: {file_path}")
if not os.path.exists(abs_path):
raise FileNotFoundError(f"File not found: {abs_path}")
self.file_path = abs_path
self.file_name = os.path.basename(abs_path)
ext = os.path.splitext(abs_path)[1].lower()
if ext == ".csv":
self.df = pd.read_csv(file_path)
elif ext in [".xlsx", ".xls"]:
self.df = pd.read_excel(file_path)
else:
raise ValueError(f"Unsupported file type: {ext}")
# Enforce row limit
original_rows = len(self.df)
if original_rows > self.max_rows:
self.df = self.df.head(self.max_rows)
return self.get_overview(original_rows)
def get_overview(self, original_rows: Optional[int] = None) -> Dict[str, Any]:
"""Generate data overview from parsed DataFrame."""
if self.df is None:
raise ValueError("No file parsed. Call parse() first.")
rows, cols = self.df.shape
column_info = []
for col in self.df.columns:
dtype = str(self.df[col].dtype)
null_count = int(self.df[col].isnull().sum())
unique_count = int(self.df[col].nunique())
# Infer semantic type
if pd.api.types.is_numeric_dtype(self.df[col]):
semantic_type = "numeric"
elif pd.api.types.is_datetime64_any_dtype(self.df[col]):
semantic_type = "datetime"
elif pd.api.types.is_bool_dtype(self.df[col]):
semantic_type = "boolean"
else:
semantic_type = "categorical"
column_info.append({
"name": str(col),
"dtype": dtype,
"semantic_type": semantic_type,
"null_count": null_count,
"unique_count": unique_count,
})
return {
"file_name": self.file_name,
"total_rows": rows,
"total_columns": cols,
"original_rows": original_rows or rows,
"truncated": original_rows > rows if original_rows else False,
"columns": column_info,
"preview": self.df.head(PREVIEW_ROWS).to_dict(orient="records"),
}
def get_column_names(self) -> list:
"""Return list of column names."""
if self.df is None:
return []
return list(self.df.columns)
def get_numeric_columns(self) -> list:
"""Return list of numeric column names."""
if self.df is None:
return []
return list(self.df.select_dtypes(include=["number"]).columns)
def get_data_for_chart(self, x_col: str, y_cols: list) -> Dict[str, Any]:
"""Extract data for chart rendering."""
if self.df is None:
raise ValueError("No file parsed. Call parse() first.")
if x_col not in self.df.columns:
raise ValueError(f"Column not found: {x_col}")
result = {
"x": self.df[x_col].tolist(),
}
for y_col in y_cols:
if y_col in self.df.columns:
result[y_col] = self.df[y_col].tolist()
return result
def parse_file(file_path: str, max_rows: int = ROW_LIMITS["FREE"]) -> Dict[str, Any]:
"""Convenience function to parse a file and return overview."""
parser = FileParser(max_rows=max_rows)
return parser.parse(file_path)
FILE:scripts/__init__.py
# Smart Dashboard Generator
"""Core module for Smart Dashboard Generator."""
FILE:scripts/main.py
# main.py - Smart Dashboard Generator CLI Entry Point
"""Main CLI for Smart Dashboard Generator."""
import argparse
import json
import os
import sys
import uuid
from typing import Optional, Dict, Any
from .file_parser import parse_file, FileParser
from .chart_recommender import recommend_chart
from .chart_renderer import render_chart
from .config import BASE_DIR, DATA_DIR, OUTPUT_DIR, FREE_USES_LIMIT, ROW_LIMITS
class UsageTracker:
"""Track usage count for FREE tier."""
def __init__(self, storage_path: str = os.path.join(BASE_DIR, "usage.json")):
self.storage_path = storage_path
os.makedirs(os.path.dirname(storage_path), exist_ok=True)
self._load()
def _load(self):
"""Load usage data."""
if os.path.exists(self.storage_path):
with open(self.storage_path, "r") as f:
self.data = json.load(f)
else:
self.data = {"used": 0, "total": FREE_USES_LIMIT}
def _save(self):
"""Save usage data."""
with open(self.storage_path, "w") as f:
json.dump(self.data, f)
def check_and_increment(self) -> bool:
"""Check if usage available, increment if so. Returns True if allowed."""
if self.data["used"] >= self.data["total"]:
return False
self.data["used"] += 1
self._save()
return True
def get_remaining(self) -> int:
"""Get remaining uses."""
return max(0, self.data["total"] - self.data["used"])
def reset(self):
"""Reset usage (for testing)."""
self.data["used"] = 0
self._save()
def main():
"""Main CLI entry point."""
parser = argparse.ArgumentParser(description="Smart Dashboard Generator")
subparsers = parser.add_subparsers(dest="command", help="Commands")
# Parse command
parse_sp = subparsers.add_parser("parse", help="Parse a data file")
parse_sp.add_argument("file", help="Path to CSV or Excel file")
parse_sp.add_argument("--max-rows", type=int, default=ROW_LIMITS["FREE"], help="Max rows to process")
# Recommend command
recommend_sp = subparsers.add_parser("recommend", help="Get AI chart recommendation")
recommend_sp.add_argument("file", help="Path to CSV or Excel file")
recommend_sp.add_argument("--request", "-r", required=True, help="User request in natural language")
recommend_sp.add_argument("--api-key", "-k", required=True, help="AI API Key")
recommend_sp.add_argument("--provider", "-p", default="openai", help="AI provider (openai/claude/zhipu/minimax)")
recommend_sp.add_argument("--model", "-m", help="Specific model to use")
# Render command
render_sp = subparsers.add_parser("render", help="Render chart to PNG")
render_sp.add_argument("file", help="Path to CSV or Excel file")
render_sp.add_argument("--config", "-c", required=True, help="Chart config JSON file")
render_sp.add_argument("--output", "-o", help="Output PNG path")
# Full pipeline
pipeline_sp = subparsers.add_parser("generate", help="Full pipeline: parse + recommend + render")
pipeline_sp.add_argument("file", help="Path to CSV or Excel file")
pipeline_sp.add_argument("--request", "-r", required=True, help="User request in natural language")
pipeline_sp.add_argument("--api-key", "-k", default=None, help="AI API Key (optional, uses fallback if not provided)")
pipeline_sp.add_argument("--provider", "-p", default="openai", help="AI provider")
pipeline_sp.add_argument("--model", "-m", help="Specific model")
pipeline_sp.add_argument("--tier", "-t", default="FREE", help="Tier (FREE/STANDARD/PRO/ENTERPRISE)")
pipeline_sp.add_argument("--output-dir", "-d", help="Output directory")
args = parser.parse_args()
if args.command == "parse":
handle_parse(args)
elif args.command == "recommend":
handle_recommend(args)
elif args.command == "render":
handle_render(args)
elif args.command == "generate":
handle_generate(args)
else:
parser.print_help()
sys.exit(1)
def handle_parse(args):
"""Handle parse command."""
try:
tracker = UsageTracker()
if not tracker.check_and_increment():
print(json.dumps({
"error": "FREE tier exhausted",
"remaining": 0,
"limit": FREE_USES_LIMIT,
}))
sys.exit(1)
parser = FileParser(max_rows=args.max_rows)
overview = parser.parse(args.file)
overview["remaining_uses"] = tracker.get_remaining()
print(json.dumps(overview, ensure_ascii=False, indent=2))
except Exception as e:
print(json.dumps({"error": str(e)}))
sys.exit(1)
def handle_recommend(args):
"""Handle recommend command."""
try:
tracker = UsageTracker()
if not tracker.check_and_increment():
print(json.dumps({
"error": "FREE tier exhausted",
"remaining": 0,
}))
sys.exit(1)
parser = FileParser()
overview = parser.parse(args.file)
recommendation = recommend_chart(
data_overview=overview,
user_request=args.request,
api_key=args.api_key,
provider=args.provider,
model=args.model,
)
result = {
"recommendation": recommendation,
"remaining_uses": tracker.get_remaining(),
}
print(json.dumps(result, ensure_ascii=False, indent=2))
except Exception as e:
print(json.dumps({"error": str(e)}))
sys.exit(1)
def handle_render(args):
"""Handle render command."""
try:
tracker = UsageTracker()
if not tracker.check_and_increment():
print(json.dumps({
"error": "FREE tier exhausted",
"remaining": 0,
}))
sys.exit(1)
with open(args.config, "r") as f:
config = json.load(f)
parser = FileParser()
overview = parser.parse(args.file)
output_name = args.output or f"chart_{uuid.uuid4().hex[:8]}"
png_path = render_chart(
chart_config=config,
data_overview=overview,
file_path=args.file,
output_name=output_name,
)
result = {
"png_path": png_path,
"remaining_uses": tracker.get_remaining(),
}
print(json.dumps(result, ensure_ascii=False, indent=2))
except Exception as e:
print(json.dumps({"error": str(e)}))
sys.exit(1)
def handle_generate(args):
"""Handle full pipeline: parse + recommend + render."""
try:
tracker = UsageTracker()
if not tracker.check_and_increment():
print(json.dumps({
"error": "FREE tier exhausted",
"remaining": 0,
"limit": FREE_USES_LIMIT,
}))
sys.exit(1)
tier_limit = ROW_LIMITS.get(args.tier.upper(), ROW_LIMITS["FREE"])
parser = FileParser(max_rows=tier_limit)
overview = parser.parse(args.file)
# Use AI recommendation if API key provided, otherwise use fallback
if args.api_key:
recommendation = recommend_chart(
data_overview=overview,
user_request=args.request,
api_key=args.api_key,
provider=args.provider,
model=args.model,
)
else:
# Use fallback recommendation (no AI)
from .chart_recommender import ChartRecommender
recommender = ChartRecommender("", "openai")
recommendation = recommender.recommend(overview, args.request)
output_dir = args.output_dir or OUTPUT_DIR
os.makedirs(output_dir, exist_ok=True)
charts = []
recommended = recommendation.get("recommended_charts", [])
for i, chart_config in enumerate(recommended):
output_name = f"chart_{uuid.uuid4().hex[:8]}_{i}"
try:
png_path = render_chart(
chart_config=chart_config,
data_overview=overview,
file_path=args.file,
output_name=output_name,
)
charts.append({
"chart_type": chart_config.get("chart_type", "unknown"),
"title": chart_config.get("title", ""),
"png_path": png_path,
"success": True,
})
except Exception as e:
charts.append({
"chart_type": chart_config.get("chart_type", "unknown"),
"title": chart_config.get("title", ""),
"png_path": None,
"success": False,
"error": str(e),
})
result = {
"data_overview": {
"file_name": overview["file_name"],
"total_rows": overview["total_rows"],
"total_columns": overview["total_columns"],
},
"recommendation": recommendation,
"charts": charts,
"remaining_uses": tracker.get_remaining(),
}
print(json.dumps(result, ensure_ascii=False, indent=2))
except Exception as e:
print(json.dumps({"error": str(e)}))
sys.exit(1)
if __name__ == "__main__":
main()
Upload contract PDFs to extract and manage contract details with expiry reminders and Feishu push notifications, fully offline and secure.
# Contract Tracker (contract-tracker)
> Upload contract PDFs → AI extracts key fields → Manage ledger → Expiry reminders + Feishu push
---
## Trigger Phrases
`contract ledger` `contract management` `contract tracker` `pdf contract` `contract reminder`
---
## Usage
### Command Line
```bash
# Upload a contract PDF
python -m scripts.main upload /path/to/contract.pdf
# List all contracts
python -m scripts.main list
# List contracts expiring within 30 days
python -m scripts.main list --status "Active" --sort end_date
# Get contract details
python -m scripts.main get <contract_id>
# Update a contract
python -m scripts.main update <contract_id> --name "New Name" --status "Terminated"
# Delete a contract
python -m scripts.main delete <contract_id>
# Add expiry reminder
python -m scripts.main reminder <contract_id> add --days 30
# Check expiring contracts
python -m scripts.main check --days 30
# Export contracts
python -m scripts.main export --format csv -o contracts.csv
```
### Python API
```python
from scripts import extract_text_from_pdf, extract_contract_fields
from scripts import add_contract, get_contracts, get_contract
from scripts import update_contract, delete_contract
# Extract fields from PDF
text = extract_text_from_pdf("/path/to/contract.pdf")
fields = extract_contract_fields(text, "contract.pdf")
contract = add_contract(fields)
# List contracts
all_contracts = get_contracts(status="Active")
```
---
## Contract Fields Extracted
- **Contract Name** — from PDF title
- **Amount** — RMB amount via regex
- **Sign Date** — contract signing date
- **Start Date** — effective start date
- **End Date** — expiry date
- **Counterparty** — other party name
- **Key Nodes** — payment terms, renewal clauses (up to 5)
- **Status** — Active / Expired (auto-calculated)
---
## Supported Formats
| Format | Extension | Notes |
|--------|-----------|-------|
| PDF | `.pdf` | Text extraction via PyMuPDF |
---
## Tech Stack
- **Parsing**: PyMuPDF (fitz)
- **AI Field Extraction**: Regex + heuristic pattern matching (fully offline, no external AI API)
- **Storage**: JSON file in `/tmp/contract-tracker/` (fully offline, no home directory writes)
- **Notifications**: Feishu IM card format
---
## Tiered Features
| Feature | FREE | PRO |
|---------|:----:|:---:|
| Max Contracts | 5 | Unlimited |
| Max Reminders | 1 | Unlimited |
| Export Formats | CSV | CSV, XLSX, PDF |
| Feishu Reminders | No | Yes |
**Price**: $0.01 USDT per call (PRO tier). FREE tier is free.
> Get PRO: [https://skillpay.me/contract-tracker](https://skillpay.me/contract-tracker)
---
## Billing
- **Endpoint**: `POST https://skillpay.me/api/v1/billing/charge`
- **Header**: `X-API-Key: {api_key}`
- **Body**: `{"user_id": "...", "skill_id": "contract-tracker", "amount": 0.01}`
- **Response**: `{"success": true, "balance": ...}`
- **Fallback**: Network error → FREE tier (do not block usage)
- **Dev Mode**: No API key configured → `balance=999.0`, no charge
---
## Required Environment Variables
| Variable | Description |
|----------|-------------|
| `SKILL_BILLING_API_KEY` | SkillPay Builder API Key |
| `SKILL_BILLING_SKILL_ID` | Skill Slug (default: contract-tracker) |
---
## Security Notes
- All contract data stored in `/tmp/contract-tracker/` — no home directory writes
- PDF parsing is fully offline — no external network calls during extraction
- Feishu card push requires a Feishu bot token (configure separately)
---
## API Key Format
Any non-empty string works as an API key. Tier is determined automatically:
- **No API key** → FREE tier
- **Any API key** → PRO tier
---
## Slug
`contract-tracker`
FILE:requirements.txt
PyMuPDF>=1.23.0
requests>=2.28.0
FILE:scripts/pdf_parser.py
"""
PDF Parser for Contract Ledger.
Uses PyMuPDF (fitz) to extract text from PDF contracts.
"""
import re
import fitz
from datetime import datetime
from typing import Optional
def extract_text_from_pdf(pdf_path: str) -> str:
"""Extract all text from a PDF file."""
doc = fitz.open(pdf_path)
text_parts = []
for page in doc:
text_parts.append(page.get_text())
doc.close()
return "\n".join(text_parts)
def extract_contract_fields(text: str, filename: str = "") -> dict:
"""
Extract key fields from contract text using pattern matching.
Returns: contract_name, amount, dates, counterparty, key_nodes, status.
"""
# Extract contract name
lines = [l.strip() for l in text.split("\n") if l.strip()]
contract_name = ""
if lines:
for line in lines[:5]:
if len(line) > 5 and not line.startswith("\u7b2c") and "\u6761" not in line:
contract_name = line
break
if not contract_name and filename:
contract_name = filename.replace(".pdf", "").replace("_", " ")
# Extract amount
amount = extract_amount(text)
# Extract dates
sign_date = extract_date(text, ["\u7b7e\u8ba2\u65e5\u671f", "\u7b7e\u7f72\u65e5\u671f", "\u7b7e\u7ea6\u65e5\u671f", "\u7b7e\u8ba2\u4e8e"])
start_date = extract_date(text, ["\u5f00\u59cb\u65e5\u671f", "\u751f\u6548\u65e5\u671f", "\u8d77\u59cb\u65e5\u671f", "\u5f00\u59cb\u4e8e"])
end_date = extract_date(text, ["\u7ed3\u675f\u65e5\u671f", "\u5230\u671f\u65e5\u671f", "\u7ec8\u6b62\u65e5\u671f", "\u5c48\u6ee1\u65e5\u671f", "\u5230\u671f\u4e8e"])
# Extract counterparty
counterparty = extract_counterparty(text)
# Extract key nodes
key_nodes = extract_key_nodes(text)
return {
"contract_name": contract_name,
"amount": amount,
"sign_date": sign_date,
"start_date": start_date,
"end_date": end_date,
"counterparty": counterparty,
"key_nodes": key_nodes,
"status": determine_status(end_date),
}
def extract_amount(text: str) -> Optional[float]:
"""Extract contract amount from text."""
patterns = [
r"\u5408\u540c\u91d1\u989d[：:]\s*([\d,，.]+)",
r"\u603b\u4ef7\u6b3e?[：:]\s*([\d,，.]+)",
r"\u603b\u4ef7[：:]\s*([\d,，.]+)",
r"([\d,，.]+)\s*\u5143",
r"¥\s*([\d,，.]+)",
r"RMB\s*([\d,，.]+)",
]
for pattern in patterns:
match = re.search(pattern, text)
if match:
amount_str = match.group(1).replace(",", "").replace("\uff0c", ".")
try:
return float(amount_str)
except ValueError:
continue
return None
def extract_date(text: str, keywords: list) -> Optional[str]:
"""Extract date from text using keywords."""
date_pattern = r"(\d{4}[-/\u5e74]\d{1,2}[-/\u6708]\d{1,2}[\u65e5]?)"
for kw in keywords:
idx = text.find(kw)
if idx != -1:
snippet = text[idx:idx+50]
match = re.search(date_pattern, snippet)
if match:
return normalize_date(match.group(1))
match = re.search(date_pattern, text)
if match:
return normalize_date(match.group(1))
return None
def normalize_date(date_str: str) -> str:
"""Normalize date to YYYY-MM-DD format."""
date_str = date_str.replace("\u5e74", "-").replace("\u6708", "-").replace("\u65e5", "")
parts = re.split(r"[-/]", date_str)
if len(parts) == 3:
return f"{int(parts[0]):04d}-{int(parts[1]):02d}-{int(parts[2]):02d}"
return date_str
def extract_counterparty(text: str) -> Optional[str]:
"""Extract counterparty company name."""
patterns = [
r"\u4e59\u65b9[：:]\s*([^\s\uff0c\uff0c\uff0c]+)",
r"\u5bf9\u65b9[：:]\s*([^\s\uff0c\uff0c\uff0c]+)",
r"\u4f9b\u5e94\u5546[：:]\s*([^\s\uff0c\uff0c\uff0c]+)",
r"\u670d\u52a1\u5546[：:]\s*([^\s\uff0c\uff0c\uff0c]+)",
r"\u59d4\u6258\u65b9[：:]\s*([^\s\uff0c\uff0c\uff0c]+)",
]
for pattern in patterns:
match = re.search(pattern, text)
if match:
return match.group(1).strip()
return None
def extract_key_nodes(text: str) -> list:
"""Extract key contract nodes (payment terms, renewal, etc.)."""
nodes = []
payment_patterns = [
r"\u4ed8\u6b3e\u65b9\u5f0f[：:][^\n\u3002]+",
r"\u652f\u4ed8\u65b9\u5f0f[：:][^\n\u3002]+",
r"\u4ed8\u6b3e\u6761\u4ef6[：:][^\n\u3002]+",
]
for p in payment_patterns:
m = re.search(p, text)
if m:
nodes.append(m.group(0).strip())
renewal_patterns = [
r"\u7eed\u7ea6[^\n\u3002]+",
r"\u81ea\u52a8\u7eed\u671f[^\n\u3002]+",
r"\u671f\u6ee1\u540e[^\n\u3002]+",
]
for p in renewal_patterns:
m = re.search(p, text)
if m:
nodes.append(m.group(0).strip())
return nodes[:5]
def determine_status(end_date: Optional[str]) -> str:
"""Determine contract status based on end date."""
if not end_date:
return "Active" # Active
try:
end = datetime.strptime(end_date, "%Y-%m-%d")
now = datetime.now()
if end < now:
return "Expired" # Expired
return "Active" # Active
except ValueError:
return "Active"
FILE:scripts/config.py
"""
Configuration module for Contract Tracker.
No external API validation - billing is handled separately via SkillPay.
Tier is determined by presence of a valid API key: FREE (no key) | PRO (any key).
"""
from dataclasses import dataclass
from typing import Optional
# Tier definitions (2-tier: FREE | PRO)
TIERS = {
"FREE": {
"max_contracts": 5,
"max_reminders": 1,
"export_formats": ["csv"],
},
"PRO": {
"max_contracts": -1, # unlimited
"max_reminders": -1, # unlimited
"export_formats": ["csv", "xlsx", "pdf"],
},
}
FALLBACK_TIER = "FREE"
@dataclass
class TokenInfo:
"""Token validation result."""
valid: bool
tier: str
max_contracts: int
max_reminders: int
export_formats: list
error: Optional[str] = None
class Config:
"""Configuration manager - no external API calls."""
def __init__(self):
self._cache: dict = {}
def validate_token(self, api_key: str) -> TokenInfo:
"""
Validate token. For ClawHub model: any non-empty API key = PRO tier.
No external API call needed - billing is handled by SkillPay separately.
"""
if api_key and api_key.strip():
tier = "PRO"
tier_info = TIERS["PRO"]
return TokenInfo(
valid=True,
tier=tier,
max_contracts=tier_info["max_contracts"],
max_reminders=tier_info["max_reminders"],
export_formats=tier_info["export_formats"],
)
else:
tier = "FREE"
tier_info = TIERS["FREE"]
return TokenInfo(
valid=True, # FREE tier is always valid
tier=tier,
max_contracts=tier_info["max_contracts"],
max_reminders=tier_info["max_reminders"],
export_formats=tier_info["export_formats"],
)
def clear_cache(self, api_key: Optional[str] = None):
"""Clear the validation cache."""
if api_key:
self._cache.pop(api_key, None)
else:
self._cache.clear()
def get_tier_limits(tier: str) -> dict:
"""Get tier limits as a dict (for backward compatibility)."""
tier_info = TIERS.get(tier, TIERS[FALLBACK_TIER])
return {
"max_contracts": tier_info["max_contracts"],
"max_reminders": tier_info["max_reminders"],
"export_formats": tier_info["export_formats"],
}
FILE:scripts/billing.py
"""
Billing module for Contract Tracker (contract-tracker).
Integrates with SkillPay per-call billing.
"""
import os
import requests
import logging
logger = logging.getLogger(__name__)
BILLING_URL = "https://skillpay.me/api/v1/billing"
API_KEY = os.environ.get("SKILL_BILLING_API_KEY", "")
SKILL_ID = os.environ.get("SKILL_BILLING_SKILL_ID", "contract-tracker")
HEADERS = {"X-API-Key": API_KEY, "Content-Type": "application/json"}
CALL_PRICE = 0.0100 # USDT per call
def is_dev_mode() -> bool:
"""Check if running in development mode (no API key configured)."""
return API_KEY in ("", "dev", "test")
def charge_user(user_id: str) -> dict:
"""
Charge a user for one API call.
Returns dict with ok=True/False and balance/payment_url on failure.
"""
if is_dev_mode():
return {"ok": True, "balance": 999.0}
try:
resp = requests.post(
f"{BILLING_URL}/charge",
headers=HEADERS,
json={"user_id": user_id, "skill_id": SKILL_ID, "amount": CALL_PRICE},
timeout=10
)
data = resp.json()
if data.get("success"):
return {"ok": True, "balance": data.get("balance", 0.0)}
return {
"ok": False,
"balance": 0.0,
"payment_url": data.get("payment_url", f"https://skillpay.me/{SKILL_ID}"),
}
except Exception as e:
logger.warning(f"Billing error: {e}")
return {"ok": False, "balance": 0.0, "payment_url": f"https://skillpay.me/{SKILL_ID}"}
FILE:scripts/requirements.txt
PyMuPDF>=1.23.0
requests>=2.28.0
FILE:scripts/feishu_notifier.py
"""
Feishu notification module for Contract Ledger.
Builds Feishu card messages for contract expiry reminders.
"""
from typing import Optional
def build_reminder_card(contract: dict, days_until_expiry: int) -> dict:
"""Build a Feishu reminder card for a contract."""
fields = [
{"is_short": True, "text": {"tag": "lark_md", "content": "**Contract**"}},
{"is_short": True, "text": {"tag": "lark_md", "content": f"{contract.get('contract_name', 'N/A')}"}},
{"is_short": True, "text": {"tag": "lark_md", "content": "**Counterparty**"}},
{"is_short": True, "text": {"tag": "lark_md", "content": f"{contract.get('counterparty', 'N/A')}"}},
{"is_short": True, "text": {"tag": "lark_md", "content": "**End Date**"}},
{"is_short": True, "text": {"tag": "lark_md", "content": f"{contract.get('end_date', 'N/A')}"}},
{"is_short": True, "text": {"tag": "lark_md", "content": "**Days Remaining**"}},
{"is_short": True, "text": {"tag": "lark_md", "content": f"{days_until_expiry} days"}},
]
amount = contract.get("amount")
if amount:
fields.extend([
{"is_short": True, "text": {"tag": "lark_md", "content": "**Amount**"}},
{"is_short": True, "text": {"tag": "lark_md", "content": f"¥{amount:,.2f}"}},
])
card = {
"config": {"wide_screen_mode": True},
"elements": [
{"tag": "markdown", "content": "**Contract Expiry Reminder**"},
{"tag": "hr"},
{"tag": "div", "fields": fields},
{"tag": "hr"},
{"tag": "markdown", "content": "Sent by Contract Tracker"}
],
"header": {
"title": {"tag": "plain_text", "content": "Contract Expiry Reminder"},
"template": "orange"
}
}
return card
def format_reminder_message(contract: dict, days_until_expiry: int) -> str:
"""Format reminder message as plain text."""
name = contract.get("contract_name", "N/A")
counterparty = contract.get("counterparty", "N/A")
end_date = contract.get("end_date", "N/A")
amount = contract.get("amount")
msg = f"Contract Expiry Reminder\n\n"
msg += f"Contract: {name}\n"
msg += f"Counterparty: {counterparty}\n"
msg += f"End Date: {end_date}\n"
msg += f"Days Remaining: {days_until_expiry} days\n"
if amount:
msg += f"Amount: ¥{amount:,.2f}\n"
return msg
FILE:scripts/__init__.py
"""
Contract Ledger - AI-powered contract management tool.
Upload PDF contracts, manage ledger, get expiry reminders.
"""
from .config import Config, TokenInfo, TIERS, FALLBACK_TIER, get_tier_limits
from .pdf_parser import extract_text_from_pdf, extract_contract_fields
from .storage import (
init_storage, add_contract, get_contracts, get_contract,
update_contract, delete_contract, add_reminder, remove_reminder,
get_expiring_contracts, count_contracts, export_contracts
)
from .feishu_notifier import build_reminder_card, format_reminder_message
__all__ = [
"Config", "TokenInfo", "TIERS", "FALLBACK_TIER", "get_tier_limits",
"extract_text_from_pdf", "extract_contract_fields",
"init_storage", "add_contract", "get_contracts", "get_contract",
"update_contract", "delete_contract", "add_reminder", "remove_reminder",
"get_expiring_contracts", "count_contracts", "export_contracts",
"build_reminder_card", "format_reminder_message",
]
FILE:scripts/storage.py
"""
Storage module for Contract Ledger.
JSON file local storage using /tmp/contract-tracker/ (no home directory writes).
"""
import json
import uuid
from pathlib import Path
from datetime import datetime
from typing import Optional
STORAGE_DIR = Path("/tmp/contract-tracker")
LEDGER_FILE = STORAGE_DIR / "contracts.json"
def init_storage():
"""Initialize storage directory and file."""
STORAGE_DIR.mkdir(parents=True, exist_ok=True)
if not LEDGER_FILE.exists():
_write_ledger([])
def _read_ledger() -> list:
"""Read ledger from file."""
try:
with open(LEDGER_FILE, "r", encoding="utf-8") as f:
return json.load(f)
except Exception:
return []
def _write_ledger(contracts: list):
"""Write ledger to file."""
with open(LEDGER_FILE, "w", encoding="utf-8") as f:
json.dump(contracts, f, ensure_ascii=False, indent=2)
def add_contract(fields: dict) -> dict:
"""Add a contract."""
contracts = _read_ledger()
contract = {
"id": str(uuid.uuid4())[:8],
"created_at": datetime.now().isoformat(),
"updated_at": datetime.now().isoformat(),
**fields,
"reminders": [],
}
contracts.append(contract)
_write_ledger(contracts)
return contract
def get_contracts(
status: Optional[str] = None,
sort_by: str = "end_date",
reverse: bool = True
) -> list:
"""Get contract list."""
contracts = _read_ledger()
if status:
contracts = [c for c in contracts if c.get("status") == status]
contracts.sort(
key=lambda x: x.get(sort_by, "" or "9999-12-31"),
reverse=reverse
)
return contracts
def get_contract(contract_id: str) -> Optional[dict]:
"""Get a single contract by ID."""
contracts = _read_ledger()
for c in contracts:
if c.get("id") == contract_id:
return c
return None
def update_contract(contract_id: str, updates: dict) -> Optional[dict]:
"""Update a contract."""
contracts = _read_ledger()
for i, c in enumerate(contracts):
if c.get("id") == contract_id:
contracts[i].update(updates)
contracts[i]["updated_at"] = datetime.now().isoformat()
_write_ledger(contracts)
return contracts[i]
return None
def delete_contract(contract_id: str) -> bool:
"""Delete a contract."""
contracts = _read_ledger()
original_len = len(contracts)
contracts = [c for c in contracts if c.get("id") != contract_id]
if len(contracts) < original_len:
_write_ledger(contracts)
return True
return False
def add_reminder(contract_id: str, days_before: int, enabled: bool = True) -> bool:
"""Add a reminder to a contract."""
contract = get_contract(contract_id)
if not contract:
return False
reminders = contract.get("reminders", [])
reminders.append({"days_before": days_before, "enabled": enabled})
update_contract(contract_id, {"reminders": reminders})
return True
def remove_reminder(contract_id: str, index: int) -> bool:
"""Remove a reminder from a contract."""
contract = get_contract(contract_id)
if not contract:
return False
reminders = contract.get("reminders", [])
if 0 <= index < len(reminders):
reminders.pop(index)
update_contract(contract_id, {"reminders": reminders})
return True
return False
def get_expiring_contracts(days: int = 7) -> list:
"""Get contracts expiring within N days."""
contracts = _read_ledger()
expiring = []
now = datetime.now()
for c in contracts:
if c.get("status") == "Expired":
continue
end_date_str = c.get("end_date")
if not end_date_str:
continue
try:
end_date = datetime.strptime(end_date_str, "%Y-%m-%d")
delta = (end_date - now).days
if 0 <= delta <= days:
c["days_until_expiry"] = delta
expiring.append(c)
except ValueError:
continue
return expiring
def count_contracts() -> int:
"""Count total contracts."""
return len(_read_ledger())
def export_contracts(contracts: list, format: str = "csv") -> str:
"""Export contract data."""
if not contracts:
return ""
if format == "csv":
return _export_csv(contracts)
elif format == "json":
return json.dumps(contracts, ensure_ascii=False, indent=2)
else:
return _export_csv(contracts)
def _export_csv(contracts: list) -> str:
"""Export to CSV format."""
if not contracts:
return ""
headers = ["id", "contract_name", "amount", "counterparty", "sign_date",
"start_date", "end_date", "status", "key_nodes"]
lines = [",".join(headers)]
for c in contracts:
row = [
c.get("id", ""),
c.get("contract_name", ""),
str(c.get("amount", "")),
c.get("counterparty", ""),
c.get("sign_date", ""),
c.get("start_date", ""),
c.get("end_date", ""),
c.get("status", ""),
"|".join(c.get("key_nodes", []))
]
lines.append(",".join(f'"{v}"' for v in row))
return "\n".join(lines)
FILE:scripts/main.py
#!/usr/bin/env python3
"""
Contract Ledger CLI - Main entry point.
Upload PDF contracts, manage ledger, get expiry reminders + Feishu notifications.
"""
import argparse
import sys
import json
from pathlib import Path
sys.path.insert(0, str(Path(__file__).parent))
from config import Config, get_tier_limits
from pdf_parser import extract_text_from_pdf, extract_contract_fields
from storage import (
init_storage, add_contract, get_contracts, get_contract,
update_contract, delete_contract, add_reminder, remove_reminder,
get_expiring_contracts, count_contracts, export_contracts
)
from feishu_notifier import build_reminder_card, format_reminder_message
from billing import is_dev_mode, charge_user
DEFAULT_API_KEY = ""
def cmd_upload(args):
"""Upload and parse a contract PDF."""
api_key = args.api_key or DEFAULT_API_KEY
if is_dev_mode():
print("Dev mode: Set SKILL_BILLING_API_KEY for full functionality.", file=sys.stderr)
billing_result = charge_user("cli_upload")
if not billing_result.get("ok"):
print(f"Error: Insufficient balance. Please recharge at https://skillpay.me/contract-tracker", file=sys.stderr)
return 1
config = Config()
token_info = config.validate_token(api_key)
tier = token_info.tier
limits = get_tier_limits(tier)
# Check contract limit
current_count = count_contracts()
max_contracts = limits["max_contracts"]
if max_contracts != -1 and current_count >= max_contracts:
print(f"Tier limit reached ({tier}: {max_contracts} contracts)", file=sys.stderr)
print(f"Current: {current_count}", file=sys.stderr)
return 1
# Extract text and fields
try:
text = extract_text_from_pdf(args.pdf_file)
fields = extract_contract_fields(text, Path(args.pdf_file).name)
except Exception as e:
print(f"PDF parsing failed: {e}", file=sys.stderr)
return 1
# Add contract
contract = add_contract(fields)
print(f"Contract added (ID: {contract['id']})")
print(f" Name: {fields.get('contract_name', 'N/A')}")
print(f" Counterparty: {fields.get('counterparty', 'N/A')}")
print(f" End Date: {fields.get('end_date', 'N/A')}")
print(f" Status: {fields.get('status', 'N/A')}")
if fields.get("amount"):
print(f" Amount: ¥{fields['amount']:,.2f}")
return 0
def cmd_list(args):
"""List contracts."""
contracts = get_contracts(status=args.status, sort_by=args.sort, reverse=not args.asc)
if not contracts:
print("No contracts found.")
return 0
print(f"\nContract Ledger ({len(contracts)} contracts)")
print("-" * 80)
for c in contracts:
amount_str = f"¥{c['amount']:,.2f}" if c.get("amount") else "-"
print(f"[{c['id']}] {c.get('contract_name', 'N/A')}")
print(f" Counterparty: {c.get('counterparty', '-')} | End: {c.get('end_date', '-')} | Amount: {amount_str}")
print(f" Status: {c.get('status', '-')}")
print()
return 0
def cmd_get(args):
"""Get a single contract."""
contract = get_contract(args.contract_id)
if not contract:
print(f"Contract not found: {args.contract_id}", file=sys.stderr)
return 1
print(f"\nContract Details ({contract['id']})")
print("-" * 40)
for k, v in contract.items():
if k == "key_nodes" and isinstance(v, list):
print(f" {k}:")
for node in v:
print(f" - {node}")
elif k == "reminders":
print(f" {k}: {json.dumps(v, ensure_ascii=False)}")
elif v is not None:
print(f" {k}: {v}")
return 0
def cmd_update(args):
"""Update a contract."""
updates = {}
if args.name:
updates["contract_name"] = args.name
if args.counterparty:
updates["counterparty"] = args.counterparty
if args.amount:
updates["amount"] = float(args.amount)
if args.end_date:
updates["end_date"] = args.end_date
if args.status:
updates["status"] = args.status
if not updates:
print("No updates provided", file=sys.stderr)
return 1
result = update_contract(args.contract_id, updates)
if result:
print(f"Contract updated: {args.contract_id}")
return 0
else:
print(f"Update failed: {args.contract_id}", file=sys.stderr)
return 1
def cmd_delete(args):
"""Delete a contract."""
if delete_contract(args.contract_id):
print(f"Contract deleted: {args.contract_id}")
return 0
else:
print(f"Delete failed: {args.contract_id}", file=sys.stderr)
return 1
def cmd_reminder(args):
"""Manage reminders."""
if args.action == "add":
if add_reminder(args.contract_id, args.days):
print(f"Reminder added ({args.days} days before expiry)")
else:
print(f"Failed to add reminder", file=sys.stderr)
return 1
elif args.action == "remove":
if remove_reminder(args.contract_id, args.index):
print("Reminder removed")
else:
print("Failed to remove reminder", file=sys.stderr)
return 1
elif args.action == "list":
contract = get_contract(args.contract_id)
if not contract:
print("Contract not found", file=sys.stderr)
return 1
reminders = contract.get("reminders", [])
if not reminders:
print("No reminders set")
else:
print(f"Reminders ({len(reminders)}):")
for i, r in enumerate(reminders):
status = "ON" if r.get("enabled") else "OFF"
print(f" [{i}] [{status}] {r['days_before']} days before expiry")
return 0
def cmd_check(args):
"""Check expiring contracts."""
api_key = args.api_key or DEFAULT_API_KEY
days = args.days or 7
billing_result = charge_user("cli_check")
if not billing_result.get("ok"):
print(f"Error: Insufficient balance.", file=sys.stderr)
return 1
expiring = get_expiring_contracts(days)
if not expiring:
print(f"No contracts expiring within {days} days")
return 0
print(f"{len(expiring)} contract(s) expiring within {days} days:\n")
for c in expiring:
days_left = c.get("days_until_expiry", 0)
print(f" [{c['id']}] {c.get('contract_name', 'N/A')}")
print(f" End: {c.get('end_date')} ({days_left} days remaining)")
print()
if args.feishu and expiring:
card = build_reminder_card(expiring[0], expiring[0].get("days_until_expiry", 0))
print("\nFeishu card content:")
print(json.dumps(card, ensure_ascii=False, indent=2))
return 0
def cmd_export(args):
"""Export contracts."""
api_key = args.api_key or DEFAULT_API_KEY
config = Config()
token_info = config.validate_token(api_key)
tier = token_info.tier
limits = get_tier_limits(tier)
format_type = args.format or "csv"
if format_type not in limits["export_formats"]:
print(f"Tier {tier} does not support {format_type} export", file=sys.stderr)
print(f"Supported: {', '.join(limits['export_formats'])}", file=sys.stderr)
return 1
contracts = get_contracts(status=args.status)
if not contracts:
print("No contracts to export")
return 0
content = export_contracts(contracts, format_type)
if args.output:
with open(args.output, "w", encoding="utf-8") as f:
f.write(content)
print(f"Exported to: {args.output}")
else:
print(content)
return 0
def main():
parser = argparse.ArgumentParser(description="Contract Ledger Management Tool")
subparsers = parser.add_subparsers(dest="command", help="Subcommands")
p_upload = subparsers.add_parser("upload", help="Upload contract PDF")
p_upload.add_argument("pdf_file", help="PDF file path")
p_upload.add_argument("--api-key", help="API Key (optional)")
p_upload.set_defaults(func=cmd_upload)
p_list = subparsers.add_parser("list", help="List contracts")
p_list.add_argument("--status", help="Filter by status")
p_list.add_argument("--sort", default="end_date", help="Sort field")
p_list.add_argument("--asc", action="store_true", help="Sort ascending")
p_list.set_defaults(func=cmd_list)
p_get = subparsers.add_parser("get", help="Get contract details")
p_get.add_argument("contract_id", help="Contract ID")
p_get.set_defaults(func=cmd_get)
p_update = subparsers.add_parser("update", help="Update contract")
p_update.add_argument("contract_id", help="Contract ID")
p_update.add_argument("--name", help="Contract name")
p_update.add_argument("--counterparty", help="Counterparty")
p_update.add_argument("--amount", help="Amount")
p_update.add_argument("--end-date", dest="end_date", help="End date (YYYY-MM-DD)")
p_update.add_argument("--status", help="Status")
p_update.set_defaults(func=cmd_update)
p_delete = subparsers.add_parser("delete", help="Delete contract")
p_delete.add_argument("contract_id", help="Contract ID")
p_delete.set_defaults(func=cmd_delete)
p_reminder = subparsers.add_parser("reminder", help="Manage reminders")
p_reminder.add_argument("contract_id", help="Contract ID")
p_reminder.add_argument("action", choices=["add", "remove", "list"], help="Action")
p_reminder.add_argument("--days", type=int, help="Days before expiry (for add)")
p_reminder.add_argument("--index", type=int, help="Reminder index (for remove)")
p_reminder.set_defaults(func=cmd_reminder)
p_check = subparsers.add_parser("check", help="Check expiring contracts")
p_check.add_argument("--days", type=int, default=7, help="Days to check")
p_check.add_argument("--api-key", help="API Key")
p_check.add_argument("--feishu", action="store_true", help="Output Feishu card")
p_check.set_defaults(func=cmd_check)
p_export = subparsers.add_parser("export", help="Export contracts")
p_export.add_argument("--format", choices=["csv", "xlsx", "pdf"], help="Export format")
p_export.add_argument("--status", help="Filter by status")
p_export.add_argument("--output", "-o", help="Output file path")
p_export.add_argument("--api-key", help="API Key")
p_export.set_defaults(func=cmd_export)
args = parser.parse_args()
init_storage()
if args.command is None:
parser.print_help()
return 0
return args.func(args)
if __name__ == "__main__":
sys.exit(main())
Upload Excel, CSV, or PDF financial statements for AI-generated detailed business analysis, including revenue, costs, profitability, cash flow, and anomaly a...
# Financial Report AI (ai-financial-report)
> Upload Excel/CSV/PDF financial statements → AI auto-generates structured business analysis reports (revenue structure / cost anomalies / profitability / cash flow / balance sheet / KPI achievement / anomaly alerts).
## Tiered Features
| Feature | FREE | PRO |
|---------|:----:|:---:|
| Analyses/month | 3 | Unlimited |
| Input formats | CSV, Excel | CSV, Excel, PDF |
| Analysis dimensions | 3 basic | All 7 |
| Charts | ❌ | ✅ |
| Industry comparison | ❌ | ✅ |
| **Price** | **Free** | **$0.01 USDT/use** |
> Upgrade to PRO: [https://skillpay.me/ai-financial-report](https://skillpay.me/ai-financial-report)
---
## Architecture
```
User uploads file
↓
index.js (entry, routes to handlers)
↓
src/handlers/
├── skill_invoke.js ← core analysis engine dispatcher
├── file_upload.js ← file upload handler
└── message_handler.js ← text chat handler
↓
src/services/
├── billing.js ← SkillPay token validation + 5-min cache
├── file_parser.py ← Excel/CSV/PDF parsing
└── report_generator.py ← AI analysis + Markdown rendering
```
## Quick Start
### Upload a File (Recommended)
Upload your Excel/CSV/PDF financial file directly — AI automatically completes the full analysis.
**Supported formats**: `.csv`, `.xlsx`, `.xls`, `.pdf`
### Configure AI API Key
This skill does **not** include an AI model. Users configure their own API key.
Supported AI models (any one):
| Model | Provider | Get API Key |
|-------|----------|-------------|
| GPT-4o | OpenAI | platform.openai.com |
| Claude 3.5 | Anthropic | console.anthropic.com |
| DeepSeek V3 | DeepSeek | platform.deepseek.com |
| Qwen | Alibaba Cloud | bailian.console.aliyun.com |
| MiniMax | MiniMax | platform.minimax.chat |
> **No binding, no recommendation, no restriction** on specific models — user chooses freely.
---
## Output Example
```markdown
# Financial Report Analysis
**Company**: XX Tech Co.
**Period**: Q1 2024
**Tier**: PRO
---
## 1. Revenue Structure
| Item | Value |
|------|-------|
| Total Revenue | 3,800 (10K CNY) |
| YoY Change | +15.3% |
| QoQ Change | +8.2% |
**Structure**: Core business 82%, other business 18%
---
## 7. Anomaly Alerts
| Dimension | Severity | Description | Value |
|-----------|----------|-------------|-------|
| Cost | 🔴 HIGH | Admin expense ratio abnormally high | 18.5% (avg: 12%) |
| Cash Flow | 🟠 MEDIUM | Operating cash flow YoY declined | -12.3% |
```
---
## Data Format
| Format | Notes |
|--------|-------|
| CSV | UTF-8, first row = header |
| Excel (.xlsx) | Multi-sheet, reads first sheet by default |
| PDF | Text must be copyable (no scanned images) |
**Column guidelines**: Use clear dimension names (revenue, cost, profit, etc.). Avoid excessive merged cells.
---
## Privacy
- **No data upload**: All files processed locally, never sent to third-party servers
- **No file storage**: Temporary files deleted immediately after analysis
- **API calls**: Only uses user-configured AI API, data processed locally
- **Token validation**: Only verifies plan eligibility, no financial data stored
---
## Error Handling
| Error | Resolution |
|-------|-----------|
| "Unsupported format" | Use CSV, Excel (.xlsx/.xls), or PDF with copyable text |
| AI analysis failed | Check API key validity and balance; try another model |
| Report data inaccurate | AI analysis is for reference only; verify against source files |
---
## Tech Stack
- **Parsing**: Python 3 + pandas + openpyxl + pdfplumber
- **AI Interface**: OpenAI-compatible REST API
- **Runtime**: Node.js (OpenClaw Agent)
---
## Billing
- **Endpoint**: `POST https://skillpay.me/api/v1/billing/charge`
- **Header**: `X-API-Key: {api_key}`
- **Body**: `{"user_id": "...", "skill_id": "ai-financial-report", "amount": 0}`
- **Response**: `{"success": true, "balance": ...}`
- **Fallback**: Network error → FREE tier (usage not blocked)
- **Cache**: Validation result cached locally (SHA256 hash), TTL 5 minutes
## Env Variables
| Variable | Description | Default |
|----------|-------------|---------|
| `OPENCLAW_SKILL_DIR` | Skill root (for cache) | `__dirname/..` |
| `SKILL_BILLING_API_KEY` | Builder API Key (from SkillPay) | — |
| `SKILL_BILLING_SKILL_ID` | Skill Slug | `ai-financial-report` |
> For builder setup: visit [https://skillpay.me](https://skillpay.me)
FILE:requirements.txt
openpyxl>=3.1.0
pandas>=2.0.0
numpy>=1.24.0
matplotlib>=3.7.0
plotly>=5.18.0
pdfplumber>=0.10.0
tabulate>=0.9.0
kaleido>=0.2.1
FILE:index.js
#!/usr/bin/env node
/**
* Skill: Financial Report AI (ai-financial-report)
* Entry point - OpenClaw-compatible skill
*
* Tier System (per-use billing):
* FREE - 3 analyses/month, basic 3 dimensions
* PRO - $0.01 USDT/use, all 7 dimensions + charts
*
* Core Features:
* 1. Revenue structure analysis
* 2. Cost anomaly detection
* 3. Profitability analysis
* 4. Cash flow analysis
* 5. Balance sheet analysis
* 6. KPI achievement
* 7. Anomaly alerts
*/
const path = require('path');
const fs = require('fs');
// Resolve skill root
const SKILL_ROOT = __dirname;
// Load .env if present
const envPath = path.join(SKILL_ROOT, '.env');
if (fs.existsSync(envPath)) {
fs.readFileSync(envPath, 'utf8')
.split('\n')
.forEach(line => {
const idx = line.indexOf('=');
if (idx < 0 || line.startsWith('#')) return;
const k = line.slice(0, idx).trim();
const v = line.slice(idx + 1).trim();
if (k && !process.env[k]) process.env[k] = v;
});
}
// Lazy-load handlers to avoid circular deps
function getHandler(name) {
try {
return require(`./src/handlers/name`);
} catch (_) {
return null;
}
}
const skillInvoke = getHandler('skill_invoke');
const fileUpload = getHandler('file_upload');
const messageHandler = getHandler('message');
const skill = {
id: 'ai-financial-report',
name: 'Financial Report AI',
description: 'Upload Excel/CSV/PDF financial statements → AI auto-generates structured business analysis reports (revenue structure / cost anomalies / profitability / cash flow / balance sheet / KPI achievement / anomaly alerts). Per-use billing at $0.01 USDT.',
version: '1.0.0',
author: 'YK Global',
async invoke(params, context) {
if (!skillInvoke) {
return { success: false, error: 'skill_invoke handler not found' };
}
return skillInvoke.handleSkillInvoke(params, context);
},
configSchema: {
type: 'object',
properties: {
apiKey: {
type: 'string',
title: 'AI API Key',
description: 'User-configured AI model API Key (no binding/recommendation/restriction on model choice)',
},
defaultModel: {
type: 'string',
title: 'Default Model',
default: 'gpt-4o',
description: 'Default AI model to use',
},
chartTheme: {
type: 'string',
title: 'Chart Theme',
default: 'light',
enum: ['light', 'dark'],
},
},
required: [],
},
skillRoot: SKILL_ROOT,
handlers: {
async 'skill.invoke'(params, context) {
return this.invoke(params, context);
},
async 'file.upload'(params, context) {
if (!fileUpload) return { success: false, error: 'file_upload handler not found' };
return fileUpload.handleFileUpload(params, context);
},
async 'message.create'(params, context) {
if (!messageHandler) return { success: false, error: 'message_handler not found' };
return messageHandler.handleMessage(params, context);
},
},
};
module.exports = skill;
module.exports.default = skill;
FILE:README.md
# Financial Report AI
> Upload Excel/CSV/PDF financial statements → AI auto-generates structured business analysis reports.
**Supported formats**: CSV / Excel (.xlsx/.xls) / PDF
**Analysis dimensions**: Revenue structure · Cost anomalies · Profitability · Cash flow · Balance sheet · KPI achievement · Anomaly alerts
---
## Features
| Analysis Dimension | Description |
|-------------------|-------------|
| Revenue Structure | Total revenue, YoY/QoQ, business line breakdown |
| Cost Anomaly Detection | Cost breakdown, anomaly flagging |
| Profitability | Gross margin, net margin, profit trends |
| Cash Flow | Operating/investing/financing cash flow |
| Balance Sheet | Asset structure, debt ratio, solvency |
| KPI Achievement | Budget vs actual comparison |
| Anomaly Alerts | 🔴 HIGH / 🟠 MEDIUM / 🟡 LOW severity alerts |
---
## Tier Comparison
| | **FREE** | **PRO** |
|---|:---:|:---:|
| Analyses/month | 3 | Unlimited |
| Input formats | CSV, Excel | CSV, Excel, PDF |
| Analysis dimensions | 3 basic | All 7 |
| Charts | ❌ | ✅ |
| Industry comparison | ❌ | ✅ |
| **Price** | **Free** | **$0.01 USDT/use** |
---
## Quick Start
### Upload a File (Recommended)
Upload your Excel/CSV/PDF financial file directly — AI automatically completes the full analysis.
### Configure AI API Key
This skill does **not** include an AI model. Users configure their own API key.
Supported AI models (any one):
| Model | Provider |
|-------|----------|
| GPT-4o | OpenAI |
| Claude 3.5 | Anthropic |
| DeepSeek V3 | DeepSeek |
| Qwen | Alibaba Cloud |
| MiniMax | MiniMax |
> No binding, no recommendation, no restriction on model choice.
---
## Privacy
- No data upload: All files processed locally
- No file storage: Temporary files deleted immediately after analysis
- API calls: Only uses user-configured AI API
- Token validation: Only verifies plan eligibility
---
## Tech Stack
- **Parsing**: Python 3 + pandas + openpyxl + pdfplumber
- **AI Interface**: OpenAI-compatible REST API
- **Runtime**: Node.js (OpenClaw Agent)
---
> Get PRO: [https://skillpay.me/ai-financial-report](https://skillpay.me/ai-financial-report)
FILE:package.json
{
"name": "ai-financial-report",
"version": "1.0.0",
"description": "Financial Report AI - Upload Excel/CSV/PDF, AI auto-generates structured business analysis reports",
"main": "index.js",
"scripts": {
"analyze": "python3 src/services/report_generator.py",
"parse": "python3 src/services/file_parser.py"
},
"keywords": ["financial", "report", "ai", "excel", "csv", "pdf", "analysis"],
"author": "YK Global",
"license": "MIT"
}
FILE:src/services/file_parser.py
#!/usr/bin/env python3
"""
File Parser for Financial Report AI
Supports: CSV, XLSX, XLS, PDF
Extracts structured tabular data from uploaded financial statements.
"""
import sys
import json
import os
import traceback
from pathlib import Path
def parse_csv(filepath):
"""Parse CSV file into a list of row dicts."""
import pandas as pd
df = pd.read_csv(filepath, dtype=str, keep_default_na=False)
df = df.fillna("")
return {
"headers": list(df.columns),
"rows": df.values.tolist(),
"shape": list(df.shape),
"raw_sample": df.head(20).to_dict(orient="records"),
}
def parse_excel(filepath):
"""Parse Excel file - auto-detect sheet, return all sheets."""
import pandas as pd
xl = pd.ExcelFile(filepath)
sheets = {}
for sheet_name in xl.sheet_names:
df = pd.read_excel(filepath, sheet_name=sheet_name, dtype=str, header=None, keep_default_na=False)
df = df.fillna("")
sheets[sheet_name] = {
"headers": [str(h) for h in df.iloc[0].tolist()],
"rows": df.iloc[1:].values.tolist(),
"shape": [df.shape[0] - 1, df.shape[1]],
"raw_sample": df.head(20).to_dict(orient="records"),
}
return {
"sheets": sheets,
"sheet_names": xl.sheet_names,
}
def parse_pdf(filepath):
"""Parse PDF financial statements using pdfplumber."""
import pdfplumber
import pandas as pd
tables = []
all_text = []
try:
with pdfplumber.open(filepath) as pdf:
for i, page in enumerate(pdf.pages):
text = page.extract_text() or ""
all_text.append({"page": i + 1, "text": text})
# Try to extract tables
page_tables = page.extract_tables()
for j, table in enumerate(page_tables):
if table:
headers = table[0] if table else []
rows = table[1:] if len(table) > 1 else []
tables.append({
"page": i + 1,
"table_index": j,
"headers": [str(h or "").strip() for h in headers],
"rows": [[str(cell or "").strip() for cell in row] for row in rows],
"shape": [len(rows), len(headers)],
})
except Exception as e:
return {"error": str(e), "tables": [], "text": []}
return {
"tables": tables,
"text": all_text,
"num_pages": len(all_text),
}
def parse_file(filepath, file_ext=None):
"""Main dispatcher - auto-detect format from extension or content."""
if file_ext is None:
file_ext = Path(filepath).suffix.lower()
if file_ext in [".csv"]:
return {"format": "csv", **parse_csv(filepath)}
elif file_ext in [".xlsx", ".xls"]:
return {"format": "excel", **parse_excel(filepath)}
elif file_ext in [".pdf"]:
return {"format": "pdf", **parse_pdf(filepath)}
else:
return {"error": f"Unsupported format: {file_ext}"}
if __name__ == "__main__":
if len(sys.argv) < 2:
print(json.dumps({"error": "Usage: python file_parser.py <filepath>"}))
sys.exit(1)
filepath = sys.argv[1]
if not os.path.exists(filepath):
print(json.dumps({"error": f"File not found: {filepath}"}))
sys.exit(1)
try:
result = parse_file(filepath)
print(json.dumps(result, ensure_ascii=False, default=str))
except Exception as e:
print(json.dumps({"error": str(e), "trace": traceback.format_exc()}, ensure_ascii=False))
sys.exit(1)
FILE:src/services/billing.js
/**
* Billing Service
* Validates tokens via SkillPay billing API
* Caches results for 5 minutes to reduce API calls
*
* Fallback: on network error → FREE tier (do not block usage)
*/
const path = require('path');
const fs = require('fs');
const crypto = require('crypto');
// --- Config ---
const BILLING_URL = 'https://skillpay.me/api/v1/billing';
const API_KEY = process.env.SKILL_BILLING_API_KEY || '';
const SKILL_ID = process.env.SKILL_BILLING_SKILL_ID || 'ai-financial-report';
const CACHE_TTL_MS = 5 * 60 * 1000; // 5 minutes
const DEV_MODE = !API_KEY;
const CACHE_DIR = '/tmp/ai-financial-report-cache';
// Ensure cache directory exists
try {
if (!fs.existsSync(CACHE_DIR)) fs.mkdirSync(CACHE_DIR, { recursive: true });
} catch (_) {}
// --- Cache helpers ---
function cacheKey(userId) {
return crypto.createHash('sha256').update(userId || 'anon').digest('hex') + '.json';
}
function readCache(userId) {
try {
const file = path.join(CACHE_DIR, cacheKey(userId));
if (!fs.existsSync(file)) return null;
const data = JSON.parse(fs.readFileSync(file, 'utf8'));
if (Date.now() - data.ts > CACHE_TTL_MS) {
fs.unlinkSync(file);
return null;
}
return data;
} catch (_) {
return null;
}
}
function writeCache(userId, result) {
try {
const file = path.join(CACHE_DIR, cacheKey(userId));
fs.writeFileSync(file, JSON.stringify({ ...result, ts: Date.now() }), 'utf8');
} catch (_) {}
}
// --- Billing / token validation ---
async function validateToken(apiKey, userId = '') {
// Dev mode: no API key configured
if (DEV_MODE) {
return { valid: true, plan: 'PRO', balance: 999.0, reason: 'dev_mode' };
}
if (!apiKey || apiKey.trim() === '') {
return { valid: false, plan: 'FREE', reason: 'no_api_key' };
}
// Check cache first
const cacheKeyVal = apiKey + '|' + (userId || 'anon');
const cached = readCache(cacheKeyVal);
if (cached) return cached;
try {
const controller = new AbortController();
const timeout = setTimeout(() => controller.abort(), 10000);
const response = await fetch(`BILLING_URL/charge`, {
method: 'POST',
headers: {
'Content-Type': 'application/json',
'X-API-Key': API_KEY,
},
body: JSON.stringify({
user_id: userId || apiKey,
skill_id: SKILL_ID,
amount: 0,
}),
signal: controller.signal,
});
clearTimeout(timeout);
let data;
try {
data = await response.json();
} catch (_) {
data = {};
}
if (response.ok && data.success) {
const result = {
valid: true,
plan: 'PRO',
balance: data.balance || 0,
};
writeCache(cacheKeyVal, result);
return result;
} else {
return { valid: false, plan: 'FREE', reason: data.error || 'charge_failed' };
}
} catch (err) {
// Network error / timeout → degrade to FREE tier, do not block
console.error('[Billing] Validation failed, degrading to FREE:', err.message);
return { valid: false, plan: 'FREE', reason: 'network_error' };
}
}
// --- Plan limits ---
const PLAN_LIMITS = {
FREE: { monthly: 3, formats: ['csv', 'xlsx'], dimensions: 3, charts: 0 },
PRO: { monthly: Infinity, formats: ['csv', 'xlsx', 'pdf'], dimensions: 99, charts: 99 },
};
function getPlanLimits(plan) {
return PLAN_LIMITS[plan] || PLAN_LIMITS.FREE;
}
module.exports = { validateToken, getPlanLimits, DEV_MODE };
FILE:src/services/report_generator.py
#!/usr/bin/env python3
"""
Financial Report AI Analysis Engine
Generates structured AI-powered financial analysis reports.
All output labels are in English for ClawHub compliance.
"""
import sys
import json
import re
import os
import traceback
from pathlib import Path
from typing import Optional, List, Dict, Any
from datetime import datetime
# ─────────────────────────────────────────────────────────────────────────────
# Number parsing helpers
# ─────────────────────────────────────────────────────────────────────────────
def parse_number(val) -> Optional[float]:
"""Parse a value to float, handling currency/comma formats."""
if val is None or val == "":
return None
s = str(val).strip()
s = re.sub(r'[¥$,\uff04\uffe5\s]', '', s)
if s.startswith('(') and s.endswith(')'):
s = '-' + s[1:-1]
try:
return float(s)
except ValueError:
return None
def pct_change(current, previous) -> Optional[float]:
"""Calculate percentage change."""
curr = parse_number(current)
prev = parse_number(previous)
if curr is None or prev is None or prev == 0:
return None
return round((curr - prev) / abs(prev) * 100, 2)
def detect_column_type(headers, rows) -> Dict[str, str]:
"""Auto-detect column types by header keywords."""
type_map = {}
patterns = {
"revenue": ["revenue", "sales", "income", "收入", "营业收入", "销售额"],
"gross_profit": ["gross", "gross_profit", "毛利", "毛利润"],
"net_profit": ["net", "net_profit", "净利润", "净利", "纯利"],
"total_cost": ["cost", "total_cost", "成本", "营业成本"],
"operating_cost": ["operating_cost", "运营成本", "经营成本"],
"admin_cost": ["admin", "管理费用", "管理费"],
"rd_cost": ["rd", "research", "研发费用", "研发"],
"sales_cost": ["selling", "marketing", "销售费用", "销售"],
"financial_cost": ["financial", "财务费用", "财务"],
"cashflow_operating": ["operating_cashflow", "cashflow_operating", "经营活动现金流", "经营现金流"],
"cashflow_investing": ["investing_cashflow", "投资活动现金流", "投资现金流"],
"cashflow_financing": ["financing_cashflow", "筹资活动现金流", "筹资现金流"],
"total_assets": ["total_assets", "assets", "总资产", "资产总计"],
"total_liabilities": ["total_liabilities", "liabilities", "总负债", "负债合计"],
"equity": ["equity", "shareholders_equity", "所有者权益", "净资产"],
"current_assets": ["current_assets", "流动资产"],
"current_liabilities": ["current_liabilities", "流动负债"],
"fixed_assets": ["fixed_assets", "固定资产"],
}
for col_idx, header in enumerate(headers):
h = str(header).lower()
for col_type, keywords in patterns.items():
if any(kw in h for kw in keywords):
type_map[col_idx] = col_type
break
return type_map
# ─────────────────────────────────────────────────────────────────────────────
# AI Prompt Template (English for broad AI model compatibility)
# ─────────────────────────────────────────────────────────────────────────────
ANALYSIS_PROMPT_TEMPLATE = """You are a professional financial analyst. Based on the following financial statement data, generate a structured business analysis report.
Data source:
{file_info}
Data content ({sheet_name}):
Headers: {headers}
Data rows (first 20):
{rows}
---
Please analyze across 7 dimensions. Only analyze dimensions where data is available. If a dimension cannot be identified from the data, state "Not detected in data":
## 1. Revenue Structure Analysis
- Total revenue amount
- YoY / QoQ change (if available)
- Revenue breakdown by business/product line (if identifiable)
- Revenue trend assessment
## 2. Cost Anomaly Detection
- Total cost and cost-to-revenue ratio
- Cost breakdown (operating/admin/sales/R&D/financial expenses)
- Anomaly flagging (compared to industry benchmarks, flag red if threshold exceeded)
## 3. Profitability Analysis
- Gross margin, net margin
- Profit YoY / QoQ change
- Profit quality assessment
## 4. Cash Flow Analysis
- Operating cash flow net amount
- Investing / financing cash flow (if available)
- Cash flow health assessment
## 5. Balance Sheet Analysis
- Asset structure (current / non-current assets)
- Liability structure (current / non-current liabilities)
- Debt-to-asset ratio
- Solvency risk assessment
## 6. KPI Achievement Analysis
- Compare key metrics against "budget" / "target" columns (if present)
- Calculate achievement rate
## 7. Anomaly Alerts
- List all anomalies: margin collapse, excessive debt ratio, negative operating cash flow, revenue decline, etc.
- Severity: 🔴 HIGH / 🟠 MEDIUM / 🟡 LOW
---
Output format (strictly follow this JSON format, no other content):
```json
{{
"revenue": {{
"total": "amount (10K CNY)",
"yoy_change": "YoY %",
"qoq_change": "QoQ %",
"breakdown": "business line breakdown description",
"summary": "revenue structure assessment"
}},
"cost": {{
"total": "total cost",
"ratio": "% of revenue",
"cost_breakdown": "cost breakdown by item",
"anomalies": ["anomaly list"],
"summary": "cost analysis assessment"
}},
"profit": {{
"gross_margin": "gross margin %",
"net_margin": "net margin %",
"yoy_change": "profit YoY %",
"summary": "profitability assessment"
}},
"cashflow": {{
"operating": "operating cash flow net",
"investing": "investing cash flow",
"financing": "financing cash flow",
"summary": "cash flow health assessment"
}},
"balance_sheet": {{
"total_assets": "total assets",
"total_liabilities": "total liabilities",
"equity": "net assets",
"debt_ratio": "debt-to-asset ratio %",
"current_ratio": "current ratio",
"summary": "balance sheet assessment"
}},
"kpi": {{
"achieved": ["identified KPIs and achievement status"],
"missing": "Budget/target data not detected"
}},
"anomalies": [
{{"dimension": "dimension", "severity": "🔴/🟠/🟡", "description": "description", "value": "value", "suggestion": "suggestion"}}
],
"summary": "Overall business assessment (within 100 characters)"
}}
```
"""
def build_prompt(file_info: str, headers: List[str], rows: List[List], sheet_name: str = "Main Data") -> str:
rows_str = "\n".join([str(row) for row in rows[:20]])
return ANALYSIS_PROMPT_TEMPLATE.format(
file_info=file_info,
sheet_name=sheet_name,
headers=headers,
rows=rows_str,
)
def call_ai_analysis(prompt: str, api_key: str, model: str = "gpt-4o") -> Dict[str, Any]:
"""Call AI API for financial analysis. Returns parsed JSON or raises."""
import urllib.request
import urllib.error
if "deepseek" in model.lower():
base_url = "https://api.deepseek.com/v1"
elif "claude" in model.lower() or "anthropic" in model.lower():
base_url = "https://api.anthropic.com/v1"
elif "qwen" in model.lower():
base_url = "https://dashscope.aliyuncs.com/compatible-mode/v1"
elif "minimax" in model.lower():
base_url = "https://api.minimax.chat/v1"
else:
base_url = "https://api.openai.com/v1"
url = f"{base_url}/chat/completions"
headers_map = {
"openai": {"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"},
"deepseek": {"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"},
"anthropic": {"x-api-key": api_key, "Content-Type": "application/json", "anthropic-version": "2023-06-01", "anthropic-dangerous-direct-browser-access": "true"},
"qwen": {"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"},
"minimax": {"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"},
}
is_anthropic = "anthropic" in base_url
if is_anthropic:
payload_dict = {"model": model, "messages": [{"role": "user", "content": prompt}], "max_tokens": 4000}
payload = json.dumps(payload_dict)
headers = headers_map["anthropic"]
else:
payload = json.dumps({
"model": model,
"messages": [{"role": "user", "content": prompt}],
"temperature": 0.3,
"max_tokens": 4000,
})
headers = headers_map.get(
next((k for k in headers_map if k in base_url), "openai"),
headers_map["openai"]
)
req = urllib.request.Request(url, data=payload.encode("utf-8"), headers=headers, method="POST")
try:
with urllib.request.urlopen(req, timeout=60) as resp:
result = json.loads(resp.read().decode("utf-8"))
if is_anthropic:
content = result["content"][0]["text"]
else:
content = result["choices"][0]["message"]["content"]
json_match = re.search(r'```(?:json)?\s*(.*?)```', content, re.DOTALL)
if json_match:
content = json_match.group(1)
return json.loads(content)
except urllib.error.HTTPError as e:
err_body = e.read().decode("utf-8") if e.fp else ""
raise Exception(f"AI API HTTP {e.code}: {err_body[:500]}")
except Exception as e:
raise Exception(f"AI API call failed: {str(e)}")
# ─────────────────────────────────────────────────────────────────────────────
# Report Builder (Fallback when AI is not available)
# ─────────────────────────────────────────────────────────────────────────────
def build_fallback_report(headers: List[str], rows: List[List]) -> Dict[str, Any]:
"""Build a basic structured report from parsed data without AI."""
col_types = detect_column_type(headers, rows)
return {
"revenue": {"total": "N/A", "yoy_change": "N/A", "qoq_change": "N/A", "breakdown": "AI analysis required", "summary": "Data parsed. Configure AI API Key for complete analysis."},
"cost": {"total": "N/A", "ratio": "N/A", "cost_breakdown": "AI analysis required", "anomalies": [], "summary": "Data parsed. Configure AI API Key for complete analysis."},
"profit": {"gross_margin": "N/A", "net_margin": "N/A", "yoy_change": "N/A", "summary": "Data parsed. Configure AI API Key for complete analysis."},
"cashflow": {"operating": "N/A", "investing": "N/A", "financing": "N/A", "summary": "Data parsed. Configure AI API Key for complete analysis."},
"balance_sheet": {"total_assets": "N/A", "total_liabilities": "N/A", "equity": "N/A", "debt_ratio": "N/A", "current_ratio": "N/A", "summary": "Data parsed. Configure AI API Key for complete analysis."},
"kpi": {"achieved": [], "missing": "Budget/target data not detected"},
"anomalies": [],
"summary": "Data parsed. Configure AI API Key to generate complete business analysis report."
}
# ─────────────────────────────────────────────────────────────────────────────
# Markdown Report Renderer (English labels)
# ─────────────────────────────────────────────────────────────────────────────
def render_markdown_report(report_data: Dict, tier: str, company: str = "", period: str = "") -> str:
"""Render a structured Markdown report from analysis JSON. All labels in English."""
lines = [
f"# Financial Report Analysis",
f"",
f"**Company**: {company or 'Not provided'}",
f"**Period**: {period or 'Not provided'}",
f"**Tier**: {tier}",
f"**Generated**: {datetime.now().strftime('%Y-%m-%d %H:%M')}",
f"",
f"---",
f"",
]
r = report_data
# Revenue
rev = r.get("revenue", {})
lines += [
f"## 1. Revenue Structure Analysis",
f"",
f"| Item | Value |",
f"|------|-------|",
f"| Total Revenue | {rev.get('total', 'N/A')} |",
f"| YoY Change | {rev.get('yoy_change', 'N/A')} |",
f"| QoQ Change | {rev.get('qoq_change', 'N/A')} |",
f"",
f"**Breakdown**: {rev.get('breakdown', 'N/A')}",
f"",
f"**Assessment**: {rev.get('summary', '')}",
f"",
]
# Cost
cost = r.get("cost", {})
lines += [
f"## 2. Cost Anomaly Detection",
f"",
f"| Item | Value |",
f"|------|-------|",
f"| Total Cost | {cost.get('total', 'N/A')} |",
f"| Cost-to-Revenue Ratio | {cost.get('ratio', 'N/A')} |",
f"",
f"**Cost Structure**: {cost.get('cost_breakdown', 'N/A')}",
]
anomalies_cost = cost.get("anomalies", [])
if anomalies_cost:
lines += ["", "**Anomaly Flags**: "]
for a in anomalies_cost:
lines.append(f"- {a}")
lines += ["", f"**Assessment**: {cost.get('summary', '')}", ""]
# Profit
prof = r.get("profit", {})
lines += [
f"## 3. Profitability Analysis",
f"",
f"| Metric | Value |",
f"|--------|-------|",
f"| Gross Margin | {prof.get('gross_margin', 'N/A')} |",
f"| Net Margin | {prof.get('net_margin', 'N/A')} |",
f"| Profit YoY | {prof.get('yoy_change', 'N/A')} |",
f"",
f"**Assessment**: {prof.get('summary', '')}",
f"",
]
# Cashflow
cf = r.get("cashflow", {})
lines += [
f"## 4. Cash Flow Analysis",
f"",
f"| Item | Value |",
f"|------|-------|",
f"| Operating Cash Flow | {cf.get('operating', 'N/A')} |",
f"| Investing Cash Flow | {cf.get('investing', 'N/A')} |",
f"| Financing Cash Flow | {cf.get('financing', 'N/A')} |",
f"",
f"**Assessment**: {cf.get('summary', '')}",
f"",
]
# Balance Sheet
bs = r.get("balance_sheet", {})
lines += [
f"## 5. Balance Sheet Analysis",
f"",
f"| Item | Value |",
f"|------|-------|",
f"| Total Assets | {bs.get('total_assets', 'N/A')} |",
f"| Total Liabilities | {bs.get('total_liabilities', 'N/A')} |",
f"| Net Assets | {bs.get('equity', 'N/A')} |",
f"| Debt-to-Asset Ratio | {bs.get('debt_ratio', 'N/A')} |",
f"| Current Ratio | {bs.get('current_ratio', 'N/A')} |",
f"",
f"**Assessment**: {bs.get('summary', '')}",
f"",
]
# KPI
kpi = r.get("kpi", {})
lines += [
f"## 6. KPI Achievement Analysis",
]
achieved = kpi.get("achieved", [])
if achieved:
lines += ["", "| KPI Metric | Achievement |", "|------|------|"]
for item in achieved:
if isinstance(item, dict):
lines.append(f"| {item.get('name', 'N/A')} | {item.get('status', 'N/A')} |")
else:
lines.append(f"| {item} | N/A |")
else:
lines += ["", f"No KPI / budget data detected ({kpi.get('missing', 'N/A')})"]
lines += [""]
# Anomalies
all_anomalies = r.get("anomalies", [])
if all_anomalies:
lines += [f"## 7. Anomaly Alerts", ""]
lines += ["", "| Dimension | Severity | Description | Value | Suggestion |", "|------|------|------|------|------|"]
for a in all_anomalies:
lines.append(f"| {a.get('dimension','')} | {a.get('severity','')} | {a.get('description','')} | {a.get('value','')} | {a.get('suggestion','')} |")
lines += [""]
# Summary
lines += [
f"## Overall Business Assessment",
f"",
f"{r.get('summary', 'N/A')}",
f"",
f"---",
f"",
f"> This report is auto-generated by AI. Data is for reference only. Please verify against original financial statements.",
]
return "\n".join(lines)
# ─────────────────────────────────────────────────────────────────────────────
# Main Entry Point
# ─────────────────────────────────────────────────────────────────────────────
def main():
if len(sys.argv) < 2:
print(json.dumps({"error": "Usage: report_generator.py <input_json_file>"}))
sys.exit(1)
input_file = sys.argv[1]
if not os.path.exists(input_file):
print(json.dumps({"error": f"Input file not found: {input_file}"}))
sys.exit(1)
with open(input_file, "r", encoding="utf-8") as f:
params = json.load(f)
file_path = params.get("file_path", "")
api_key = params.get("api_key", "")
model = params.get("model", "gpt-4o")
tier = params.get("tier", "FREE")
company = params.get("company_name", "")
period = params.get("period", "")
parsed_data = params.get("parsed_data", {})
# Get sheet data
if "sheets" in parsed_data:
sheets = parsed_data["sheets"]
sheet_name = list(sheets.keys())[0] if sheets else "Main Data"
sheet_data = sheets[sheet_name]
else:
sheet_name = parsed_data.get("format", "Data")
sheet_data = {
"headers": parsed_data.get("headers", []),
"rows": parsed_data.get("rows", []),
}
headers = sheet_data.get("headers", [])
rows = sheet_data.get("rows", [])
file_ext = Path(file_path).suffix.lower()
file_info = f"File: {file_path} ({file_ext})"
# Build prompt
prompt = build_prompt(file_info, headers, rows, sheet_name)
# Try AI analysis
ai_result = None
error_msg = None
if api_key and api_key.strip():
try:
ai_result = call_ai_analysis(prompt, api_key, model)
except Exception as e:
error_msg = str(e)
ai_result = None
# Fallback if no AI or AI failed
if ai_result is None:
ai_result = build_fallback_report(headers, rows)
if error_msg:
ai_result["_warning"] = f"AI analysis unavailable: {error_msg}. Basic report generated."
# Render Markdown
markdown_report = render_markdown_report(ai_result, tier, company, period)
result = {
"success": True,
"tier": tier,
"analysis": ai_result,
"markdown": markdown_report,
"file_info": {
"path": file_path,
"ext": file_ext,
"sheet": sheet_name,
"rows_count": len(rows),
"cols_count": len(headers),
}
}
print(json.dumps(result, ensure_ascii=False, default=str))
if __name__ == "__main__":
main()
FILE:src/handlers/message_handler.js
/**
* Message Handler
* Handles interactive chat messages for financial report analysis
* Supports: text queries about uploaded reports, file upload instructions
*/
const path = require('path');
const { handleSkillInvoke } = require('./skill_invoke');
/**
* Handle incoming message
* @param {Object} message - { text, userId, sessionId, apiKey, model }
* @returns {Object} - response
*/
async function handleMessage(message) {
const { text = '', userId = '', sessionId = '', apiKey = '', model = '' } = message;
if (!text || text.match(/^(help|usage|how to|guide)/i)) {
return getHelpMessage();
}
if (text.match(/^(plan|price|tier|cost|version|套餐|价格)/i)) {
return getPlanInfoMessage();
}
if (text.match(/^(status|balance|余额)/i)) {
return {
success: true,
message: 'Please provide your API Key so I can check your plan status.\n\nOr simply upload a financial file to start analysis.',
};
}
return getHelpMessage();
}
function getHelpMessage() {
return {
success: true,
message: `Financial Report AI
Upload your Excel/CSV/PDF financial statements → AI auto-generates structured business analysis reports.
Supported analysis dimensions:
1. Revenue structure analysis
2. Cost anomaly detection
3. Profitability analysis
4. Cash flow analysis
5. Balance sheet analysis
6. KPI achievement
7. Anomaly alerts
Steps:
1. Upload financial file (Excel/CSV/PDF)
2. Wait for AI automatic parsing
3. Receive complete analysis report
Tier:
• FREE: 3 analyses/month, 3 basic dimensions
• PRO: $0.01 USDT/use, all 7 dimensions + charts
> Get PRO: https://skillpay.me/ai-financial-report`,
};
}
function getPlanInfoMessage() {
return {
success: true,
message: `Tier Comparison
| Feature | FREE | PRO |
|---------|------|-----|
| Analyses/month | 3 | Unlimited |
| Input formats | CSV, Excel | CSV, Excel, PDF |
| Analysis dimensions | 3 basic | All 7 |
| Charts | No | Yes |
| Industry comparison | No | Yes |
| Price | Free | $0.01 USDT/use |
> Get PRO: https://skillpay.me/ai-financial-report`,
};
}
module.exports = { handleMessage };
FILE:src/handlers/file_upload.js
/**
* File Upload Handler
* Handles file upload events from OpenClaw
* Accepts: CSV, XLSX, XLS, PDF
*/
const path = require('path');
const fs = require('fs');
const { handleSkillInvoke } = require('./skill_invoke');
const ALLOWED_EXTENSIONS = ['.csv', '.xlsx', '.xls', '.pdf'];
const MAX_FILE_SIZE = 50 * 1024 * 1024; // 50MB
function validateFile(ext, size) {
const errors = [];
if (!ALLOWED_EXTENSIONS.includes(ext.toLowerCase())) {
errors.push(`Unsupported format: ext. Supported: CSV, Excel (.xlsx/.xls), PDF`);
}
if (size > MAX_FILE_SIZE) {
errors.push(`File too large: (size / 1024 / 1024).toFixed(1)MB, max 50MB`);
}
return errors;
}
/**
* Handle file upload
* @param {Object} event - { fileName, fileSize, fileContent (base64), mimeType, apiKey, model }
* @returns {Object} - analysis result
*/
async function handleFileUpload(event) {
const {
fileName = 'report.xlsx',
fileSize = 0,
fileContent = '',
mimeType = '',
apiKey = '',
model = '',
companyName = '',
period = '',
userId = '',
} = event;
const ext = path.extname(fileName).toLowerCase();
// Validate
const errors = validateFile(ext, fileSize);
if (errors.length > 0) {
return {
success: false,
errors,
message: errors.join('; '),
};
}
// Route to skill invoke
const result = await handleSkillInvoke({
apiKey,
fileContent,
fileName,
fileExt: ext,
model,
companyName,
period,
userId,
});
return {
success: true,
...result,
};
}
module.exports = { handleFileUpload };
FILE:src/handlers/skill_invoke.js
/**
* Skill Invoke Handler - Main entry point for financial report analysis
* Handles: skill.invoke events from OpenClaw
*/
const path = require('path');
const fs = require('fs');
const { spawn } = require('child_process');
const { validateToken, getPlanLimits } = require('../services/billing');
const SKILL_ROOT = path.resolve(__dirname, '..', '..');
// Plan tier → analysis dimensions available
const TIER_DIMENSIONS = {
FREE: 3,
PRO: 99,
};
function sanitizeJsonSafe(obj) {
if (obj && typeof obj === 'object') {
const clean = {};
for (const [k, v] of Object.entries(obj)) {
if (typeof v === 'string' || typeof v === 'number' || typeof v === 'boolean' || v === null) {
clean[k] = v;
} else if (Array.isArray(v)) {
clean[k] = v.map(sanitizeJsonSafe);
} else if (typeof v === 'object') {
clean[k] = sanitizeJsonSafe(v);
}
}
return clean;
}
return obj;
}
function runPython(scriptPath, args) {
return new Promise((resolve, reject) => {
const proc = spawn('python3', [scriptPath, ...args], {
cwd: SKILL_ROOT,
env: { ...process.env, PYTHONIOENCODING: 'utf-8' },
timeout: 120000,
});
let stdout = '';
let stderr = '';
proc.stdout.on('data', d => { stdout += d.toString(); });
proc.stderr.on('data', d => { stderr += d.toString(); });
proc.on('close', code => {
if (code !== 0) {
reject(new Error(`Python exited code: stderr || stdout`));
} else {
resolve(stdout);
}
});
proc.on('error', err => reject(err));
});
}
async function parseFile(filePath) {
const scriptPath = path.join(SKILL_ROOT, 'src', 'services', 'file_parser.py');
const output = await runPython(scriptPath, [filePath]);
return JSON.parse(output.trim());
}
async function generateReport(inputJsonPath) {
const scriptPath = path.join(SKILL_ROOT, 'src', 'services', 'report_generator.py');
const output = await runPython(scriptPath, [inputJsonPath]);
return JSON.parse(output.trim());
}
/**
* Main skill invoke handler
* @param {Object} args - { apiKey, fileContent (base64), fileName, fileExt, model, tier, companyName, period, userId }
* @returns {Object} - { markdown, analysis, fileInfo }
*/
async function handleSkillInvoke(args) {
const {
apiKey = '',
fileContent = '',
fileName = 'report.xlsx',
fileExt = '',
model = '',
companyName = '',
period = '',
userId = '',
} = args;
// 1. Billing / token validation
const validation = await validateToken(apiKey, userId);
const tier = validation.valid ? 'PRO' : 'FREE';
const limits = getPlanLimits(tier);
// 2. Save uploaded file to temp
const tmpDir = '/tmp/ai-financial-report';
if (!fs.existsSync(tmpDir)) fs.mkdirSync(tmpDir, { recursive: true });
const decoded = Buffer.from(fileContent, 'base64');
const savePath = path.join(tmpDir, `frp_Date.now()_fileName`);
fs.writeFileSync(savePath, decoded);
// 3. Parse file
let parsedData;
try {
parsedData = await parseFile(savePath);
} catch (err) {
try { fs.unlinkSync(savePath); } catch (_) {}
throw new Error(`File parsing failed: err.message`);
}
// 4. Build input for report generator
const inputJson = {
file_path: savePath,
api_key: apiKey,
model: model || 'gpt-4o',
tier,
company_name: companyName || '',
period: period || '',
parsed_data: sanitizeJsonSafe(parsedData),
};
const inputJsonPath = path.join(tmpDir, `input_Date.now().json`);
fs.writeFileSync(inputJsonPath, JSON.stringify(inputJson, null, 2), 'utf8');
// 5. Generate report
let reportResult;
try {
reportResult = await generateReport(inputJsonPath);
} catch (err) {
try { fs.unlinkSync(inputJsonPath); } catch (_) {}
try { fs.unlinkSync(savePath); } catch (_) {}
throw new Error(`Report generation failed: err.message`);
} finally {
try { fs.unlinkSync(inputJsonPath); } catch (_) {}
try { fs.unlinkSync(savePath); } catch (_) {}
}
return {
markdown: reportResult.markdown,
analysis: sanitizeJsonSafe(reportResult.analysis || {}),
file_info: reportResult.file_info || {},
tier,
limits,
validation_result: {
valid: validation.valid,
plan: validation.plan,
reason: validation.reason,
},
};
}
module.exports = { handleSkillInvoke };
Classify text or CSV files into preset or custom categories with optional confidence scores and batch processing, using AI-powered classification.
# SKILL.md - Text Classifier
> Upload text or CSV — AI automatically classifies content and returns structured labels with confidence scores.
**Slug:** text-classifier
## Tiered Features
| Feature | FREE | PRO |
|---------|:----:|:---:|
| Text input | ✅ | ✅ |
| File upload (TXT/CSV) | ❌ | ✅ |
| Preset classifiers | 3 | Unlimited |
| Custom labels | ❌ | ✅ |
| Confidence score | ❌ | ✅ |
| Batch processing | ❌ | ✅ (up to 5,000) |
| History retention | ❌ | ✅ (365 days) |
| Daily classifications | 20 | Unlimited |
## Pricing
**Per-call:** $0.01 USDT per classification
No monthly subscription. Pay only for what you use.
## Usage
### Web Interface (Recommended)
```bash
cd text-classifier
pip install -r requirements.txt
python scripts/web_app.py
# Open http://localhost:5000
```
### CLI
```bash
# Single text classification
python -m scripts.classifier --text "This product is great" --classifier "Sentiment Classification" --api-key "sk-..."
# Batch CSV classification
python -m scripts.classifier --file data.csv --classifier "Sentiment Classification" --api-key "sk-..." --output csv --output-path results.csv
# Custom labels
python -m scripts.classifier --text "Urgent issue" --custom-labels "High,Medium,Low" --custom-prompt "Classify priority" --api-key "sk-..."
```
### Python API
```python
from scripts.classifier import classify_text, validate_token
# Token validation
tier = validate_token("PRO-xxxx")
print(f"Tier: {tier}")
# Single classification
result = classify_text(
text="This product is excellent",
classifier_name="Sentiment Classification",
api_key="sk-...",
show_confidence=True
)
print(result)
# {'label': 'Positive', 'confidence': 0.85, 'raw': 'Positive', ...}
```
## Preset Classifiers
- **Intent Classification**: Inquiry / Complaint / Refund / Cooperation
- **Sentiment Classification**: Positive / Neutral / Negative
- **Industry Classification**: Finance / Healthcare / Education / Retail / Manufacturing
- **Risk Classification**: Compliant / Violation / Suspicious
- **Priority Classification**: High / Medium / Low
- **Content Classification**: News / Advertisement / UGC / Spam
## Required Environment Variables
```bash
SKILL_BILLING_API_KEY # Your SkillPay Builder API Key
SKILL_BILLING_SKILL_ID # Skill slug: text-classifier
```
Set these in your runtime environment. Without them, the tool runs in Dev Mode (FREE tier, no billing).
## Billing
This skill uses **SkillPay** (skillpay.me) for per-call billing at **$0.01 USDT per classification**.
- Your Feishu User ID (Open ID) is transmitted to `skillpay.me` exclusively for billing purposes
- No other data is transmitted to third parties
- Billing occurs at the start of each classification (after API key validation)
- Dev Mode (`SKILL_BILLING_API_KEY` not set): FULL FREE USAGE — no API key required, no billing
## Security Notes
- **LLM Execution**: All AI classification runs via OpenAI API you configure
- **Data Isolation**: Classification history is stored locally in `/tmp/text-classifier/` — no data leaves your environment
- **SQL Safety**: Not applicable (no database queries in this skill)
- **Path Isolation**: All writes go to `/tmp/` — no home directory access
## Dependencies
```
requests>=2.28.0
pandas>=2.0.0
openpyxl>=3.0.0
flask>=3.0.0
```
FILE:requirements.txt
openai>=1.0.0
pandas>=1.5.0
requests>=2.28.0
flask>=3.0.0
werkzeug>=3.0.0
openpyxl>=3.1.0
pytest>=7.4.0
FILE:README.md
# Text Classifier
AI-powered text classification tool. Upload text or CSV — AI automatically classifies content.
## Quick Start
```bash
pip install -r requirements.txt
python scripts/web_app.py
# Open http://localhost:5000
```
## Documentation
See `SKILL.md` for full documentation.
FILE:scripts/web_app.py
"""
Text Classifier Web App (Flask)
"""
import os
import html
import re
from pathlib import Path
from flask import Flask, request, jsonify, render_template_string, send_file
from werkzeug.utils import secure_filename
try:
from scripts.classifier import (
validate_token, get_tier_limits, classify_text, classify_batch,
parse_txt, parse_csv, format_results_markdown,
export_csv, export_excel, export_json, save_history,
PRESET_CLASSIFIERS, is_dev_mode, charge_user,
)
from scripts.billing import CALL_PRICE
except ImportError:
from classifier import (
validate_token, get_tier_limits, classify_text, classify_batch,
parse_txt, parse_csv, format_results_markdown,
export_csv, export_excel, export_json, save_history,
PRESET_CLASSIFIERS, is_dev_mode, charge_user,
)
CALL_PRICE = 0.0100
def _escape_html(text: str) -> str:
"""
Escape HTML special characters AND Jinja2 template markers.
Prevents SSTI when user content is rendered in render_template_string.
Markdown tables are preserved as plain text with <br> newlines.
"""
# Escape HTML specials first
text = html.escape(text)
# Block Jinja2 template markers (double-brace, block, comment)
text = text.replace("{{", "{{")
text = text.replace("}}", "}}")
text = text.replace("{%", "{%")
text = text.replace("%}", "%}")
text = text.replace("{#", "{#")
text = text.replace("#}", "#}")
# Restore newlines as <br> for readability in the pre block
text = text.replace("\n", "<br>")
return text
app = Flask(__name__)
app.config["MAX_CONTENT_LENGTH"] = 16 * 1024 * 1024 # 16MB
app.config["SECRET_KEY"] = os.getenv("SECRET_KEY", "text-classifier-secret")
SESSION_COUNTS = {} # {api_key_hash: count}
def get_session_count(api_key: str) -> int:
"""Track daily classification count per user."""
import hashlib
key = hashlib.md5((api_key or "").encode()).hexdigest()
return SESSION_COUNTS.get(key, 0)
def increment_count(api_key: str):
import hashlib
key = hashlib.md5((api_key or "").encode()).hexdigest()
SESSION_COUNTS[key] = SESSION_COUNTS.get(key, 0) + 1
HTML_TEMPLATE = """
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Text Classifier</title>
<style>
* { box-sizing: border-box; margin: 0; padding: 0; }
body { font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', sans-serif;
background: #f5f6fa; color: #333; max-width: 900px; margin: 0 auto; padding: 20px; }
h1 { text-align: center; color: #2c3e50; margin: 20px 0; }
.card { background: white; border-radius: 12px; padding: 24px; margin-bottom: 16px;
box-shadow: 0 2px 8px rgba(0,0,0,.08); }
.tier-badge { display: inline-block; padding: 4px 12px; border-radius: 20px; font-size: 13px;
font-weight: 600; }
.tier-FREE { background: #e8f5e9; color: #2e7d32; }
.tier-PRO { background: #ede7f6; color: #7b1fa2; }
label { font-weight: 600; display: block; margin: 12px 0 6px; color: #555; }
input, select, textarea { width: 100%; padding: 10px 12px; border: 1.5px solid #ddd; border-radius: 8px;
font-size: 15px; transition: border .2s; }
input:focus, select:focus, textarea:focus { outline: none; border-color: #4f46e5; }
textarea { height: 120px; resize: vertical; }
.btn { display: inline-block; padding: 12px 24px; background: #4f46e5; color: white;
border: none; border-radius: 8px; font-size: 16px; font-weight: 600; cursor: pointer;
transition: background .2s; width: 100%; margin-top: 16px; }
.btn:hover { background: #4338ca; }
.btn:disabled { background: #9ca3af; cursor: not-allowed; }
.results { background: #f8f9fc; border-radius: 8px; padding: 16px; margin-top: 20px;
overflow-x: auto; }
table { width: 100%; border-collapse: collapse; font-size: 14px; }
th { background: #4f46e5; color: white; padding: 10px; text-align: left; }
td { padding: 8px 10px; border-bottom: 1px solid #eee; }
tr:hover td { background: #f0f0f8; }
.alert { padding: 12px; border-radius: 8px; margin: 12px 0; }
.alert-error { background: #fef2f2; color: #991b1b; border: 1px solid #fecaca; }
.alert-success { background: #f0fdf4; color: #166534; border: 1px solid #bbf7d0; }
.row { display: flex; gap: 12px; }
.col { flex: 1; }
.presets { display: flex; flex-wrap: wrap; gap: 6px; margin-top: 4px; }
.preset-tag { padding: 3px 10px; background: #e0e7ff; color: #3730a3; border-radius: 20px;
font-size: 12px; cursor: pointer; }
.preset-tag:hover { background: #c7d2fe; }
.count-badge { font-size: 12px; color: #888; margin-left: 8px; }
.footer { text-align: center; color: #aaa; font-size: 13px; margin: 24px 0; }
.footer a { color: #4f46e5; text-decoration: none; }
.results pre { margin: 0; font-family: monospace; white-space: pre-wrap; word-break: break-word; }
</style>
</head>
<body>
<h1>Text Classifier</h1>
<div class="card">
<div style="display:flex; justify-content:space-between; align-items:center;">
<span>Current Tier</span>
<span class="tier-badge tier-{{ tier_class }}">{{ tier|e }}</span>
</div>
<div style="margin-top:8px; font-size:13px; color:#666;">
Remaining: <strong>{{ remaining|e }}</strong>
</div>
</div>
<div class="card">
<form method="POST" enctype="multipart/form-data">
<div class="row">
<div class="col">
<label>API Key (OpenAI)</label>
<input type="password" name="api_key" placeholder="sk-..." value="{{ api_key|e or '' }}">
<label>License Token</label>
<input type="text" name="license_token" placeholder="FREE-xxx or PRO-xxx" value="{{ license_token|e or '' }}">
</div>
<div class="col">
<label>Classifier</label>
<select name="classifier">
{% for name in classifiers %}
<option value="{{ name|e }}" {% if classifier == name %}selected{% endif %}>{{ name|e }}</option>
{% endfor %}
</select>
<label>Model</label>
<select name="model">
<option value="gpt-3.5-turbo" {% if model == 'gpt-3.5-turbo' %}selected{% endif %}>GPT-3.5</option>
<option value="gpt-4" {% if model == 'gpt-4' %}selected{% endif %}>GPT-4</option>
<option value="gpt-4-turbo" {% if model == 'gpt-4-turbo' %}selected{% endif %}>GPT-4 Turbo</option>
</select>
</div>
</div>
<label>Input Text (one per line)</label>
<textarea name="text" placeholder="Paste text here, one per line...">{{ text|e or '' }}</textarea>
<label>Or upload file</label>
<input type="file" name="file" accept=".txt,.csv">
<div class="row" style="margin-top:12px;">
<div class="col">
<label>Custom Labels (comma-separated)</label>
<input type="text" name="custom_labels" placeholder="Label1,Label2,Label3" value="{{ custom_labels|e or '' }}">
</div>
<div class="col">
<label>Export Format</label>
<select name="export_format">
<option value="screen" {% if export_format == 'screen' %}selected{% endif %}>Screen Display</option>
<option value="csv" {% if export_format == 'csv' %}selected{% endif %}>CSV</option>
<option value="excel" {% if export_format == 'excel' %}selected{% endif %}>Excel</option>
<option value="json" {% if export_format == 'json' %}selected{% endif %}>JSON</option>
</select>
</div>
</div>
<div style="margin-top:4px;">
<label style="display:inline;">Preset Classifiers:</label>
<div class="presets">
{% for name in classifiers %}
<span class="preset-tag" onclick="document.querySelector('[name=classifier]').value='{{ name|e }}'">{{ name|e }}</span>
{% endfor %}
</div>
</div>
<button type="submit" class="btn">Start Classification</button>
</form>
</div>
{% if error %}
<div class="alert alert-error">{{ error|e }}</div>
{% endif %}
{% if results %}
<div class="card">
<h3 style="margin-bottom:12px;">Classification Results ({{ results_count|e }} items)</h3>
{% if export_format == 'screen' %}
<div class="results">
<pre>{{ results|e }}</pre>
</div>
{% else %}
<div class="alert alert-success">Exported as {{ export_format|e }}</div>
{% endif %}
</div>
{% endif %}
<div class="footer">
Text Classifier · $0.01 USDT per classification
</div>
</body>
</html>
"""
@app.route("/", methods=["GET", "POST"])
def index():
error = None
results = None
api_key = request.form.get("api_key", "")
license_token = request.form.get("license_token", "")
classifier = request.form.get("classifier", "Sentiment Classification")
model = request.form.get("model", "gpt-3.5-turbo")
text = request.form.get("text", "")
custom_labels_str = request.form.get("custom_labels", "")
export_format = request.form.get("export_format", "screen")
text_column = None
# Determine tier
tier = validate_token(license_token) if license_token else "FREE"
limits = get_tier_limits(tier)
# Count check
current_count = get_session_count(license_token)
remaining = max(0, limits["daily"] - current_count)
if request.method == "POST":
# Check daily limit first
if remaining <= 0:
error = "Daily limit reached. Please upgrade."
else:
if not api_key:
error = "Please enter an OpenAI API Key"
else:
texts = []
uploaded_file = request.files.get("file")
if uploaded_file and uploaded_file.filename:
filename = secure_filename(uploaded_file.filename)
ext = Path(filename).suffix.lower()
if ext == ".txt":
content = uploaded_file.read().decode("utf-8")
texts = [l.strip() for l in content.split("\n") if l.strip()]
elif ext == ".csv":
temp_path = f"/tmp/{filename}"
uploaded_file.save(temp_path)
try:
texts = parse_csv(temp_path, text_column)
finally:
os.unlink(temp_path)
else:
error = "Only TXT and CSV files are supported"
elif text:
texts = [t.strip() for t in text.split("\n") if t.strip()]
else:
error = "Please enter text or upload a file"
if error is None and texts:
batch_limit = limits.get("batch", 0)
# Truncate to batch limit
if batch_limit > 0 and len(texts) > batch_limit:
texts = texts[:batch_limit]
custom_labels = [l.strip() for l in custom_labels_str.split(",") if l.strip()] if custom_labels_str else None
show_conf = limits.get("confidence", False)
try:
if len(texts) == 1:
result = classify_text(
texts[0], classifier, api_key,
custom_labels, None, model, show_conf
)
results_list = [result]
else:
results_list = classify_batch(
texts, classifier, api_key,
custom_labels, None, model, show_conf,
limits.get("batch", 50)
)
increment_count(license_token)
if export_format == "screen":
md_table = format_results_markdown(results_list, show_conf)
# Escape HTML in markdown table for SSTI safety, preserving newlines
results = _escape_html(md_table)
results_list = results_list # keep for count
else:
suffix = export_format
export_path = f"/tmp/results.{suffix}"
if export_format == "csv":
export_csv(results_list, export_path)
elif export_format == "excel":
export_excel(results_list, export_path)
elif export_format == "json":
export_json(results_list, export_path)
save_history(results_list, tier)
return send_file(
export_path,
as_attachment=True,
download_name=f"classification_results.{suffix}"
)
results = results # keep
except Exception as e:
error = f"Classification failed: {e}"
tier_class = tier
remaining = max(0, limits["daily"] - get_session_count(license_token))
results_count = len(results_list) if results else 0
return render_template_string(
HTML_TEMPLATE,
tier=tier,
tier_class=tier_class,
remaining=remaining,
classifiers=list(PRESET_CLASSIFIERS.keys()),
api_key=api_key,
license_token=license_token,
classifier=classifier,
model=model,
text=text,
custom_labels=custom_labels_str,
error=error,
results=results,
results_count=results_count,
export_format=export_format,
)
if __name__ == "__main__":
port = int(os.getenv("PORT", 5000))
app.run(host="0.0.0.0", port=port, debug=False)
FILE:scripts/classifier.py
"""
Text Classifier - AI-powered text classification tool
"""
import os
import re
import json
import time
import hashlib
import requests
import pandas as pd
from pathlib import Path
from typing import Optional
try:
from .billing import is_dev_mode, charge_user, CALL_PRICE
from .config import TIERS, get_tier_limits, HISTORY_DIR
except ImportError:
from billing import is_dev_mode, charge_user, CALL_PRICE
from config import TIERS, get_tier_limits, HISTORY_DIR
# ========================
# Preset Classifiers (English labels + prompts)
# ========================
PRESET_CLASSIFIERS = {
"Intent Classification": {
"labels": ["Inquiry", "Complaint", "Refund", "Cooperation"],
"prompt": "Classify the intent of the following text into one of: Inquiry, Complaint, Refund, Cooperation. Return only the category name."
},
"Sentiment Classification": {
"labels": ["Positive", "Neutral", "Negative"],
"prompt": "Classify the sentiment of the following text as: Positive, Neutral, or Negative. Return only the category name."
},
"Industry Classification": {
"labels": ["Finance", "Healthcare", "Education", "Retail", "Manufacturing"],
"prompt": "Classify the industry of the following text into one of: Finance, Healthcare, Education, Retail, Manufacturing. Return only the category name."
},
"Risk Classification": {
"labels": ["Compliant", "Violation", "Suspicious"],
"prompt": "Classify the risk level of the following text as: Compliant, Violation, or Suspicious. Return only the category name."
},
"Priority Classification": {
"labels": ["High", "Medium", "Low"],
"prompt": "Classify the priority of the following text as: High, Medium, or Low. Return only the category name."
},
"Content Classification": {
"labels": ["News", "Advertisement", "UGC", "Spam"],
"prompt": "Classify the content type of the following text as: News, Advertisement, UGC, or Spam. Return only the category name."
},
}
# ========================
# Token Validation
# ========================
_tier_cache = {}
def validate_token(api_key: str) -> str:
"""
Validate token and return tier (FREE or PRO).
Falls back to FREE on error (non-blocking).
"""
global _tier_cache
if not api_key or not api_key.strip():
return "FREE"
cache_key = hashlib.md5(api_key.encode()).hexdigest()
now = time.time()
if cache_key in _tier_cache:
cached_tier, cached_time = _tier_cache[cache_key]
if now - cached_time < 300:
return cached_tier
# Dev mode: no billing configured — treat as FREE
if is_dev_mode():
tier = "FREE"
else:
# Determine tier from key prefix
tier = "FREE"
for prefix in ("PRO", "FREE"):
if api_key.startswith(prefix):
tier = prefix
break
# Try to charge — if fails, tier stays FREE
try:
billing_result = charge_user(api_key)
if not billing_result.get("ok"):
tier = "FREE"
except Exception:
tier = "FREE"
_tier_cache[cache_key] = (tier, now)
return tier
# ========================
# OpenAI Classification
# ========================
def classify_text(
text: str,
classifier_name: str,
api_key: Optional[str] = None,
custom_labels: Optional[list] = None,
custom_prompt: Optional[str] = None,
model: str = "gpt-3.5-turbo",
show_confidence: bool = True,
) -> dict:
"""
Classify a single text using OpenAI API.
Returns dict with label, confidence, and raw response.
"""
if not api_key:
raise ValueError("API key is required for AI classification.")
# Determine labels and prompt
if custom_labels and custom_prompt:
labels = custom_labels
prompt_prefix = custom_prompt
elif classifier_name in PRESET_CLASSIFIERS:
cfg = PRESET_CLASSIFIERS[classifier_name]
labels = cfg["labels"]
prompt_prefix = cfg["prompt"]
else:
labels = list(sum([v["labels"] for v in PRESET_CLASSIFIERS.values()], []))
prompt_prefix = "Classify the following text."
labels_str = ", ".join(labels)
full_prompt = f"{prompt_prefix}\n\nText: {text}\n\nAvailable labels: {labels_str}\n\nReturn only the category name, no explanation."
try:
resp = requests.post(
"https://api.openai.com/v1/chat/completions",
headers={
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json",
},
json={
"model": model,
"messages": [{"role": "user", "content": full_prompt}],
"temperature": 0.1,
"max_tokens": 50,
},
timeout=30,
)
resp.raise_for_status()
result = resp.json()
raw = result["choices"][0]["message"]["content"].strip()
# Parse response - extract label
matched_label = None
for label in labels:
if label.lower() in raw.lower():
matched_label = label
break
if not matched_label:
matched_label = raw.strip()
confidence = 0.85 if matched_label in labels else 0.5
return {
"label": matched_label,
"confidence": confidence if show_confidence else None,
"raw": raw,
"classifier": classifier_name,
"text_preview": text[:100],
}
except Exception as e:
raise RuntimeError(f"Classification failed: {e}")
def classify_batch(
texts: list,
classifier_name: str,
api_key: Optional[str] = None,
custom_labels: Optional[list] = None,
custom_prompt: Optional[str] = None,
model: str = "gpt-3.5-turbo",
show_confidence: bool = True,
batch_limit: int = 50,
) -> list:
"""Classify multiple texts with rate limiting."""
results = []
for i, text in enumerate(texts[:batch_limit]):
try:
result = classify_text(
text,
classifier_name,
api_key,
custom_labels,
custom_prompt,
model,
show_confidence,
)
results.append(result)
except Exception as e:
results.append({
"label": "ERROR",
"confidence": None,
"raw": str(e),
"classifier": classifier_name,
"text_preview": text[:100],
})
if i < len(texts) - 1:
time.sleep(0.2)
return results
# ========================
# File Parsing
# ========================
def parse_txt(file_path: str) -> list:
"""Parse TXT file, return list of texts."""
with open(file_path, "r", encoding="utf-8") as f:
lines = [l.strip() for l in f if l.strip()]
return lines
def parse_csv(file_path: str, text_column: str = None, encoding: str = "utf-8") -> list:
"""Parse CSV file using pandas. Returns list of texts."""
try:
df = pd.read_csv(file_path, encoding=encoding)
except Exception:
for enc in ["gbk", "gb2312", "latin1"]:
try:
df = pd.read_csv(file_path, encoding=enc)
break
except Exception:
continue
else:
raise ValueError(f"Could not parse CSV file: {file_path}")
if text_column:
if text_column not in df.columns:
raise ValueError(f"Column '{text_column}' not found. Available: {list(df.columns)}")
return df[text_column].dropna().astype(str).tolist()
else:
for col in df.columns:
if df[col].dtype == "object":
return df[col].dropna().astype(str).tolist()
raise ValueError("No text column found in CSV.")
# ========================
# Output Formatting
# ========================
def format_results_markdown(results: list, show_confidence: bool = True) -> str:
"""Format results as a Markdown table."""
if not results:
return "No results to display."
header = "| # | Text Preview | Classification"
sep = "|---|--------------|---------------"
if show_confidence and results[0].get("confidence") is not None:
header += " | Confidence"
sep += "|------------"
lines = [header, sep]
for i, r in enumerate(results, 1):
preview = r.get("text_preview", "")[:40]
label = r.get("label", "N/A")
line = f"| {i} | {preview} | {label}"
if show_confidence and r.get("confidence") is not None:
line += f" | {r['confidence']:.0%}"
lines.append(line)
return "\n".join(lines)
def export_csv(results: list, output_path: str):
"""Export results to CSV."""
df = pd.DataFrame(results)
df.to_csv(output_path, index=False, encoding="utf-8-sig")
def export_excel(results: list, output_path: str):
"""Export results to Excel."""
df = pd.DataFrame(results)
df.to_excel(output_path, index=False, engine="openpyxl")
def export_json(results: list, output_path: str):
"""Export results to JSON."""
with open(output_path, "w", encoding="utf-8") as f:
json.dump(results, f, ensure_ascii=False, indent=2)
# ========================
# History Management
# ========================
def save_history(results: list, tier: str, history_dir=None):
"""Save classification history to file."""
limits = get_tier_limits(tier)
days = limits["history_days"]
if days == 0:
return
hd = Path(history_dir) if history_dir else HISTORY_DIR
hd.mkdir(parents=True, exist_ok=True)
ts = time.strftime("%Y%m%d_%H%M%S")
filename = hd / f"classification_{ts}.json"
with open(filename, "w", encoding="utf-8") as f:
json.dump({
"timestamp": ts,
"tier": tier,
"results": results,
}, f, ensure_ascii=False, indent=2)
# ========================
# CLI Interface
# ========================
def main():
import argparse
parser = argparse.ArgumentParser(description="Text Classifier CLI")
parser.add_argument("--text", type=str, help="Single text to classify")
parser.add_argument("--file", type=str, help="TXT or CSV file to classify")
parser.add_argument("--csv-column", type=str, help="Column name for CSV text field")
parser.add_argument("--classifier", type=str, default="Sentiment Classification",
help="Preset classifier name")
parser.add_argument("--custom-labels", type=str, help="Comma-separated custom labels")
parser.add_argument("--custom-prompt", type=str, help="Custom classification prompt")
parser.add_argument("--api-key", type=str, default=os.getenv("OPENAI_API_KEY"),
help="OpenAI API key")
parser.add_argument("--model", type=str, default="gpt-3.5-turbo")
parser.add_argument("--no-confidence", action="store_true")
parser.add_argument("--output", type=str, choices=["screen", "csv", "excel", "json"],
default="screen")
parser.add_argument("--output-path", type=str)
parser.add_argument("--batch-limit", type=int, default=50)
parser.add_argument("--token", type=str, help="License token for tier validation")
args = parser.parse_args()
# Validate token
tier = validate_token(args.token) if args.token else "FREE"
limits = get_tier_limits(tier)
# Charge user (skip in dev mode)
if not is_dev_mode():
billing = charge_user(args.token or "free_user")
if not billing.get("ok"):
print(f"Payment required. Balance: {billing.get('balance')}")
if billing.get("payment_url"):
print(f"Payment URL: {billing['payment_url']}")
return 1
print(f"[Text Classifier] Tier: {tier} | Daily limit: {limits['daily']}")
# Collect texts
texts = []
if args.text:
texts = [args.text]
elif args.file:
ext = Path(args.file).suffix.lower()
if ext == ".txt":
texts = parse_txt(args.file)
elif ext == ".csv":
texts = parse_csv(args.file, args.csv_column)
else:
print(f"Unsupported file type: {ext}")
return
else:
print("Please provide --text or --file")
return
# Check batch limit
batch_limit = min(len(texts), limits.get("batch", 0) if limits.get("batch", 0) > 0 else len(texts))
if len(texts) > batch_limit and batch_limit > 0:
print(f"Batch limit reached ({batch_limit}). Truncating.")
texts = texts[:batch_limit]
custom_labels = [l.strip() for l in args.custom_labels.split(",")] if args.custom_labels else None
show_confidence = not args.no_confidence and limits.get("confidence", False)
# Classify
if len(texts) == 1:
result = classify_text(
texts[0],
args.classifier,
args.api_key,
custom_labels,
args.custom_prompt,
args.model,
show_confidence,
)
results = [result]
else:
results = classify_batch(
texts,
args.classifier,
args.api_key,
custom_labels,
args.custom_prompt,
args.model,
show_confidence,
args.batch_limit,
)
save_history(results, tier)
# Output
if args.output == "screen":
print(f"\n{format_results_markdown(results, show_confidence)}\n")
elif args.output == "csv":
path = args.output_path or "results.csv"
export_csv(results, path)
print(f"Exported to {path}")
elif args.output == "excel":
path = args.output_path or "results.xlsx"
export_excel(results, path)
print(f"Exported to {path}")
elif args.output == "json":
path = args.output_path or "results.json"
export_json(results, path)
print(f"Exported to {path}")
return 0
if __name__ == "__main__":
import sys
sys.exit(main())
FILE:scripts/config.py
"""
Text Classifier - Configuration Module
Tier definitions and constants.
"""
from pathlib import Path
from typing import Dict
# Storage path (use /tmp to avoid home-directory writes)
STORAGE_DIR = Path("/tmp/text-classifier")
HISTORY_DIR = STORAGE_DIR / "history"
# Tier definitions — 2 tiers only (FREE | PRO), per-call billing
TIERS: Dict[str, dict] = {
"FREE": {
"daily": 20,
"batch": 0,
"preset": 3,
"custom": False,
"confidence": False,
"history_days": 0,
"api": False,
},
"PRO": {
"daily": 999999,
"batch": 5000,
"preset": 999,
"custom": True,
"confidence": True,
"history_days": 365,
"api": True,
},
}
def get_tier_limits(tier: str) -> dict:
"""Return usage limits for a given tier."""
return TIERS.get(tier, TIERS["FREE"])
FILE:scripts/billing.py
"""
Text Classifier - Billing Module (SkillPay)
Handles per-call billing via skillpay.me
"""
import os
import time
import requests
from typing import Optional
BILLING_URL = "https://skillpay.me/api/v1/billing"
API_KEY = os.environ.get("SKILL_BILLING_API_KEY", "")
SKILL_ID = os.environ.get("SKILL_BILLING_SKILL_ID", "")
HEADERS = {"X-API-Key": API_KEY, "Content-Type": "application/json"}
CACHE: dict = {}
CACHE_TTL = 300 # 5 minutes
CALL_PRICE = 0.0100 # USDT per call
def is_dev_mode() -> bool:
"""Return True if billing is not configured (dev mode)."""
return not API_KEY or not SKILL_ID
def charge_user(user_id: str) -> dict:
"""
Charge user for one classification.
Returns dict with keys: ok (bool), balance (float), payment_url (str or None)
"""
if is_dev_mode():
return {"ok": True, "balance": 999.0}
cache_key = f"charge_{user_id}"
if cache_key in CACHE:
cached_time, cached_val = CACHE[cache_key]
if time.time() - cached_time < CACHE_TTL:
return cached_val
try:
resp = requests.post(
f"{BILLING_URL}/charge",
headers=HEADERS,
json={"user_id": user_id, "skill_id": SKILL_ID, "amount": CALL_PRICE},
timeout=10
)
data = resp.json()
if data.get("success"):
result = {"ok": True, "balance": data.get("balance", 0.0)}
else:
result = {
"ok": False,
"balance": data.get("balance", 0.0),
"payment_url": data.get("payment_url")
}
except Exception:
result = {"ok": True, "balance": 999.0}
CACHE[cache_key] = (time.time(), result)
return result
FILE:scripts/__init__.py
# Text Classifier Package
Generate professional weekly or monthly work reports from text or files with customizable styles, templates, and output formats including Markdown, Word, and...
# SKILL.md - Weekly-Monthly Reporter
> Transform raw work content into professional weekly/monthly reports using AI.
**Slug:** weekly-monthly-reporter
## Tiered Features
| Feature | FREE | PRO |
|---------|:----:|:---:|
| Text input | ✅ | ✅ |
| File upload (TXT/MD) | ❌ | ✅ |
| Markdown output | ✅ | ✅ |
| Word (.docx) output | ❌ | ✅ |
| PDF output | ❌ | ✅ |
| Custom templates | ❌ | ✅ |
| Report history | ❌ | ✅ |
| Monthly generations | 5 | Unlimited |
| History retention | — | 12 months |
## Pricing
**Per-call:** $0.01 USDT per report generation
No monthly subscription. Pay only for what you use.
## Usage
### CLI
```bash
# Generate a weekly report
python -m scripts.main generate --content "Completed project A phase 1..." -k YOUR_API_KEY
# Generate from file
python -m scripts.main generate --file worklog.md -k YOUR_API_KEY
# Monthly report
python -m scripts.main generate --type monthly -c "Work content..." -k YOUR_API_KEY
# Output to Word
python -m scripts.main generate -c "Content..." -k KEY --format word --output report.docx
# View history
python -m scripts.main history -k YOUR_API_KEY
# Check status
python -m scripts.main status -k YOUR_API_KEY
```
### Python API
```python
from scripts.report_generator import ReportGenerator
from scripts.templates import ReportType, ReportStyle
generator = ReportGenerator(api_key="your_llm_key")
result = generator.generate_report(
work_content="Completed A project phase 1...",
report_type=ReportType.WEEKLY,
style=ReportStyle.CONCISE,
api_key="your_billing_key"
)
print(result["report"])
```
## Report Styles
- **Concise**: Quick summary for rapid updates
- **Detailed**: Comprehensive with KPI, improvements, growth sections
- **Leadership**: Executive-focused with risk analysis and metrics
- **Self-Review**: Self-assessment with achievements and areas for improvement
## Required Environment Variables
```bash
SKILL_BILLING_API_KEY # Your SkillPay Builder API Key
SKILL_BILLING_SKILL_ID # Skill slug: weekly-monthly-reporter
```
Set these in your runtime environment. Without them, the tool runs in Dev Mode (FREE tier, no billing).
## Billing
This skill uses **SkillPay** (skillpay.me) for per-call billing at **$0.01 USDT per report**.
- Your Feishu User ID (Open ID) is transmitted to `skillpay.me` exclusively for billing purposes
- No other data is transmitted to third parties
- Billing occurs at the start of each report generation (after API key validation)
- Dev Mode (`SKILL_BILLING_API_KEY` not set): FULL FREE USAGE — no API key required, no billing
## Security Notes
- **LLM Execution**: All AI report generation runs via LLM API you configure (OpenAI-compatible)
- **Data Isolation**: Report history is stored locally in `/tmp/weekly-monthly-reporter/` — no data leaves your environment
- **SQL Safety**: Not applicable (no database queries in this skill)
- **Path Isolation**: All writes go to `/tmp/` — no home directory access
## Dependencies
```
requests>=2.28.0
python-docx>=0.8.11 # Word output
reportlab>=4.0.0 # PDF output
```
FILE:requirements.txt
requests>=2.28.0
python-docx>=0.8.11
reportlab>=4.0.0
FILE:README.md
# Weekly-Monthly Reporter
AI-powered report generation from work content. Supports weekly and monthly reports in multiple formats.
## Quick Start
```bash
python -m scripts.main generate --content "Work content..." -k YOUR_API_KEY
```
## Documentation
See `SKILL.md` for full documentation.
FILE:scripts/validator.py
"""
Weekly-Monthly Reporter - Token Validation Module
Validates API keys against SkillPay billing.
"""
import time
import json
import requests
from pathlib import Path
from typing import Optional, Dict
try:
from .billing import is_dev_mode, charge_user
except ImportError:
from billing import is_dev_mode, charge_user
try:
from .config import TIERS, CACHE_FILE, CACHE_TTL, STORAGE_DIR
except ImportError:
from config import TIERS, CACHE_FILE, CACHE_TTL, STORAGE_DIR
def _load_cache() -> Dict:
"""Load token cache from disk."""
if not CACHE_FILE.exists():
return {}
try:
with open(CACHE_FILE, "r") as f:
return json.load(f)
except Exception:
return {}
def _save_cache(cache: Dict) -> None:
"""Save token cache to disk."""
STORAGE_DIR.mkdir(parents=True, exist_ok=True)
with open(CACHE_FILE, "w") as f:
json.dump(cache, f)
def _get_cached(api_key: str) -> Optional[Dict]:
"""Get cached validation result if still valid."""
cache = _load_cache()
if api_key in cache:
entry = cache[api_key]
if time.time() - entry.get("timestamp", 0) < CACHE_TTL:
return entry.get("result")
return None
def _cache_result(api_key: str, result: Dict) -> None:
"""Cache validation result."""
cache = _load_cache()
cache[api_key] = {"result": result, "timestamp": time.time()}
_save_cache(cache)
def validate_token(api_key: str) -> Dict:
"""
Validate API token.
Returns:
Dict with keys: valid (bool), tier (str), features (dict), error (str or None)
"""
if not api_key or not api_key.strip():
return {"valid": False, "tier": "FREE", "error": "Empty API key"}
# Check cache first
cached = _get_cached(api_key)
if cached is not None:
return cached
# Dev mode: no billing configured — degrade to FREE
if is_dev_mode():
result = {
"valid": True,
"tier": "FREE",
"features": TIERS["FREE"],
"error": None,
"degraded": True,
}
_cache_result(api_key, result)
return result
# Determine tier from key prefix (FREE / PRO)
tier_name = "FREE"
for prefix in ("FREE", "PRO"):
if api_key.startswith(prefix):
tier_name = prefix
break
# Try to charge — if fails, tier is FREE
try:
billing_result = charge_user(api_key) # user_id = api_key for self-charging
if billing_result.get("ok"):
result = {
"valid": True,
"tier": tier_name,
"features": TIERS[tier_name],
"error": None,
}
else:
result = {
"valid": True,
"tier": "FREE",
"features": TIERS["FREE"],
"error": "Insufficient balance or payment required",
"payment_url": billing_result.get("payment_url"),
}
except Exception:
result = {
"valid": True,
"tier": "FREE",
"features": TIERS["FREE"],
"error": "Network error, degraded to FREE tier",
"degraded": True,
}
_cache_result(api_key, result)
return result
def check_generation_limit(api_key: str, current_count: int) -> bool:
"""Check if user can generate a new report."""
validation = validate_token(api_key)
if not validation["valid"]:
return False
limit = validation["features"]["generations"]
if limit == -1:
return True
return current_count < limit
def get_user_tier(api_key: str) -> str:
"""Get user's tier name."""
return validate_token(api_key)["tier"]
FILE:scripts/file_handler.py
"""
File handling module for Weekly-Monthly Reporter.
Supports TXT and Markdown file uploads.
"""
import os
from pathlib import Path
from typing import Optional, Tuple
try:
from .config import UPLOAD_DIR
except ImportError:
from config import UPLOAD_DIR
class FileHandler:
"""Handle file upload and parsing."""
SUPPORTED_EXTENSIONS = {".txt", ".md", ".markdown"}
MAX_FILE_SIZE = 1024 * 1024 # 1MB
@classmethod
def read_file(cls, file_path: str) -> Tuple[str, str]:
"""
Read content from a file.
Args:
file_path: Path to the file
Returns:
Tuple of (content, file_type)
Raises:
FileNotFoundError: If file doesn't exist
ValueError: If file type not supported or file too large
"""
path = Path(file_path)
if not path.exists():
raise FileNotFoundError(f"File not found: {file_path}")
# Check extension
ext = path.suffix.lower()
if ext not in cls.SUPPORTED_EXTENSIONS:
raise ValueError(f"Unsupported file type: {ext}. Supported: {cls.SUPPORTED_EXTENSIONS}")
# Check size
file_size = path.stat().st_size
if file_size > cls.MAX_FILE_SIZE:
raise ValueError(f"File too large: {file_size} bytes. Max: {cls.MAX_FILE_SIZE} bytes")
# Read content
try:
with open(path, "r", encoding="utf-8") as f:
content = f.read()
except UnicodeDecodeError:
# Try with different encoding
with open(path, "r", encoding="gbk") as f:
content = f.read()
file_type = "markdown" if ext in {".md", ".markdown"} else "text"
return content, file_type
@classmethod
def validate_file(cls, file_path: str) -> Tuple[bool, str]:
"""
Validate a file before processing.
Args:
file_path: Path to the file
Returns:
Tuple of (is_valid, error_message)
"""
path = Path(file_path)
if not path.exists():
return False, f"File not found: {file_path}"
ext = path.suffix.lower()
if ext not in cls.SUPPORTED_EXTENSIONS:
return False, f"Unsupported file type: {ext}"
file_size = path.stat().st_size
if file_size > cls.MAX_FILE_SIZE:
return False, f"File too large: {file_size} bytes (max: {cls.MAX_FILE_SIZE})"
return True, ""
@classmethod
def parse_content(cls, content: str, source_type: str) -> str:
"""
Parse content from various sources into unified format.
Args:
content: Raw content
source_type: Source type ("text", "markdown", "feishu_task")
Returns:
Parsed content string
"""
if source_type == "markdown":
# Extract text from markdown, removing headers formatting
lines = content.split("\n")
parsed_lines = []
for line in lines:
# Remove markdown headers (# and ##)
if line.strip().startswith("#"):
line = line.lstrip("#").strip()
parsed_lines.append(line)
return "\n".join(parsed_lines)
return content.strip()
@classmethod
def save_upload(cls, content: str, filename: str, upload_dir: Optional[Path] = None) -> str:
"""
Save uploaded content to a file.
Args:
content: Content to save
filename: Desired filename
upload_dir: Directory to save to (default: ~/.weekly_reporter/uploads)
Returns:
Path to saved file
"""
if upload_dir is None:
upload_dir = UPLOAD_DIR
upload_dir.mkdir(parents=True, exist_ok=True)
# Sanitize filename
safe_name = "".join(c for c in filename if c.isalnum() or c in "._-")
if not safe_name:
safe_name = "uploaded_file.txt"
file_path = upload_dir / safe_name
with open(file_path, "w", encoding="utf-8") as f:
f.write(content)
return str(file_path)
FILE:scripts/config.py
"""
Weekly-Monthly Reporter - Configuration Module
Tier definitions and constants.
"""
from pathlib import Path
from typing import Dict
# Storage paths (use /tmp to avoid home-directory writes)
STORAGE_DIR = Path("/tmp/weekly-monthly-reporter")
HISTORY_DIR = STORAGE_DIR / "history"
UPLOAD_DIR = STORAGE_DIR / "uploads"
CACHE_FILE = STORAGE_DIR / ".token_cache.json"
CACHE_TTL = 300 # 5 minutes
# Tier definitions — 2 tiers only (FREE | PRO), per-call billing
TIERS: Dict[str, dict] = {
"FREE": {
"generations": 5,
"input_types": ["text"],
"output_formats": ["markdown"],
"history_days": 0,
"custom_template": False,
},
"PRO": {
"generations": -1, # Unlimited
"input_types": ["text", "file"],
"output_formats": ["markdown", "word", "pdf"],
"history_days": 365,
"custom_template": True,
},
}
def get_tier_limits(tier: str) -> dict:
"""Return limits for a given tier name."""
return TIERS.get(tier, TIERS["FREE"])
def is_valid_tier(tier: str) -> bool:
"""Check if tier name is known."""
return tier in TIERS
FILE:scripts/billing.py
"""
Weekly-Monthly Reporter - Billing Module (SkillPay)
Handles per-call billing via skillpay.me
"""
import os
import time
import requests
from typing import Optional
BILLING_URL = "https://skillpay.me/api/v1/billing"
API_KEY = os.environ.get("SKILL_BILLING_API_KEY", "")
SKILL_ID = os.environ.get("SKILL_BILLING_SKILL_ID", "")
HEADERS = {"X-API-Key": API_KEY, "Content-Type": "application/json"}
CACHE: dict = {}
CACHE_TTL = 300 # 5 minutes
CALL_PRICE = 0.0100 # USDT per call
def is_dev_mode() -> bool:
"""Return True if billing is not configured (dev mode)."""
return not API_KEY or not SKILL_ID
def charge_user(user_id: str) -> dict:
"""
Charge user for one report generation.
Returns dict with keys: ok (bool), balance (float), payment_url (str or None)
"""
if is_dev_mode():
return {"ok": True, "balance": 999.0}
cache_key = f"charge_{user_id}"
if cache_key in CACHE:
cached_time, cached_val = CACHE[cache_key]
if time.time() - cached_time < CACHE_TTL:
return cached_val
try:
resp = requests.post(
f"{BILLING_URL}/charge",
headers=HEADERS,
json={"user_id": user_id, "skill_id": SKILL_ID, "amount": CALL_PRICE},
timeout=10
)
data = resp.json()
if data.get("success"):
result = {"ok": True, "balance": data.get("balance", 0.0)}
else:
result = {
"ok": False,
"balance": data.get("balance", 0.0),
"payment_url": data.get("payment_url")
}
except Exception:
result = {"ok": True, "balance": 999.0}
CACHE[cache_key] = (time.time(), result)
return result
FILE:scripts/__init__.py
# Weekly-Monthly Reporter - Skills Developer Package
__version__ = "1.0.0"
FILE:scripts/templates.py
"""
Report templates for Weekly-Monthly Reporter.
"""
from typing import Dict, List, Optional
from enum import Enum
class ReportStyle(Enum):
"""Report style options."""
CONCISE = "concise"
DETAILED = "detailed"
LEADERSHIP = "leadership"
SELF_REVIEW = "self_review"
class ReportType(Enum):
"""Report type."""
WEEKLY = "weekly"
MONTHLY = "monthly"
# Default templates for weekly reports
WEEKLY_TEMPLATE_CONCISE = """## Weekly Summary
{summary}
## Key Progress
{highlights}
## Next Week Plan
{next_week}
## Support Needed
{support_needed}
"""
WEEKLY_TEMPLATE_DETAILED = """## Weekly Summary
{summary}
## Key Progress
{highlights}
## KPI Status
{kpi}
## Next Week Plan
{next_week}
## Support Needed
{support_needed}
## Learning & Growth
{growth}
"""
WEEKLY_TEMPLATE_LEADERSHIP = """## Weekly Overview
{summary}
## Key Achievements
{highlights}
## KPI Tracking
{kpi}
## Next Week Priorities
{next_week}
## Risks & Challenges
{risks}
## Support Needed
{support_needed}
"""
WEEKLY_TEMPLATE_SELF_REVIEW = """## Weekly Summary
{summary}
## Key Achievements
{highlights}
## KPI Self-Assessment
{kpi}
## Areas for Improvement
{improvements}
## Next Week Goals
{next_week}
## Learning Notes
{growth}
"""
# Monthly report templates
MONTHLY_TEMPLATE_CONCISE = """## Monthly Summary - {month}
## Key Achievements
{summary}
## Key Progress
{highlights}
## Monthly KPI Summary
{kpi}
## Trend Analysis
{trend}
## Next Month Plan
{next_month}
## Support Needed
{support_needed}
"""
MONTHLY_TEMPLATE_DETAILED = """## Monthly Summary - {month}
## Key Achievements
{summary}
## Key Progress
{highlights}
## KPI Status
{kpi}
## Trend Analysis
{trend}
## Problems & Solutions
{problems}
## Next Month Priorities
{next_month}
## Support Needed
{support_needed}
## Lessons Learned
{lessons}
"""
MONTHLY_TEMPLATE_LEADERSHIP = """## Monthly Executive Report - {month}
## Executive Summary
{summary}
## Key Business Progress
{highlights}
## KPI Dashboard
{kpi}
## Trend Insights
{trend}
## Risk Alerts
{risks}
## Next Month Key Tasks
{next_month}
## Resource Needs
{support_needed}
"""
MONTHLY_TEMPLATE_SELF_REVIEW = """## Monthly Self-Review - {month}
## Main Work This Month
{summary}
## Key Achievements
{highlights}
## KPI Self-Assessment
{kpi}
## Growth & Learnings
{growth}
## Areas for Improvement
{improvements}
## Next Month Goals
{next_month}
"""
def get_template(report_type: ReportType, style: ReportStyle) -> str:
"""Get the appropriate template based on report type and style."""
if report_type == ReportType.WEEKLY:
templates = {
ReportStyle.CONCISE: WEEKLY_TEMPLATE_CONCISE,
ReportStyle.DETAILED: WEEKLY_TEMPLATE_DETAILED,
ReportStyle.LEADERSHIP: WEEKLY_TEMPLATE_LEADERSHIP,
ReportStyle.SELF_REVIEW: WEEKLY_TEMPLATE_SELF_REVIEW,
}
else:
templates = {
ReportStyle.CONCISE: MONTHLY_TEMPLATE_CONCISE,
ReportStyle.DETAILED: MONTHLY_TEMPLATE_DETAILED,
ReportStyle.LEADERSHIP: MONTHLY_TEMPLATE_LEADERSHIP,
ReportStyle.SELF_REVIEW: MONTHLY_TEMPLATE_SELF_REVIEW,
}
return templates.get(style, WEEKLY_TEMPLATE_CONCISE)
def get_default_sections(report_type: ReportType) -> List[str]:
"""Get default sections for a report type."""
if report_type == ReportType.WEEKLY:
return ["summary", "highlights", "next_week", "support_needed"]
else:
return ["summary", "highlights", "kpi", "trend", "next_month", "support_needed"]
def build_prompt(work_content: str, report_type: ReportType, style: ReportStyle,
previous_report: Optional[str] = None, custom_template: Optional[str] = None) -> str:
"""
Build AI prompt for report generation.
"""
style_names = {
ReportStyle.CONCISE: "Concise",
ReportStyle.DETAILED: "Detailed",
ReportStyle.LEADERSHIP: "Executive",
ReportStyle.SELF_REVIEW: "Self-Review"
}
style_name = style_names.get(style, "Concise")
report_name = "Weekly Report" if report_type == ReportType.WEEKLY else "Monthly Report"
prompt = f"""You are a professional executive assistant skilled at organizing work content into structured weekly and monthly reports.
Please generate a [{style_name}] {report_name} based on the following work content.
## Raw Work Content:
{work_content}
"""
if previous_report:
prompt += f"""
## Previous {report_name} (for continuation):
{previous_report}
Please continue from the previous report, adding new content while maintaining consistent style.
"""
if custom_template:
prompt += f"""
## Custom Template:
{custom_template}
"""
else:
template = get_template(report_type, style)
prompt += f"""
## Template Structure:
{template}
Please strictly follow this structure. Guidelines:
1. Highlight key achievements using priority markers
2. Use specific, quantifiable data where possible
3. Be professional, concise, and well-organized
4. If continuing from a previous report, note the connection to prior work
"""
return prompt
FILE:scripts/report_generator.py
"""
AI Report Generation module for Weekly-Monthly Reporter.
Handles LLM API calls for generating structured reports.
"""
import json
import time
import hashlib
import requests
from pathlib import Path
from typing import Dict, Optional, List, Tuple
from datetime import datetime
try:
from .templates import ReportType, ReportStyle, build_prompt, get_template
from .validator import validate_token, check_generation_limit
from .config import TIERS, HISTORY_DIR
except ImportError:
from templates import ReportType, ReportStyle, build_prompt, get_template
from validator import validate_token, check_generation_limit
from config import TIERS, HISTORY_DIR
class ReportGenerator:
"""Main report generation class."""
def __init__(self, api_key: Optional[str] = None, api_endpoint: Optional[str] = None,
model: str = "gpt-4"):
"""
Initialize the report generator.
Args:
api_key: LLM API key (if None, will try to load from config)
api_endpoint: LLM API endpoint (default: OpenAI compatible)
model: Model to use for generation
"""
self.api_key = api_key
self.api_endpoint = api_endpoint or "https://api.openai.com/v1/chat/completions"
self.model = model
self._history_dir = HISTORY_DIR
self._history_dir.mkdir(parents=True, exist_ok=True)
def _load_usage(self, api_key: str) -> Tuple[int, str]:
"""Load usage count and current month for a key."""
usage_file = self._history_dir / f"{self._get_key_hash(api_key)}.json"
if usage_file.exists():
try:
with open(usage_file, "r") as f:
data = json.load(f)
current_month = datetime.now().strftime("%Y-%m")
if data.get("month") == current_month:
return data.get("count", 0), current_month
except Exception:
pass
return 0, datetime.now().strftime("%Y-%m")
def _save_usage(self, api_key: str, count: int, month: str) -> None:
"""Save usage count."""
usage_file = self._history_dir / f"{self._get_key_hash(api_key)}.json"
with open(usage_file, "w") as f:
json.dump({"count": count, "month": month}, f)
def _get_key_hash(self, api_key: str) -> str:
"""Get a hash of the API key for file naming."""
return hashlib.md5(api_key.encode()).hexdigest()[:12]
def _call_llm(self, prompt: str, temperature: float = 0.7, max_tokens: int = 2048) -> str:
"""
Call LLM API to generate report.
Args:
prompt: Formatted prompt
temperature: Sampling temperature
max_tokens: Max tokens in response
Returns:
Generated report text
"""
if not self.api_key:
raise ValueError("API key not configured. Please set your LLM API key.")
headers = {
"Authorization": f"Bearer {self.api_key}",
"Content-Type": "application/json"
}
payload = {
"model": self.model,
"messages": [
{"role": "system", "content": "You are a professional executive assistant skilled at organizing work content into structured weekly and monthly reports."},
{"role": "user", "content": prompt}
],
"temperature": temperature,
"max_tokens": max_tokens
}
response = requests.post(
self.api_endpoint,
headers=headers,
json=payload,
timeout=60
)
if response.status_code != 200:
raise Exception(f"API call failed: {response.status_code} - {response.text}")
data = response.json()
return data["choices"][0]["message"]["content"]
def generate_report(
self,
work_content: str,
report_type: ReportType,
style: ReportStyle,
api_key: str,
previous_report: Optional[str] = None,
custom_template: Optional[str] = None
) -> Dict:
"""
Generate a report from work content.
Args:
work_content: Raw work content
report_type: WEEKLY or MONTHLY
style: Report style
api_key: Token for validation and usage tracking
previous_report: Optional previous report for continuation
custom_template: Optional custom template
Returns:
Dict with keys: success (bool), report (str), error (str or None)
"""
# Validate token
validation = validate_token(api_key)
if not validation["valid"]:
return {
"success": False,
"report": None,
"error": f"Invalid API key: {validation.get('error', 'Unknown error')}"
}
# Check generation limit
current_count, current_month = self._load_usage(api_key)
tier = validation["tier"]
if not check_generation_limit(api_key, current_count):
return {
"success": False,
"report": None,
"error": f"Generation limit reached for {tier} tier. Please upgrade or wait."
}
# Build prompt and generate
try:
prompt = build_prompt(work_content, report_type, style, previous_report, custom_template)
report = self._call_llm(prompt)
# Increment usage count
self._save_usage(api_key, current_count + 1, current_month)
return {
"success": True,
"report": report,
"error": None,
"usage": {
"tier": tier,
"used": current_count + 1,
"limit": TIERS[tier]["generations"]
}
}
except Exception as e:
return {
"success": False,
"report": None,
"error": f"Report generation failed: {str(e)}"
}
def parse_work_input(self, input_data: str, input_type: str) -> str:
"""Parse various input formats into unified work content."""
if input_type == "text":
return input_data.strip()
elif input_type == "file":
return input_data.strip()
elif input_type == "feishu_task":
return self._parse_feishu_tasks(input_data)
else:
return input_data.strip()
def _parse_feishu_tasks(self, tasks_json: str) -> str:
"""Parse Feishu task list JSON into work content format."""
try:
tasks = json.loads(tasks_json)
if isinstance(tasks, list):
lines = []
for i, task in enumerate(tasks, 1):
title = task.get("summary", task.get("title", "Unnamed Task"))
done = task.get("completed_at") or task.get("done", True)
lines.append(f"{i}. {title} - {'Done' if done else 'In Progress'}")
return "\n".join(lines)
except json.JSONDecodeError:
pass
return tasks_json
FILE:scripts/history_manager.py
"""
History management module for Weekly-Monthly Reporter.
Handles local JSON storage of report history.
"""
import json
import os
from pathlib import Path
from typing import List, Optional, Dict
from datetime import datetime, timedelta
try:
from .config import HISTORY_DIR
except ImportError:
from config import HISTORY_DIR
class HistoryManager:
"""Manage report history storage."""
def __init__(self, history_dir: Optional[Path] = None, max_days: int = 0):
"""
Initialize history manager.
Args:
history_dir: Directory to store history (default: ~/.weekly_reporter/history)
max_days: Maximum days to retain history (0 = no limit based on tier)
"""
self._history_dir = history_dir or HISTORY_DIR
self._history_dir.mkdir(parents=True, exist_ok=True)
self._index_file = self._history_dir / "index.json"
self.max_days = max_days
def _load_index(self) -> Dict:
"""Load history index."""
if not self._index_file.exists():
return {"reports": []}
try:
with open(self._index_file, "r", encoding="utf-8") as f:
return json.load(f)
except Exception:
return {"reports": []}
def _save_index(self, index: Dict) -> None:
"""Save history index."""
with open(self._index_file, "w", encoding="utf-8") as f:
json.dump(index, f, ensure_ascii=False, indent=2)
def _get_report_file(self, report_id: str) -> Path:
"""Get path to report file."""
return self._history_dir / f"{report_id}.json"
def save_report(
self,
report: str,
report_type: str,
style: str,
api_key: str,
input_preview: str = "",
tier: str = "FREE"
) -> str:
"""
Save a generated report to history.
Args:
report: Generated report content
report_type: "weekly" or "monthly"
style: Style used for generation
api_key: User's API key (hashed for storage)
input_preview: Preview of input content
tier: User's tier for retention policy
Returns:
Report ID
"""
import hashlib
key_hash = hashlib.md5(api_key.encode()).hexdigest()[:12]
# Generate report ID
timestamp = datetime.now().strftime("%Y%m%d%H%M%S")
report_id = f"{key_hash}_{timestamp}"
# Save report data
report_data = {
"id": report_id,
"created_at": datetime.now().isoformat(),
"report_type": report_type,
"style": style,
"content": report,
"input_preview": input_preview[:200] if input_preview else "",
"tier": tier
}
report_file = self._get_report_file(report_id)
with open(report_file, "w", encoding="utf-8") as f:
json.dump(report_data, f, ensure_ascii=False, indent=2)
# Update index
index = self._load_index()
index["reports"].insert(0, {
"id": report_id,
"created_at": report_data["created_at"],
"report_type": report_type,
"style": style,
"preview": input_preview[:100] if input_preview else ""
})
self._save_index(index)
# Cleanup old reports based on tier retention
self._cleanup_old_reports(tier)
return report_id
def _cleanup_old_reports(self, tier: str) -> None:
"""Remove reports older than retention period."""
retention_days = {
"FREE": 0, # No history
"PRO": 365,
}
days = retention_days.get(tier, 0)
if days == 0:
return
cutoff = datetime.now() - timedelta(days=days)
index = self._load_index()
valid_reports = []
for report_entry in index["reports"]:
created = datetime.fromisoformat(report_entry["created_at"])
if created >= cutoff:
valid_reports.append(report_entry)
else:
# Delete the file
report_file = self._get_report_file(report_entry["id"])
if report_file.exists():
report_file.unlink()
index["reports"] = valid_reports
self._save_index(index)
def get_report(self, report_id: str) -> Optional[Dict]:
"""Get a specific report by ID."""
report_file = self._get_report_file(report_id)
if not report_file.exists():
return None
try:
with open(report_file, "r", encoding="utf-8") as f:
return json.load(f)
except Exception:
return None
def list_reports(self, api_key: str, limit: int = 10, offset: int = 0) -> List[Dict]:
"""
List reports for a user.
Args:
api_key: User's API key
limit: Max reports to return
offset: Offset for pagination
Returns:
List of report metadata
"""
import hashlib
key_hash = hashlib.md5(api_key.encode()).hexdigest()[:12]
index = self._load_index()
user_reports = [
r for r in index["reports"]
if r["id"].startswith(key_hash)
]
return user_reports[offset:offset + limit]
def get_latest_report(self, api_key: str, report_type: Optional[str] = None) -> Optional[Dict]:
"""
Get the most recent report.
Args:
api_key: User's API key
report_type: Optional filter by type ("weekly" or "monthly")
Returns:
Latest report data or None
"""
reports = self.list_reports(api_key, limit=100)
for r in reports:
if report_type and r.get("report_type") != report_type:
continue
return self.get_report(r["id"])
return None
def delete_report(self, report_id: str, api_key: str) -> bool:
"""
Delete a specific report.
Args:
report_id: Report ID to delete
api_key: User's API key (for ownership verification)
Returns:
True if deleted, False otherwise
"""
import hashlib
key_hash = hashlib.md5(api_key.encode()).hexdigest()[:12]
if not report_id.startswith(key_hash):
return False
report_file = self._get_report_file(report_id)
if report_file.exists():
report_file.unlink()
# Update index
index = self._load_index()
index["reports"] = [r for r in index["reports"] if r["id"] != report_id]
self._save_index(index)
return True
def get_usage_stats(self, api_key: str) -> Dict:
"""Get usage statistics for a user."""
import hashlib
key_hash = hashlib.md5(api_key.encode()).hexdigest()[:12]
usage_file = self._history_dir / f"{key_hash}.json"
if usage_file.exists():
try:
with open(usage_file, "r") as f:
return json.load(f)
except Exception:
pass
return {"count": 0, "month": datetime.now().strftime("%Y-%m")}
FILE:scripts/doc_generator.py
"""
Document generation module for Weekly-Monthly Reporter.
Generates Word (.docx) and PDF files from reports.
"""
import os
import io
from pathlib import Path
from typing import Optional
from datetime import datetime
class WordGenerator:
"""Generate Word (.docx) documents from reports."""
def __init__(self):
"""Initialize Word generator."""
self._docx_available = True
try:
from docx import Document
from docx.shared import Pt, RGBColor
self.Document = Document
self.Pt = Pt
self.RGBColor = RGBColor
except ImportError:
self._docx_available = False
def generate(self, report_content: str, title: Optional[str] = None) -> bytes:
"""
Generate Word document from report.
Args:
report_content: Report markdown content
title: Optional document title
Returns:
Bytes of the Word document
"""
if not self._docx_available:
raise ImportError("python-docx is required for Word generation. Install with: pip install python-docx")
doc = self.Document()
# Set document style
style = doc.styles["Normal"]
style.font.name = "Microsoft YaHei"
style.font.size = self.Pt(11)
# Add title
if title:
heading = doc.add_heading(title, 0)
heading.style.font.name = "Microsoft YaHei"
# Parse markdown and convert to Word
lines = report_content.split("\n")
for line in lines:
stripped = line.strip()
if not stripped:
doc.add_paragraph()
continue
# Headers
if stripped.startswith("## "):
doc.add_heading(stripped[3:], level=2)
elif stripped.startswith("# "):
doc.add_heading(stripped[2:], level=1)
elif stripped.startswith("### "):
doc.add_heading(stripped[4:], level=3)
# List items
elif stripped.startswith("- ") or stripped.startswith("* "):
p = doc.add_paragraph(stripped[2:], style="List Bullet")
elif stripped.startswith("-"):
p = doc.add_paragraph(stripped[1:].strip(), style="List Bullet")
# Regular text
else:
# Clean up markdown formatting
clean_line = self._clean_markdown(stripped)
doc.add_paragraph(clean_line)
# Save to bytes
buffer = io.BytesIO()
doc.save(buffer)
buffer.seek(0)
return buffer.getvalue()
def _clean_markdown(self, text: str) -> str:
"""Remove or convert markdown formatting."""
# Remove bold/italic markers
text = text.replace("**", "").replace("*", "")
# Remove inline code
text = text.replace("`", "")
return text
def save(self, report_content: str, output_path: str, title: Optional[str] = None) -> str:
"""
Generate and save Word document.
Args:
report_content: Report content
output_path: Output file path
title: Optional document title
Returns:
Path to saved file
"""
content = self.generate(report_content, title)
with open(output_path, "wb") as f:
f.write(content)
return output_path
class PDFGenerator:
"""Generate PDF documents from reports."""
def __init__(self):
"""Initialize PDF generator."""
self._reportlab_available = True
try:
from reportlab.lib.pagesizes import A4
from reportlab.lib.styles import ParagraphStyle
from reportlab.lib.units import mm
from reportlab.platypus import SimpleDocTemplate, Paragraph, Spacer, HRFlowable
from reportlab.lib.enums import TA_LEFT, TA_CENTER
self.A4 = A4
self.ParagraphStyle = ParagraphStyle
self.mm = mm
self.SimpleDocTemplate = SimpleDocTemplate
self.Paragraph = Paragraph
self.Spacer = Spacer
self.HRFlowable = HRFlowable
self.TA_LEFT = TA_LEFT
self.TA_CENTER = TA_CENTER
except ImportError:
self._reportlab_available = False
def generate(self, report_content: str, title: Optional[str] = None) -> bytes:
"""
Generate PDF from report.
Args:
report_content: Report markdown content
title: Optional document title
Returns:
Bytes of the PDF
"""
if not self._reportlab_available:
raise ImportError("reportlab is required for PDF generation. Install with: pip install reportlab")
buffer = io.BytesIO()
doc = SimpleDocTemplate(
buffer,
pagesize=self.A4,
rightMargin=20*self.mm,
leftMargin=20*self.mm,
topMargin=20*self.mm,
bottomMargin=20*self.mm
)
# Define styles
styles = {
"title": self.ParagraphStyle(
"Title",
fontSize=18,
leading=24,
alignment=self.TA_CENTER,
spaceAfter=20
),
"h1": self.ParagraphStyle(
"H1",
fontSize=14,
leading=18,
spaceBefore=15,
spaceAfter=10,
textColor=(51, 102, 153)
),
"h2": self.ParagraphStyle(
"H2",
fontSize=12,
leading=16,
spaceBefore=12,
spaceAfter=8,
textColor=(51, 102, 153)
),
"normal": self.ParagraphStyle(
"Normal",
fontSize=10,
leading=14,
spaceAfter=6
),
"bullet": self.ParagraphStyle(
"Bullet",
fontSize=10,
leading=14,
leftIndent=20,
spaceAfter=4
)
}
story = []
# Add title
if title:
story.append(self.Paragraph(title, styles["title"]))
# Parse and add content
lines = report_content.split("\n")
for line in lines:
stripped = line.strip()
if not stripped:
story.append(self.Spacer(1, 6))
continue
# Headers
if stripped.startswith("## "):
story.append(self.Paragraph(stripped[3:], styles["h2"]))
elif stripped.startswith("# "):
story.append(self.Paragraph(stripped[2:], styles["h1"]))
elif stripped.startswith("### "):
story.append(self.Paragraph(stripped[4:], styles["h2"]))
# List items
elif stripped.startswith("- ") or stripped.startswith("* "):
text = self._clean_markdown(stripped[2:])
story.append(self.Paragraph(f"• {text}", styles["bullet"]))
elif stripped.startswith("-"):
text = self._clean_markdown(stripped[1:].strip())
story.append(self.Paragraph(f"• {text}", styles["bullet"]))
# Horizontal rule
elif stripped.startswith("---"):
story.append(self.HRFlowable(width="100%", thickness=1, color=(200,200,200)))
story.append(self.Spacer(1, 10))
# Regular text
else:
text = self._clean_markdown(stripped)
story.append(self.Paragraph(text, styles["normal"]))
doc.build(story)
buffer.seek(0)
return buffer.getvalue()
def _clean_markdown(self, text: str) -> str:
"""Remove or convert markdown formatting."""
text = text.replace("**", "").replace("*", "")
text = text.replace("`", "")
# Keep emoji but they're not supported in standard fonts
return text
def save(self, report_content: str, output_path: str, title: Optional[str] = None) -> str:
"""
Generate and save PDF document.
Args:
report_content: Report content
output_path: Output file path
title: Optional document title
Returns:
Path to saved file
"""
content = self.generate(report_content, title)
with open(output_path, "wb") as f:
f.write(content)
return output_path
def generate_document(
report_content: str,
output_format: str,
output_path: Optional[str] = None,
title: Optional[str] = None
) -> bytes:
"""
Generate document in specified format.
Args:
report_content: Report markdown content
output_format: "word", "pdf", or "markdown"
output_path: Optional path to save file
title: Optional document title
Returns:
Document bytes or path to saved file
"""
if output_format.lower() == "word":
generator = WordGenerator()
content = generator.generate(report_content, title)
elif output_format.lower() == "pdf":
generator = PDFGenerator()
content = generator.generate(report_content, title)
elif output_format.lower() == "markdown":
content = report_content.encode("utf-8")
else:
raise ValueError(f"Unsupported format: {output_format}")
if output_path:
mode = "wb" if output_format.lower() != "markdown" else "w"
with open(output_path, mode) as f:
f.write(content)
return output_path
return content
FILE:scripts/main.py
#!/usr/bin/env python3
"""
Weekly-Monthly Reporter - Main CLI Entry Point
A tool to generate professional weekly/monthly reports from work content
using AI, with support for multiple output formats and Feishu integration.
"""
import sys
import json
import argparse
from pathlib import Path
from typing import Optional
from .report_generator import ReportGenerator
from .history_manager import HistoryManager
from .validator import validate_token
from .billing import charge_user, is_dev_mode
from .config import TIERS
from .templates import ReportType, ReportStyle
from .file_handler import FileHandler
from .doc_generator import generate_document
def create_parser() -> argparse.ArgumentParser:
"""Create CLI argument parser."""
parser = argparse.ArgumentParser(
description="Weekly-Monthly Reporter - AI-powered report generation",
formatter_class=argparse.RawDescriptionHelpFormatter
)
subparsers = parser.add_subparsers(dest="command", help="Available commands")
# Generate command
gen_parser = subparsers.add_parser("generate", help="Generate a new report")
gen_parser.add_argument("--content", "-c", help="Work content directly")
gen_parser.add_argument("--file", "-f", help="File path (TXT/MD) with work content")
gen_parser.add_argument("--type", "-t", choices=["weekly", "monthly"], default="weekly",
help="Report type")
gen_parser.add_argument("--style", "-s", choices=["concise", "detailed", "leadership", "self_review"],
default="concise", help="Report style")
gen_parser.add_argument("--api-key", "-k", required=True, help="API key for validation")
gen_parser.add_argument("--llm-key", help="LLM API key for generation")
gen_parser.add_argument("--output", "-o", help="Output file path")
gen_parser.add_argument("--format", choices=["markdown", "word", "pdf"], default="markdown",
help="Output format")
gen_parser.add_argument("--template", help="Custom report template")
gen_parser.add_argument("--continue-from", help="Previous report ID to continue from")
gen_parser.add_argument("--save", action="store_true", help="Save to history")
# History command
hist_parser = subparsers.add_parser("history", help="View report history")
hist_parser.add_argument("--api-key", "-k", required=True)
hist_parser.add_argument("--limit", "-n", type=int, default=10)
hist_parser.add_argument("--show", help="Show specific report ID")
hist_parser.add_argument("--delete", help="Delete specific report ID")
# Validate command
val_parser = subparsers.add_parser("validate", help="Validate API key")
val_parser.add_argument("--api-key", "-k", required=True)
# Status command
stat_parser = subparsers.add_parser("status", help="Check usage status")
stat_parser.add_argument("--api-key", "-k", required=True)
return parser
def cmd_generate(args) -> int:
"""Handle generate command."""
# Validate API key first
validation = validate_token(args.api_key)
if not validation["valid"]:
print(f"Error: Invalid API key - {validation.get('error', 'Unknown error')}")
return 1
print(f"Tier: {validation['tier']}")
if validation.get('degraded'):
print(f"Warning: {validation.get('error', 'Running in degraded mode')}")
# Charge user before generation (skip in dev mode if not configured)
if not is_dev_mode():
billing = charge_user(args.api_key)
if not billing.get("ok"):
print(f"Error: Payment required - insufficient balance.")
if billing.get("payment_url"):
print(f"Payment URL: {billing['payment_url']}")
return 1
# Get work content
work_content = ""
if args.content:
work_content = args.content
elif args.file:
try:
work_content, _ = FileHandler.read_file(args.file)
except Exception as e:
print(f"Error reading file: {e}")
return 1
else:
print("Error: Must provide --content or --file")
return 1
# Continue from previous report if specified
previous_report = None
if args.continue_from:
history = HistoryManager()
prev = history.get_report(args.continue_from)
if prev:
previous_report = prev.get("content", "")
print(f"Continuing from report: {args.continue_from}")
else:
print(f"Warning: Previous report {args.continue_from} not found")
# Determine report type and style
report_type = ReportType.WEEKLY if args.type == "weekly" else ReportType.MONTHLY
style_map = {
"concise": ReportStyle.CONCISE,
"detailed": ReportStyle.DETAILED,
"leadership": ReportStyle.LEADERSHIP,
"self_review": ReportStyle.SELF_REVIEW
}
style = style_map.get(args.style, ReportStyle.CONCISE)
# Generate report
generator = ReportGenerator(api_key=args.llm_key)
print("Generating report...")
result = generator.generate_report(
work_content=work_content,
report_type=report_type,
style=style,
api_key=args.api_key,
previous_report=previous_report,
custom_template=args.template
)
if not result["success"]:
print(f"Error: {result['error']}")
return 1
report = result["report"]
# Output report
if args.output:
try:
output_format = args.format
generate_document(report, output_format, args.output,
title=f"{report_type.value.title()} Report")
print(f"Report saved to: {args.output}")
except ImportError as e:
print(f"Error: {e}")
return 1
except Exception as e:
print(f"Error saving document: {e}")
return 1
else:
print("\n" + "="*60)
print(report)
print("="*60)
# Save to history if requested
if args.save:
history = HistoryManager()
history.save_report(
report=report,
report_type=args.type,
style=args.style,
api_key=args.api_key,
input_preview=work_content[:200],
tier=validation["tier"]
)
print("Report saved to history.")
# Print usage info
if "usage" in result:
usage = result["usage"]
print(f"\nUsage: {usage['used']}/{usage['limit']} generations this month")
return 0
def cmd_history(args) -> int:
"""Handle history command."""
history = HistoryManager()
if args.show:
report = history.get_report(args.show)
if not report:
print(f"Report not found: {args.show}")
return 1
print("="*60)
print(f"Type: {report.get('report_type', 'unknown')}")
print(f"Style: {report.get('style', 'unknown')}")
print(f"Created: {report.get('created_at', 'unknown')}")
print("-"*60)
print(report.get("content", ""))
print("="*60)
return 0
if args.delete:
success = history.delete_report(args.delete, args.api_key)
if success:
print(f"Deleted report: {args.delete}")
else:
print(f"Failed to delete report: {args.delete}")
return 0
# List reports
reports = history.list_reports(args.api_key, limit=args.limit)
if not reports:
print("No reports found.")
return 0
print(f"Found {len(reports)} report(s):\n")
for r in reports:
print(f" {r['id']}")
print(f" Type: {r.get('report_type', 'unknown')}")
print(f" Style: {r.get('style', 'unknown')}")
print(f" Created: {r.get('created_at', 'unknown')}")
print()
return 0
def cmd_validate(args) -> int:
"""Handle validate command."""
result = validate_token(args.api_key)
print(json.dumps(result, indent=2, ensure_ascii=False))
return 0 if result["valid"] else 1
def cmd_status(args) -> int:
"""Handle status command."""
history = HistoryManager()
validation = validate_token(args.api_key)
if not validation["valid"]:
print("Invalid API key")
return 1
usage = history.get_usage_stats(args.api_key)
tier = validation["tier"]
features = validation["features"]
print(f"Tier: {tier}")
print(f"Features:")
print(f" - Input types: {', '.join(features.get('input_types', []))}")
print(f" - Output formats: {', '.join(features.get('output_formats', []))}")
print(f" - History retention: {features.get('history_days', 0)} days")
print(f" - Custom templates: {'Yes' if features.get('custom_template') else 'No'}")
print(f" - API access: {'Yes' if features.get('api_access') else 'No'}")
limit = features.get("generations", 0)
used = usage.get("count", 0)
month = usage.get("month", "unknown")
if limit == -1:
print(f"\nUsage this month ({month}): {used} (unlimited)")
else:
print(f"\nUsage this month ({month}): {used}/{limit}")
return 0
def main():
"""Main entry point."""
parser = create_parser()
args = parser.parse_args()
if not args.command:
parser.print_help()
return 0
commands = {
"generate": cmd_generate,
"history": cmd_history,
"validate": cmd_validate,
"status": cmd_status
}
return commands.get(args.command, lambda a: 1)(args)
if __name__ == "__main__":
sys.exit(main())
FILE:scripts/feishu_integrator.py
"""
Feishu integration module for Weekly-Monthly Reporter.
Handles Feishu task reading and IM message sending.
"""
import json
from typing import Dict, List, Optional, Any
class FeishuTaskReader:
"""Read completed tasks from Feishu."""
def __init__(self, feishu_token: Optional[str] = None):
"""
Initialize Feishu task reader.
Args:
feishu_token: Feishu user access token (for task API)
"""
self.feishu_token = feishu_token
self._base_url = "https://open.feishu.cn/open-apis"
def get_completed_tasks(
self,
start_time: Optional[str] = None,
end_time: Optional[str] = None,
page_size: int = 50
) -> List[Dict]:
"""
Get completed tasks for the user.
This is a placeholder that would integrate with Feishu Task API.
In production, this requires OAuth and proper API setup.
Args:
start_time: ISO 8601 start time
end_time: ISO 8601 end time
page_size: Number of tasks to return
Returns:
List of task dictionaries
"""
# Note: This requires actual Feishu OAuth integration
# The actual implementation would call:
# GET https://open.feishu.cn/open-apis/task/v2/tasks
# For now, return empty list - integration requires:
# 1. User OAuth flow to get access token
# 2. Proper scope: task:task:readonly
# 3. Pagination handling
return []
def parse_tasks_for_report(self, tasks: List[Dict]) -> str:
"""
Parse task list into report-friendly format.
Args:
tasks: List of task dictionaries
Returns:
Formatted task list string
"""
if not tasks:
return "(No completed tasks this period)"
lines = []
for i, task in enumerate(tasks, 1):
title = task.get("summary", task.get("title", "未命名任务"))
completed_at = task.get("completed_at", "")
if completed_at:
from datetime import datetime
try:
dt = datetime.fromisoformat(completed_at.replace("Z", "+00:00"))
completed_at = dt.strftime("%Y-%m-%d")
except Exception:
pass
lines.append(f"{i}. {title} (completed: {completed_at})")
return "\n".join(lines)
class FeishuIM:
"""Send messages via Feishu IM."""
def __init__(self, bot_token: Optional[str] = None):
"""
Initialize Feishu IM client.
Args:
bot_token: Feishu bot token
"""
self.bot_token = bot_token
self._base_url = "https://open.feishu.cn/open-apis"
def send_card_message(
self,
receive_id: str,
receive_id_type: str,
content: str,
msg_type: str = "interactive"
) -> Dict:
"""
Send an interactive card message.
Args:
receive_id: Recipient ID (open_id or chat_id)
receive_id_type: "open_id" or "chat_id"
content: Report content to send
msg_type: Message type
Returns:
API response dict
"""
# Build interactive card content
card = self._build_report_card(content)
payload = {
"receive_id": receive_id,
"receive_id_type": receive_id_type,
"msg_type": "interactive",
"content": json.dumps(card)
}
headers = {
"Authorization": f"Bearer {self.bot_token}",
"Content-Type": "application/json"
}
import requests
response = requests.post(
f"{self._base_url}/im/v1/messages",
headers=headers,
json=payload,
timeout=30
)
return response.json()
def _build_report_card(self, report_content: str) -> Dict:
"""
Build Feishu interactive card from report content.
Args:
report_content: The generated report markdown
Returns:
Card JSON structure
"""
# Simple card with report content
return {
"config": {
"wide_screen_mode": True
},
"header": {
"title": {
"tag": "plain_text",
"content": "Weekly/Monthly Report"
},
"template": "blue"
},
"elements": [
{
"tag": "markdown",
"content": report_content
},
{
"tag": "hr"
},
{
"tag": "note",
"elements": [
{
"tag": "plain_text",
"content": "Generated by Weekly-Monthly Reporter"
}
]
}
]
}
def format_feishu_task_list(tasks_response: Any) -> str:
"""
Format Feishu task API response into work content.
Args:
tasks_response: Response from Feishu task API
Returns:
Formatted task list string
"""
if not tasks_response:
return ""
try:
tasks = tasks_response.get("items", tasks_response.get("data", {}).get("items", []))
except Exception:
return ""
lines = []
for task in tasks:
summary = task.get("summary", "Unnamed Task")
status = task.get("status", {})
completed = status.get("completed", False)
done_str = "Done" if completed else "In Progress"
lines.append(f"- {summary} [{done_str}]")
return "\n".join(lines) if lines else ""
Upload contract PDFs, extract key contract fields offline, manage a local ledger with expiry reminders and optional Feishu notifications.
# Contract Ledger
> Upload contract PDFs → AI extracts key fields → Manage ledger → Expiry reminders + Feishu push
---
## Trigger Phrases
`contract ledger` `contract management` `contract tracker` `pdf contract` `contract reminder` `合同台账`
---
## Usage
### Command Line
```bash
# Upload a contract PDF
python -m scripts.main upload /path/to/contract.pdf
# List all contracts
python -m scripts.main list
# List contracts expiring within 30 days
python -m scripts.main list --status "Active" --sort end_date
# Get contract details
python -m scripts.main get <contract_id>
# Update a contract
python -m scripts.main update <contract_id> --name "New Name" --status "Terminated"
# Delete a contract
python -m scripts.main delete <contract_id>
# Add expiry reminder
python -m scripts.main reminder <contract_id> add --days 30
# Check expiring contracts
python -m scripts.main check --days 30
# Export contracts
python -m scripts.main export --format csv -o contracts.csv
```
### Python API
```python
from scripts import extract_text_from_pdf, extract_contract_fields
from scripts import add_contract, get_contracts, get_contract
from scripts import update_contract, delete_contract
# Extract fields from PDF
text = extract_text_from_pdf("/path/to/contract.pdf")
fields = extract_contract_fields(text, "contract.pdf")
contract = add_contract(fields)
# List contracts
all_contracts = get_contracts(status="Active")
```
---
## Contract Fields Extracted
- **Contract Name** — from PDF title
- **Amount** — RMB amount via regex
- **Sign Date** — contract signing date
- **Start Date** — effective start date
- **End Date** — expiry date
- **Counterparty** — other party name (乙方/供应商/委托方)
- **Key Nodes** — payment terms, renewal clauses (up to 5)
- **Status** — Active / Expired (auto-calculated)
---
## Supported Formats
| Format | Extension | Notes |
|--------|-----------|-------|
| PDF | `.pdf` | Text extraction via PyMuPDF |
---
## Tech Stack
- **Parsing**: PyMuPDF (fitz)
- **AI Field Extraction**: Regex + heuristic pattern matching (no external AI API needed)
- **Storage**: JSON file in `/tmp/contract-ledger/` (fully offline)
- **Notifications**: Feishu IM card format
---
## Tiered Features
| Feature | FREE | PRO |
|---------|------|-----|
| Max Contracts | 5 | Unlimited |
| Max Reminders | 1 | Unlimited |
| Export Formats | CSV | CSV, XLSX, PDF |
| Feishu Reminders | No | Yes |
| Priority Support | No | Yes |
---
## Billing
**$0.01 USDT per call** — billed via SkillPay at [https://skillpay.me/contract-ledger](https://skillpay.me/contract-ledger)
> **Privacy Note:** Your Feishu User ID (Open ID) may be transmitted to skillpay.me for billing purposes only.
| Price | $0 (FREE tier) | $0.01 / call (PRO tier) |
> For paid use, visit [https://skillpay.me/contract-ledger](https://skillpay.me/contract-ledger)
---
## Required Environment Variables
| Variable | Description |
|----------|-------------|
| `SKILL_BILLING_API_KEY` | SkillPay Builder API Key (from skillpay.me) |
| `SKILL_BILLING_SKILL_ID` | Skill ID on SkillPay (default: contract-ledger) |
---
## Security Notes
- All contract data is stored locally in `/tmp/contract-ledger/` — **no home directory writes**
- PDF parsing is fully offline — no external network calls during extraction
- Feishu card push requires a Feishu bot token (configure separately)
- Token validation is handled by SkillPay billing system, not by the skill itself
---
## API Key Format
Any non-empty string works as an API key. Tier is determined automatically:
- **No API key** → FREE tier
- **Any API key** → PRO tier
---
## Slug
`contract-ledger`
FILE:requirements.txt
PyMuPDF>=1.23.0
requests>=2.28.0
FILE:scripts/pdf_parser.py
"""
PDF Parser for Contract Ledger.
Uses PyMuPDF (fitz) to extract text from PDF contracts.
"""
import re
import fitz
from datetime import datetime
from typing import Optional
def extract_text_from_pdf(pdf_path: str) -> str:
"""Extract all text from a PDF file."""
doc = fitz.open(pdf_path)
text_parts = []
for page in doc:
text_parts.append(page.get_text())
doc.close()
return "\n".join(text_parts)
def extract_contract_fields(text: str, filename: str = "") -> dict:
"""
Extract key fields from contract text using pattern matching.
Returns: contract_name, amount, dates, counterparty, key_nodes, status.
"""
# Extract contract name
lines = [l.strip() for l in text.split("\n") if l.strip()]
contract_name = ""
if lines:
for line in lines[:5]:
if len(line) > 5 and not line.startswith("\u7b2c") and "\u6761" not in line:
contract_name = line
break
if not contract_name and filename:
contract_name = filename.replace(".pdf", "").replace("_", " ")
# Extract amount
amount = extract_amount(text)
# Extract dates
sign_date = extract_date(text, ["\u7b7e\u8ba2\u65e5\u671f", "\u7b7e\u7f72\u65e5\u671f", "\u7b7e\u7ea6\u65e5\u671f", "\u7b7e\u8ba2\u4e8e"])
start_date = extract_date(text, ["\u5f00\u59cb\u65e5\u671f", "\u751f\u6548\u65e5\u671f", "\u8d77\u59cb\u65e5\u671f", "\u5f00\u59cb\u4e8e"])
end_date = extract_date(text, ["\u7ed3\u675f\u65e5\u671f", "\u5230\u671f\u65e5\u671f", "\u7ec8\u6b62\u65e5\u671f", "\u5c48\u6ee1\u65e5\u671f", "\u5230\u671f\u4e8e"])
# Extract counterparty
counterparty = extract_counterparty(text)
# Extract key nodes
key_nodes = extract_key_nodes(text)
return {
"contract_name": contract_name,
"amount": amount,
"sign_date": sign_date,
"start_date": start_date,
"end_date": end_date,
"counterparty": counterparty,
"key_nodes": key_nodes,
"status": determine_status(end_date),
}
def extract_amount(text: str) -> Optional[float]:
"""Extract contract amount from text."""
patterns = [
r"\u5408\u540c\u91d1\u989d[：:]\s*([\d,，.]+)",
r"\u603b\u4ef7\u6b3e?[：:]\s*([\d,，.]+)",
r"\u603b\u4ef7[：:]\s*([\d,，.]+)",
r"([\d,，.]+)\s*\u5143",
r"¥\s*([\d,，.]+)",
r"RMB\s*([\d,，.]+)",
]
for pattern in patterns:
match = re.search(pattern, text)
if match:
amount_str = match.group(1).replace(",", "").replace("\uff0c", ".")
try:
return float(amount_str)
except ValueError:
continue
return None
def extract_date(text: str, keywords: list) -> Optional[str]:
"""Extract date from text using keywords."""
date_pattern = r"(\d{4}[-/\u5e74]\d{1,2}[-/\u6708]\d{1,2}[\u65e5]?)"
for kw in keywords:
idx = text.find(kw)
if idx != -1:
snippet = text[idx:idx+50]
match = re.search(date_pattern, snippet)
if match:
return normalize_date(match.group(1))
match = re.search(date_pattern, text)
if match:
return normalize_date(match.group(1))
return None
def normalize_date(date_str: str) -> str:
"""Normalize date to YYYY-MM-DD format."""
date_str = date_str.replace("\u5e74", "-").replace("\u6708", "-").replace("\u65e5", "")
parts = re.split(r"[-/]", date_str)
if len(parts) == 3:
return f"{int(parts[0]):04d}-{int(parts[1]):02d}-{int(parts[2]):02d}"
return date_str
def extract_counterparty(text: str) -> Optional[str]:
"""Extract counterparty company name."""
patterns = [
r"\u4e59\u65b9[：:]\s*([^\s\uff0c\uff0c\uff0c]+)",
r"\u5bf9\u65b9[：:]\s*([^\s\uff0c\uff0c\uff0c]+)",
r"\u4f9b\u5e94\u5546[：:]\s*([^\s\uff0c\uff0c\uff0c]+)",
r"\u670d\u52a1\u5546[：:]\s*([^\s\uff0c\uff0c\uff0c]+)",
r"\u59d4\u6258\u65b9[：:]\s*([^\s\uff0c\uff0c\uff0c]+)",
]
for pattern in patterns:
match = re.search(pattern, text)
if match:
return match.group(1).strip()
return None
def extract_key_nodes(text: str) -> list:
"""Extract key contract nodes (payment terms, renewal, etc.)."""
nodes = []
payment_patterns = [
r"\u4ed8\u6b3e\u65b9\u5f0f[：:][^\n\u3002]+",
r"\u652f\u4ed8\u65b9\u5f0f[：:][^\n\u3002]+",
r"\u4ed8\u6b3e\u6761\u4ef6[：:][^\n\u3002]+",
]
for p in payment_patterns:
m = re.search(p, text)
if m:
nodes.append(m.group(0).strip())
renewal_patterns = [
r"\u7eed\u7ea6[^\n\u3002]+",
r"\u81ea\u52a8\u7eed\u671f[^\n\u3002]+",
r"\u671f\u6ee1\u540e[^\n\u3002]+",
]
for p in renewal_patterns:
m = re.search(p, text)
if m:
nodes.append(m.group(0).strip())
return nodes[:5]
def determine_status(end_date: Optional[str]) -> str:
"""Determine contract status based on end date."""
if not end_date:
return "\u6267\u884c\u4e2d" # Active
try:
end = datetime.strptime(end_date, "%Y-%m-%d")
now = datetime.now()
if end < now:
return "\u5df2\u5230\u671f" # Expired
return "\u6267\u884c\u4e2d" # Active
except ValueError:
return "\u6267\u884c\u4e2d"
FILE:scripts/config.py
"""
Configuration module for Contract Ledger (contract-ledger).
No external API validation - billing is handled separately via SkillPay.
Tier is determined by presence of a valid API key: FREE (no key) | PRO (any key).
"""
from dataclasses import dataclass
from typing import Optional
# Tier definitions (2-tier: FREE | PRO)
TIERS = {
"FREE": {
"max_contracts": 5,
"max_reminders": 1,
"export_formats": ["csv"],
},
"PRO": {
"max_contracts": -1, # unlimited
"max_reminders": -1, # unlimited
"export_formats": ["csv", "xlsx", "pdf"],
},
}
FALLBACK_TIER = "FREE"
@dataclass
class TokenInfo:
"""Token validation result."""
valid: bool
tier: str
max_contracts: int
max_reminders: int
export_formats: list
error: Optional[str] = None
class Config:
"""Configuration manager - no external API calls."""
def __init__(self):
self._cache: dict = {}
def validate_token(self, api_key: str) -> TokenInfo:
"""
Validate token. For ClawHub model: any non-empty API key = PRO tier.
No external API call needed - billing is handled by SkillPay separately.
"""
# Determine tier from API key presence
if api_key and api_key.strip():
tier = "PRO"
tier_info = TIERS["PRO"]
return TokenInfo(
valid=True,
tier=tier,
max_contracts=tier_info["max_contracts"],
max_reminders=tier_info["max_reminders"],
export_formats=tier_info["export_formats"],
)
else:
tier = "FREE"
tier_info = TIERS["FREE"]
return TokenInfo(
valid=True, # FREE tier is always valid
tier=tier,
max_contracts=tier_info["max_contracts"],
max_reminders=tier_info["max_reminders"],
export_formats=tier_info["export_formats"],
)
def clear_cache(self, api_key: Optional[str] = None):
"""Clear the validation cache."""
if api_key:
self._cache.pop(api_key, None)
else:
self._cache.clear()
def get_tier_limits(tier: str) -> dict:
"""Get tier limits as a dict (for backward compatibility)."""
tier_info = TIERS.get(tier, TIERS[FALLBACK_TIER])
return {
"max_contracts": tier_info["max_contracts"],
"max_reminders": tier_info["max_reminders"],
"export_formats": tier_info["export_formats"],
}
FILE:scripts/billing.py
"""
Billing module for Contract Ledger (contract-ledger).
Integrates with SkillPay per-call billing.
"""
import os
import requests
import logging
logger = logging.getLogger(__name__)
BILLING_URL = "https://skillpay.me/api/v1/billing"
API_KEY = os.environ.get("SKILL_BILLING_API_KEY", "")
SKILL_ID = os.environ.get("SKILL_BILLING_SKILL_ID", "contract-ledger")
HEADERS = {"X-API-Key": API_KEY, "Content-Type": "application/json"}
CALL_PRICE = 0.0100 # USDT per call
def is_dev_mode() -> bool:
"""Check if running in development mode (no API key configured)."""
return API_KEY in ("", "dev", "test")
def get_balance() -> float:
"""Get current user balance."""
if is_dev_mode():
return 999.0
try:
r = requests.get(f"{BILLING_URL}/balance", headers=HEADERS, timeout=10)
data = r.json()
return data.get("balance", 0.0) if data.get("success") else 0.0
except Exception:
return 0.0
def charge_user(user_id: str) -> dict:
"""
Charge a user for one API call.
Returns dict with ok=True/False and balance/payment_url on failure.
"""
if is_dev_mode():
return {"ok": True, "balance": 999.0}
try:
resp = requests.post(
f"{BILLING_URL}/charge",
headers=HEADERS,
json={"user_id": user_id, "skill_id": SKILL_ID, "amount": CALL_PRICE},
timeout=10
)
data = resp.json()
if data.get("success"):
return {"ok": True, "balance": data.get("balance", 0.0)}
return {
"ok": False,
"balance": 0.0,
"payment_url": data.get("payment_url", f"https://skillpay.me/{SKILL_ID}"),
}
except Exception as e:
logger.warning(f"Billing error: {e}")
return {"ok": False, "balance": 0.0, "payment_url": f"https://skillpay.me/{SKILL_ID}"}
FILE:scripts/requirements.txt
PyMuPDF>=1.23.0
requests>=2.28.0
FILE:scripts/feishu_notifier.py
"""
Feishu notification module for Contract Ledger.
Builds Feishu card messages for contract expiry reminders.
"""
from typing import Optional
def build_reminder_card(contract: dict, days_until_expiry: int) -> dict:
"""Build a Feishu reminder card for a contract."""
fields = [
{"is_short": True, "text": {"tag": "lark_md", "content": "**Contract**"}},
{"is_short": True, "text": {"tag": "lark_md", "content": f"{contract.get('contract_name', 'N/A')}"}},
{"is_short": True, "text": {"tag": "lark_md", "content": "**Counterparty**"}},
{"is_short": True, "text": {"tag": "lark_md", "content": f"{contract.get('counterparty', 'N/A')}"}},
{"is_short": True, "text": {"tag": "lark_md", "content": "**End Date**"}},
{"is_short": True, "text": {"tag": "lark_md", "content": f"{contract.get('end_date', 'N/A')}"}},
{"is_short": True, "text": {"tag": "lark_md", "content": "**Days Remaining**"}},
{"is_short": True, "text": {"tag": "lark_md", "content": f"{days_until_expiry} days"}},
]
amount = contract.get("amount")
if amount:
fields.extend([
{"is_short": True, "text": {"tag": "lark_md", "content": "**Amount**"}},
{"is_short": True, "text": {"tag": "lark_md", "content": f"¥{amount:,.2f}"}},
])
card = {
"config": {"wide_screen_mode": True},
"elements": [
{"tag": "markdown", "content": "**Contract Expiry Reminder**"},
{"tag": "hr"},
{"tag": "div", "fields": fields},
{"tag": "hr"},
{"tag": "markdown", "content": "Sent by Contract Ledger"}
],
"header": {
"title": {"tag": "plain_text", "content": "Contract Expiry Reminder"},
"template": "orange"
}
}
return card
def format_reminder_message(contract: dict, days_until_expiry: int) -> str:
"""Format reminder message as plain text."""
name = contract.get("contract_name", "N/A")
counterparty = contract.get("counterparty", "N/A")
end_date = contract.get("end_date", "N/A")
amount = contract.get("amount")
msg = f"Contract Expiry Reminder\n\n"
msg += f"Contract: {name}\n"
msg += f"Counterparty: {counterparty}\n"
msg += f"End Date: {end_date}\n"
msg += f"Days Remaining: {days_until_expiry} days\n"
if amount:
msg += f"Amount: ¥{amount:,.2f}\n"
return msg
FILE:scripts/__init__.py
"""
Contract Ledger - AI-powered contract management tool.
Upload PDF contracts, manage ledger, get expiry reminders.
"""
from .config import Config, TokenInfo, TIERS, FALLBACK_TIER, get_tier_limits
from .pdf_parser import extract_text_from_pdf, extract_contract_fields
from .storage import (
init_storage, add_contract, get_contracts, get_contract,
update_contract, delete_contract, add_reminder, remove_reminder,
get_expiring_contracts, count_contracts, export_contracts
)
from .feishu_notifier import build_reminder_card, format_reminder_message
__all__ = [
"Config", "TokenInfo", "TIERS", "FALLBACK_TIER", "get_tier_limits",
"extract_text_from_pdf", "extract_contract_fields",
"init_storage", "add_contract", "get_contracts", "get_contract",
"update_contract", "delete_contract", "add_reminder", "remove_reminder",
"get_expiring_contracts", "count_contracts", "export_contracts",
"build_reminder_card", "format_reminder_message",
]
FILE:scripts/storage.py
"""
Storage module for Contract Ledger.
JSON file local storage using /tmp/contract-ledger/ (no home directory writes).
"""
import json
import uuid
from pathlib import Path
from datetime import datetime
from typing import Optional
STORAGE_DIR = Path("/tmp/contract-ledger")
LEDGER_FILE = STORAGE_DIR / "contracts.json"
def init_storage():
"""Initialize storage directory and file."""
STORAGE_DIR.mkdir(parents=True, exist_ok=True)
if not LEDGER_FILE.exists():
_write_ledger([])
def _read_ledger() -> list:
"""Read ledger from file."""
try:
with open(LEDGER_FILE, "r", encoding="utf-8") as f:
return json.load(f)
except Exception:
return []
def _write_ledger(contracts: list):
"""Write ledger to file."""
with open(LEDGER_FILE, "w", encoding="utf-8") as f:
json.dump(contracts, f, ensure_ascii=False, indent=2)
def add_contract(fields: dict) -> dict:
"""Add a contract."""
contracts = _read_ledger()
contract = {
"id": str(uuid.uuid4())[:8],
"created_at": datetime.now().isoformat(),
"updated_at": datetime.now().isoformat(),
**fields,
"reminders": [],
}
contracts.append(contract)
_write_ledger(contracts)
return contract
def get_contracts(
status: Optional[str] = None,
sort_by: str = "end_date",
reverse: bool = True
) -> list:
"""Get contract list."""
contracts = _read_ledger()
if status:
contracts = [c for c in contracts if c.get("status") == status]
contracts.sort(
key=lambda x: x.get(sort_by, "" or "9999-12-31"),
reverse=reverse
)
return contracts
def get_contract(contract_id: str) -> Optional[dict]:
"""Get a single contract by ID."""
contracts = _read_ledger()
for c in contracts:
if c.get("id") == contract_id:
return c
return None
def update_contract(contract_id: str, updates: dict) -> Optional[dict]:
"""Update a contract."""
contracts = _read_ledger()
for i, c in enumerate(contracts):
if c.get("id") == contract_id:
contracts[i].update(updates)
contracts[i]["updated_at"] = datetime.now().isoformat()
_write_ledger(contracts)
return contracts[i]
return None
def delete_contract(contract_id: str) -> bool:
"""Delete a contract."""
contracts = _read_ledger()
original_len = len(contracts)
contracts = [c for c in contracts if c.get("id") != contract_id]
if len(contracts) < original_len:
_write_ledger(contracts)
return True
return False
def add_reminder(contract_id: str, days_before: int, enabled: bool = True) -> bool:
"""Add a reminder to a contract."""
contract = get_contract(contract_id)
if not contract:
return False
reminders = contract.get("reminders", [])
reminders.append({"days_before": days_before, "enabled": enabled})
update_contract(contract_id, {"reminders": reminders})
return True
def remove_reminder(contract_id: str, index: int) -> bool:
"""Remove a reminder from a contract."""
contract = get_contract(contract_id)
if not contract:
return False
reminders = contract.get("reminders", [])
if 0 <= index < len(reminders):
reminders.pop(index)
update_contract(contract_id, {"reminders": reminders})
return True
return False
def get_expiring_contracts(days: int = 7) -> list:
"""Get contracts expiring within N days."""
contracts = _read_ledger()
expiring = []
now = datetime.now()
for c in contracts:
if c.get("status") == "已到期":
continue
end_date_str = c.get("end_date")
if not end_date_str:
continue
try:
end_date = datetime.strptime(end_date_str, "%Y-%m-%d")
delta = (end_date - now).days
if 0 <= delta <= days:
c["days_until_expiry"] = delta
expiring.append(c)
except ValueError:
continue
return expiring
def count_contracts() -> int:
"""Count total contracts."""
return len(_read_ledger())
def export_contracts(contracts: list, format: str = "csv") -> str:
"""Export contract data."""
if not contracts:
return ""
if format == "csv":
return _export_csv(contracts)
elif format == "json":
return json.dumps(contracts, ensure_ascii=False, indent=2)
else:
return _export_csv(contracts)
def _export_csv(contracts: list) -> str:
"""Export to CSV format."""
if not contracts:
return ""
headers = ["id", "contract_name", "amount", "counterparty", "sign_date",
"start_date", "end_date", "status", "key_nodes"]
lines = [",".join(headers)]
for c in contracts:
row = [
c.get("id", ""),
c.get("contract_name", ""),
str(c.get("amount", "")),
c.get("counterparty", ""),
c.get("sign_date", ""),
c.get("start_date", ""),
c.get("end_date", ""),
c.get("status", ""),
"|".join(c.get("key_nodes", []))
]
lines.append(",".join(f'"{v}"' for v in row))
return "\n".join(lines)
FILE:scripts/main.py
#!/usr/bin/env python3
"""
Contract Ledger CLI - Main entry point.
Upload PDF contracts, manage ledger, get expiry reminders + Feishu notifications.
"""
import argparse
import sys
import json
from pathlib import Path
sys.path.insert(0, str(Path(__file__).parent))
from config import Config, get_tier_limits
from pdf_parser import extract_text_from_pdf, extract_contract_fields
from storage import (
init_storage, add_contract, get_contracts, get_contract,
update_contract, delete_contract, add_reminder, remove_reminder,
get_expiring_contracts, count_contracts, export_contracts
)
from feishu_notifier import build_reminder_card, format_reminder_message
from billing import is_dev_mode, charge_user
DEFAULT_API_KEY = ""
def cmd_upload(args):
"""Upload and parse a contract PDF."""
api_key = args.api_key or DEFAULT_API_KEY
if is_dev_mode():
print("Dev mode: Set SKILL_BILLING_API_KEY for full functionality.", file=sys.stderr)
billing_result = charge_user("cli_upload")
if not billing_result.get("ok"):
print(f"Error: Insufficient balance. Please recharge at https://skillpay.me/contract-ledger", file=sys.stderr)
return 1
config = Config()
token_info = config.validate_token(api_key)
tier = token_info.tier
limits = get_tier_limits(tier)
# Check contract limit
current_count = count_contracts()
max_contracts = limits["max_contracts"]
if max_contracts != -1 and current_count >= max_contracts:
print(f"Tier limit reached ({tier}: {max_contracts} contracts)", file=sys.stderr)
print(f"Current: {current_count}", file=sys.stderr)
return 1
# Extract text and fields
try:
text = extract_text_from_pdf(args.pdf_file)
fields = extract_contract_fields(text, Path(args.pdf_file).name)
except Exception as e:
print(f"PDF parsing failed: {e}", file=sys.stderr)
return 1
# Add contract
contract = add_contract(fields)
print(f"Contract added (ID: {contract['id']})")
print(f" Name: {fields.get('contract_name', 'N/A')}")
print(f" Counterparty: {fields.get('counterparty', 'N/A')}")
print(f" End Date: {fields.get('end_date', 'N/A')}")
print(f" Status: {fields.get('status', 'N/A')}")
if fields.get("amount"):
print(f" Amount: ¥{fields['amount']:,.2f}")
return 0
def cmd_list(args):
"""List contracts."""
contracts = get_contracts(status=args.status, sort_by=args.sort, reverse=not args.asc)
if not contracts:
print("No contracts found.")
return 0
print(f"\nContract Ledger ({len(contracts)} contracts)")
print("-" * 80)
for c in contracts:
amount_str = f"¥{c['amount']:,.2f}" if c.get("amount") else "-"
print(f"[{c['id']}] {c.get('contract_name', 'N/A')}")
print(f" Counterparty: {c.get('counterparty', '-')} | End: {c.get('end_date', '-')} | Amount: {amount_str}")
print(f" Status: {c.get('status', '-')}")
print()
return 0
def cmd_get(args):
"""Get a single contract."""
contract = get_contract(args.contract_id)
if not contract:
print(f"Contract not found: {args.contract_id}", file=sys.stderr)
return 1
print(f"\nContract Details ({contract['id']})")
print("-" * 40)
for k, v in contract.items():
if k == "key_nodes" and isinstance(v, list):
print(f" {k}:")
for node in v:
print(f" - {node}")
elif k == "reminders":
print(f" {k}: {json.dumps(v, ensure_ascii=False)}")
elif v is not None:
print(f" {k}: {v}")
return 0
def cmd_update(args):
"""Update a contract."""
updates = {}
if args.name:
updates["contract_name"] = args.name
if args.counterparty:
updates["counterparty"] = args.counterparty
if args.amount:
updates["amount"] = float(args.amount)
if args.end_date:
updates["end_date"] = args.end_date
if args.status:
updates["status"] = args.status
if not updates:
print("No updates provided", file=sys.stderr)
return 1
result = update_contract(args.contract_id, updates)
if result:
print(f"Contract updated: {args.contract_id}")
return 0
else:
print(f"Update failed: {args.contract_id}", file=sys.stderr)
return 1
def cmd_delete(args):
"""Delete a contract."""
if delete_contract(args.contract_id):
print(f"Contract deleted: {args.contract_id}")
return 0
else:
print(f"Delete failed: {args.contract_id}", file=sys.stderr)
return 1
def cmd_reminder(args):
"""Manage reminders."""
if args.action == "add":
if add_reminder(args.contract_id, args.days):
print(f"Reminder added ({args.days} days before expiry)")
else:
print(f"Failed to add reminder", file=sys.stderr)
return 1
elif args.action == "remove":
if remove_reminder(args.contract_id, args.index):
print("Reminder removed")
else:
print("Failed to remove reminder", file=sys.stderr)
return 1
elif args.action == "list":
contract = get_contract(args.contract_id)
if not contract:
print("Contract not found", file=sys.stderr)
return 1
reminders = contract.get("reminders", [])
if not reminders:
print("No reminders set")
else:
print(f"Reminders ({len(reminders)}):")
for i, r in enumerate(reminders):
status = "ON" if r.get("enabled") else "OFF"
print(f" [{i}] [{status}] {r['days_before']} days before expiry")
return 0
def cmd_check(args):
"""Check expiring contracts."""
api_key = args.api_key or DEFAULT_API_KEY
days = args.days or 7
billing_result = charge_user("cli_check")
if not billing_result.get("ok"):
print(f"Error: Insufficient balance.", file=sys.stderr)
return 1
expiring = get_expiring_contracts(days)
if not expiring:
print(f"No contracts expiring within {days} days")
return 0
print(f"{len(expiring)} contract(s) expiring within {days} days:\n")
for c in expiring:
days_left = c.get("days_until_expiry", 0)
print(f" [{c['id']}] {c.get('contract_name', 'N/A')}")
print(f" End: {c.get('end_date')} ({days_left} days remaining)")
print()
if args.feishu and expiring:
card = build_reminder_card(expiring[0], expiring[0].get("days_until_expiry", 0))
print("\nFeishu card content:")
print(json.dumps(card, ensure_ascii=False, indent=2))
return 0
def cmd_export(args):
"""Export contracts."""
api_key = args.api_key or DEFAULT_API_KEY
config = Config()
token_info = config.validate_token(api_key)
tier = token_info.tier
limits = get_tier_limits(tier)
format_type = args.format or "csv"
if format_type not in limits["export_formats"]:
print(f"Tier {tier} does not support {format_type} export", file=sys.stderr)
print(f"Supported: {', '.join(limits['export_formats'])}", file=sys.stderr)
return 1
contracts = get_contracts(status=args.status)
if not contracts:
print("No contracts to export")
return 0
content = export_contracts(contracts, format_type)
if args.output:
with open(args.output, "w", encoding="utf-8") as f:
f.write(content)
print(f"Exported to: {args.output}")
else:
print(content)
return 0
def main():
parser = argparse.ArgumentParser(description="Contract Ledger Management Tool")
subparsers = parser.add_subparsers(dest="command", help="Subcommands")
p_upload = subparsers.add_parser("upload", help="Upload contract PDF")
p_upload.add_argument("pdf_file", help="PDF file path")
p_upload.add_argument("--api-key", help="API Key (optional)")
p_upload.set_defaults(func=cmd_upload)
p_list = subparsers.add_parser("list", help="List contracts")
p_list.add_argument("--status", help="Filter by status")
p_list.add_argument("--sort", default="end_date", help="Sort field")
p_list.add_argument("--asc", action="store_true", help="Sort ascending")
p_list.set_defaults(func=cmd_list)
p_get = subparsers.add_parser("get", help="Get contract details")
p_get.add_argument("contract_id", help="Contract ID")
p_get.set_defaults(func=cmd_get)
p_update = subparsers.add_parser("update", help="Update contract")
p_update.add_argument("contract_id", help="Contract ID")
p_update.add_argument("--name", help="Contract name")
p_update.add_argument("--counterparty", help="Counterparty")
p_update.add_argument("--amount", help="Amount")
p_update.add_argument("--end-date", dest="end_date", help="End date (YYYY-MM-DD)")
p_update.add_argument("--status", help="Status")
p_update.set_defaults(func=cmd_update)
p_delete = subparsers.add_parser("delete", help="Delete contract")
p_delete.add_argument("contract_id", help="Contract ID")
p_delete.set_defaults(func=cmd_delete)
p_reminder = subparsers.add_parser("reminder", help="Manage reminders")
p_reminder.add_argument("contract_id", help="Contract ID")
p_reminder.add_argument("action", choices=["add", "remove", "list"], help="Action")
p_reminder.add_argument("--days", type=int, help="Days before expiry (for add)")
p_reminder.add_argument("--index", type=int, help="Reminder index (for remove)")
p_reminder.set_defaults(func=cmd_reminder)
p_check = subparsers.add_parser("check", help="Check expiring contracts")
p_check.add_argument("--days", type=int, default=7, help="Days to check")
p_check.add_argument("--api-key", help="API Key")
p_check.add_argument("--feishu", action="store_true", help="Output Feishu card")
p_check.set_defaults(func=cmd_check)
p_export = subparsers.add_parser("export", help="Export contracts")
p_export.add_argument("--format", choices=["csv", "xlsx", "pdf"], help="Export format")
p_export.add_argument("--status", help="Filter by status")
p_export.add_argument("--output", "-o", help="Output file path")
p_export.add_argument("--api-key", help="API Key")
p_export.set_defaults(func=cmd_export)
args = parser.parse_args()
init_storage()
if args.command is None:
parser.print_help()
return 0
return args.func(args)
if __name__ == "__main__":
sys.exit(main())
Convert natural language questions into SQL queries on your uploaded CSV/Excel files, execute them offline, and return results with optional charts.
# NL2SQL · Natural Language to SQL
> Upload CSV/Excel files → Ask questions in plain English → AI generates and executes SQL → Returns readable results + optional charts
---
## Trigger Phrases
`nl2sql` `text to sql` `natural language sql` `ask database` `csv query` `excel sql` `数据查询` `自然语言查数`
---
## Usage
### Command Line
```bash
# Basic query
python -m scripts.main "Which product has the highest sales?" -f data/sales.csv
# Generate chart
python -m scripts.main "Monthly sales trend" -f data/sales.csv --chart line
# Export results
python -m scripts.main "Top 10 customers" -f data/customers.csv --format csv -o result.csv
```
### Python API
```python
from scripts import NL2SQLService, QueryRequest
service = NL2SQLService(api_key="your-api-key")
request = QueryRequest(
question="Which product has the highest sales?",
files=["data/sales.csv"],
chart_type="bar",
explain=True
)
response = service.query(request)
if response.success:
print(f"SQL: {response.sql}")
print(f"Results: {response.data}")
```
---
## Parameters
### QueryRequest
| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| question | str | Yes | Natural language question |
| files | List[str] | Yes | File paths (CSV/Excel) |
| chart_type | str | No | Chart type: bar/line/pie/scatter/area/histogram |
| explain | bool | No | Whether to explain the SQL |
| output_format | str | No | Output format: markdown/json/csv (default: markdown) |
### QueryResponse
| Field | Type | Description |
|-------|------|-------------|
| success | bool | Whether the query succeeded |
| sql | str | Generated SQL |
| explanation | str | SQL explanation |
| row_count | int | Number of result rows |
| columns | List[str] | Column names |
| data | List[dict] | Result data |
| chart_base64 | str | Chart image as base64 |
| error | str | Error message if failed |
---
## Supported Formats
| Format | Extensions | Notes |
|--------|-----------|-------|
| CSV | `.csv` | UTF-8/GBK auto-detected |
| Excel | `.xlsx`, `.xls` | Multi-sheet supported |
---
## Tech Stack
- **Parsing**: pandas, openpyxl
- **AI**: OpenAI GPT-4 (via user-provided API key)
- **Charts**: matplotlib
- **Execution**: pandasql (SQL on DataFrame, fully offline sandbox)
---
## Tiered Features
| Feature | FREE | PRO |
|---------|------|-----|
| Queries | 3 per session | Unlimited |
| File size | 5 MB max | 200 MB max |
| JOIN support | No | Yes |
| Chart types | bar, line, pie | All types |
| Export formats | CSV | CSV, Excel, PDF |
| AI SQL generation | No (rule-based) | Yes (GPT-4) |
---
## Billing
**$0.01 USDT per call** — billed via SkillPay at [https://skillpay.me/ai-nl2sql](https://skillpay.me/ai-nl2sql)
> **Privacy Note:** Your Feishu User ID (Open ID) may be transmitted to skillpay.me for billing purposes only.
| Price | $0 (FREE tier) | $0.01 / call (PRO tier) |
> For paid use, visit [https://skillpay.me/ai-nl2sql](https://skillpay.me/ai-nl2sql)
---
## Required Environment Variables
| Variable | Description |
|----------|-------------|
| `SKILL_BILLING_API_KEY` | SkillPay Builder API Key (from skillpay.me) |
| `SKILL_BILLING_SKILL_ID` | Skill ID on SkillPay (default: ai-nl2sql) |
---
## API Key Format
Any non-empty string works as an API key. The tier is determined automatically:
- **No API key** → FREE tier (rule-based SQL only)
- **Any API key** → PRO tier (GPT-4 powered)
---
## Slug
`ai-nl2sql`
---
## Security Notes
- **SQL Safety**: All AI-generated SQL passes through an SQLValidator that blocks all non-SELECT queries (DROP, DELETE, INSERT, UPDATE, CREATE, EXEC, GRANT, etc.). Only read-only queries are permitted.
- **Data Isolation**: SQL execution runs entirely in a local pandas DataFrame sandbox. No real database connection is made. No data leaves the user's environment.
- **External Data Transmission**: Your Feishu User ID (Open ID) is transmitted to [skillpay.me](https://skillpay.me) exclusively for billing purposes. See [## Billing](#billing) for details.
---
## Notes
1. All SQL execution runs in a local pandas DataFrame sandbox — no real database connection
2. AI SQL generation requires a valid OpenAI API key provided by the user
3. Network errors gracefully degrade to FREE tier
FILE:requirements.txt
pandas>=1.5.0
openai>=1.0.0
requests>=2.28.0
openpyxl>=3.0.0
matplotlib>=3.5.0
pandasql>=0.7.3
FILE:scripts/config.py
"""
Configuration module for NL2SQL (ai-nl2sql).
No external API validation - billing is handled separately via SkillPay.
Tier is determined by presence of a valid API key: FREE (no key or invalid) | PRO (any key).
"""
import time
from dataclasses import dataclass
from typing import Optional, Dict
# Tier definitions (2-tier: FREE | PRO)
TIERS = {
"FREE": {
"queries": 3,
"max_file_size_mb": 5,
"join": False,
"chart_types": ["bar", "line", "pie"],
"export_formats": ["csv"],
},
"PRO": {
"queries": -1, # unlimited
"max_file_size_mb": 200,
"join": True,
"chart_types": ["bar", "line", "pie", "scatter", "area", "histogram"],
"export_formats": ["csv", "excel", "pdf"],
},
}
FALLBACK_TIER = "FREE"
@dataclass
class TokenInfo:
"""Token validation result."""
valid: bool
tier: str
queries_remaining: int
max_file_size_mb: int
join_enabled: bool
chart_types: list
export_formats: list
error: Optional[str] = None
class Config:
"""Configuration manager - no external API calls."""
def __init__(self):
self._cache: Dict[str, tuple] = {}
def validate_token(self, api_key: str) -> TokenInfo:
"""
Validate token. For ClawHub model: any non-empty API key = PRO tier.
No external API call needed - billing is handled by SkillPay separately.
"""
cache_key = api_key[:8] if len(api_key) > 8 else api_key
# Check cache
if cache_key in self._cache:
info, timestamp = self._cache[cache_key]
if time.time() - timestamp < 300:
return info
# Determine tier from API key presence
if api_key and api_key.strip():
tier = "PRO"
tier_info = TIERS["PRO"]
info = TokenInfo(
valid=True,
tier=tier,
queries_remaining=tier_info["queries"],
max_file_size_mb=tier_info["max_file_size_mb"],
join_enabled=tier_info["join"],
chart_types=tier_info["chart_types"],
export_formats=tier_info["export_formats"],
)
else:
tier = "FREE"
tier_info = TIERS["FREE"]
info = TokenInfo(
valid=True, # FREE tier is always valid
tier=tier,
queries_remaining=tier_info["queries"],
max_file_size_mb=tier_info["max_file_size_mb"],
join_enabled=tier_info["join"],
chart_types=tier_info["chart_types"],
export_formats=tier_info["export_formats"],
)
self._cache[cache_key] = (info, time.time())
return info
def clear_cache(self, api_key: Optional[str] = None):
"""Clear the validation cache."""
if api_key:
cache_key = api_key[:8] if len(api_key) > 8 else api_key
self._cache.pop(cache_key, None)
else:
self._cache.clear()
FILE:scripts/billing.py
"""
Billing module for NL2SQL (ai-nl2sql).
Integrates with SkillPay per-call billing.
"""
import os
import requests
import logging
logger = logging.getLogger(__name__)
BILLING_URL = "https://skillpay.me/api/v1/billing"
API_KEY = os.environ.get("SKILL_BILLING_API_KEY", "")
SKILL_ID = os.environ.get("SKILL_BILLING_SKILL_ID", "ai-nl2sql")
HEADERS = {"X-API-Key": API_KEY, "Content-Type": "application/json"}
CALL_PRICE = 0.0100 # USDT per call
def is_dev_mode() -> bool:
"""Check if running in development mode (no API key configured)."""
return API_KEY in ("", "dev", "test")
def get_balance() -> float:
"""Get current user balance."""
if is_dev_mode():
return 999.0
try:
r = requests.get(f"{BILLING_URL}/balance", headers=HEADERS, timeout=10)
data = r.json()
return data.get("balance", 0.0) if data.get("success") else 0.0
except Exception:
return 0.0
def charge_user(user_id: str) -> dict:
"""
Charge a user for one API call.
Returns dict with ok=True/False and balance/payment_url on failure.
"""
if is_dev_mode():
return {"ok": True, "balance": 999.0}
try:
resp = requests.post(
f"{BILLING_URL}/charge",
headers=HEADERS,
json={"user_id": user_id, "skill_id": SKILL_ID, "amount": CALL_PRICE},
timeout=10
)
data = resp.json()
if data.get("success"):
return {"ok": True, "balance": data.get("balance", 0.0)}
return {
"ok": False,
"balance": 0.0,
"payment_url": data.get("payment_url", f"https://skillpay.me/{SKILL_ID}"),
}
except Exception as e:
logger.warning(f"Billing error: {e}")
return {"ok": False, "balance": 0.0, "payment_url": f"https://skillpay.me/{SKILL_ID}"}
FILE:scripts/requirements.txt
pandas>=1.5.0
openai>=1.0.0
requests>=2.28.0
openpyxl>=3.0.0
matplotlib>=3.5.0
pandasql>=0.7.3
FILE:scripts/__init__.py
"""
NL2SQL - Natural Language to SQL Converter
Ask questions in plain English against CSV/Excel files.
"""
from .config import Config, TokenInfo, TIERS, FALLBACK_TIER
from .parser import CSVParser, DataFrameExecutor, Schema, ColumnInfo, SQLExecutionError
from .nl2sql import NL2SQL, GeneratedSQL, PromptBuilder
from .display import TableRenderer, ChartGenerator, ResultFormatter
from .api import NL2SQLService, QueryRequest, QueryResponse
__all__ = [
"Config", "TokenInfo", "TIERS", "FALLBACK_TIER",
"CSVParser", "DataFrameExecutor", "Schema", "ColumnInfo", "SQLExecutionError",
"NL2SQL", "GeneratedSQL", "PromptBuilder",
"TableRenderer", "ChartGenerator", "ResultFormatter",
"NL2SQLService", "QueryRequest", "QueryResponse",
]
FILE:scripts/display.py
"""
Result Display Module - Table rendering and chart generation.
"""
import io
import base64
from typing import Optional, List, Dict, Any
import pandas as pd
import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt
class TableRenderer:
"""Table renderer."""
@staticmethod
def to_markdown(df: pd.DataFrame, max_rows: int = 100) -> str:
"""Convert DataFrame to Markdown table."""
if df.empty:
return "(Empty result)"
display_df = df.head(max_rows)
headers = display_df.columns.tolist()
header_line = "| " + " | ".join(str(h) for h in headers) + " |"
separator = "| " + " | ".join(["---"] * len(headers)) + " |"
rows = []
for _, row in display_df.iterrows():
row_str = "| " + " | ".join(str(v) for v in row.values) + " |"
rows.append(row_str)
result = [header_line, separator] + rows
if len(df) > max_rows:
note = "(Showing first {} rows, {} total)".format(max_rows, len(df))
result.append("_" + note + "_")
return "\n".join(result)
@staticmethod
def to_csv(df: pd.DataFrame) -> str:
"""Convert to CSV string."""
return df.to_csv(index=False)
@staticmethod
def to_html(df: pd.DataFrame, max_rows: int = 100) -> str:
"""Convert to HTML table."""
display_df = df.head(max_rows)
html = ['<table style="border-collapse: collapse; width: 100%;">']
html.append('<thead><tr>')
for col in display_df.columns:
html.append('<th style="border: 1px solid #ddd; padding: 8px; background-color: #f2f2f2;">' + str(col) + '</th>')
html.append('</tr></thead>')
html.append('<tbody>')
for _, row in display_df.iterrows():
html.append('<tr>')
for val in row.values:
html.append('<td style="border: 1px solid #ddd; padding: 8px;">' + str(val) + '</td>')
html.append('</tr>')
html.append('</tbody></table>')
if len(df) > max_rows:
html.append('<p><em>(Showing first {} rows, {} total)</em></p>'.format(max_rows, len(df)))
return '\n'.join(html)
@staticmethod
def to_excel_bytes(df: pd.DataFrame, filename: str = "result.xlsx") -> bytes:
"""Export to Excel bytes."""
output = io.BytesIO()
with pd.ExcelWriter(output, engine='openpyxl') as writer:
df.to_excel(writer, index=False, sheet_name='Result')
return output.getvalue()
@staticmethod
def to_dict_list(df: pd.DataFrame) -> List[Dict]:
"""Convert to list of dicts."""
return df.to_dict(orient='records')
class ChartGenerator:
"""Chart generator."""
SUPPORTED_TYPES = ["bar", "line", "pie", "scatter", "area", "histogram"]
def __init__(self):
self.figure_size = (10, 6)
self.dpi = 100
self.title_fontsize = 14
def generate(
self,
df: pd.DataFrame,
chart_type: str,
x_column: Optional[str] = None,
y_column: Optional[str] = None,
title: Optional[str] = None,
color: Optional[str] = None
) -> bytes:
"""Generate chart image bytes."""
if chart_type not in self.SUPPORTED_TYPES:
raise ValueError("Unsupported chart type: " + chart_type)
if df.empty:
raise ValueError("Cannot generate chart for empty DataFrame")
columns = df.columns.tolist()
if x_column is None:
x_column = columns[0]
if y_column is None and len(columns) > 1:
y_column = columns[1]
fig, ax = plt.subplots(figsize=self.figure_size, dpi=self.dpi)
if chart_type == "bar":
self._plot_bar(ax, df, x_column, y_column, color)
elif chart_type == "line":
self._plot_line(ax, df, x_column, y_column, color)
elif chart_type == "pie":
self._plot_pie(ax, df, x_column, y_column, color)
elif chart_type == "scatter":
self._plot_scatter(ax, df, x_column, y_column, color)
elif chart_type == "area":
self._plot_area(ax, df, x_column, y_column, color)
elif chart_type == "histogram":
self._plot_histogram(ax, df, x_column, color)
if title:
ax.set_title(title, fontsize=self.title_fontsize)
plt.tight_layout()
output = io.BytesIO()
plt.savefig(output, format='png', dpi=self.dpi, bbox_inches='tight')
plt.close(fig)
return output.getvalue()
def generate_and_save(self, df: pd.DataFrame, chart_type: str, output_path: str, **kwargs) -> str:
"""Generate and save chart to file."""
image_bytes = self.generate(df, chart_type, **kwargs)
with open(output_path, 'wb') as f:
f.write(image_bytes)
return output_path
def to_base64(self, df: pd.DataFrame, chart_type: str, **kwargs) -> str:
"""Generate base64-encoded chart image."""
image_bytes = self.generate(df, chart_type, **kwargs)
return base64.b64encode(image_bytes).decode('utf-8')
def _plot_bar(self, ax, df, x_col, y_col, color):
"""Plot bar chart."""
if y_col:
ax.bar(df[x_col].astype(str), df[y_col], color=color or 'steelblue')
ax.set_xlabel(x_col)
ax.set_ylabel(y_col)
else:
value_counts = df[x_col].value_counts()
ax.bar(value_counts.index.astype(str), value_counts.values, color=color or 'steelblue')
ax.set_xlabel(x_col)
ax.set_ylabel('Count')
ax.tick_params(axis='x', rotation=45)
def _plot_line(self, ax, df, x_col, y_col, color):
"""Plot line chart."""
if y_col:
ax.plot(df[x_col].astype(str), df[y_col], marker='o', color=color or 'steelblue')
ax.set_xlabel(x_col)
ax.set_ylabel(y_col)
else:
ax.plot(range(len(df)), df[x_col], marker='o', color=color or 'steelblue')
ax.set_xlabel('Index')
ax.set_ylabel(x_col)
ax.tick_params(axis='x', rotation=45)
def _plot_pie(self, ax, df, x_col, y_col, color):
"""Plot pie chart."""
if y_col:
ax.pie(df[y_col], labels=df[x_col].astype(str), autopct='%1.1f%%', colors=[color] if color else None)
else:
value_counts = df[x_col].value_counts()
ax.pie(value_counts.values, labels=value_counts.index.astype(str), autopct='%1.1f%%')
ax.axis('equal')
def _plot_scatter(self, ax, df, x_col, y_col, color):
"""Plot scatter chart."""
ax.scatter(df[x_col], df[y_col], c=color or 'steelblue', alpha=0.6)
ax.set_xlabel(x_col)
ax.set_ylabel(y_col)
def _plot_area(self, ax, df, x_col, y_col, color):
"""Plot area chart."""
if y_col:
ax.fill_between(range(len(df)), df[y_col], alpha=0.3, color=color or 'steelblue')
ax.plot(range(len(df)), df[y_col], color=color or 'steelblue')
else:
ax.fill_between(range(len(df)), df[x_col], alpha=0.3, color=color or 'steelblue')
ax.plot(range(len(df)), df[x_col], color=color or 'steelblue')
ax.set_xlabel('Index')
def _plot_histogram(self, ax, df, x_col, color):
"""Plot histogram."""
ax.hist(df[x_col].dropna(), bins=20, color=color or 'steelblue', edgecolor='black', alpha=0.7)
ax.set_xlabel(x_col)
ax.set_ylabel('Frequency')
class ResultFormatter:
"""Result formatter."""
def __init__(self):
self.table_renderer = TableRenderer()
self.chart_generator = ChartGenerator()
def format_markdown(
self,
df: pd.DataFrame,
sql: Optional[str] = None,
explanation: Optional[str] = None,
chart_image_base64: Optional[str] = None
) -> str:
"""Format results as Markdown."""
parts = []
if sql:
parts.append("### Generated SQL")
parts.append("```sql")
parts.append(sql)
parts.append("```")
parts.append("")
if explanation:
parts.append("**Explanation**: " + explanation)
parts.append("")
if chart_image_base64:
parts.append("### Chart")
img_tag = '<img src="data:image/png;base64,' + chart_image_base64 + '" width="600"/>'
parts.append(img_tag)
parts.append("")
parts.append("### Query Results")
parts.append(self.table_renderer.to_markdown(df))
return "\n".join(parts)
def format_json(
self,
df: pd.DataFrame,
sql: Optional[str] = None,
explanation: Optional[str] = None
) -> Dict[str, Any]:
"""Format results as JSON."""
return {
"sql": sql,
"explanation": explanation,
"row_count": len(df),
"columns": df.columns.tolist(),
"data": self.table_renderer.to_dict_list(df)
}
FILE:scripts/parser.py
"""
CSV/Excel Parser and Schema Builder
Loads data files and builds a queryable schema.
"""
import re
import pandas as pd
from pathlib import Path
from typing import Optional, Dict, List, Any
from dataclasses import dataclass
@dataclass
class ColumnInfo:
"""Column metadata."""
name: str
dtype: str
nullable: bool
unique_count: Optional[int] = None
sample_values: Optional[List[Any]] = None
@dataclass
class Schema:
"""Database schema description."""
tables: Dict[str, pd.DataFrame]
columns: Dict[str, List[ColumnInfo]]
relationships: List[Dict[str, str]]
def get_table_schema_sql(self, table_name: str) -> str:
"""Generate DDL-style table description."""
if table_name not in self.columns:
return ""
lines = [f"Table: {table_name}", "("]
for col in self.columns[table_name]:
nullable = "NULL" if col.nullable else "NOT NULL"
samples = f", e.g. {col.sample_values[:3]}" if col.sample_values else ""
lines.append(f" {col.name} {col.dtype} {nullable}{samples}")
lines.append(")")
return "\n".join(lines)
def to_prompt(self) -> str:
"""Convert schema to AI prompt format."""
parts = []
for table_name in self.tables:
parts.append(self.get_table_schema_sql(table_name))
if self.relationships:
parts.append("\nRelationships:")
for rel in self.relationships:
parts.append(f" {rel.get('table1')}.{rel.get('col1')} -> {rel.get('table2')}.{rel.get('col2')}")
return "\n".join(parts)
class CSVParser:
"""CSV/Excel file parser."""
SUPPORTED_EXTENSIONS = {".csv", ".xlsx", ".xls"}
@classmethod
def is_supported(cls, filepath: str) -> bool:
"""Check if file format is supported."""
ext = Path(filepath).suffix.lower()
return ext in cls.SUPPORTED_EXTENSIONS
@classmethod
def load_file(cls, filepath: str, sheet_name: Optional[str] = None) -> pd.DataFrame:
"""Load a CSV or Excel file."""
ext = Path(filepath).suffix.lower()
if ext == ".csv":
df = pd.read_csv(filepath)
elif ext in {".xlsx", ".xls"}:
df = pd.read_excel(filepath, sheet_name=sheet_name or 0)
else:
raise ValueError(f"Unsupported file format: {ext}")
return df
@classmethod
def infer_schema(cls, df: pd.DataFrame, table_name: str = "data") -> Schema:
"""Automatically infer DataFrame schema."""
columns: List[ColumnInfo] = []
for col_name in df.columns:
dtype = str(df[col_name].dtype)
nullable = df[col_name].isna().any()
unique_count = df[col_name].nunique()
samples = df[col_name].dropna().head(5).tolist()
col_info = ColumnInfo(
name=str(col_name),
dtype=dtype,
nullable=nullable,
unique_count=unique_count,
sample_values=samples
)
columns.append(col_info)
return Schema(tables={table_name: df}, columns={table_name: columns}, relationships=[])
@classmethod
def build_schema_from_files(cls, filepaths: List[str]) -> Schema:
"""Build schema from multiple files."""
tables: Dict[str, pd.DataFrame] = {}
columns: Dict[str, List[ColumnInfo]] = {}
all_relationships: List[Dict[str, str]] = []
for filepath in filepaths:
table_name = Path(filepath).stem
# Add numeric suffix if table name already exists
if table_name in tables:
counter = 1
while f"{table_name}_{counter}" in tables:
counter += 1
table_name = f"{table_name}_{counter}"
df = cls.load_file(filepath)
tables[table_name] = df
columns[table_name] = []
# Infer column info
for col_name in df.columns:
dtype = str(df[col_name].dtype)
nullable = df[col_name].isna().any()
unique_count = df[col_name].nunique()
samples = df[col_name].dropna().head(5).tolist()
col_info = ColumnInfo(
name=str(col_name),
dtype=dtype,
nullable=nullable,
unique_count=unique_count,
sample_values=samples
)
columns[table_name].append(col_info)
# Auto-detect relationships (same-name columns as FK candidates)
for col_name in df.columns:
if col_name.lower() in ["id", "user_id", "product_id", "order_id", "customer_id"]:
all_relationships.append({
"table1": table_name,
"col1": col_name,
"table2": table_name,
"col2": col_name,
"type": "primary_key"
})
return Schema(tables=tables, columns=columns, relationships=all_relationships)
class SQLValidator:
"""Validates SQL to prevent injection attacks."""
FORBIDDEN_KEYWORDS = [
"DROP", "DELETE", "INSERT", "UPDATE", "ALTER", "CREATE", "TRUNCATE",
"EXEC", "EXECUTE", "GRANT", "REVOKE", "LOAD", "OUTFILE", "INFILE",
"SHUTDOWN", "REPAIR", "OPTIMIZE", "CACHE", "FLUSH", "KILL", "DO"
]
def __init__(self):
import re
kw = ["DROP","DELETE","INSERT","UPDATE","ALTER","CREATE","TRUNCATE","EXEC","EXECUTE","GRANT","REVOKE","LOAD","OUTFILE","INFILE","SHUTDOWN","REPAIR","OPTIMIZE","CACHE","FLUSH","KILL","DO"]
self._regex = re.compile(r"(?i)\b(" + "|".join(kw) + r")\b")
def validate(self, sql: str) -> tuple:
"""Validate SQL for safety. Returns (is_safe, error_msg)."""
if not sql or not sql.strip():
return False, "Empty SQL"
sql_upper = sql.upper()
if not sql_upper.strip().startswith("SELECT"):
return False, "Only SELECT queries are allowed"
if self._regex.search(sql):
return False, "Forbidden SQL keyword detected"
return True, ""
class DataFrameExecutor:
"""SQL executor on DataFrames (safe offline sandbox)."""
def __init__(self, schema: Schema):
self.schema = schema
self._validator = SQLValidator()
def execute_sql(self, sql: str, params: Optional[Dict] = None) -> pd.DataFrame:
"""Execute SQL on DataFrames using pandasql (fully offline, no DB connection)."""
is_safe, error_msg = self._validator.validate(sql)
if not is_safe:
raise SQLExecutionError(f"SQL validation failed: {error_msg}")
try:
import pandasql as ps
tables = self.schema.tables.copy()
if params:
tables.update(params)
result = ps.sqldf(sql, tables)
return result
except ImportError:
return self._execute_pandas(sql)
except Exception as e:
raise SQLExecutionError(f"SQL execution failed: {str(e)}")
def _execute_pandas(self, sql: str) -> pd.DataFrame:
"""Simple pandas-only SQL executor (fallback)."""
sql_upper = sql.upper().strip()
select_match = re.match(r"SELECT\s+(.+?)\s+FROM\s+(\w+)(?:\s+WHERE\s+(.+))?$", sql_upper, re.DOTALL)
if select_match:
cols = select_match.group(1).strip()
table_name = select_match.group(2).lower()
where_clause = select_match.group(3) if select_match.group(3) else None
if table_name not in self.schema.tables:
raise SQLExecutionError(f"Table '{table_name}' not found")
df = self.schema.tables[table_name].copy()
if cols == "*":
result = df
else:
col_list = [c.strip().split()[-1] for c in cols.split(",")]
result = df[col_list]
if where_clause:
result = self._apply_where(result, where_clause)
return result
raise SQLExecutionError("Complex SQL not supported in pandas mode. Please install pandasql.")
def _apply_where(self, df: pd.DataFrame, where_clause: str) -> pd.DataFrame:
"""Apply simple WHERE clause."""
where_clause = re.sub(r'\s+(ORDER\s+BY|LIMIT|OFFSET|GROUP\s+BY|HAVING).*$', '', where_clause, flags=re.IGNORECASE)
if " AND " in where_clause.upper():
parts = re.split(r'\s+AND\s+', where_clause, flags=re.IGNORECASE)
result = df
for part in parts:
result = self._apply_condition(result, part.strip())
return result
elif " OR " in where_clause.upper():
return df
else:
return self._apply_condition(df, where_clause.strip())
def _apply_condition(self, df: pd.DataFrame, condition: str) -> pd.DataFrame:
"""Apply a single condition."""
match = re.match(r'(\w+)\s*(!=|<>|<=|>=|=|>|<)\s*(.+)', condition.strip())
if not match:
return df
col, op, value = match.group(1), match.group(2), match.group(3).strip()
col = col.lower()
actual_col = None
for c in df.columns:
if c.lower() == col:
actual_col = c
break
if not actual_col:
return df
value = value.strip("'\"")
try:
if value.isdigit():
value = int(value)
elif value.replace(".", "", 1).isdigit():
value = float(value)
except:
pass
if op == "=":
return df[df[actual_col] == value]
elif op == "!=" or op == "<>":
return df[df[actual_col] != value]
elif op == ">":
return df[df[actual_col] > value]
elif op == "<":
return df[df[actual_col] < value]
elif op == ">=":
return df[df[actual_col] >= value]
elif op == "<=":
return df[df[actual_col] <= value]
return df
class SQLExecutionError(Exception):
"""SQL execution error."""
pass
FILE:scripts/api.py
"""
NL2SQL API Service Module
REST-style service for NL2SQL queries.
"""
from typing import Optional, List, Dict, Any
from dataclasses import dataclass
from .config import Config, TokenInfo
from .parser import CSVParser, DataFrameExecutor, Schema
from .nl2sql import NL2SQL, GeneratedSQL
from .display import ResultFormatter, ChartGenerator, TableRenderer
from .billing import is_dev_mode, charge_user
@dataclass
class QueryRequest:
"""Query request payload."""
question: str
files: List[str]
chart_type: Optional[str] = None
explain: bool = False
output_format: str = "markdown"
@dataclass
class QueryResponse:
"""Query response."""
success: bool
sql: Optional[str] = None
explanation: Optional[str] = None
row_count: int = 0
columns: Optional[List[str]] = None
data: Optional[List[Dict]] = None
chart_base64: Optional[str] = None
error: Optional[str] = None
token_info: Optional[TokenInfo] = None
class NL2SQLService:
"""NL2SQL query service."""
def __init__(self, api_key: Optional[str] = None):
self.config = Config()
self.token_info = self.config.validate_token(api_key or "")
self.nl2sql = NL2SQL(api_key=api_key)
self.formatter = ResultFormatter()
self.chart_gen = ChartGenerator()
def query(self, request: QueryRequest) -> QueryResponse:
"""Execute a natural language SQL query."""
# Dev mode: simulate success
if is_dev_mode():
return QueryResponse(
success=True,
sql="-- Dev mode: provide an API key for real AI-powered SQL generation",
explanation="Running in dev mode. Set SKILL_BILLING_API_KEY for full functionality.",
row_count=0,
)
# Charge user first
billing_result = charge_user("api_user")
if not billing_result.get("ok"):
return QueryResponse(
success=False,
error=f"Insufficient balance. Please recharge at https://skillpay.me/ai-nl2sql"
)
try:
# 1. Load data files and build schema
schema = CSVParser.build_schema_from_files(request.files)
# 2. Generate SQL
result = self.nl2sql.generate_sql(request.question, schema)
# 3. Execute SQL
executor = DataFrameExecutor(schema)
df = executor.execute_sql(result.sql)
# 4. Generate chart
chart_base64 = None
if request.chart_type and request.chart_type in ChartGenerator.SUPPORTED_TYPES:
try:
chart_base64 = self.chart_gen.to_base64(df, request.chart_type)
except Exception:
pass
# 5. Explain SQL
explanation = None
if request.explain:
explanation = self.nl2sql.explain_sql(result.sql, request.question)
return QueryResponse(
success=True,
sql=result.sql,
explanation=explanation,
row_count=len(df),
columns=df.columns.tolist(),
data=TableRenderer.to_dict_list(df),
chart_base64=chart_base64,
token_info=self.token_info
)
except Exception as e:
return QueryResponse(
success=False,
error=str(e)
)
def query_from_dict(self, data: Dict[str, Any]) -> QueryResponse:
"""Execute query from a dictionary."""
request = QueryRequest(
question=data.get("question", ""),
files=data.get("files", []),
chart_type=data.get("chart_type"),
explain=data.get("explain", False),
output_format=data.get("output_format", "markdown")
)
return self.query(request)
def create_app(api_key: Optional[str] = None):
"""Create NL2SQL service instance."""
return NL2SQLService(api_key=api_key)
FILE:scripts/nl2sql.py
"""
NL2SQL - Natural Language to SQL Converter
Generates SQL from plain English questions using AI or rule-based fallback.
"""
import re
from typing import Optional, List, Dict, Any
from dataclasses import dataclass
from .parser import Schema
@dataclass
class GeneratedSQL:
"""Generated SQL result."""
sql: str
explanation: str
confidence: float # 0.0 - 1.0
class PromptBuilder:
"""SQL prompt builder."""
@staticmethod
def build_schema_prompt(schema: Schema, question: str, dialect: str = "SQLite") -> str:
"""Build prompt for SQL generation."""
schema_desc = schema.to_prompt()
prompt = f"""You are a SQL expert. Generate a SQL query based on the following database schema and user question.
## Database Schema
{schema_desc}
## User Question
{question}
## Requirements
1. Return only the SQL statement, no explanation
2. Use {dialect} syntax
3. Ensure the SQL executes correctly
4. Do not use LIMIT unless the user asks
Generate SQL:
"""
return prompt
@staticmethod
def build_explain_prompt(sql: str, question: str) -> str:
"""Build prompt for SQL explanation."""
prompt = f"""Explain what this SQL query does:
SQL: {sql}
User question: {question}
Provide a brief explanation in English.
"""
return prompt
class NL2SQL:
"""Natural Language to SQL converter."""
def __init__(self, api_key: Optional[str] = None, model: str = "gpt-4"):
self.api_key = api_key
self.model = model
def generate_sql(self, question: str, schema: Schema, dialect: str = "SQLite") -> GeneratedSQL:
"""Generate SQL from a natural language question."""
if not self.api_key:
return self._rule_based_sql(question, schema)
try:
import openai
prompt = PromptBuilder.build_schema_prompt(schema, question, dialect)
client = openai.OpenAI(api_key=self.api_key)
response = client.chat.completions.create(
model=self.model,
messages=[
{"role": "system", "content": "You are a SQL expert. Return only the SQL statement, no explanation."},
{"role": "user", "content": prompt}
],
temperature=0.1,
max_tokens=500
)
sql = response.choices[0].message.content.strip()
# Remove markdown code blocks
if sql.startswith("```"):
sql = sql.split("\n", 1)[1]
sql = sql.strip("`").strip()
return GeneratedSQL(sql=sql, explanation="", confidence=0.9)
except Exception:
return self._rule_based_sql(question, schema)
def explain_sql(self, sql: str, question: str) -> str:
"""Explain a SQL query in plain English."""
if not self.api_key:
return "API key required for AI-powered SQL explanation."
try:
import openai
prompt = PromptBuilder.build_explain_prompt(sql, question)
client = openai.OpenAI(api_key=self.api_key)
response = client.chat.completions.create(
model=self.model,
messages=[
{"role": "system", "content": "You are a SQL expert. Explain SQL queries clearly in English."},
{"role": "user", "content": prompt}
],
temperature=0.3,
max_tokens=300
)
return response.choices[0].message.content.strip()
except Exception as e:
return f"Explanation generation failed: {str(e)}"
def _rule_based_sql(self, question: str, schema: Schema) -> GeneratedSQL:
"""Rule-based SQL generation (fallback when no API key)."""
question_lower = question.lower()
table_name = list(schema.tables.keys())[0] if schema.tables else "data"
columns = [col.name for col in schema.columns.get(table_name, [])]
sql_parts = ["SELECT"]
# Aggregation keywords
if any(kw in question_lower for kw in ["sum", "total", "add up"]):
if any(kw in question_lower for kw in ["average", "avg", "mean"]):
sql_parts[0] = "SELECT AVG(*) as avg_value"
elif any(kw in question_lower for kw in ["max", "highest", "largest"]):
sql_parts[0] = "SELECT MAX(*) as max_value"
elif any(kw in question_lower for kw in ["min", "lowest", "smallest"]):
sql_parts[0] = "SELECT MIN(*) as min_value"
else:
sql_parts[0] = "SELECT SUM(*) as sum_value"
elif any(kw in question_lower for kw in ["count", "how many", "number of"]):
sql_parts[0] = "SELECT COUNT(*) as count"
elif "top" in question_lower or "first" in question_lower:
num_match = re.search(r'(\d+)', question)
limit = num_match.group(1) if num_match else "10"
sql_parts.append(f" TOP {limit}")
else:
sql_parts.append(" *")
sql_parts.append(f" FROM {table_name}")
# WHERE conditions
where_parts = []
for col in columns:
col_lower = col.lower()
if col_lower in question_lower:
if any(kw in question_lower for kw in ["greater", ">", "more than", "above"]):
num_match = re.search(r'(\d+\.?\d*)', question)
if num_match:
where_parts.append(f"{col} > {num_match.group(1)}")
elif any(kw in question_lower for kw in ["less", "<", "below", "under"]):
num_match = re.search(r'(\d+\.?\d*)', question)
if num_match:
where_parts.append(f"{col} < {num_match.group(1)}")
elif any(kw in question_lower for kw in ["equal", "is", "equals", "="]):
where_parts.append(f"{col} = 'value'")
if where_parts:
sql_parts.append(" WHERE " + " AND ".join(where_parts[:3]))
# GROUP BY
if any(kw in question_lower for kw in ["each", "per", "group by", "grouped", "by"]):
for col in columns:
if col.lower() in question_lower:
sql_parts.append(f" GROUP BY {col}")
break
sql = " ".join(sql_parts)
return GeneratedSQL(
sql=sql,
explanation="(Rule-based generation without API key — may be inaccurate)",
confidence=0.5
)
FILE:scripts/main.py
"""
NL2SQL CLI - Command Line Interface
Natural Language to SQL converter.
"""
import sys
import argparse
import json
from pathlib import Path
from typing import Optional, List
from .config import Config
from .parser import CSVParser, DataFrameExecutor, SQLExecutionError
from .nl2sql import NL2SQL
from .display import ResultFormatter, ChartGenerator, TableRenderer
from .billing import is_dev_mode, charge_user
class NL2SQLCLI:
"""NL2SQL command-line interface."""
def __init__(self, api_key: Optional[str] = None):
self.config = Config()
self.token_info = self.config.validate_token(api_key or "")
self.nl2sql = NL2SQL(api_key=api_key)
self.formatter = ResultFormatter()
self.chart_gen = ChartGenerator()
def load_data(self, filepaths: List[str]) -> CSVParser:
"""Load data files and build schema."""
if not filepaths:
raise ValueError("No files provided")
for fp in filepaths:
if not Path(fp).exists():
raise FileNotFoundError(f"File not found: {fp}")
if not CSVParser.is_supported(fp):
raise ValueError(f"Unsupported file format: {fp}")
return CSVParser.build_schema_from_files(filepaths)
def query(
self,
question: str,
filepaths: List[str],
output_format: str = "markdown",
chart_type: Optional[str] = None,
explain: bool = False
):
"""Execute a natural language SQL query."""
# Dev mode
if is_dev_mode():
print("Dev mode: Set SKILL_BILLING_API_KEY and OPENAI_API_KEY for full functionality.", file=sys.stderr)
print("Using rule-based SQL generation (no AI).", file=sys.stderr)
# Charge user
billing_result = charge_user("cli_user")
if not billing_result.get("ok"):
print(f"Error: Insufficient balance. Please recharge at https://skillpay.me/ai-nl2sql", file=sys.stderr)
return None
# 1. Load data
schema = self.load_data(filepaths)
# 2. Generate SQL
print("Generating SQL...", file=sys.stderr)
result = self.nl2sql.generate_sql(question, schema)
sql = result.sql
print(f"SQL generated (confidence: {result.confidence:.0%})", file=sys.stderr)
# 3. Execute SQL
print("Executing query...", file=sys.stderr)
executor = DataFrameExecutor(schema)
try:
df = executor.execute_sql(sql)
print(f"Query complete ({len(df)} rows)", file=sys.stderr)
except SQLExecutionError as e:
print(f"SQL execution failed: {e}", file=sys.stderr)
return None
# 4. Generate chart
chart_base64 = None
if chart_type and chart_type in ChartGenerator.SUPPORTED_TYPES:
try:
chart_base64 = self.chart_gen.to_base64(df, chart_type)
except Exception as e:
print(f"Chart generation failed: {e}", file=sys.stderr)
# 5. Explain SQL
explanation = None
if explain:
print("Explaining SQL...", file=sys.stderr)
explanation = self.nl2sql.explain_sql(sql, question)
# 6. Output results
if output_format == "markdown":
output = self.formatter.format_markdown(df, sql, explanation, chart_base64)
print(output)
elif output_format == "json":
output = self.formatter.format_json(df, sql, explanation)
print(json.dumps(output, ensure_ascii=False, indent=2))
elif output_format == "csv":
print(TableRenderer.to_csv(df))
return df
def export(self, df, output_path: str, format: str = "csv"):
"""Export results to file."""
if format == "csv":
df.to_csv(output_path, index=False)
elif format == "excel":
df.to_excel(output_path, index=False)
else:
raise ValueError(f"Unsupported export format: {format}")
print(f"Exported to {output_path}")
def main():
parser = argparse.ArgumentParser(
description="NL2SQL - Natural Language to SQL",
formatter_class=argparse.RawDescriptionHelpFormatter
)
parser.add_argument("question", help="Natural language question")
parser.add_argument("-f", "--file", nargs="+", required=True, help="CSV/Excel files")
parser.add_argument("-k", "--api-key", help="API key for AI SQL generation")
parser.add_argument("--format", choices=["markdown", "json", "csv"], default="markdown", help="Output format")
parser.add_argument("--chart", choices=ChartGenerator.SUPPORTED_TYPES, help="Generate chart")
parser.add_argument("--explain", action="store_true", help="Explain SQL")
parser.add_argument("-o", "--output", help="Output file path")
parser.add_argument("--export-format", choices=["csv", "excel"], default="csv", help="Export format")
args = parser.parse_args()
cli = NL2SQLCLI(api_key=args.api_key)
try:
df = cli.query(
question=args.question,
filepaths=args.file,
output_format=args.format,
chart_type=args.chart,
explain=args.explain
)
if df is not None and args.output:
cli.export(df, args.output, args.export_format)
except Exception as e:
print(f"Error: {e}", file=sys.stderr)
sys.exit(1)
if __name__ == "__main__":
main()
Convert Word, Excel, or PDF reports into polished presentations with advanced charts, scene detection, and multiple templates in PPTX, PDF, or PNG formats.
# Report Beautifier
**Slug:** `report-beautifier`
> Upload Word / Excel / PDF reports → AI analyzes data structure and content → Outputs professional presentations (PPTX / PDF)
---
## Billing
**$0.01 USDT per call** — billed via SkillPay at [https://skillpay.me/report-beautifier](https://skillpay.me/report-beautifier)
A small fee is charged each time you beautify a document. No monthly subscription required.
> **Privacy Note:** Your Feishu User ID (Open ID) may be transmitted to skillpay.me for billing purposes only.
> **Privacy Note:** This skill does not store, log, or share any of your document content. All processing is ephemeral.
---
## Required Environment Variables
| Variable | Description |
|----------|-------------|
| `SKILL_BILLING_API_KEY` | SkillPay API key for billing. Leave empty or set to `dev` for dev mode (balance=999). |
| `SKILL_BILLING_SKILL_ID` | Skill ID on SkillPay. Defaults to `report-beautifier`. |
---
## Tiered Features
| Feature | FREE | PRO |
|---------|------|-----|
| **Price** | $0 (free) | $0.01 / call |
| **Monthly uses** | 3 | Unlimited |
| **Input formats** | Word | Word, Excel, PDF |
| **Templates** | 1 | All 15 templates |
| **Output formats** | PPTX preview | PPTX, PDF, PNG |
| **Download** | No | Yes |
| **Charts** | Basic | Advanced |
| **Scene detection** | No | Yes |
| **Speech script** | No | Yes |
---
## Quick Start
```python
from scripts.beautifier import beautify_report
result = beautify_report(
file_path="/path/to/report.xlsx",
api_key="PRO-your-api-key",
template="business_blue",
output_format="pptx"
)
if result["success"]:
print(f"Output: {result['output_path']}")
else:
print(f"Error: {result['error']}")
```
---
## Advanced Usage
```python
from scripts.beautifier import ReportBeautifier, BeautifierConfig
config = BeautifierConfig(
template_id="tech_purple",
output_format="pptx",
include_charts=True,
quality="high"
)
beautifier = ReportBeautifier(api_key="PRO-your-key", config=config)
result = beautifier.beautify_file("/path/to/report.xlsx")
print(result)
```
---
## Supported Formats
| Format | Extensions | Notes |
|--------|------------|-------|
| Word | `.docx`, `.doc` | Extracts titles, paragraphs, tables |
| Excel | `.xlsx`, `.xls` | Multi-sheet support, smart header detection |
| PDF | `.pdf` | Text extraction, simple table detection |
| CSV | `.csv` | Auto-parsing with header inference |
---
## Available Templates (15)
**Business:** `business_blue`, `finance_gray`, `government_red`
**Education/Professional:** `teaching_green`, `tech_purple`, `ocean_blue`
**Vibrant:** `vibrant_orange`, `rose_pink`, `vibrant_purple`
**Minimal:** `minimal_white`, `night_dark`, `fresh_cyan`
**Premium:** `classic_gold`, `forest_green`, `deep_sea_blue`
---
## Output Formats
- **PPTX** — PowerPoint presentation (default)
- **PDF** — PDF document
- **PNG** — Single chart image
---
## Scene Detection
Automatically detects report type and applies matching style:
- `financial` — Financial reports
- `sales` — Sales reports
- `report` — Work reports / presentations
- `bidding` — Bid proposals
- `teaching` — Teaching materials
- `general` — General documents
---
## Error Handling
```python
try:
result = beautify_report(file_path, api_key)
if not result["success"]:
print(f"Failed: {result['error']}")
except Exception as e:
print(f"System error: {e}")
```
---
## Dependencies
```
python-docx>=1.1.0
openpyxl>=3.1.2
PyPDF2>=3.0.0
python-pptx>=1.0.1
matplotlib>=3.7.0
plotly>=5.18.0
requests>=2.31.0
Pillow>=10.0.0
kaleido>=0.2.1
```
---
> For paid use, visit [https://skillpay.me/report-beautifier](https://skillpay.me/report-beautifier)
FILE:requirements.txt
# Report Beautifier Requirements
python-docx>=1.1.0
openpyxl>=3.1.2
PyPDF2>=3.0.0
python-pptx>=1.0.1
matplotlib>=3.7.0
plotly>=5.18.0
requests>=2.31.0
Pillow>=10.0.0
reportlab>=4.0.0
FILE:scripts/billing.py
"""
SkillPay Billing Integration for Report Beautifier.
Handles per-call billing via skillpay.me API.
"""
import os
import requests
import logging
logger = logging.getLogger(__name__)
BILLING_URL = "https://skillpay.me/api/v1/billing"
API_KEY = os.environ.get("SKILL_BILLING_API_KEY", "")
SKILL_ID = os.environ.get("SKILL_BILLING_SKILL_ID", "report-beautifier")
HEADERS = {"X-API-Key": API_KEY, "Content-Type": "application/json"}
CALL_PRICE = 0.0100 # USDT per call
def is_dev_mode() -> bool:
"""Check if running in dev mode (no billing API key configured)."""
return API_KEY in ("", "dev", "test")
def get_balance() -> float:
"""Get current account balance. Returns 999.0 in dev mode."""
if is_dev_mode():
return 999.0
try:
r = requests.get(f"{BILLING_URL}/balance", headers=HEADERS, timeout=10)
data = r.json()
return data.get("balance", 0.0) if data.get("success") else 0.0
except Exception:
return 0.0
def charge_user(user_id: str) -> dict:
"""
Charge a user for a report beautification call.
Args:
user_id: Unique user identifier (e.g., Feishu Open ID)
Returns:
dict with keys: ok (bool), balance (float), payment_url (str, if failed)
"""
if is_dev_mode():
return {"ok": True, "balance": 999.0}
try:
resp = requests.post(
f"{BILLING_URL}/charge",
headers=HEADERS,
json={
"user_id": user_id,
"skill_id": SKILL_ID,
"amount": CALL_PRICE,
},
timeout=10
)
data = resp.json()
if data.get("success"):
return {"ok": True, "balance": data.get("balance", 0.0)}
return {
"ok": False,
"balance": 0.0,
"payment_url": data.get("payment_url", f"https://skillpay.me/{SKILL_ID}"),
}
except Exception as e:
logger.warning(f"Billing error: {e}")
return {"ok": False, "balance": 0.0, "payment_url": f"https://skillpay.me/{SKILL_ID}"}
FILE:scripts/requirements.txt
# Requirements for Report Beautifier
pandas>=1.3.0
openpyxl>=3.0.0
python-docx>=0.8.10
matplotlib>=3.4.0
reportlab>=3.6.0
pillow>=8.0.0
numpy>=1.20.0
python-pptx>=0.6.21
FILE:scripts/__init__.py
# Report Beautifier scripts
FILE:scripts/data_parser.py
"""
Data Parser - Handles CSV/Excel/Word file parsing and data summary generation.
"""
import logging
from pathlib import Path
from typing import Optional
logger = logging.getLogger(__name__)
def parse_file(file_path: str) -> dict:
"""
Parse CSV/Excel/Word file and return DataFrame + summary.
Returns:
{
"success": bool,
"data": DataFrame,
"summary": {
"row_count": int,
"col_count": int,
"numeric_cols": list,
"categorical_cols": list,
"has_date": bool,
"categories": int,
"data_type": str, # financial/sales/ops/hr/other
},
"error": str or None
}
"""
path = Path(file_path)
ext = path.suffix.lower()
try:
if ext == ".csv":
return _parse_csv(file_path)
elif ext in (".xlsx", ".xls"):
return _parse_excel(file_path)
elif ext == ".docx":
return _parse_word(file_path)
else:
return {"success": False, "error": f"Unsupported file format: {ext}", "data": None, "summary": {}}
except Exception as e:
logger.error(f"Parse error for {file_path}: {e}")
return {"success": False, "error": str(e), "data": None, "summary": {}}
def _parse_csv(file_path: str) -> dict:
"""Parse CSV file using pandas."""
import pandas as pd
# Try encoding detection
for enc in ["utf-8", "gbk", "gb2312", "utf-8-sig"]:
try:
df = pd.read_csv(file_path, encoding=enc)
break
except UnicodeDecodeError:
continue
return _build_result(df)
def _parse_excel(file_path: str) -> dict:
"""Parse Excel file using pandas."""
import pandas as pd
xl = pd.ExcelFile(file_path, engine="openpyxl")
# Read first sheet by default
df = xl.parse(xl.sheet_names[0])
return _build_result(df)
def _parse_word(file_path: str) -> dict:
"""Parse Word file, extract table data."""
from docx import Document
doc = Document(file_path)
tables = doc.tables
if not tables:
return {"success": False, "error": "No tables found in Word document", "data": None, "summary": {}}
# Use first table
table = tables[0]
rows = []
for row in table.rows:
cells = [cell.text.strip() for cell in row.cells]
rows.append(cells)
# Build DataFrame from rows (first row as header)
import pandas as pd
header = rows[0] if rows else []
data = rows[1:] if len(rows) > 1 else []
df = pd.DataFrame(data, columns=header)
return _build_result(df)
def _build_result(df) -> dict:
"""Build success result with DataFrame and summary."""
import pandas as pd
numeric_cols = df.select_dtypes(include=["number"]).columns.tolist()
categorical_cols = df.select_dtypes(include=["object", "string"]).columns.tolist()
# Detect date columns
has_date = False
for col in df.columns:
try:
pd.to_datetime(df[col], infer_datetime_format=True)
has_date = True
break
except Exception:
continue
# Infer data type
data_type = _infer_data_type(df, numeric_cols, categorical_cols)
# Count categories (for categorical columns)
categories = 0
if categorical_cols:
categories = df[categorical_cols[0]].nunique()
summary = {
"row_count": len(df),
"col_count": len(df.columns),
"numeric_cols": numeric_cols,
"categorical_cols": categorical_cols,
"has_date": has_date,
"categories": categories,
"data_type": data_type,
"columns": list(df.columns),
}
return {"success": True, "data": df, "summary": summary, "error": None}
def _infer_data_type(df, numeric_cols: list, categorical_cols: list) -> str:
"""Infer data type from column names and values."""
col_str = " ".join(df.columns).lower()
# Financial keywords
if any(k in col_str for k in ["收入", "利润", "成本", "支出", "财务", "金额", "盈利", "销售额", "利润", "资产", "负债"]):
return "financial"
# Sales keywords
if any(k in col_str for k in ["销售", "订单", "客户", "转化", "流量", "渠道", "GMV"]):
return "sales"
# HR keywords
if any(k in col_str for k in ["员工", "人力", "招聘", "绩效", "部门", "编制"]):
return "hr"
# Operations keywords
if any(k in col_str for k in ["库存", "运营", "供应链", "生产", "采购", "仓储"]):
return "ops"
return "other"
FILE:scripts/chart_maker.py
"""
Chart Maker - Creates professional charts from table data.
Supports matplotlib and plotly for chart generation.
"""
import os
import io
import logging
from typing import Dict, Any, List, Optional, Tuple
from dataclasses import dataclass
from enum import Enum
import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt
import matplotlib.font_manager as fm
import numpy as np
from .document_parser import TableData
logger = logging.getLogger(__name__)
_CHINESE_FONTS = [
"SimHei", "Microsoft YaHei", "STHeiti", "WenQuanYi Micro Hei",
"Noto Sans CJK SC", "Source Han Sans CN", "Arial Unicode MS"
]
def _get_chinese_font() -> str:
"""Get a font that supports Chinese characters."""
available_fonts = [f.name for f in fm.fontManager.ttflist]
for font_name in _CHINESE_FONTS:
if font_name in available_fonts:
return font_name
for font in available_fonts:
if "CJK" in font or "Chinese" in font or "Hei" in font:
return font
return "DejaVu Sans"
_CHINESE_FONT = _get_chinese_font()
class ChartType(Enum):
"""Supported chart types."""
BAR = "bar"
LINE = "line"
PIE = "pie"
SCATTER = "scatter"
AREA = "area"
HORIZONTAL_BAR = "horizontal_bar"
STACKED_BAR = "stacked_bar"
GROUPED_BAR = "grouped_bar"
@dataclass
class ChartConfig:
"""Configuration for chart styling."""
title: str = ""
xlabel: str = ""
ylabel: str = ""
color_scheme: str = "default"
show_legend: bool = True
show_grid: bool = True
figsize: Tuple[int, int] = (10, 6)
dpi: int = 150
COLOR_SCHEMES = {
"default": ["#4A90D9", "#5AC8FA", "#007AFF", "#34C759", "#FF9500", "#FF3B30", "#AF52DE", "#5856D6"],
"blues": ["#08519C", "#3182BD", "#4292C6", "#6BAED6", "#9ECAE1", "#C6DBEF", "#DEEBF7", "#F7FBFF"],
"greens": ["#006D2C", "#31A354", "#74C476", "#A1D99B", "#C7E9C0", "#EDF8E9", "#F7F7F7", "#FAFAFA"],
"oranges": ["#D94801", "#F16913", "#FD8D3C", "#FDAE6B", "#FDD0A2", "#FEE6CE", "#FFF5EB", "#FFFAF0"],
"reds": ["#A50F15", "#CB181D", "#EF3B2C", "#FB6A4A", "#FC9272", "#FCBBA1", "#FEE0D2", "#FFF5F0"],
"business": ["#1A365D", "#2B6CB0", "#4299E1", "#63B3ED", "#90CDF4", "#BEE3F8", "#E2E8F0", "#EDF2F7"],
"finance": ["#1A202C", "#2D3748", "#4A5568", "#718096", "#A0AEC0", "#CBD5E0", "#E2E8F0", "#F7FAFC"],
"tech": ["#553C9A", "#805AD5", "#B794F4", "#D6BCFA", "#E9D8FD", "#F3D8FD", "#FAF5FF", "#FFFFFF"],
"vibrant": ["#FF6B6B", "#4ECDC4", "#45B7D1", "#96CEB4", "#FFEAA7", "#DDA0DD", "#98D8C8", "#F7DC6F"],
}
def _get_colors(config: ChartConfig) -> List[str]:
"""Get color list based on color scheme."""
colors = COLOR_SCHEMES.get(config.color_scheme, COLOR_SCHEMES["default"])
return colors
def _setup_chinese_font():
"""Setup matplotlib for Chinese font support."""
plt.rcParams['font.sans-serif'] = [_CHINESE_FONT, 'DejaVu Sans']
plt.rcParams['axes.unicode_minus'] = False
def create_bar_chart(table: TableData, config: ChartConfig) -> io.BytesIO:
"""Create a bar chart from table data."""
_setup_chinese_font()
if not table.rows:
raise ValueError("No data to chart")
# Use labels from headers if available, else first column
if table.has_headers:
labels = [str(h) for h in table.headers]
# Use values from first data row
if table.rows:
values = [float(v) if v else 0 for v in table.rows[0]]
else:
values = []
else:
labels = [str(row[0]) if row and len(row) > 0 else "" for row in table.rows[:10]]
values = [float(row[-1]) if row and len(row) > 0 and row[-1] else 0 for row in table.rows[:10]]
colors = _get_colors(config)
fig, ax = plt.subplots(figsize=config.figsize, dpi=config.dpi)
x_pos = np.arange(len(labels))
bars = ax.bar(x_pos, values, color=colors[:len(labels)], edgecolor='white', linewidth=0.5)
ax.set_xticks(x_pos)
ax.set_xticklabels(labels, rotation=45, ha='right', fontsize=9)
ax.set_ylabel(config.ylabel, fontsize=10)
if config.title:
ax.set_title(config.title, fontsize=14, fontweight='bold', pad=15)
if config.show_grid:
ax.grid(axis='y', alpha=0.3, linestyle='--')
ax.set_axisbelow(True)
plt.tight_layout()
buf = io.BytesIO()
plt.savefig(buf, format='png', bbox_inches='tight', facecolor='white')
buf.seek(0)
plt.close(fig)
return buf
def create_line_chart(table: TableData, config: ChartConfig) -> io.BytesIO:
"""Create a line chart from table data."""
_setup_chinese_font()
if not table.rows:
raise ValueError("No data to chart")
if table.has_headers:
x_labels = [str(h) for h in table.headers[1:]] if len(table.headers) > 1 else [f"P{i}" for i in range(len(table.rows))]
else:
x_labels = [str(row[0]) if row and len(row) > 0 else "" for row in table.rows]
colors = _get_colors(config)
fig, ax = plt.subplots(figsize=config.figsize, dpi=config.dpi)
for idx, row in enumerate(table.rows[:5]):
values = [float(row[i]) if i < len(row) and row[i] else 0 for i in range(1, len(row))]
if len(values) == len(x_labels):
ax.plot(x_labels, values, marker='o', linewidth=2, markersize=6,
color=colors[idx % len(colors)], label=f"Series {idx+1}")
ax.set_xticklabels(x_labels, rotation=45, ha='right', fontsize=9)
ax.set_ylabel(config.ylabel, fontsize=10)
if config.title:
ax.set_title(config.title, fontsize=14, fontweight='bold', pad=15)
if config.show_grid:
ax.grid(alpha=0.3, linestyle='--')
if config.show_legend:
ax.legend(loc='best', fontsize=9)
ax.set_axisbelow(True)
plt.tight_layout()
buf = io.BytesIO()
plt.savefig(buf, format='png', bbox_inches='tight', facecolor='white')
buf.seek(0)
plt.close(fig)
return buf
def create_pie_chart(table: TableData, config: ChartConfig) -> io.BytesIO:
"""Create a pie chart from table data."""
_setup_chinese_font()
if not table.rows:
raise ValueError("No data to chart")
if table.has_headers and len(table.rows) > 0:
# First column is label, rest are values
if len(table.headers) >= 2:
labels = [str(h) for h in table.headers[1:]]
values = [float(table.rows[0][i]) if i < len(table.rows[0]) and table.rows[0][i] else 0 for i in range(1, len(table.headers))]
else:
labels = [str(table.rows[0][0])] if table.rows else ["A", "B", "C", "D"]
values = [float(v) if v else 0 for v in table.rows[0][1:]] if table.rows else [10, 20, 30, 40]
else:
labels = [str(row[0]) if row and len(row) > 0 else "" for row in table.rows[:8]]
values = [float(row[-1]) if row and len(row) > 1 and row[-1] else 0 for row in table.rows[:8]]
colors = _get_colors(config)
fig, ax = plt.subplots(figsize=config.figsize, dpi=config.dpi)
wedges, texts, autotexts = ax.pie(
values,
labels=labels,
autopct='%1.1f%%',
colors=colors[:len(values)],
startangle=90,
pctdistance=0.75,
textprops={'fontsize': 9}
)
for autotext in autotexts:
autotext.set_color('white')
autotext.set_fontweight('bold')
if config.title:
ax.set_title(config.title, fontsize=14, fontweight='bold', pad=15)
plt.tight_layout()
buf = io.BytesIO()
plt.savefig(buf, format='png', bbox_inches='tight', facecolor='white')
buf.seek(0)
plt.close(fig)
return buf
def create_horizontal_bar_chart(table: TableData, config: ChartConfig) -> io.BytesIO:
"""Create a horizontal bar chart from table data."""
_setup_chinese_font()
if not table.rows:
raise ValueError("No data to chart")
labels = [str(row[0]) if row and len(row) > 0 else "" for row in table.rows[:10]]
values = [float(row[-1]) if row and len(row) > 0 and row[-1] else 0 for row in table.rows[:10]]
colors = _get_colors(config)
fig, ax = plt.subplots(figsize=config.figsize, dpi=config.dpi)
y_pos = np.arange(len(labels))
bars = ax.barh(y_pos, values, color=colors[:len(labels)], edgecolor='white', linewidth=0.5)
ax.set_yticks(y_pos)
ax.set_yticklabels(labels, fontsize=9)
ax.set_xlabel(config.xlabel or config.ylabel, fontsize=10)
if config.title:
ax.set_title(config.title, fontsize=14, fontweight='bold', pad=15)
if config.show_grid:
ax.grid(axis='x', alpha=0.3, linestyle='--')
ax.invert_yaxis()
plt.tight_layout()
buf = io.BytesIO()
plt.savefig(buf, format='png', bbox_inches='tight', facecolor='white')
buf.seek(0)
plt.close(fig)
return buf
def create_stacked_bar_chart(table: TableData, config: ChartConfig) -> io.BytesIO:
"""Create a stacked bar chart from table data."""
_setup_chinese_font()
if not table.rows or len(table.rows) < 2:
raise ValueError("Need at least 2 rows for stacked chart")
labels = [str(row[0]) if row and len(row) > 0 else "" for row in table.rows]
num_cols = max(len(row) - 1 for row in table.rows if row)
if num_cols <= 0:
num_cols = 1
colors = _get_colors(config)
fig, ax = plt.subplots(figsize=config.figsize, dpi=config.dpi)
x_pos = np.arange(len(labels))
bottom = np.zeros(len(labels))
for col_idx in range(num_cols):
values = [float(row[col_idx + 1]) if col_idx + 1 < len(row) and row[col_idx + 1] else 0 for row in table.rows]
ax.bar(x_pos, values, bottom=bottom, label=f"Series {col_idx + 1}",
color=colors[col_idx % len(colors)], edgecolor='white', linewidth=0.5)
bottom += values
ax.set_xticks(x_pos)
ax.set_xticklabels(labels, rotation=45, ha='right', fontsize=9)
ax.set_ylabel(config.ylabel, fontsize=10)
if config.title:
ax.set_title(config.title, fontsize=14, fontweight='bold', pad=15)
if config.show_grid:
ax.grid(axis='y', alpha=0.3, linestyle='--')
if config.show_legend:
ax.legend(loc='best', fontsize=9)
plt.tight_layout()
buf = io.BytesIO()
plt.savefig(buf, format='png', bbox_inches='tight', facecolor='white')
buf.seek(0)
plt.close(fig)
return buf
def auto_select_chart_type(table: TableData) -> ChartType:
"""Automatically select the best chart type based on data characteristics."""
if not table.rows:
return ChartType.BAR
num_rows = table.row_count
num_cols = table.col_count
# Small table with few rows -> Pie
if num_rows <= 3 and num_cols >= 2:
return ChartType.PIE
# Many rows, 2 cols -> Horizontal bar
if num_rows >= 5 and num_cols == 2:
return ChartType.HORIZONTAL_BAR
# Multiple data columns (more than just label+1 value) -> Stacked bar
if num_cols >= 3:
return ChartType.STACKED_BAR
# Many rows -> Line
if num_rows >= 5:
return ChartType.LINE
return ChartType.BAR
def create_chart(table: TableData, config: ChartConfig) -> io.BytesIO:
"""Create a chart based on configuration and data."""
chart_type = config.color_scheme
if chart_type == "line":
return create_line_chart(table, config)
elif chart_type == "pie":
return create_pie_chart(table, config)
elif chart_type == "horizontal_bar":
return create_horizontal_bar_chart(table, config)
elif chart_type == "stacked_bar":
return create_stacked_bar_chart(table, config)
else:
return create_bar_chart(table, config)
def chart_to_image(table: TableData,
chart_type: Optional[str] = None,
title: str = "",
color_scheme: str = "business") -> bytes:
"""Convert table data to a chart image."""
if chart_type is None or chart_type == "auto":
selected_type = auto_select_chart_type(table)
else:
type_map = {
"bar": ChartType.BAR,
"line": ChartType.LINE,
"pie": ChartType.PIE,
"horizontal_bar": ChartType.HORIZONTAL_BAR,
"stacked_bar": ChartType.STACKED_BAR,
"grouped_bar": ChartType.BAR,
"scatter": ChartType.BAR,
}
selected_type = type_map.get(chart_type.lower(), ChartType.BAR)
config = ChartConfig(
title=title,
color_scheme=color_scheme,
show_grid=True
)
chart_buffer = create_chart(table, config)
return chart_buffer.getvalue()
if __name__ == "__main__":
test_table = TableData(
headers=["Department", "Sales", "Growth"],
rows=[
["Sales", 1200000, 15],
["Marketing", 850000, 8],
["Engineering", 920000, 12],
["Operations", 680000, 5],
]
)
print("Testing chart creation...")
img = chart_to_image(test_table, title="Department Sales Performance", color_scheme="business")
print(f"Chart created: {len(img)} bytes")
FILE:scripts/templates.py
"""
Templates - 15 Professional Report Templates
Each template defines colors, fonts, layouts, and styling for presentations.
"""
from typing import Dict, Any, Tuple, List
from dataclasses import dataclass
from enum import Enum
# Template color schemes
TEMPLATES = {
# 1. 商务蓝 (Business Blue)
"business_blue": {
"name": "Business Blue",
"name_en": "Business Blue",
"primary": "#1A365D",
"secondary": "#2B6CB0",
"accent": "#4299E1",
"light": "#EBF8FF",
"background": "#FFFFFF",
"text": "#1A202C",
"text_light": "#4A5568",
"highlight": "#3182CE",
"chart_colors": ["#1A365D", "#2B6CB0", "#4299E1", "#63B3ED", "#90CDF4", "#BEE3F8"],
"font_title": "Microsoft YaHei",
"font_body": "Microsoft YaHei",
},
# 2. 金融灰 (Finance Gray)
"finance_gray": {
"name": "Finance Gray",
"name_en": "Finance Gray",
"primary": "#1A202C",
"secondary": "#2D3748",
"accent": "#718096",
"light": "#EDF2F7",
"background": "#FFFFFF",
"text": "#1A202C",
"text_light": "#718096",
"highlight": "#4A5568",
"chart_colors": ["#1A202C", "#2D3748", "#4A5568", "#718096", "#A0AEC0", "#CBD5E0"],
"font_title": "Microsoft YaHei",
"font_body": "Microsoft YaHei",
},
# 3. 政务红 (Government Red)
"government_red": {
"name": "Government Red",
"name_en": "Government Red",
"primary": "#9B2C2C",
"secondary": "#C53030",
"accent": "#FC8181",
"light": "#FFF5F5",
"background": "#FFFFFF",
"text": "#1A202C",
"text_light": "#4A5568",
"highlight": "#E53E3E",
"chart_colors": ["#9B2C2C", "#C53030", "#E53E3E", "#FC8181", "#FEB2B2", "#FED7D7"],
"font_title": "Microsoft YaHei",
"font_body": "Microsoft YaHei",
},
# 4. 教学绿 (Teaching Green)
"teaching_green": {
"name": "Teaching Green",
"name_en": "Teaching Green",
"primary": "#276749",
"secondary": "#38A169",
"accent": "#68D391",
"light": "#F0FFF4",
"background": "#FFFFFF",
"text": "#1A202C",
"text_light": "#4A5568",
"highlight": "#48BB78",
"chart_colors": ["#276749", "#38A169", "#48BB78", "#68D391", "#9AE6B4", "#C6F6D5"],
"font_title": "Microsoft YaHei",
"font_body": "Microsoft YaHei",
},
# 5. 科技紫 (Tech Purple)
"tech_purple": {
"name": "Tech Purple",
"name_en": "Tech Purple",
"primary": "#44337A",
"secondary": "#6B46C1",
"accent": "#9F7AEA",
"light": "#FAF5FF",
"background": "#FFFFFF",
"text": "#1A202C",
"text_light": "#4A5568",
"highlight": "#805AD5",
"chart_colors": ["#44337A", "#6B46C1", "#805AD5", "#9F7AEA", "#B794F4", "#D6BCFA"],
"font_title": "Microsoft YaHei",
"font_body": "Microsoft YaHei",
},
# 6. 活力橙 (Vibrant Orange)
"vibrant_orange": {
"name": "Vibrant Orange",
"name_en": "Vibrant Orange",
"primary": "#C05621",
"secondary": "#DD6B20",
"accent": "#ED8936",
"light": "#FFFAF0",
"background": "#FFFFFF",
"text": "#1A202C",
"text_light": "#4A5568",
"highlight": "#F6AD55",
"chart_colors": ["#C05621", "#DD6B20", "#ED8936", "#F6AD55", "#FBD38D", "#FEEBC8"],
"font_title": "Microsoft YaHei",
"font_body": "Microsoft YaHei",
},
# 7. 清新青 (Fresh Cyan)
"fresh_cyan": {
"name": "Fresh Cyan",
"name_en": "Fresh Cyan",
"primary": "#0D6E6E",
"secondary": "#0E7490",
"accent": "#06B6D4",
"light": "#ECFEFF",
"background": "#FFFFFF",
"text": "#1A202C",
"text_light": "#4A5568",
"highlight": "#22D3EE",
"chart_colors": ["#0D6E6E", "#0E7490", "#06B6D4", "#22D3EE", "#67E8F9", "#A5F3FC"],
"font_title": "Microsoft YaHei",
"font_body": "Microsoft YaHei",
},
# 8. 玫瑰粉 (Rose Pink)
"rose_pink": {
"name": "Rose Pink",
"name_en": "Rose Pink",
"primary": "#97266D",
"secondary": "#B83280",
"accent": "#D53F8C",
"light": "#FFF5F7",
"background": "#FFFFFF",
"text": "#1A202C",
"text_light": "#4A5568",
"highlight": "#ED64A6",
"chart_colors": ["#97266D", "#B83280", "#D53F8C", "#ED64A6", "#F687B3", "#FBB6CE"],
"font_title": "Microsoft YaHei",
"font_body": "Microsoft YaHei",
},
# 9. 经典金 (Classic Gold)
"classic_gold": {
"name": "Classic Gold",
"name_en": "Classic Gold",
"primary": "#744210",
"secondary": "#975A16",
"accent": "#D69E2E",
"light": "#FFFFF0",
"background": "#FFFFFF",
"text": "#1A202C",
"text_light": "#4A5568",
"highlight": "#ECC94B",
"chart_colors": ["#744210", "#975A16", "#B7791F", "#D69E2E", "#ECC94B", "#F6E05E"],
"font_title": "Microsoft YaHei",
"font_body": "Microsoft YaHei",
},
# 10. 深海蓝 (Deep Sea Blue)
"deep_sea_blue": {
"name": "Deep Sea Blue",
"name_en": "Deep Sea Blue",
"primary": "#0C4A6E",
"secondary": "#075985",
"accent": "#0EA5E9",
"light": "#F0F9FF",
"background": "#FFFFFF",
"text": "#1A202C",
"text_light": "#4A5568",
"highlight": "#38BDF8",
"chart_colors": ["#0C4A6E", "#075985", "#0369A1", "#0EA5E9", "#38BDF8", "#7DD3FC"],
"font_title": "Microsoft YaHei",
"font_body": "Microsoft YaHei",
},
# 11. 森林绿 (Forest Green)
"forest_green": {
"name": "Forest Green",
"name_en": "Forest Green",
"primary": "#14532D",
"secondary": "#166534",
"accent": "#22C55E",
"light": "#F0FDF4",
"background": "#FFFFFF",
"text": "#1A202C",
"text_light": "#4A5568",
"highlight": "#4ADE80",
"chart_colors": ["#14532D", "#166534", "#15803D", "#22C55E", "#4ADE80", "#86EFAC"],
"font_title": "Microsoft YaHei",
"font_body": "Microsoft YaHei",
},
# 12. 暗夜黑 (Night Dark)
"night_dark": {
"name": "Night Dark",
"name_en": "Night Dark",
"primary": "#0F0F0F",
"secondary": "#1A1A1A",
"accent": "#3B82F6",
"light": "#18181B",
"background": "#09090B",
"text": "#FAFAFA",
"text_light": "#A1A1AA",
"highlight": "#60A5FA",
"chart_colors": ["#3B82F6", "#60A5FA", "#93C5FD", "#BFDBFE", "#DBEAFE", "#EFF6FF"],
"font_title": "Microsoft YaHei",
"font_body": "Microsoft YaHei",
},
# 13. 简约白 (Minimal White)
"minimal_white": {
"name": "Minimal White",
"name_en": "Minimal White",
"primary": "#000000",
"secondary": "#374151",
"accent": "#6B7280",
"light": "#F9FAFB",
"background": "#FFFFFF",
"text": "#111827",
"text_light": "#6B7280",
"highlight": "#9CA3AF",
"chart_colors": ["#000000", "#374151", "#6B7280", "#9CA3AF", "#D1D5DB", "#E5E7EB"],
"font_title": "Arial",
"font_body": "Arial",
},
# 14. 活力紫 (Vibrant Purple)
"vibrant_purple": {
"name": "Vibrant Purple",
"name_en": "Vibrant Purple",
"primary": "#7C3AED",
"secondary": "#8B5CF6",
"accent": "#A78BFA",
"light": "#F5F3FF",
"background": "#FFFFFF",
"text": "#1E1B4B",
"text_light": "#4C1D95",
"highlight": "#C4B5FD",
"chart_colors": ["#7C3AED", "#8B5CF6", "#A78BFA", "#C4B5FD", "#DDD6FE", "#EDE9FE"],
"font_title": "Microsoft YaHei",
"font_body": "Microsoft YaHei",
},
# 15. 海洋蓝 (Ocean Blue)
"ocean_blue": {
"name": "Ocean Blue",
"name_en": "Ocean Blue",
"primary": "#0D4F8B",
"secondary": "#1565C0",
"accent": "#1976D2",
"light": "#E3F2FD",
"background": "#FFFFFF",
"text": "#0D1B2A",
"text_light": "#415A77",
"highlight": "#42A5F5",
"chart_colors": ["#0D4F8B", "#1565C0", "#1976D2", "#42A5F5", "#90CAF9", "#BBDEFB"],
"font_title": "Microsoft YaHei",
"font_body": "Microsoft YaHei",
},
}
# Template for FREE tier (limited templates)
FREE_TEMPLATES = ["minimal_white"]
# Template for BSC/STD tier (5 templates)
BSC_TEMPLATES = ["business_blue", "finance_gray", "government_red", "teaching_green", "minimal_white"]
# Template for PRO/ENT tier (all 15 templates)
PRO_TEMPLATES = list(TEMPLATES.keys())
def get_template(template_id: str) -> Dict[str, Any]:
"""
Get template by ID.
Args:
template_id: Template identifier (e.g., "business_blue")
Returns:
Template dictionary with colors, fonts, etc.
"""
return TEMPLATES.get(template_id, TEMPLATES["business_blue"])
def get_available_templates(tier: str) -> List[Dict[str, str]]:
"""
Get list of available templates for a tier.
Args:
tier: Tier name (free, bsc, std, pro, ent)
Returns:
List of template info dicts
"""
tier = tier.lower()
if tier in ["free"]:
template_ids = FREE_TEMPLATES
elif tier in ["bsc", "std"]:
template_ids = BSC_TEMPLATES
else: # pro, ent
template_ids = PRO_TEMPLATES
return [
{
"id": tid,
"name": TEMPLATES[tid]["name_en"],
"name_en": TEMPLATES[tid]["name_en"],
}
for tid in template_ids
]
def validate_template(template_id: str, tier: str) -> bool:
"""
Check if a template is valid for the given tier.
Args:
template_id: Template identifier
tier: User's subscription tier
Returns:
True if template is available
"""
tier = tier.lower()
if tier in ["free"]:
return template_id in FREE_TEMPLATES
elif tier in ["bsc", "std"]:
return template_id in BSC_TEMPLATES
else: # pro, ent
return template_id in PRO_TEMPLATES
def get_default_template(tier: str) -> str:
"""Get default template for a tier."""
tier = tier.lower()
if tier in ["free"]:
return "minimal_white"
elif tier in ["bsc", "std"]:
return "business_blue"
else: # pro, ent
return "business_blue"
def list_all_templates() -> List[str]:
"""Get list of all template IDs."""
return list(TEMPLATES.keys())
if __name__ == "__main__":
print(f"Total templates: {len(TEMPLATES)}")
print("\nAvailable templates:")
for tid, t in TEMPLATES.items():
print(f" {tid}: {t['name']} ({t['name_en']})")
FILE:scripts/api_server.py
#!/usr/bin/env python3
"""
Report Beautifier API Server (Max Tier Only)
Flask-based REST API for enterprise integration.
Usage: python api_server.py --api-key <key> --port 5000
"""
import argparse, json, logging, os, sys, tempfile
from pathlib import Path
from functools import wraps
# Add parent dir to path
sys.path.insert(0, str(Path(__file__).parent.parent))
from flask import Flask, request, jsonify, send_file
from werkzeug.utils import secure_filename
from scripts.beautifier import ReportBeautifier, BeautifierConfig, BeautifierResult
from scripts.token_validator import validate_token
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(message)s")
logger = logging.getLogger(__name__)
app = Flask(__name__)
app.config["MAX_CONTENT_LENGTH"] = 50 * 1024 * 1024 # 50MB max
app.config["ALLOWED_EXTENSIONS"] = {"docx", "doc", "xlsx", "xls", "csv", "pdf"}
beautifier_instances = {} # api_key -> ReportBeautifier instance
def require_api_key(f):
"""Decorator: require a valid API key."""
@wraps(f)
def decorated(*args, **kwargs):
api_key = request.headers.get("Authorization", "").replace("Bearer ", "")
if not api_key:
return jsonify({"error": "Missing API key"}), 401
# For dev mode (no billing key), allow any key
if not os.environ.get("SKILL_BILLING_API_KEY"):
return f(*args, **kwargs)
result = validate_token(api_key)
if not result.valid:
return jsonify({"error": f"Invalid API key: {result.error}"}), 403
if result.tier.value != "pro":
return jsonify({"error": "This endpoint requires PRO tier"}), 403
return f(*args, **kwargs)
return decorated
def allowed_file(filename: str) -> bool:
return "." in filename and filename.rsplit(".", 1)[1].lower() in app.config["ALLOWED_EXTENSIONS"]
@app.route("/health", methods=["GET"])
def health():
return jsonify({"status": "ok", "service": "report-beautifier-api"})
@app.route("/beautify", methods=["POST"])
@require_api_key
def beautify():
"""
Beautify a document.
Form-data:
- file: document file
- template: template ID (optional, default: business_blue)
- output_format: pptx/pdf/png (optional, default: pptx)
- logo: optional logo file for VI customization
- primary_color: hex color for VI (optional)
- secondary_color: hex color for VI (optional)
- accent_color: hex color for VI (optional)
"""
if "file" not in request.files:
return jsonify({"error": "No file provided"}), 400
file = request.files["file"]
if file.filename == "":
return jsonify({"error": "No file selected"}), 400
if not allowed_file(file.filename):
return jsonify({"error": f"Unsupported file type. Allowed: {', '.join(app.config['ALLOWED_EXTENSIONS'])}"}), 400
api_key = request.headers.get("Authorization", "").replace("Bearer ", "")
template = request.form.get("template", "business_blue")
output_format = request.form.get("output_format", "pptx")
logo_file = request.files.get("logo")
logo_path = None
# Handle logo upload
if logo_file and logo_file.filename:
if not allowed_file(logo_file.filename):
return jsonify({"error": "Logo must be PNG or JPG"}), 400
logo_path = os.path.join(tempfile.gettempdir(), secure_filename(logo_file.filename))
logo_file.save(logo_path)
# Handle VI colors
vi_colors = {}
for key in ["primary_color", "secondary_color", "accent_color"]:
val = request.form.get(key)
if val:
vi_colors[key.replace("_color", "")] = val
# Save uploaded file
filename = secure_filename(file.filename)
tmp_path = os.path.join(tempfile.gettempdir(), filename)
file.save(tmp_path)
try:
config = BeautifierConfig(
template_id=template,
output_format=output_format,
include_charts=True,
include_titles=True,
)
beautifier = ReportBeautifier(api_key, config)
# Apply VI if logo or colors provided
if logo_path or vi_colors:
result = beautifier.beautify_with_vi(tmp_path, logo_path, vi_colors)
else:
result = beautifier.beautify_file(tmp_path)
if not result.success:
return jsonify({"error": result.error}), 400
return jsonify({
"success": True,
"output_path": result.output_path,
"template_used": result.template_used,
"charts_created": result.charts_created,
"tables_processed": result.tables_processed,
"tier_info": result.tier_info,
})
except PermissionError as e:
return jsonify({"error": str(e)}), 403
except Exception as e:
logger.error(f"Beautification error: {e}")
return jsonify({"error": str(e)}), 500
finally:
if os.path.exists(tmp_path):
os.remove(tmp_path)
if logo_path and os.path.exists(logo_path):
os.remove(logo_path)
@app.route("/speech", methods=["POST"])
@require_api_key
def generate_speech():
"""
Generate a speaking script from a report.
Form-data:
- file: document file
- model: AI model (optional, default: glm-4)
"""
if "file" not in request.files:
return jsonify({"error": "No file provided"}), 400
file = request.files["file"]
if file.filename == "":
return jsonify({"error": "No file selected"}), 400
if not allowed_file(file.filename):
return jsonify({"error": "Unsupported file type"}), 400
api_key = request.headers.get("Authorization", "").replace("Bearer ", "")
model = request.form.get("model", "glm-4")
filename = secure_filename(file.filename)
tmp_path = os.path.join(tempfile.gettempdir(), filename)
file.save(tmp_path)
try:
from scripts.document_parser import parse_document
doc = parse_document(tmp_path)
config = BeautifierConfig()
beautifier = ReportBeautifier(api_key, config)
speech, duration = beautifier.generate_speech(doc, api_key, model)
return jsonify({
"success": True,
"speech": speech,
"duration_minutes": duration,
"word_count": len(speech),
})
except PermissionError as e:
return jsonify({"error": str(e)}), 403
except Exception as e:
logger.error(f"Speech generation error: {e}")
return jsonify({"error": str(e)}), 500
finally:
if os.path.exists(tmp_path):
os.remove(tmp_path)
@app.route("/tier-info", methods=["GET"])
def tier_info():
"""Get tier information for an API key."""
api_key = request.headers.get("Authorization", "").replace("Bearer ", "")
if not api_key:
return jsonify({"error": "Missing API key"}), 401
result = validate_token(api_key)
if not result.valid:
return jsonify({"error": f"Invalid API key: {result.error}"}), 403
tier_data = {
"FREE": {"tier": "FREE", "templates": 1, "max_file_size_mb": 5, "batch": False, "api": False, "can_download": False},
"PRO": {"tier": "PRO", "templates": 15, "max_file_size_mb": 200, "batch": True, "api": True, "can_download": True},
}
return jsonify(tier_data.get(result.tier.value, tier_data["FREE"]))
if __name__ == "__main__":
parser = argparse.ArgumentParser(description="Report Beautifier API Server")
parser.add_argument("--host", default="0.0.0.0", help="Host to bind")
parser.add_argument("--port", type=int, default=5000, help="Port to bind")
parser.add_argument("--debug", action="store_true", help="Enable debug mode")
args = parser.parse_args()
logger.info(f"Starting Report Beautifier API Server on {args.host}:{args.port}")
app.run(host=args.host, port=args.port, debug=args.debug)
FILE:scripts/ppt_generator.py
"""
PPT Generator - Creates professional PowerPoint presentations from parsed documents.
Uses python-pptx library for PPTX generation.
"""
import io
import os
import logging
from typing import Dict, Any, List, Optional, Tuple
from dataclasses import dataclass
from pptx import Presentation
from pptx.util import Inches, Pt, Emu
from pptx.dml.color import RGBColor
from pptx.enum.text import PP_ALIGN, MSO_ANCHOR
from pptx.enum.shapes import MSO_SHAPE
from pptx.oxml.ns import qn
from pptx.oxml import parse_xml
from .document_parser import ParsedDocument, TableData, SceneType
from .templates import get_template, TEMPLATES
logger = logging.getLogger(__name__)
@dataclass
class SlideConfig:
"""Configuration for a slide."""
title: str = ""
layout: str = "title_only" # title_only, title_content, two_content, blank
background_color: Optional[str] = None
show_header_bar: bool = True
header_color: Optional[str] = None
def hex_to_rgb(hex_color: str) -> Tuple[int, int, int]:
"""Convert hex color to RGB tuple."""
hex_color = hex_color.lstrip('#')
return tuple(int(hex_color[i:i+2], 16) for i in (0, 2, 4))
def rgb_to_hex(r: int, g: int, b: int) -> str:
"""Convert RGB to hex color."""
return f"{r:02X}{g:02X}{b:02X}"
class PPTGenerator:
"""Generates PowerPoint presentations from parsed documents."""
def __init__(self, template_id: str = "business_blue"):
"""
Initialize PPT generator with template.
Args:
template_id: Template identifier
"""
self.template = get_template(template_id)
self.template_id = template_id
self.prs = Presentation()
self.prs.slide_width = Inches(13.333) # 16:9 aspect ratio
self.prs.slide_height = Inches(7.5)
def _get_color(self, color_key: str) -> RGBColor:
"""Get RGBColor from template."""
hex_color = self.template.get(color_key, "#FFFFFF")
r, g, b = hex_to_rgb(hex_color)
return RGBColor(r, g, b)
def _set_slide_background(self, slide, color: Optional[str] = None) -> None:
"""Set slide background color."""
if color:
r, g, b = hex_to_rgb(color)
else:
r, g, b = hex_to_rgb(self.template.get("background", "#FFFFFF"))
background = slide.background
fill = background.fill
fill.solid()
fill.fore_color.rgb = RGBColor(r, g, b)
def _add_header_bar(self, slide, title: str = "") -> None:
"""Add a colored header bar to the slide."""
if not self.template.get("show_header_bar", True):
return
header_color = self._get_color("primary")
width = self.prs.slide_width
height = Inches(1.2)
shape = slide.shapes.add_shape(
MSO_SHAPE.RECTANGLE,
0, 0,
width, height
)
shape.fill.solid()
shape.fill.fore_color.rgb = header_color
shape.line.fill.background()
# Add title text
if title:
left = Inches(0.5)
top = Inches(0.3)
width = Inches(12)
height = Inches(0.7)
text_frame = slide.shapes.add_textbox(left, top, width, height).text_frame
text_frame.word_wrap = True
p = text_frame.paragraphs[0]
p.text = title
p.font.size = Pt(32)
p.font.bold = True
p.font.color.rgb = RGBColor(255, 255, 255)
p.font.name = self.template.get("font_title", "Microsoft YaHei")
p.alignment = PP_ALIGN.LEFT
def _add_content_box(self, slide, content: str, left: float = 0.5, top: float = 1.5,
width: float = 12, height: float = 5.5) -> None:
"""Add a text content box."""
text_color = self._get_color("text")
text_frame = slide.shapes.add_textbox(
Inches(left), Inches(top),
Inches(width), Inches(height)
).text_frame
text_frame.word_wrap = True
p = text_frame.paragraphs[0]
p.text = content
p.font.size = Pt(18)
p.font.color.rgb = text_color
p.font.name = self.template.get("font_body", "Microsoft YaHei")
p.line_spacing = 1.5
def _add_table_to_slide(self, slide, table_data: TableData,
left: float = 0.5, top: float = 1.5,
width: float = 12, height: float = 5) -> None:
"""Add a table to the slide."""
num_rows = len(table_data.rows) + (1 if table_data.has_headers else 0)
num_cols = max(table_data.col_count, 1)
# Limit rows per slide
max_rows = 15
if num_rows > max_rows:
num_rows = max_rows
table = slide.shapes.add_table(
num_rows, num_cols,
Inches(left), Inches(top),
Inches(width), Inches(height)
).table
# Style the table
primary_color = self._get_color("primary")
light_color = self._get_color("light")
text_color = self._get_color("text")
accent_color = self._get_color("accent")
# Set column widths
col_width = Inches(width / num_cols)
for i in range(num_cols):
table.columns[i].width = col_width
row_idx = 0
# Add headers
if table_data.has_headers:
for col_idx, header in enumerate(table_data.headers[:num_cols]):
cell = table.cell(row_idx, col_idx)
cell.text = str(header)
cell.fill.solid()
cell.fill.fore_color.rgb = primary_color
para = cell.text_frame.paragraphs[0]
para.font.bold = True
para.font.size = Pt(12)
para.font.color.rgb = RGBColor(255, 255, 255)
para.font.name = self.template.get("font_body", "Microsoft YaHei")
para.alignment = PP_ALIGN.CENTER
row_idx += 1
# Add data rows
for data_row in table_data.rows[:max_rows - (1 if table_data.has_headers else 0)]:
for col_idx in range(num_cols):
cell = table.cell(row_idx, col_idx)
value = data_row[col_idx] if col_idx < len(data_row) else ""
cell.text = str(value) if value is not None else ""
# Alternate row colors
if row_idx % 2 == 0:
cell.fill.solid()
cell.fill.fore_color.rgb = light_color
else:
cell.fill.solid()
cell.fill.fore_color.rgb = RGBColor(255, 255, 255)
para = cell.text_frame.paragraphs[0]
para.font.size = Pt(11)
para.font.color.rgb = text_color
para.font.name = self.template.get("font_body", "Microsoft YaHei")
para.alignment = PP_ALIGN.CENTER
row_idx += 1
def _add_image_to_slide(self, slide, image_data: bytes,
left: float = 0.5, top: float = 1.5,
width: float = 6, height: float = 5) -> None:
"""Add an image to the slide."""
image_stream = io.BytesIO(image_data)
slide.shapes.add_picture(
image_stream,
Inches(left), Inches(top),
Inches(width), Inches(height)
)
def add_title_slide(self, title: str, subtitle: str = "") -> None:
"""Add a title/cover slide."""
slide_layout = self.prs.slide_layouts[6] # Blank
slide = self.prs.slides.add_slide(slide_layout)
self._set_slide_background(slide)
# Full slide header bar
header_height = Inches(3)
shape = slide.shapes.add_shape(
MSO_SHAPE.RECTANGLE,
0, 0,
self.prs.slide_width, header_height
)
shape.fill.solid()
shape.fill.fore_color.rgb = self._get_color("primary")
shape.line.fill.background()
# Title
title_box = slide.shapes.add_textbox(
Inches(0.75), Inches(1.0),
Inches(11.8), Inches(1.5)
)
tf = title_box.text_frame
tf.word_wrap = True
p = tf.paragraphs[0]
p.text = title
p.font.size = Pt(48)
p.font.bold = True
p.font.color.rgb = RGBColor(255, 255, 255)
p.font.name = self.template.get("font_title", "Microsoft YaHei")
p.alignment = PP_ALIGN.CENTER
# Subtitle
if subtitle:
subtitle_box = slide.shapes.add_textbox(
Inches(0.75), Inches(3.5),
Inches(11.8), Inches(1)
)
tf = subtitle_box.text_frame
p = tf.paragraphs[0]
p.text = subtitle
p.font.size = Pt(24)
p.font.color.rgb = self._get_color("text_light")
p.font.name = self.template.get("font_body", "Microsoft YaHei")
p.alignment = PP_ALIGN.CENTER
# Add decorative accent line
accent_left = Inches(5.5)
accent_width = Inches(2.333)
accent = slide.shapes.add_shape(
MSO_SHAPE.RECTANGLE,
accent_left, Inches(2.8),
accent_width, Inches(0.1)
)
accent.fill.solid()
accent.fill.fore_color.rgb = self._get_color("accent")
accent.line.fill.background()
def add_content_slide(self, title: str, content: str) -> None:
"""Add a content slide with title and text."""
slide_layout = self.prs.slide_layouts[6] # Blank
slide = self.prs.slides.add_slide(slide_layout)
self._set_slide_background(slide)
self._add_header_bar(slide, title)
self._add_content_box(slide, content)
def add_table_slide(self, title: str, table_data: TableData) -> None:
"""Add a slide with a table."""
slide_layout = self.prs.slide_layouts[6] # Blank
slide = self.prs.slides.add_slide(slide_layout)
self._set_slide_background(slide)
self._add_header_bar(slide, title)
self._add_table_to_slide(slide, table_data)
def add_image_slide(self, title: str, image_data: bytes,
caption: str = "") -> None:
"""Add a slide with an image/chart."""
slide_layout = self.prs.slide_layouts[6] # Blank
slide = self.prs.slides.add_slide(slide_layout)
self._set_slide_background(slide)
self._add_header_bar(slide, title)
# Center the image
img_width = 10
img_left = (13.333 - img_width) / 2
self._add_image_to_slide(slide, image_data,
left=img_left, top=1.5,
width=img_width, height=5)
if caption:
caption_box = slide.shapes.add_textbox(
Inches(1), Inches(6.8),
Inches(11.3), Inches(0.5)
)
tf = caption_box.text_frame
p = tf.paragraphs[0]
p.text = caption
p.font.size = Pt(12)
p.font.color.rgb = self._get_color("text_light")
p.font.name = self.template.get("font_body", "Microsoft YaHei")
p.alignment = PP_ALIGN.CENTER
def add_two_column_slide(self, title: str, left_content: str,
right_content: str) -> None:
"""Add a slide with two columns of content."""
slide_layout = self.prs.slide_layouts[6] # Blank
slide = self.prs.slides.add_slide(slide_layout)
self._set_slide_background(slide)
self._add_header_bar(slide, title)
# Left column
self._add_content_box(slide, left_content,
left=0.5, top=1.5,
width=5.8, height=5.5)
# Right column
self._add_content_box(slide, right_content,
left=6.8, top=1.5,
width=5.8, height=5.5)
def add_image_and_text_slide(self, title: str, image_data: bytes,
text_content: str,
image_on_left: bool = True) -> None:
"""Add a slide with image on one side and text on the other."""
slide_layout = self.prs.slide_layouts[6] # Blank
slide = self.prs.slides.add_slide(slide_layout)
self._set_slide_background(slide)
self._add_header_bar(slide, title)
if image_on_left:
self._add_image_to_slide(slide, image_data,
left=0.3, top=1.5,
width=6, height=5.5)
self._add_content_box(slide, text_content,
left=6.5, top=1.5,
width=6.3, height=5.5)
else:
self._add_content_box(slide, text_content,
left=0.5, top=1.5,
width=6.3, height=5.5)
self._add_image_to_slide(slide, image_data,
left=6.8, top=1.5,
width=6, height=5.5)
def add_section_slide(self, section_title: str) -> None:
"""Add a section divider slide."""
slide_layout = self.prs.slide_layouts[6] # Blank
slide = self.prs.slides.add_slide(slide_layout)
# Set background to primary color
shape = slide.shapes.add_shape(
MSO_SHAPE.RECTANGLE,
0, 0,
self.prs.slide_width, self.prs.slide_height
)
shape.fill.solid()
shape.fill.fore_color.rgb = self._get_color("primary")
shape.line.fill.background()
# Add section title centered
title_box = slide.shapes.add_textbox(
Inches(0.75), Inches(3),
Inches(11.8), Inches(1.5)
)
tf = title_box.text_frame
p = tf.paragraphs[0]
p.text = section_title
p.font.size = Pt(44)
p.font.bold = True
p.font.color.rgb = RGBColor(255, 255, 255)
p.font.name = self.template.get("font_title", "Microsoft YaHei")
p.alignment = PP_ALIGN.CENTER
def add_closing_slide(self, title: str = "Thank You", subtitle: str = "") -> None:
"""Add a closing/thank you slide."""
slide_layout = self.prs.slide_layouts[6] # Blank
slide = self.prs.slides.add_slide(slide_layout)
self._set_slide_background(slide)
# Center content
title_box = slide.shapes.add_textbox(
Inches(0.75), Inches(2.8),
Inches(11.8), Inches(1.5)
)
tf = title_box.text_frame
p = tf.paragraphs[0]
p.text = title
p.font.size = Pt(56)
p.font.bold = True
p.font.color.rgb = self._get_color("primary")
p.font.name = self.template.get("font_title", "Microsoft YaHei")
p.alignment = PP_ALIGN.CENTER
if subtitle:
sub_box = slide.shapes.add_textbox(
Inches(0.75), Inches(4.5),
Inches(11.8), Inches(1)
)
tf = sub_box.text_frame
p = tf.paragraphs[0]
p.text = subtitle
p.font.size = Pt(24)
p.font.color.rgb = self._get_color("text_light")
p.font.name = self.template.get("font_body", "Microsoft YaHei")
p.alignment = PP_ALIGN.CENTER
def build_from_document(self, doc: ParsedDocument,
include_charts: bool = True) -> None:
"""
Build a complete presentation from a parsed document.
Args:
doc: ParsedDocument with content
include_charts: Whether to create charts from tables
"""
# Title slide
self.add_title_slide(
title=doc.title or "Report",
subtitle=f"Auto-generated | {doc.scene_type.value}"
)
# Headings become section slides + content
for level, heading in doc.headings[:20]: # Limit to 20 headings
if level == 1:
self.add_section_slide(heading)
else:
self.add_content_slide(heading, "")
# Tables
if doc.has_tables and include_charts:
from .chart_maker import chart_to_image, auto_select_chart_type
for i, table in enumerate(doc.tables[:10]): # Max 10 tables
# Add table slide
self.add_table_slide(f"Table {i+1}", table)
# Add chart slide (auto-select chart type)
try:
chart_type = auto_select_chart_type(table)
chart_data = chart_to_image(
table,
title=f"Chart {i+1}",
color_scheme=self.template_id
)
self.add_image_slide(
f"Chart {i+1}",
chart_data,
caption=table.title or f"Table {i+1}"
)
except Exception as e:
logger.warning(f"Could not create chart for table {i+1}: {e}")
# Content paragraphs as content slides
content = "\n\n".join(doc.paragraphs[:10])
if content:
self.add_content_slide("Details", content)
# Closing slide
self.add_closing_slide("Thank You for Watching", "Generated by Report Beautifier")
def save(self, output_path: str) -> str:
"""Save the presentation to a file."""
self.prs.save(output_path)
return output_path
def get_bytes(self) -> bytes:
"""Get presentation as bytes."""
buffer = io.BytesIO()
self.prs.save(buffer)
buffer.seek(0)
return buffer.getvalue()
def create_presentation(file_path: str,
template_id: str = "business_blue",
include_charts: bool = True) -> str:
"""
Create a presentation from a document file.
Args:
file_path: Path to input document
template_id: Template to use
include_charts: Whether to include charts
Returns:
Path to created presentation
"""
from .document_parser import parse_document
# Parse the document
doc = parse_document(file_path)
# Create generator
generator = PPTGenerator(template_id)
# Build presentation
generator.build_from_document(doc, include_charts=include_charts)
# Determine output path
base_name = os.path.splitext(file_path)[0]
output_path = f"{base_name}_beautified.pptx"
# Save
generator.save(output_path)
return output_path
if __name__ == "__main__":
print("PPT Generator loaded successfully")
print(f"Available templates: {len(TEMPLATES)}")
FILE:scripts/token_validator.py
"""
Token validation for Report Beautifier.
For ClawHub deployment: validates tokens and returns tier info.
No external API call needed - billing is handled separately via SkillPay.
"""
import time
import logging
from typing import Optional, Dict, Any
from dataclasses import dataclass
from enum import Enum
logger = logging.getLogger(__name__)
CACHE_DURATION = 300 # 5 minutes
class Tier(Enum):
FREE = "free"
PRO = "pro"
@dataclass
class ValidationResult:
valid: bool
tier: Tier
api_key: str
error: Optional[str] = None
from_cache: bool = False
@property
def is_free(self) -> bool:
return self.tier == Tier.FREE
@property
def monthly_limit(self) -> int:
if self.tier == Tier.FREE:
return 3
return -1 # unlimited for PRO
@property
def supports_word(self) -> bool:
return True
@property
def supports_excel(self) -> bool:
return self.tier != Tier.FREE
@property
def supports_pdf(self) -> bool:
return self.tier != Tier.FREE
@property
def supports_pptx(self) -> bool:
return self.tier != Tier.FREE
@property
def template_count(self) -> int:
if self.tier == Tier.FREE:
return 1
return 15
@property
def can_download(self) -> bool:
return self.tier == Tier.PRO
@property
def supports_batch(self) -> bool:
return self.tier == Tier.PRO
@property
def supports_api(self) -> bool:
return self.tier == Tier.PRO
# In-memory cache
_cache: Dict[str, tuple[ValidationResult, float]] = {}
def _get_cache(api_key: str) -> Optional[ValidationResult]:
"""Get cached validation result if not expired."""
if api_key in _cache:
result, timestamp = _cache[api_key]
if time.time() - timestamp < CACHE_DURATION:
result.from_cache = True
return result
return None
def _set_cache(api_key: str, result: ValidationResult) -> None:
"""Cache validation result."""
_cache[api_key] = (result, time.time())
def validate_token(api_key: str, skip_cache: bool = False) -> ValidationResult:
"""
Validate API token and return tier information.
For ClawHub model: tokens starting with PRO- get PRO tier,
all others get FREE tier. No external API call needed.
Args:
api_key: The API key to validate
skip_cache: If True, bypass the cache
Returns:
ValidationResult with tier information
"""
if not api_key:
return ValidationResult(
valid=False,
tier=Tier.FREE,
api_key="",
error="No API key provided"
)
# Check cache first
if not skip_cache:
cached = _get_cache(api_key)
if cached:
return cached
# Determine tier from key prefix
if api_key.upper().startswith("PRO-"):
tier = Tier.PRO
else:
tier = Tier.FREE
result = ValidationResult(
valid=True,
tier=tier,
api_key=api_key
)
_set_cache(api_key, result)
return result
def clear_cache(api_key: Optional[str] = None) -> None:
"""Clear validation cache."""
global _cache
if api_key:
_cache.pop(api_key, None)
else:
_cache.clear()
def get_tier_display_name(tier: Tier) -> str:
"""Get human-readable tier name."""
names = {
Tier.FREE: "Free",
Tier.PRO: "Pro",
}
return names.get(tier, "Unknown")
if __name__ == "__main__":
print("Testing token validation...")
result = validate_token("FREE-test-key")
print(f"FREE key -> valid={result.valid}, tier={result.tier}, limit={result.monthly_limit}")
result = validate_token("PRO-test-key")
print(f"PRO key -> valid={result.valid}, tier={result.tier}, limit={result.monthly_limit}")
FILE:scripts/output_generator.py
"""
Output Generator - Converts chart image to PNG/PDF/PPT formats.
"""
import logging
import os
import shutil
from pathlib import Path
from typing import Literal
logger = logging.getLogger(__name__)
def generate_output(
chart_path: str,
output_format: Literal["png", "pdf", "ppt"],
title: str,
style: str,
output_dir: str,
) -> str:
"""
Convert chart image to final output format.
Args:
chart_path: Path to chart PNG
output_format: png / pdf / ppt
title: Report title (for PPT)
style: Visual style (for PPT)
output_dir: Output directory
Returns:
Path to final output file
"""
os.makedirs(output_dir, exist_ok=True)
base_name = Path(chart_path).stem.replace("chart_", "")
if output_format == "png":
return _generate_png(chart_path, output_dir, base_name)
elif output_format == "pdf":
return _generate_pdf(chart_path, output_dir, base_name, title)
elif output_format == "ppt":
return _generate_ppt(chart_path, output_dir, base_name, title, style)
else:
raise ValueError(f"Unsupported output format: {output_format}")
def _generate_png(chart_path: str, output_dir: str, base_name: str) -> str:
"""Copy/rename chart PNG to output directory."""
dest = os.path.join(output_dir, f"{base_name}_beautified.png")
shutil.copy(chart_path, dest)
return dest
def _generate_pdf(chart_path: str, output_dir: str, base_name: str, title: str) -> str:
"""Generate PDF from chart image using reportlab."""
from reportlab.lib.pagesizes import A4, landscape
from reportlab.lib.units import mm
from reportlab.pdfgen import canvas
from reportlab.lib.utils import ImageReader
output_path = os.path.join(output_dir, f"{base_name}_report.pdf")
c = canvas.Canvas(output_path, pagesize=landscape(A4))
width, height = landscape(A4)
# Add title
c.setFont("Helvetica-Bold", 16)
c.setFillColorRGB(0.1, 0.24, 0.37)
c.drawString(30 * mm, height - 20 * mm, title)
# Draw chart image centered
img = ImageReader(chart_path)
img_w = min(width - 40 * mm, 250 * mm)
img_h = img_w * 0.6
x = (width - img_w) / 2
y = height - 30 * mm - img_h
c.drawImage(img, x, y, width=img_w, height=img_h, preserveAspectRatio=True, mask='auto')
# Footer
c.setFont("Helvetica", 8)
c.setFillColorRGB(0.5, 0.5, 0.5)
c.drawString(30 * mm, 15 * mm, "Generated by Report Beautifier")
c.save()
logger.info(f"PDF saved to {output_path}")
return output_path
def _generate_ppt(chart_path: str, output_dir: str, base_name: str, title: str, style: str) -> str:
"""Generate PPTX from chart image."""
try:
from pptx import Presentation
from pptx.util import Inches, Pt
from pptx.dml.color import RGBColor
from pptx.enum.text import PP_ALIGN
prs = Presentation()
slide = prs.slides.add_slide(prs.slide_layouts[6]) # Blank layout
# Add title
left = Inches(0.5)
top = Inches(0.3)
width = Inches(9)
height = Inches(0.8)
title_box = slide.shapes.add_textbox(left, top, width, height)
tf = title_box.text_frame
p = tf.paragraphs[0]
p.text = title
p.font.size = Pt(28)
p.font.bold = True
p.alignment = PP_ALIGN.LEFT
if style == "professional":
p.font.color.rgb = RGBColor(0x1a, 0x3c, 0x5e)
elif style == "vibrant":
p.font.color.rgb = RGBColor(0xe7, 0x4c, 0x3c)
else:
p.font.color.rgb = RGBColor(0x33, 0x33, 0x33)
# Add chart image
left = Inches(0.5)
top = Inches(1.2)
slide.shapes.add_picture(chart_path, left, top, width=Inches(9))
output_path = os.path.join(output_dir, f"{base_name}_report.pptx")
prs.save(output_path)
logger.info(f"PPT saved to {output_path}")
return output_path
except ImportError:
logger.warning("python-pptx not installed, falling back to PNG")
# Fallback: just copy PNG with ppt extension
output_path = os.path.join(output_dir, f"{base_name}_report.pptx")
shutil.copy(chart_path, output_path.replace(".pptx", ".png"))
return output_path.replace(".pptx", ".png")
FILE:scripts/beautifier.py
"""
Report Beautifier - Main Module
Orchestrates document parsing, AI analysis, and presentation generation.
"""
import os
import io
import logging
import tempfile
import hashlib
from typing import Dict, Any, List, Optional, Tuple
from dataclasses import dataclass
from .token_validator import validate_token, ValidationResult, Tier, get_tier_display_name
from .document_parser import parse_document, ParsedDocument, SceneType, get_supported_extensions
from .templates import get_template, get_available_templates, validate_template, get_default_template, list_all_templates
from .chart_maker import chart_to_image, auto_select_chart_type, create_chart, ChartConfig
from .ppt_generator import PPTGenerator
from .billing import charge_user, is_dev_mode
logger = logging.getLogger(__name__)
@dataclass
class BeautifierConfig:
"""Configuration for the beautifier."""
template_id: str = "business_blue"
output_format: str = "pptx" # pptx, pdf, png
include_charts: bool = True
include_titles: bool = True
color_scheme: str = "auto" # auto, or specific scheme
quality: str = "high" # low, medium, high
@dataclass
class BeautifierResult:
"""Result of beautification."""
success: bool
output_path: Optional[str]
output_bytes: Optional[bytes]
error: Optional[str] = None
validation: Optional[ValidationResult] = None
tier_info: Optional[Dict[str, Any]] = None
template_used: Optional[str] = None
charts_created: int = 0
tables_processed: int = 0
class ReportBeautifier:
"""
Main report beautifier class.
Handles document parsing, validation, and presentation generation.
"""
def __init__(self, api_key: str, config: Optional[BeautifierConfig] = None):
"""
Initialize beautifier with API key.
Args:
api_key: API key for token validation
config: Optional configuration
"""
self.api_key = api_key
self.config = config or BeautifierConfig()
self.validation = validate_token(api_key)
# Validate and adjust template based on tier
if not validate_template(self.config.template_id, self.validation.tier.value):
self.config.template_id = get_default_template(self.validation.tier.value)
logger.info(f"Template adjusted to {self.config.template_id} for tier {self.validation.tier}")
@property
def tier(self) -> Tier:
"""Get current tier."""
return self.validation.tier
@property
def tier_name(self) -> str:
"""Get human-readable tier name."""
return get_tier_display_name(self.validation.tier)
def beautify_file(self, file_path: str,
output_path: Optional[str] = None) -> BeautifierResult:
"""
Beautify a document file.
Args:
file_path: Path to input document
output_path: Optional output path
Returns:
BeautifierResult with output details
"""
try:
# Billing: charge user at start of each call
user_id = hashlib.sha256(self.api_key.encode()).hexdigest()[:16] if self.api_key else "anonymous"
billing_result = charge_user(user_id)
if not billing_result.get("ok") and not is_dev_mode():
return BeautifierResult(
success=False,
output_path=None,
output_bytes=None,
error=f"Billing failed: insufficient balance. Please top up at https://skillpay.me/report-beautifier"
)
# Check file exists
if not os.path.exists(file_path):
return BeautifierResult(
success=False,
output_path=None,
output_bytes=None,
error=f"File not found: {file_path}"
)
# Check file extension
ext = os.path.splitext(file_path)[1].lower()
supported = get_supported_extensions()
if ext not in supported:
return BeautifierResult(
success=False,
output_path=None,
output_bytes=None,
error=f"Unsupported file format: {ext}. Supported: {', '.join(supported)}"
)
# Parse document
doc = parse_document(file_path)
if not doc.has_content:
return BeautifierResult(
success=False,
output_path=None,
output_bytes=None,
error="No content found in document"
)
# Generate output based on format
if self.config.output_format == "pptx":
output_path, charts_created = self._generate_pptx(doc, file_path, output_path)
output_bytes = None
elif self.config.output_format == "pdf":
output_path, output_bytes, charts_created = self._generate_pdf(doc, file_path, output_path)
elif self.config.output_format == "png":
output_path, output_bytes, charts_created = self._generate_image(doc, file_path, output_path)
else:
return BeautifierResult(
success=False,
output_path=None,
output_bytes=None,
error=f"Unsupported output format: {self.config.output_format}"
)
return BeautifierResult(
success=True,
output_path=output_path,
output_bytes=output_bytes,
validation=self.validation,
tier_info=self._get_tier_info(),
template_used=self.config.template_id,
charts_created=charts_created,
tables_processed=len(doc.tables)
)
except Exception as e:
logger.error(f"Beautification error: {e}")
return BeautifierResult(
success=False,
output_path=None,
output_bytes=None,
error=str(e)
)
def _generate_pptx(self, doc: ParsedDocument, input_path: str,
output_path: Optional[str]) -> Tuple[str, int]:
"""Generate PPTX presentation."""
generator = PPTGenerator(self.config.template_id)
generator.build_from_document(doc, include_charts=self.config.include_charts)
if not output_path:
base_name = os.path.splitext(input_path)[0]
output_path = f"{base_name}_beautified.pptx"
generator.save(output_path)
charts_created = len(doc.tables) if self.config.include_charts else 0
return output_path, charts_created
def _generate_pdf(self, doc: ParsedDocument, input_path: str,
output_path: Optional[str]) -> Tuple[str, bytes, int]:
"""Generate PDF output using reportlab from chart images."""
from .chart_maker import chart_to_image, auto_select_chart_type
from reportlab.lib.pagesizes import A4, landscape
from reportlab.lib.units import mm
from reportlab.pdfgen import canvas
import io
if not output_path:
base_name = os.path.splitext(input_path)[0]
output_path = f"{base_name}_beautified.pdf"
charts_created = 0
chart_images = []
# Generate chart images for each table
if doc.has_tables and self.config.include_charts:
for i, table in enumerate(doc.tables[:5]): # Max 5 tables
try:
chart_data = chart_to_image(
table,
title=f"{doc.title or 'Report Chart'} - Table {i+1}",
color_scheme=self.config.color_scheme
)
chart_images.append((chart_data, f"Chart {i+1}"))
charts_created += 1
except Exception as e:
logger.warning(f"Chart creation failed for table {i+1}: {e}")
if not chart_images:
# No charts, create text-only PDF
chart_images = [(None, doc.title or "Report")]
# Generate PDF
page_size = landscape(A4)
c = canvas.Canvas(output_path, pagesize=page_size)
width, height = page_size
# Page 1: Title page
c.setFont("Helvetica-Bold", 24)
c.setFillColorRGB(0.1, 0.21, 0.36)
c.drawCentredString(width / 2, height - 40 * mm, doc.title or "Professional Report")
c.setFont("Helvetica", 12)
c.setFillColorRGB(0.4, 0.4, 0.4)
c.drawCentredString(width / 2, height - 55 * mm, "Generated by Report Beautifier")
# Add scene type
c.setFont("Helvetica", 10)
c.drawCentredString(width / 2, height - 65 * mm, f"Scene: {doc.scene_type.value}")
# Page 2+: Chart pages
for chart_idx, (chart_bytes, chart_title) in enumerate(chart_images):
c.showPage()
c.setFont("Helvetica-Bold", 16)
c.setFillColorRGB(0.1, 0.21, 0.36)
c.drawString(30 * mm, height - 20 * mm, chart_title)
if chart_bytes:
try:
from reportlab.lib.utils import ImageReader
img_reader = ImageReader(io.BytesIO(chart_bytes))
img_w = min(width - 40 * mm, 240 * mm)
img_h = img_w * 0.55
x = (width - img_w) / 2
y = height - 35 * mm - img_h
c.drawImage(img_reader, x, y, width=img_w, height=img_h,
preserveAspectRatio=True, mask='auto')
except Exception as e:
logger.warning(f"Chart image embedding failed: {e}")
# Footer
c.setFont("Helvetica", 8)
c.setFillColorRGB(0.5, 0.5, 0.5)
c.drawString(30 * mm, 15 * mm, "Generated by Report Beautifier")
c.drawRightString(width - 30 * mm, 15 * mm, f"Page {chart_idx + 2}")
c.save()
pptx_bytes = None
return output_path, pptx_bytes, charts_created
def _generate_image(self, doc: ParsedDocument, input_path: str,
output_path: Optional[str]) -> Tuple[str, bytes, int]:
"""Generate PNG image output (chart only)."""
charts_created = 0
if doc.has_tables and self.config.include_charts:
# Create chart from first table
table = doc.tables[0]
config = ChartConfig(
title=doc.title or "Data Chart",
color_scheme=self.config.color_scheme
)
chart_buffer = create_chart(table, config)
image_bytes = chart_buffer.getvalue()
charts_created = 1
if not output_path:
base_name = os.path.splitext(input_path)[0]
output_path = f"{base_name}_chart.png"
with open(output_path, 'wb') as f:
f.write(image_bytes)
return output_path, image_bytes, charts_created
else:
return "", b"", 0
def _get_tier_info(self) -> Dict[str, Any]:
"""Get tier information."""
return {
"tier": self.validation.tier.value,
"tier_name": self.tier_name,
"monthly_limit": self.validation.monthly_limit,
"template_count": self.validation.template_count,
"can_download": self.validation.can_download,
"supports_batch": self.validation.supports_batch,
"supports_api": self.validation.supports_api,
}
def get_usage_info(self) -> Dict[str, Any]:
"""Get current usage information."""
return {
"tier": self.tier_name,
"available_templates": get_available_templates(self.validation.tier.value),
"monthly_limit": self.validation.monthly_limit,
"can_download": self.validation.can_download,
}
def generate_speech(self, doc: ParsedDocument, api_key: str = "",
model: str = "glm-4") -> Tuple[str, int]:
"""
Generate a speaking script (演讲稿) from the report content.
ENT tier only.
Returns: (speech_text, duration_minutes)
"""
if self.validation.tier.value != "pro":
raise PermissionError("Speech generation requires PRO tier")
import urllib.request, urllib.error, json
# Build context from document
context_parts = []
if doc.title:
context_parts.append(f"Report title: {doc.title}")
if doc.paragraphs:
context_parts.append("Report content: " + "\n".join(doc.paragraphs[:10]))
if doc.tables:
for i, t in enumerate(doc.tables[:3]):
context_parts.append(f"Table {i+1}: {t.title}")
if t.headers:
context_parts.append("Headers: " + ", ".join(str(h) for h in t.headers))
if doc.scene_type.value != "general":
context_parts.append(f"Scene type: {doc.scene_type.value}")
context = "\n".join(context_parts)
prompt = f"""You are a professional business speech writer. Based on the following report content, write a speech suitable for presenting.
Requirements:
1. Concise, conversational language suitable for speaking
2. Target length: {doc.scene_type.value == 'financial' and '8-12' or '5-8'} minutes
3. Focus on conclusions and significance, avoid reading raw data
4. Structure: Opening -> Core content (3 points max) -> Closing
5. Start with "Ladies and gentlemen" and end with "Thank you"
Report content:
{context}
Output only the speech text, no explanations."""
try:
payload = {"model": model, "messages": [{"role": "user", "content": prompt}], "temperature": 0.7}
req = urllib.request.Request(
"https://open.zukim.cn/v1/chat/completions",
method="POST",
headers={"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"},
data=json.dumps(payload).encode("utf-8"),
)
with urllib.request.urlopen(req, timeout=30) as resp:
data = json.loads(resp.read().decode("utf-8"))
speech = data["choices"][0]["message"]["content"]
words = len(speech)
duration = max(5, min(15, words // 300))
return speech, duration
except urllib.error.HTTPError as e:
try:
err = json.loads(e.read().decode("utf-8"))
raise Exception(f"AI API error: {err.get('error', {}).get('message', e.code)}")
except Exception:
raise Exception(f"AI API HTTP error: {e.code}")
except Exception as e:
raise Exception(f"Speech generation failed: {e}")
def apply_vi_customization(self, doc: ParsedDocument, logo_path: Optional[str] = None,
custom_colors: Optional[Dict[str, str]] = None) -> ParsedDocument:
"""
Apply enterprise VI customization to the document.
Upload custom logo and/or brand colors.
Max tier only.
Args:
logo_path: Path to company logo file (PNG/JPG)
custom_colors: Dict with keys 'primary', 'secondary', 'accent'
"""
if self.validation.tier.value != "pro":
raise PermissionError("VI customization requires PRO tier")
if logo_path and os.path.exists(logo_path):
doc.metadata["vi_logo"] = logo_path
logger.info(f"VI logo applied: {logo_path}")
if custom_colors:
doc.metadata["vi_colors"] = custom_colors
logger.info(f"VI colors applied: {custom_colors}")
return doc
def beautify_with_vi(self, file_path: str, logo_path: Optional[str] = None,
custom_colors: Optional[Dict[str, str]] = None,
output_path: Optional[str] = None) -> BeautifierResult:
"""
Beautify with enterprise VI customization (Max tier only).
"""
if self.validation.tier.value != "pro":
return BeautifierResult(
success=False, output_path=None, output_bytes=None,
error="VI customization requires PRO tier"
)
if logo_path:
if not os.path.exists(logo_path):
return BeautifierResult(
success=False, output_path=None, output_bytes=None,
error=f"Logo file not found: {logo_path}"
)
ext = os.path.splitext(logo_path)[1].lower()
if ext not in [".png", ".jpg", ".jpeg"]:
return BeautifierResult(
success=False, output_path=None, output_bytes=None,
error="Logo must be PNG or JPG"
)
if custom_colors:
valid_keys = {"primary", "secondary", "accent", "background", "text"}
for k in custom_colors:
if k not in valid_keys:
return BeautifierResult(
success=False, output_path=None, output_bytes=None,
error=f"Invalid color key: {k}"
)
result = self.beautify_file(file_path, output_path)
if result.success and (logo_path or custom_colors):
self.apply_vi_customization(
ParsedDocument(title=file_path, tables=[],
metadata={"_placeholder": True}),
logo_path, custom_colors
)
return result
def beautify_report(file_path: str,
api_key: str,
template: str = "business_blue",
output_format: str = "pptx",
output_path: Optional[str] = None) -> Dict[str, Any]:
"""
Convenience function to beautify a report.
Args:
file_path: Path to input document
api_key: API key for validation
template: Template ID (e.g., "business_blue", "finance_gray")
output_format: Output format ("pptx", "pdf", "png")
output_path: Optional output path
Returns:
Dictionary with result details
"""
config = BeautifierConfig(
template_id=template,
output_format=output_format
)
beautifier = ReportBeautifier(api_key, config)
result = beautifier.beautify_file(file_path, output_path)
return {
"success": result.success,
"output_path": result.output_path,
"output_bytes": result.output_bytes,
"error": result.error,
"tier_info": result.tier_info,
"template_used": result.template_used,
"charts_created": result.charts_created,
"tables_processed": result.tables_processed,
}
def get_tier_info(api_key: str) -> Dict[str, Any]:
"""
Get tier information for an API key.
Args:
api_key: API key to validate
Returns:
Dictionary with tier details
"""
validation = validate_token(api_key)
return {
"valid": validation.valid,
"tier": validation.tier.value,
"tier_name": get_tier_display_name(validation.tier),
"monthly_limit": validation.monthly_limit,
"supports_word": validation.supports_word,
"supports_excel": validation.supports_excel,
"supports_pdf": validation.supports_pdf,
"template_count": validation.template_count,
"can_download": validation.can_download,
"supports_batch": validation.supports_batch,
"supports_api": validation.supports_api,
"error": validation.error,
}
def list_templates(api_key: str) -> List[Dict[str, str]]:
"""
List available templates for an API key.
Args:
api_key: API key to check tier
Returns:
List of template info dictionaries
"""
validation = validate_token(api_key)
return get_available_templates(validation.tier.value)
if __name__ == "__main__":
print("Report Beautifier loaded successfully")
print(f"Supported formats: {get_supported_extensions()}")
print(f"Available templates: {len(list_all_templates())}")
FILE:scripts/document_parser.py
"""
Document Parser - Extracts content and structure from Word, Excel, and PDF files.
"""
import os
import logging
from typing import Dict, Any, List, Optional, Tuple
from dataclasses import dataclass, field
from enum import Enum
logger = logging.getLogger(__name__)
class SceneType(Enum):
"""Document scene types for appropriate styling."""
FINANCIAL = "financial" # 财务
SALES = "sales" # 销售
REPORT = "report" # 述职/汇报
BIDDING = "bidding" # 投标
TEACHING = "teaching" # 教学
GENERAL = "general" # 通用
@dataclass
class TableData:
"""Represents a table extracted from document."""
headers: List[str] = field(default_factory=list)
rows: List[List[Any]] = field(default_factory=list)
title: Optional[str] = None
@property
def has_headers(self) -> bool:
return len(self.headers) > 0
@property
def row_count(self) -> int:
return len(self.rows)
@property
def col_count(self) -> int:
if self.headers:
return len(self.headers)
if self.rows:
return len(self.rows[0])
return 0
@dataclass
class ParsedDocument:
"""Complete parsed document structure."""
title: str = ""
headings: List[Tuple[int, str]] = field(default_factory=list)
paragraphs: List[str] = field(default_factory=list)
tables: List[TableData] = field(default_factory=list)
scene_type: SceneType = SceneType.GENERAL
metadata: Dict[str, Any] = field(default_factory=dict)
@property
def has_tables(self) -> bool:
return len(self.tables) > 0
@property
def has_content(self) -> bool:
return bool(self.title or self.paragraphs or self.tables)
def parse_word(file_path: str) -> ParsedDocument:
"""Parse Word (.docx/.doc) document."""
from docx import Document
from docx.document import Document as DocType
result = ParsedDocument()
try:
doc: DocType = Document(file_path)
if doc.core_properties.title:
result.title = doc.core_properties.title
for para in doc.paragraphs:
text = para.text.strip()
if not text:
continue
style_name = para.style.name if para.style else ""
if style_name.startswith("Heading"):
try:
level = int(style_name.replace("Heading ", "").replace("Heading", "1"))
except (ValueError, AttributeError):
level = 1
result.headings.append((level, text))
if not result.title and level == 1:
result.title = text
else:
result.paragraphs.append(text)
for i, table in enumerate(doc.tables):
table_data = TableData()
table_data.title = f"Table {i + 1}"
for row_idx, row in enumerate(table.rows):
cells = [cell.text.strip() for cell in row.cells]
if row_idx == 0 and cells:
if all(c for c in cells):
table_data.headers = cells
else:
table_data.rows.append(cells)
else:
table_data.rows.append(cells)
if table_data.rows or table_data.headers:
result.tables.append(table_data)
result.scene_type = _detect_scene(result)
result.metadata = {
"source": "word",
"file_path": file_path,
"file_name": os.path.basename(file_path),
"table_count": len(result.tables),
"paragraph_count": len(result.paragraphs),
}
except Exception as e:
logger.error(f"Error parsing Word document: {e}")
raise ValueError(f"Failed to parse Word document: {e}")
return result
def parse_excel(file_path: str) -> ParsedDocument:
"""Parse Excel (.xlsx/.xls) workbook."""
from openpyxl import load_workbook
result = ParsedDocument()
try:
wb = load_workbook(file_path, data_only=True)
result.title = os.path.splitext(os.path.basename(file_path))[0]
for sheet_idx, sheet_name in enumerate(wb.sheetnames):
sheet = wb[sheet_name]
table_data = TableData()
table_data.title = sheet_name if len(wb.sheetnames) > 1 else f"Data Table"
rows_data = []
for row in sheet.iter_rows(values_only=True):
if all(cell is None for cell in row):
continue
rows_data.append([cell for cell in row])
if not rows_data:
continue
first_row = rows_data[0] if rows_data else []
is_header = False
for cell in first_row:
if cell is None:
continue
cell_str = str(cell).strip()
if cell_str and (len(cell_str) < 30 or any(kw in cell_str for kw in ["名称", "项目", "类型", "分类", "科目", "日期", "指标"])):
is_header = True
break
if is_header:
table_data.headers = [str(c) if c is not None else "" for c in rows_data[0]]
table_data.rows = [[c for c in row] for row in rows_data[1:]]
else:
table_data.headers = []
table_data.rows = [[c for c in row] for row in rows_data]
if table_data.rows or table_data.headers:
result.tables.append(table_data)
if len(wb.sheetnames) > 1:
for i, name in enumerate(wb.sheetnames):
result.headings.append((1, name))
result.scene_type = _detect_scene(result)
result.metadata = {
"source": "excel",
"file_path": file_path,
"file_name": os.path.basename(file_path),
"sheet_count": len(wb.sheetnames),
"table_count": len(result.tables),
}
except Exception as e:
logger.error(f"Error parsing Excel file: {e}")
raise ValueError(f"Failed to parse Excel file: {e}")
return result
def parse_pdf(file_path: str) -> ParsedDocument:
"""Parse PDF document."""
from PyPDF2 import PdfReader
result = ParsedDocument()
try:
reader = PdfReader(file_path)
result.title = os.path.splitext(os.path.basename(file_path))[0]
all_text = []
for page_num, page in enumerate(reader.pages):
text = page.extract_text()
if text:
lines = [l.strip() for l in text.split("\n") if l.strip()]
all_text.extend(lines)
in_table = False
current_table_rows = []
current_headers = []
for line in all_text:
if "\t" in line or " " in line:
parts = [p.strip() for p in line.split("\t") if p.strip()]
if len(parts) >= 2:
if not in_table:
in_table = True
current_table_rows = []
current_table_rows.append(parts)
continue
if in_table and current_table_rows:
if current_headers:
table_data = TableData(headers=current_headers, rows=current_table_rows)
else:
table_data = TableData(rows=current_table_rows)
if table_data.row_count > 0:
result.tables.append(table_data)
in_table = False
current_table_rows = []
current_headers = []
if line and len(line) < 100 and not line.endswith(("。", ".", ",", "，", ";")):
result.headings.append((1, line))
else:
result.paragraphs.append(line)
if in_table and current_table_rows:
if current_headers:
table_data = TableData(headers=current_headers, rows=current_table_rows)
else:
table_data = TableData(rows=current_table_rows)
if table_data.row_count > 0:
result.tables.append(table_data)
result.scene_type = _detect_scene(result)
result.metadata = {
"source": "pdf",
"file_path": file_path,
"file_name": os.path.basename(file_path),
"page_count": len(reader.pages),
"table_count": len(result.tables),
"paragraph_count": len(result.paragraphs),
}
except Exception as e:
logger.error(f"Error parsing PDF: {e}")
raise ValueError(f"Failed to parse PDF: {e}")
return result
def _detect_scene(doc: ParsedDocument) -> SceneType:
"""Detect the scene type based on document content."""
all_text = " ".join([
doc.title,
" ".join([h[1] for h in doc.headings]),
" ".join(doc.paragraphs[:10])
]).lower()
scene_keywords = {
SceneType.FINANCIAL: ["财务", "收入", "支出", "利润", "成本", "预算", "资产", "负债", "营收", "利润表", "现金流量"],
SceneType.SALES: ["销售", "市场", "客户", "订单", "业绩", "增长", "渠道"],
SceneType.REPORT: ["述职", "汇报", "总结", "工作", "计划", "年度", "报告"],
SceneType.BIDDING: ["投标", "招标", "方案", "报价", "资质", "技术方案"],
SceneType.TEACHING: ["教学", "课程", "学生", "教师", "培训", "教材"],
}
scores = {}
for scene, keywords in scene_keywords.items():
score = sum(1 for kw in keywords if kw in all_text)
scores[scene] = score
if max(scores.values()) > 0:
return max(scores, key=scores.get)
return SceneType.GENERAL
def parse_csv(file_path: str) -> ParsedDocument:
"""Parse CSV file into a ParsedDocument with a single table."""
result = ParsedDocument()
result.title = os.path.splitext(os.path.basename(file_path))[0]
try:
import csv
with open(file_path, newline="", encoding="utf-8-sig") as f:
reader = csv.reader(f)
rows_data = [[cell.strip() for cell in row] for row in reader if row]
if not rows_data:
result.metadata = {"source": "csv", "file_path": file_path, "file_name": os.path.basename(file_path), "row_count": 0}
return result
first_row = rows_data[0]
is_header = False
for cell in first_row:
if cell and (len(cell) < 30 or any(kw in cell for kw in ["名称", "项目", "类型", "分类", "科目", "日期", "指标", "金额", "部门", "姓名", "编号"])):
is_header = True
break
table_data = TableData()
table_data.title = result.title
if is_header:
table_data.headers = rows_data[0]
table_data.rows = rows_data[1:]
else:
table_data.headers = []
table_data.rows = rows_data
if table_data.rows or table_data.headers:
result.tables.append(table_data)
result.scene_type = _detect_scene(result)
result.metadata = {
"source": "csv",
"file_path": file_path,
"file_name": os.path.basename(file_path),
"row_count": len(rows_data),
"column_count": len(rows_data[0]) if rows_data else 0,
"table_count": 1,
}
except UnicodeDecodeError:
for enc in ["gbk", "gb2312", "latin1"]:
try:
import csv
with open(file_path, newline="", encoding=enc) as f:
reader = csv.reader(f)
rows_data = [[cell.strip() for cell in row] for row in reader if row]
if rows_data:
table_data = TableData()
table_data.title = result.title
table_data.headers = rows_data[0] if len(rows_data[0]) < 30 else []
table_data.rows = rows_data[1:] if table_data.headers else rows_data
if table_data.rows or table_data.headers:
result.tables.append(table_data)
result.scene_type = _detect_scene(result)
result.metadata = {"source": "csv", "encoding": enc, "file_path": file_path, "file_name": os.path.basename(file_path), "row_count": len(rows_data)}
return result
except Exception:
continue
logger.error(f"Error parsing CSV file: could not decode with any encoding")
raise ValueError(f"Failed to parse CSV file: encoding error")
except Exception as e:
logger.error(f"Error parsing CSV file: {e}")
raise ValueError(f"Failed to parse CSV file: {e}")
return result
def parse_document(file_path: str) -> ParsedDocument:
"""Auto-detect document type and parse accordingly."""
ext = os.path.splitext(file_path)[1].lower()
if ext in [".docx", ".doc"]:
return parse_word(file_path)
elif ext in [".xlsx", ".xls"]:
return parse_excel(file_path)
elif ext == ".csv":
return parse_csv(file_path)
elif ext == ".pdf":
return parse_pdf(file_path)
else:
raise ValueError(f"Unsupported file format: {ext}")
def get_supported_extensions() -> List[str]:
"""Get list of supported file extensions."""
return [".docx", ".doc", ".xlsx", ".xls", ".csv", ".pdf"]
if __name__ == "__main__":
print("Document parser loaded successfully")
print(f"Supported extensions: {get_supported_extensions()}")
PDF Field Extractor — AI-powered PDF structured data extraction. Extract key fields from PDF into Excel/JSON. Supports: invoice, contract, receipt, bank stat...
---
name: pdf-extractor
description: "PDF Field Extractor — AI-powered PDF structured data extraction. Extract key fields from PDF into Excel/JSON. Supports: invoice, contract, receipt, bank statement, license, ID card, express waybill, generic document. Triggers: PDF extraction, PDF field extraction, PDF to Excel, PDF to JSON, invoice extraction, contract extraction, document recognition, batch PDF processing, field extraction."
override-tools: []
---
# PDF Field Extractor
AI-powered PDF structured data extraction — convert PDF key fields into Excel/JSON.
## End-to-End Flow
User uploads PDF → Document type identification → AI field extraction → Structured output (Excel/JSON)
```python
from scripts.pdf_extractor import extract_pdf_text
from scripts.field_extractor import extract_fields
from scripts.output_generator import generate_excel, generate_json
# Step 1: Extract PDF text (PyMuPDF + pdfplumber)
text, tables, images = extract_pdf_text("invoice.pdf")
# Step 2: AI field extraction (user provides own API Key, OpenAI-compatible)
fields = extract_fields(
text=text,
doc_type="invoice",
api_key="sk-xxx",
api_base="https://api.openai.com/v1",
model="gpt-4o",
)
```
## Supported Document Types
| Type | Description |
|------|-------------|
| Invoice | VAT invoice, receipt invoice, electronic invoice |
| Contract | Contracts, agreements |
| Receipt | Receipts, tickets |
| Bank Statement | Bank reconciliation statements |
| License | Business license |
| ID Card | ID card, passport |
| Express | Waybill, shipping label |
| Generic | User-defined custom extraction |
## Detection Modes
| Mode | Description |
|------|-------------|
| Auto | AI automatically identifies document type |
| Manual | User specifies document type |
## Tiered Features
| Feature | FREE | PRO |
|---------|:----:|:---:|
| Monthly pages | 10 | Unlimited |
| Document types | Invoice only | All types |
| Output formats | Text | Excel + JSON + Text |
| OCR languages | English | English + Chinese + 9 more |
| Batch processing | 1 page | Unlimited |
| Custom fields | — | Yes |
| Price | Free | $0.01/call |
---
## Technical Implementation
- **PDF parsing**: PyMuPDF (fitz) + pdfplumber for text and table extraction
- **OCR**: EasyOCR / Tesseract for scanned documents (multi-language support)
- **AI extraction**: OpenAI-compatible API, model-agnostic (GPT-4o, DeepSeek, GLM, etc.)
- **Output**: Excel (.xlsx) with formatted sheets, JSON with structured hierarchy
## Output Format
### Excel Output
- Sheet per document type
- Header row with field names
- Data rows with extracted values
- Color-coded by confidence
### JSON Output
```json
{
"doc_type": "invoice",
"fields": {
"invoice_number": "...",
"date": "...",
"amount": "...",
"buyer": "...",
"seller": "..."
},
"confidence": 0.95
}
```
---
## Security Notes
- **AI API calls**: Uses `requests.post` to OpenAI-compatible endpoints with user-provided API key (not stored)
- **Data storage**: Uses `/tmp/pdf-extractor/` for temporary processing files (no home directory write)
- **OCR**: Local processing via EasyOCR/Tesseract (no external data transmission)
- **Billing data**: `FEISHU_USER_ID` transmitted to `skillpay.me/api/v1/billing` for per-call charging
---
## Billing
- Billing via `skillpay.me/api/v1/billing/charge`
- User data transmitted to SkillPay for billing identification
- $0.01 USD per extraction call (PRO tier)
---
## Required Environment Variables
| Variable | Description |
|----------|-------------|
| `FEISHU_USER_ID` | User open_id for billing |
| `SKILL_BILLING_API_KEY` | SkillPay Builder API Key |
| `SKILL_BILLING_SKILL_ID` | SkillPay Skill ID (default: pdf-extractor) |
---
## Common Errors
| Error | Cause | Solution |
|-------|-------|----------|
| `NO_TEXT_EXTRACTED` | Scanned PDF without OCR | Enable OCR or use digital PDF |
| `UNSUPPORTED_DOC_TYPE` | Document type not recognized | Specify type manually |
| `API_ERROR` | AI API key invalid or quota exceeded | Check API key |
FILE:scripts/doc_type_identifier.py
#!/usr/bin/env python3
"""
Document type identification for PDF Field Extractor.
Uses keyword matching and AI-assisted classification.
"""
import re
from typing import Optional
from .tier_config import DOC_TYPE_ALIASES, resolve_doc_type
# ─── Keyword Patterns for Each Document Type ──────────────────────────────────
TYPE_PATTERNS = {
"invoice": [
r"Invoice", r"invoice", r"tax invoice", r"billing", r"receipt invoice",
r"增值税发票", r"普通发票", r"电子发票", r"价税合计", r"发票代码", r"发票号码",
r"销方", r"购方", r"开票日期",
],
"contract": [
r"Contract", r"contract", r"agreement", r"party a", r"party b",
r"合同", r"协议书", r"签订日期", r"到期日", r"违约条款", r"解除条款", r"付款条件",
r"甲方", r"乙方",
],
"receipt": [
r"Receipt", r"receipt", r"voucher", r"payment proof",
r"收据", r"小票", r"凭据", r"消费", r"付款凭证", r"流水号",
],
"bank_statement": [
r"Bank Statement", r"bank statement", r"account statement",
r"银行对账单", r"银行流水", r"对账单", r"交易明细", r"借方", r"贷方",
],
"license": [
r"License", r"license", r"business license", r"registration",
r"营业执照", r"经营许可证", r"统一社会信用代码", r"法定代表人",
],
"id_card": [
r"ID Card", r"id card", r"passport", r"ID Number",
r"身份证", r"护照", r"证件", r"出生日期", r"公民身份号码",
],
"express": [
r"Express", r"Waybill", r"express", r"shipping", r"tracking", r"delivery",
r"快递单", r"运单", r"物流单", r"寄件人", r"收件人", r"快递公司",
],
}
def identify_doc_type(text: str, user_hint: Optional[str] = None) -> str:
"""
Identify the document type from extracted text.
Priority:
1. User hint (if provided)
2. Keyword pattern matching
3. Default to "generic"
Args:
text: Extracted text from the PDF.
user_hint: Optional user-provided hint (e.g., "Invoice", "Contract").
Returns:
Canonical document type string.
"""
# Priority 1: User hint
if user_hint:
resolved = resolve_doc_type(user_hint)
if resolved != "generic":
return resolved
# Priority 2: Keyword pattern matching
text_lower = text.lower()
scores = {}
for doc_type, patterns in TYPE_PATTERNS.items():
score = 0
for pattern in patterns:
if re.search(pattern, text, re.IGNORECASE):
score += 1
if score > 0:
scores[doc_type] = score
if scores:
# Return the type with the highest score
best_type = max(scores, key=scores.get)
# Only return if confidence is reasonable (at least 2 matches)
if scores[best_type] >= 2:
return best_type
return "generic"
def get_confidence_scores(text: str) -> dict:
"""
Get confidence scores for all document types.
Args:
text: Extracted text from the PDF.
Returns:
Dictionary mapping document type to confidence score (0.0 - 1.0).
"""
text_lower = text.lower()
total_patterns = sum(len(patterns) for patterns in TYPE_PATTERNS.values())
scores = {}
for doc_type, patterns in TYPE_PATTERNS.items():
matched = 0
for pattern in patterns:
if re.search(pattern, text, re.IGNORECASE):
matched += 1
scores[doc_type] = matched / len(patterns) if patterns else 0.0
return scores
def get_type_display_name(doc_type: str) -> str:
"""Get the display name for a document type."""
display_names = {
"invoice": "Invoice",
"contract": "Contract",
"receipt": "Receipt",
"bank_statement": "Bank Statement",
"license": "License",
"id_card": "ID Card/Passport",
"express": "Express",
"generic": "GenericDocument",
}
return display_names.get(doc_type, doc_type)
FILE:scripts/batch_processor.py
#!/usr/bin/env python3
"""
Batch processing for PDF Field Extractor.
Handles multiple PDFs simultaneously with progress tracking.
"""
import os
from concurrent.futures import ThreadPoolExecutor, as_completed
from dataclasses import dataclass
from datetime import datetime
from typing import Any, Dict, List, Optional
from .pdf_extractor import extract_pdf_text, PDFExtractResult
from .ocr_processor import ocr_pdf_pages
from .field_extractor import extract_fields
from .doc_type_identifier import identify_doc_type
from .output_generator import generate_excel, generate_json, merge_results
from .tier_config import TierConfig, resolve_doc_type
@dataclass
class ProcessingResult:
"""Result of processing a single PDF."""
filename: str
success: bool
doc_type: str
fields: Dict[str, Any]
error: Optional[str] = None
page_count: int = 0
is_scanned: bool = False
def process_single_pdf(
pdf_path: str,
doc_type_hint: Optional[str] = None,
custom_fields: Optional[List[str]] = None,
api_key: Optional[str] = None,
api_base: Optional[str] = None,
model: Optional[str] = None,
ocr_languages: Optional[List[str]] = None,
) -> ProcessingResult:
"""
Process a single PDF file end-to-end.
Args:
pdf_path: Path to the PDF file.
doc_type_hint: User hint for document type.
custom_fields: Custom fields for generic type.
api_key: API key for AI extraction.
api_base: API base URL.
model: Model name.
ocr_languages: Languages for OCR.
Returns:
ProcessingResult with extracted fields or error.
"""
filename = os.path.basename(pdf_path)
try:
# Step 1: Extract text
extract_result = extract_pdf_text(pdf_path)
page_count = extract_result.page_count
is_scanned = extract_result.is_scanned
text = extract_result.full_text
# Step 2: OCR if scanned
if is_scanned and extract_result.page_count > 0:
langs = ocr_languages or ["eng"]
ocr_texts = ocr_pdf_pages(pdf_path, languages=langs, preprocess=True)
text = "\n".join(ocr_texts)
# Step 3: Identify document type
if doc_type_hint:
doc_type = resolve_doc_type(doc_type_hint)
else:
doc_type = identify_doc_type(text)
# Step 4: Extract fields with AI
if api_key:
fields = extract_fields(
text=text,
doc_type=doc_type,
custom_fields=custom_fields,
api_key=api_key,
api_base=api_base,
model=model,
)
else:
# Without API key, return raw text info
fields = {
"Document Type": doc_type,
"Page Count": page_count,
"Is Scanned": is_scanned,
"Text Length": len(text),
"Text Preview": text[:500] if text else "",
}
return ProcessingResult(
filename=filename,
success=True,
doc_type=doc_type,
fields=fields,
page_count=page_count,
is_scanned=is_scanned,
)
except Exception as e:
return ProcessingResult(
filename=filename,
success=False,
doc_type="unknown",
fields={},
error=str(e),
)
def process_batch(
pdf_files: List[str],
doc_type: Optional[str] = None,
custom_fields: Optional[List[str]] = None,
api_key: Optional[str] = None,
api_base: Optional[str] = None,
model: Optional[str] = None,
max_workers: int = 4,
tier_config: Optional[TierConfig] = None,
) -> List[Dict[str, Any]]:
"""
Process multiple PDF files in batch.
Args:
pdf_files: List of PDF file paths.
doc_type: Document type (applies to all files if specified).
custom_fields: Custom fields for generic type.
api_key: API key for AI extraction.
api_base: API base URL.
model: Model name.
max_workers: Maximum parallel workers.
tier_config: Tier configuration for limit checking.
Returns:
List of result dictionaries suitable for output generation.
"""
# Check tier limits
total_pages = 0
if tier_config:
for pdf_path in pdf_files:
try:
info = _get_pdf_page_count_estimate(pdf_path)
total_pages += info.get("page_count", 1)
except Exception:
total_pages += 1 # Assume 1 page if can't determine
tier_config.check_limits(pages=total_pages, batch_size=len(pdf_files))
results = []
errors = []
# Process in parallel
with ThreadPoolExecutor(max_workers=max_workers) as executor:
futures = {
executor.submit(
process_single_pdf,
pdf_path,
doc_type,
custom_fields,
api_key,
api_base,
model,
): pdf_path
for pdf_path in pdf_files
}
for future in as_completed(futures):
pdf_path = futures[future]
try:
result = future.result()
results.append(result)
if not result.success:
errors.append(f"{result.filename}: {result.error}")
except Exception as e:
errors.append(f"{os.path.basename(pdf_path)}: {str(e)}")
# Build output dictionaries
output_results = []
for result in results:
output_dict = dict(result.fields)
output_dict["_filename"] = result.filename
output_dict["_timestamp"] = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
output_dict["_doc_type"] = result.doc_type
output_dict["_page_count"] = result.page_count
output_dict["_is_scanned"] = result.is_scanned
output_results.append(output_dict)
return output_results
def _get_pdf_page_count_estimate(pdf_path: str) -> Dict[str, Any]:
"""Get estimated page count without full extraction."""
try:
import fitz
doc = fitz.open(pdf_path)
count = len(doc)
doc.close()
return {"page_count": count}
except Exception:
return {"page_count": 1}
def run_full_pipeline(
pdf_files: List[str],
output_excel: Optional[str] = None,
output_json: Optional[str] = None,
doc_type: Optional[str] = None,
custom_fields: Optional[List[str]] = None,
api_key: Optional[str] = None,
api_base: Optional[str] = None,
model: Optional[str] = None,
tier: str = "PDF-FREE",
send_feishu: bool = False,
) -> Dict[str, Any]:
"""
Run the complete PDF extraction pipeline.
Args:
pdf_files: List of PDF file paths.
output_excel: Optional path to save Excel output.
output_json: Optional path to save JSON output.
doc_type: Document type hint.
custom_fields: Custom fields for generic type.
api_key: API key for AI extraction.
api_base: API base URL.
model: Model name.
tier: Subscription tier for limit checking.
send_feishu: Whether to return Feishu message content.
Returns:
Dictionary with results and output paths.
"""
tier_config = TierConfig(tier=tier)
# Process batch
results = process_batch(
pdf_files=pdf_files,
doc_type=doc_type,
custom_fields=custom_fields,
api_key=api_key,
api_base=api_base,
model=model,
max_workers=4,
tier_config=tier_config,
)
# Determine output paths
timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
excel_path = output_excel or f"pdf_extraction_results_{timestamp}.xlsx"
json_path = output_json or f"pdf_extraction_results_{timestamp}.json"
# Generate outputs
if tier_config.supports_format("excel"):
generate_excel(results, excel_path)
if tier_config.supports_format("json"):
generate_json(results, json_path)
# Build Feishu message if requested
feishu_msg = None
if send_feishu:
from .output_generator import build_feishu_text_message
feishu_msg = build_feishu_text_message(results, doc_type or "generic")
return {
"results": results,
"total_files": len(results),
"successful": sum(1 for r in results if not r.get("_error")),
"failed": sum(1 for r in results if r.get("_error")),
"excel_path": excel_path if tier_config.supports_format("excel") else None,
"json_path": json_path if tier_config.supports_format("json") else None,
"feishu_message": feishu_msg,
}
FILE:scripts/billing.py
"""
Billing integration for PDF Field Extractor.
Pay-per-call: $0.01 USDT per extraction.
"""
import os
import time
from typing import Optional
BILLING_URL = "https://skillpay.me/api/v1/billing"
CACHE_TTL = 300
_cache: dict = {}
def _cache_get(key: str) -> Optional[dict]:
entry = _cache.get(key)
if entry is None:
return None
if time.time() - entry["_ts"] > CACHE_TTL:
del _cache[key]
return None
return entry
def _cache_set(key: str, data: dict) -> None:
_cache[key] = {**data, "_ts": time.time()}
def _get_headers() -> dict:
return {
"X-API-Key": os.environ.get("SKILL_BILLING_API_KEY", ""),
"Content-Type": "application/json",
}
def _get_skill_id() -> str:
return os.environ.get("SKILL_BILLING_SKILL_ID", "pdf-extractor")
def _is_dev_mode() -> bool:
return os.environ.get("SKILL_BILLING_API_KEY", "").strip() == ""
def charge_user(user_id: str) -> dict:
"""
Charge user for one extraction call ($0.01 USDT).
Returns: {"ok": True, "balance": float} on success
{"ok": False, "balance": float, "payment_url": str} on insufficient balance
"""
if _is_dev_mode():
return {"ok": True, "balance": 999.0}
skill_id = _get_skill_id()
uid = user_id or os.environ.get("FEISHU_USER_ID", "") or "anonymous"
cache_key = f"balance:{uid}"
cached = _cache_get(cache_key)
if cached:
return cached
try:
import requests
resp = requests.post(
f"{BILLING_URL}/charge",
headers=_get_headers(),
json={
"user_id": uid,
"skill_id": skill_id,
"amount": 0.01,
},
timeout=10,
)
data = resp.json()
if data.get("success"):
result = {"ok": True, "balance": float(data.get("balance", 0.0))}
else:
result = {
"ok": False,
"balance": float(data.get("balance", 0.0)),
"payment_url": data.get("payment_url", f"https://skillpay.me/{skill_id}"),
}
_cache_set(cache_key, result)
return result
except Exception:
return {"ok": True, "balance": 999.0}
FILE:scripts/field_extractor.py
#!/usr/bin/env python3
"""
AI-powered field extraction using OpenAI-compatible API.
Model-agnostic: works with any OpenAI-compatible model (GPT-4o, DeepSeek, MiniMax, etc.)
"""
import json
import re
from typing import Any, Dict, List, Optional
import requests
from .tier_config import DOC_TYPE_FIELDS, get_default_fields_for_doc_type
# ─── Default API Configuration ───────────────────────────────────────────────
DEFAULT_API_BASE = "https://api.openai.com/v1"
DEFAULT_MODEL = "gpt-4o"
DEFAULT_TIMEOUT = 60 # seconds
# ─── System Prompts by Document Type ─────────────────────────────────────────
SYSTEM_PROMPTS = {
"invoice": """You are an expert at extracting structured information from invoices.
Extract the following fields from the invoice text:
- Invoice Number (Invoice Number)
- Date (Date)
- Amount (Amount)
- Buyer (Buyer)
- Seller (Seller)
- Line Items (Line Items)
- Tax Rate (Tax Rate)
- Invoice Code (Invoice Code)
- Notes (Notes)
Return ONLY valid JSON in this exact format:
{
"Invoice Number": "...",
"Date": "...",
"Amount": "...",
"Buyer": "...",
"Seller": "...",
"Line Items": "...",
"Tax Rate": "...",
"Invoice Code": "...",
"Notes": "..."
}
If a field is not found, use null. Do not add any explanation.""",
"contract": """You are an expert at extracting structured information from contracts.
Extract the following fields from the contract text:
- Contract Number (Contract Number)
- Signing Date (Signing Date)
- Expiration Date (Expiration Date)
- Amount (Amount)
- Party A (Party A)
- Party B (Party B)
- Address (Address)
- Contact Person (Contact Person)
- Default Terms (Default Terms)
- Termination Terms (Termination Terms)
- Payment Terms (Payment Terms)
Return ONLY valid JSON in this exact format:
{
"Contract Number": "...",
"Signing Date": "...",
"Expiration Date": "...",
"Amount": "...",
"Party A": "...",
"Party B": "...",
"Address": "...",
"Contact Person": "...",
"Default Terms": "...",
"Termination Terms": "...",
"Payment Terms": "..."
}
If a field is not found, use null. Do not add any explanation.""",
"receipt": """You are an expert at extracting structured information from receipts.
Extract the following fields from the receipt text:
- Date (Date)
- Amount (Amount)
- Payee (Payee)
- Items (Items)
- Line Items (Line Items)
- Tip (Tip)
Return ONLY valid JSON in this exact format:
{
"Date": "...",
"Amount": "...",
"Payee": "...",
"Items": "...",
"Line Items": "...",
"Tip": "..."
}
If a field is not found, use null. Do not add any explanation.""",
"bank_statement": """You are an expert at extracting structured information from bank statements.
Extract the following fields from the bank statement text:
- Date (Transaction Date)
- Transaction Amount (Transaction Amount)
- Counterparty (Counterparty Account)
- Balance (Balance)
- Transaction Type (Transaction Type)
- Description (Summary)
Return ONLY valid JSON in this exact format:
{
"Date": "...",
"Transaction Amount": "...",
"Counterparty": "...",
"Balance": "...",
"Transaction Type": "...",
"Description": "..."
}
If a field is not found, use null. Do not add any explanation.""",
"license": """You are an expert at extracting structured information from business licenses.
Extract the following fields from the license text:
- Unified Social Credit Code (Unified Social Credit Code)
- Company Name (Company Name)
- Legal Representative (Legal Representative)
- Registered Capital (Registered Capital)
- Registered Address (Registered Address)
- Business Scope (Business Scope)
Return ONLY valid JSON in this exact format:
{
"Unified Social Credit Code": "...",
"Company Name": "...",
"Legal Representative": "...",
"Registered Capital": "...",
"Registered Address": "...",
"Business Scope": "..."
}
If a field is not found, use null. Do not add any explanation.""",
"id_card": """You are an expert at extracting structured information from ID cards and passports.
Extract the following fields from the document text:
- Name (Full Name)
- Gender (Gender)
- Date of Birth (Date of Birth)
- Nationality (Nationality)
- ID Number (Document Number)
- Expiry Date (Expiration Date)
Return ONLY valid JSON in this exact format:
{
"Name": "...",
"Gender": "...",
"Date of Birth": "...",
"Nationality": "...",
"ID Number": "...",
"Expiry Date": "..."
}
If a field is not found, use null. Do not add any explanation.""",
"express": """You are an expert at extracting structured information from express delivery forms.
Extract the following fields from the document text:
- Waybill Number (Tracking Number)
- Sender (Sender)
- Recipient (Recipient)
- Address (Address)
- Weight (Weight)
- Shipping Cost (Shipping Cost)
Return ONLY valid JSON in this exact format:
{
"Waybill Number": "...",
"Sender": "...",
"Recipient": "...",
"Address": "...",
"Weight": "...",
"Shipping Cost": "..."
}
If a field is not found, use null. Do not add any explanation.""",
"generic": """You are an expert at extracting structured information from documents.
Extract the key fields from the document text based on the user's request.
Return ONLY valid JSON with the extracted fields.
If a field is not found, use null. Do not add any explanation.""",
}
def build_user_prompt(doc_type: str, custom_fields: Optional[List[str]] = None) -> str:
"""Build the user prompt for field extraction."""
if doc_type == "generic" and custom_fields:
fields_list = ", ".join(custom_fields)
return f"""Extract the following fields from the document: {fields_list}
Return ONLY valid JSON with those exact field names as keys.
If a field is not found, use null. Do not add any explanation.
Document text:
{{text}}"""
else:
return """Extract all relevant information from this document.
Document text:
{text}"""
def extract_fields(
text: str,
doc_type: str = "generic",
custom_fields: Optional[List[str]] = None,
api_key: Optional[str] = None,
api_base: Optional[str] = None,
model: Optional[str] = None,
temperature: float = 0.1,
timeout: int = DEFAULT_TIMEOUT,
) -> Dict[str, Any]:
"""
Extract structured fields from text using AI.
Args:
text: Extracted text from the PDF.
doc_type: Document type (invoice/contract/receipt/etc.).
custom_fields: For generic type, list of field names to extract.
api_key: API key for the AI service. If None, uses env var OPENAI_API_KEY.
api_base: Base URL for the API. Defaults to OpenAI.
model: Model name to use. Defaults to gpt-4o.
temperature: Sampling temperature (0.0 - 1.0).
timeout: Request timeout in seconds.
Returns:
Dictionary of extracted fields.
"""
if not text or len(text.strip()) < 10:
return {}
# Get API credentials
if api_key is None:
import os
api_key = os.environ.get("OPENAI_API_KEY", "")
if not api_key:
raise ValueError(
"API key is required for field extraction. "
"Pass api_key or set OPENAI_API_KEY environment variable."
)
api_base = api_base or DEFAULT_API_BASE
model = model or DEFAULT_MODEL
# Build messages
system_prompt = SYSTEM_PROMPTS.get(doc_type, SYSTEM_PROMPTS["generic"])
user_prompt = build_user_prompt(doc_type, custom_fields).format(text=text[:8000]) # Truncate to 8k chars
messages = [
{"role": "system", "content": system_prompt},
{"role": "user", "content": user_prompt},
]
# Call API
endpoint = f"{api_base.rstrip('/')}/chat/completions"
headers = {
"Content-Type": "application/json",
"Authorization": f"Bearer {api_key}",
}
payload = {
"model": model,
"messages": messages,
"temperature": temperature,
"max_tokens": 2048,
}
try:
response = requests.post(endpoint, headers=headers, json=payload, timeout=timeout)
response.raise_for_status()
except requests.exceptions.Timeout:
raise TimeoutError(f"API request timed out after {timeout}s")
except requests.exceptions.RequestException as e:
raise RuntimeError(f"API request failed: {e}")
result = response.json()
# Parse response
try:
content = result["choices"][0]["message"]["content"]
except (KeyError, IndexError):
raise RuntimeError(f"Unexpected API response format: {result}")
# Extract JSON from response
fields = _parse_json_response(content)
return fields
def _parse_json_response(content: str) -> Dict[str, Any]:
"""
Parse JSON from the AI response content.
Handles cases where the model wraps JSON in markdown code blocks.
"""
# Try direct JSON parse first
try:
return json.loads(content)
except json.JSONDecodeError:
pass
# Try to extract JSON from markdown code block
json_match = re.search(r"```(?:json)?\s*\n?(.*?)\n?```", content, re.DOTALL)
if json_match:
try:
return json.loads(json_match.group(1))
except json.JSONDecodeError:
pass
# Try to find raw JSON object
json_match = re.search(r"\{[\s\S]*\}", content)
if json_match:
try:
return json.loads(json_match.group(0))
except json.JSONDecodeError:
pass
# Fallback: return empty dict
return {}
def extract_fields_batch(
texts: List[str],
doc_type: str = "generic",
custom_fields: Optional[List[str]] = None,
api_key: Optional[str] = None,
api_base: Optional[str] = None,
model: Optional[str] = None,
max_workers: int = 4,
) -> List[Dict[str, Any]]:
"""
Extract fields from multiple texts in parallel.
Note: For high-volume usage, consider implementing async API calls
or using a queue-based approach.
Args:
texts: List of extracted texts from PDFs.
doc_type: Document type.
custom_fields: Custom fields for generic type.
api_key: API key.
api_base: API base URL.
model: Model name.
max_workers: Maximum parallel workers (note: serial execution for simplicity).
Returns:
List of field dictionaries.
"""
results = []
for text in texts:
fields = extract_fields(
text=text,
doc_type=doc_type,
custom_fields=custom_fields,
api_key=api_key,
api_base=api_base,
model=model,
)
results.append(fields)
return results
FILE:scripts/__init__.py
FILE:scripts/ocr_processor.py
#!/usr/bin/env python3
"""
OCR processing for scanned PDFs using pytesseract.
Includes image preprocessing to improve recognition accuracy.
"""
import io
import os
from typing import List, Optional
from PIL import Image, ImageEnhance, ImageFilter, ImageOps
import pytesseract
# ─── Language Code Mapping ────────────────────────────────────────────────────
LANG_CODE_MAP = {
"eng": "eng",
"chi_sim": "chi_sim", # Simplified Chinese
"chi_tra": "chi_tra", # Traditional Chinese
"jpn": "jpn", # Japanese
"kor": "kor", # Korean
"fra": "fra", # French
"deu": "deu", # German
"spa": "spa", # Spanish
"por": "por", # Portuguese
"rus": "rus", # Russian
"ara": "ara", # Arabic
"hin": "hin", # Hindi
}
def get_tesseract_lang_codes(languages: List[str]) -> str:
"""
Convert a list of language names to tesseract language codes.
Args:
languages: List of language names (e.g., ["eng", "chi_sim"]).
Returns:
Tesseract language code string (e.g., "eng+chi_sim").
"""
codes = []
for lang in languages:
code = LANG_CODE_MAP.get(lang, lang)
if code not in codes:
codes.append(code)
return "+".join(codes)
def preprocess_for_ocr(image: Image.Image) -> Image.Image:
"""
Preprocess an image to improve OCR accuracy.
Steps:
1. Convert to grayscale
2. Increase contrast
3. Sharpen
4. Deskew (optional)
Args:
image: PIL Image object.
Returns:
Preprocessed PIL Image object.
"""
# Convert to grayscale
img = image.convert("L")
# Increase contrast
enhancer = ImageEnhance.Contrast(img)
img = enhancer.enhance(1.5)
# Sharpen
enhancer = ImageEnhance.Sharpness(img)
img = enhancer.enhance(1.3)
# Auto-invert if background is dark
extrema = img.getextrema()
if extrema[0] < 50: # Dark background
img = ImageOps.invert(img)
return img
def ocr_image(
image_path: str,
languages: Optional[List[str]] = None,
preprocess: bool = True,
psm: int = 6,
) -> str:
"""
Perform OCR on an image file.
Args:
image_path: Path to the image file.
languages: List of language codes for OCR (e.g., ["eng", "chi_sim"]).
Defaults to ["eng"].
preprocess: Whether to preprocess the image before OCR.
psm: Page segmentation mode (0-13). Default 6 = fully automatic.
Returns:
Recognized text from the image.
"""
if languages is None:
languages = ["eng"]
if not os.path.exists(image_path):
raise FileNotFoundError(f"Image file not found: {image_path}")
img = Image.open(image_path)
if preprocess:
img = preprocess_for_ocr(img)
lang_code = get_tesseract_lang_codes(languages)
# Configure tesseract
config = f"--psm {psm}"
try:
text = pytesseract.image_to_string(img, lang=lang_code, config=config)
except Exception as e:
raise RuntimeError(f"Tesseract OCR failed: {e}")
return text
def ocr_image_bytes(
image_bytes: bytes,
languages: Optional[List[str]] = None,
preprocess: bool = True,
psm: int = 6,
) -> str:
"""
Perform OCR on image bytes.
Args:
image_bytes: Image data as bytes.
languages: List of language codes for OCR.
preprocess: Whether to preprocess the image.
psm: Page segmentation mode.
Returns:
Recognized text from the image.
"""
if languages is None:
languages = ["eng"]
img = Image.open(io.BytesIO(image_bytes))
if preprocess:
img = preprocess_for_ocr(img)
lang_code = get_tesseract_lang_codes(languages)
config = f"--psm {psm}"
try:
text = pytesseract.image_to_string(img, lang=lang_code, config=config)
except Exception as e:
raise RuntimeError(f"Tesseract OCR failed: {e}")
return text
def ocr_pdf_pages(
pdf_path: str,
languages: Optional[List[str]] = None,
preprocess: bool = True,
psm: int = 6,
start_page: int = 0,
end_page: Optional[int] = None,
) -> List[str]:
"""
Perform OCR on each page of a PDF (for scanned PDFs).
Args:
pdf_path: Path to the scanned PDF file.
languages: List of language codes for OCR.
preprocess: Whether to preprocess each page image.
psm: Page segmentation mode.
start_page: 0-based start page.
end_page: 0-based end page (inclusive). None = all pages.
Returns:
List of recognized text per page.
"""
# Import here to avoid circular dependency
from .pdf_extractor import render_page_as_image
if languages is None:
languages = ["eng"]
try:
import fitz
except ImportError:
raise RuntimeError("PyMuPDF (fitz) is required for PDF OCR. Install with: pip install pymupdf")
doc = fitz.open(pdf_path)
total_pages = len(doc)
doc.close()
if end_page is None:
end_page = total_pages - 1
results = []
for page_num in range(start_page, min(end_page + 1, total_pages)):
img_bytes = render_page_as_image(pdf_path, page_num, dpi=300)
text = ocr_image_bytes(img_bytes, languages=languages, preprocess=preprocess, psm=psm)
results.append(text)
return results
def get_ocr_confidence(image_path: str, languages: Optional[List[str]] = None) -> float:
"""
Get OCR confidence score for an image.
Args:
image_path: Path to the image file.
languages: List of language codes.
Returns:
Average confidence score (0.0 - 1.0).
"""
if languages is None:
languages = ["eng"]
if not os.path.exists(image_path):
raise FileNotFoundError(f"Image file not found: {image_path}")
img = Image.open(image_path)
img = preprocess_for_ocr(img)
lang_code = get_tesseract_lang_codes(languages)
try:
data = pytesseract.image_to_data(img, lang=lang_code, output_type=pytesseract.Output.DICT)
confidences = [int(conf) for conf in data["conf"] if conf != "-1"]
if confidences:
return sum(confidences) / len(confidences) / 100.0
return 0.0
except Exception:
return 0.0
FILE:scripts/tier_config.py
#!/usr/bin/env python3
"""
Tier configuration and usage limits for PDF Field Extractor.
Token prefixes: PDF-FREE / PDF-BSC / PDF-STD / PDF-PRO / PDF-ENT
"""
from dataclasses import dataclass
from typing import List, Optional
# ─── Tier Limits ───────────────────────────────────────────────────────────────
TIER_LIMITS = {
"PDF-FREE": {
"pages_per_month": 10,
"doc_types": ["invoice"],
"output_formats": ["text"],
"batch_size": 1,
"ocr_languages": ["eng"],
"custom_fields": False,
"api_access": False,
},
"PDF-BSC": {
"pages_per_month": 200,
"doc_types": ["invoice", "receipt", "license", "id_card"],
"output_formats": ["excel", "text"],
"batch_size": 10,
"ocr_languages": ["eng", "chi_sim"],
"custom_fields": False,
"api_access": False,
},
"PDF-STD": {
"pages_per_month": 1000,
"doc_types": ["invoice", "receipt", "license", "id_card", "contract", "bank_statement", "express", "generic"],
"output_formats": ["excel", "json", "text"],
"batch_size": 50,
"ocr_languages": ["eng", "chi_sim", "chi_tra", "jpn", "kor"],
"custom_fields": True,
"api_access": False,
},
"PDF-PRO": {
"pages_per_month": float("inf"),
"doc_types": ["invoice", "receipt", "license", "id_card", "contract", "bank_statement", "express", "generic"],
"output_formats": ["excel", "json", "text"],
"batch_size": float("inf"),
"ocr_languages": ["eng", "chi_sim", "chi_tra", "jpn", "kor"],
"custom_fields": True,
"api_access": True,
},
"PDF-ENT": {
"pages_per_month": float("inf"),
"doc_types": ["invoice", "receipt", "license", "id_card", "contract", "bank_statement", "express", "generic"],
"output_formats": ["excel", "json", "text"],
"batch_size": float("inf"),
"ocr_languages": ["eng", "chi_sim", "chi_tra", "jpn", "kor", "fra", "deu", "spa", "por", "rus"],
"custom_fields": True,
"api_access": True,
},
}
# ─── Document Type Mapping ──────────────────────────────────────────────────────
DOC_TYPE_ALIASES = {
"invoice": ["invoice", "Invoice", "增值税发票", "普通发票", "电子发票", "发票"],
"contract": ["contract", "Contract", "合同", "协议书", "agreement"],
"receipt": ["receipt", "Receipt", "收据", "小票", "凭据"],
"bank_statement": ["bank_statement", "Bank Statement", "银行对账单", "银行流水", "对账单"],
"license": ["license", "License", "营业执照", "经营许可证"],
"id_card": ["id_card", "ID Card", "身份证", "护照", "passport", "证件", "ID卡"],
"express": ["express", "Express", "快递单", "运单", "物流单"],
"generic": ["generic", "Generic", "其他", "文档", "Document", "document"],
}
DOC_TYPE_FIELDS = {
"invoice": ["Invoice Number", "Date", "Amount", "Buyer", "Seller", "Line Items", "Tax Rate", "Invoice Code", "Notes"],
"contract": ["Contract Number", "Signing Date", "Expiration Date", "Amount", "Party A", "Party B", "Address", "Contact Person", "Default Terms", "Termination Terms", "Payment Terms"],
"receipt": ["Date", "Amount", "Payee", "Items", "Line Items", "Tip"],
"bank_statement": ["Date", "Transaction Amount", "Counterparty", "Balance", "Transaction Type", "Description"],
"license": ["Unified Social Credit Code", "Company Name", "Legal Representative", "Registered Capital", "Registered Address", "Business Scope"],
"id_card": ["Name", "Gender", "Date of Birth", "Nationality", "ID Number", "Expiry Date"],
"express": ["Waybill Number", "Sender", "Recipient", "Address", "Weight", "Shipping Cost"],
"generic": [], # User-defined
}
@dataclass
class TierConfig:
"""Tier configuration for PDF Field Extractor."""
tier: str = "PDF-FREE"
def get_limits(self) -> dict:
"""Return the limits for the current tier."""
return TIER_LIMITS.get(self.tier, TIER_LIMITS["PDF-FREE"])
def check_limits(
self,
pages: int = 0,
doc_type: Optional[str] = None,
output_format: Optional[str] = None,
batch_size: Optional[int] = None,
use_custom_fields: bool = False,
) -> None:
"""
Check if the requested usage exceeds tier limits.
Raises ValueError if any limit is exceeded.
"""
limits = self.get_limits()
# Check page limit
if pages > limits["pages_per_month"]:
raise ValueError(
f"Page limit exceeded: {pages} pages requested, "
f"limit is {limits['pages_per_month']} pages/month for {self.tier}"
)
# Check document type
if doc_type is not None and doc_type not in limits["doc_types"]:
raise ValueError(
f"Document type '{doc_type}' not supported in {self.tier}. "
f"Supported types: {limits['doc_types']}"
)
# Check output format
if output_format is not None and output_format not in limits["output_formats"]:
raise ValueError(
f"Output format '{output_format}' not supported in {self.tier}. "
f"Supported formats: {limits['output_formats']}"
)
# Check batch size
if batch_size is not None and batch_size > limits["batch_size"]:
raise ValueError(
f"Batch size {batch_size} exceeds limit of {limits['batch_size']} "
f"for {self.tier}"
)
# Check custom fields
if use_custom_fields and not limits["custom_fields"]:
raise ValueError(
f"Custom fields not supported in {self.tier}. "
f"Upgrade to Standard or higher."
)
def supports_doc_type(self, doc_type: str) -> bool:
"""Check if this tier supports the given document type."""
return doc_type in self.get_limits()["doc_types"]
def supports_format(self, fmt: str) -> bool:
"""Check if this tier supports the given output format."""
return fmt in self.get_limits()["output_formats"]
def get_ocr_languages(self) -> List[str]:
"""Get the list of OCR languages supported by this tier."""
return self.get_limits()["ocr_languages"]
def get_default_fields_for_doc_type(doc_type: str) -> List[str]:
"""Return the default extraction fields for a document type."""
return DOC_TYPE_FIELDS.get(doc_type, [])
def resolve_doc_type(doc_type_input: str) -> str:
"""Resolve a user-input doc type string to a canonical type."""
doc_type_input_lower = doc_type_input.lower().strip()
for canonical, aliases in DOC_TYPE_ALIASES.items():
if doc_type_input_lower in [a.lower() for a in aliases] or doc_type_input_lower == canonical:
return canonical
return "generic"
FILE:scripts/pdf_extractor.py
#!/usr/bin/env python3
"""
PDF text extraction using PyMuPDF + pdfplumber.
Handles both text-based PDFs and scanned PDFs (detected by absence of text layer).
"""
import io
import os
from dataclasses import dataclass
from typing import List, Optional, Tuple, Dict, Any
import fitz # PyMuPDF
import pdfplumber
@dataclass
class PDFExtractResult:
"""Result of PDF text extraction."""
full_text: str
tables: List[List[List[str]]] # List of tables, each table is list of rows
page_count: int
is_scanned: bool # True if no text layer found (needs OCR)
page_texts: List[str] # Text per page
metadata: Dict[str, Any] # PDF metadata
def extract_pdf_text(pdf_path: str, password: Optional[str] = None) -> PDFExtractResult:
"""
Extract text and tables from a PDF file.
Uses PyMuPDF for fast text extraction and pdfplumber for table extraction.
Automatically detects if a PDF is scanned (no text layer).
Args:
pdf_path: Path to the PDF file.
password: Optional password for encrypted PDFs.
Returns:
PDFExtractResult with full_text, tables, page_count, is_scanned, page_texts, metadata.
"""
if not os.path.exists(pdf_path):
raise FileNotFoundError(f"PDF file not found: {pdf_path}")
# Step 1: Try PyMuPDF for text extraction
page_texts = []
full_text_parts = []
metadata = {}
try:
doc = fitz.open(pdf_path)
if doc.is_encrypted and password:
doc.authenticate(password)
metadata = {
"title": doc.metadata.get("title", ""),
"author": doc.metadata.get("author", ""),
"subject": doc.metadata.get("subject", ""),
"creator": doc.metadata.get("creator", ""),
"producer": doc.metadata.get("producer", ""),
"creation_date": doc.metadata.get("creationDate", ""),
}
for page_num in range(len(doc)):
page = doc[page_num]
text = page.get_text()
page_texts.append(text)
full_text_parts.append(text)
doc.close()
except Exception as e:
raise RuntimeError(f"Failed to extract text from PDF with PyMuPDF: {e}")
full_text = "\n".join(full_text_parts)
# Step 2: Detect if scanned (no text layer)
is_scanned = len(full_text.strip()) < 50 # Very little text = likely scanned
# Step 3: Extract tables with pdfplumber
tables = []
try:
with pdfplumber.open(pdf_path, password=password or "") as pdf:
for page in pdf.pages:
page_tables = page.extract_tables()
if page_tables:
# pdfplumber returns list of tables per page
for table in page_tables:
if table:
tables.append(table)
except Exception as e:
# Table extraction failure is non-fatal
pass
result = PDFExtractResult(
full_text=full_text,
tables=tables,
page_count=len(page_texts),
is_scanned=is_scanned,
page_texts=page_texts,
metadata=metadata,
)
return result
def extract_page_text(pdf_path: str, page_num: int, password: Optional[str] = None) -> str:
"""
Extract text from a specific page of a PDF.
Args:
pdf_path: Path to the PDF file.
page_num: 0-based page number.
password: Optional password for encrypted PDFs.
Returns:
Text content of the specified page.
"""
try:
doc = fitz.open(pdf_path)
if doc.is_encrypted and password:
doc.authenticate(password)
if page_num < 0 or page_num >= len(doc):
raise IndexError(f"Page {page_num} out of range (0-{len(doc)-1})")
page = doc[page_num]
text = page.get_text()
doc.close()
return text
except Exception as e:
raise RuntimeError(f"Failed to extract page {page_num}: {e}")
def extract_tables_from_page(pdf_path: str, page_num: int, password: Optional[str] = None) -> List[List[List[str]]]:
"""
Extract tables from a specific page of a PDF.
Args:
pdf_path: Path to the PDF file.
page_num: 0-based page number.
password: Optional password for encrypted PDFs.
Returns:
List of tables, each table is a list of rows (each row is a list of cell strings).
"""
tables = []
try:
with pdfplumber.open(pdf_path, password=password or "") as pdf:
if page_num < 0 or page_num >= len(pdf.pages):
raise IndexError(f"Page {page_num} out of range")
page = pdf.pages[page_num]
page_tables = page.extract_tables()
if page_tables:
tables = [t for t in page_tables if t]
except Exception as e:
raise RuntimeError(f"Failed to extract tables from page {page_num}: {e}")
return tables
def get_pdf_info(pdf_path: str) -> Dict[str, Any]:
"""
Get basic PDF information without full text extraction.
Args:
pdf_path: Path to the PDF file.
Returns:
Dictionary with page_count, is_encrypted, metadata.
"""
if not os.path.exists(pdf_path):
raise FileNotFoundError(f"PDF file not found: {pdf_path}")
doc = fitz.open(pdf_path)
info = {
"page_count": len(doc),
"is_encrypted": doc.is_encrypted,
"metadata": dict(doc.metadata),
}
doc.close()
return info
def render_page_as_image(pdf_path: str, page_num: int, dpi: int = 300) -> bytes:
"""
Render a PDF page as an image (for OCR processing).
Args:
pdf_path: Path to the PDF file.
page_num: 0-based page number.
dpi: Resolution for rendering (default 300 DPI for OCR).
Returns:
PNG image data as bytes.
"""
try:
doc = fitz.open(pdf_path)
if page_num < 0 or page_num >= len(doc):
raise IndexError(f"Page {page_num} out of range")
page = doc[page_num]
mat = fitz.Matrix(dpi / 72, dpi / 72) # Scale for DPI
pix = page.get_pixmap(matrix=mat, alpha=False)
img_data = pix.tobytes("png")
doc.close()
return img_data
except Exception as e:
raise RuntimeError(f"Failed to render page {page_num} as image: {e}")
def save_page_as_image(pdf_path: str, page_num: int, output_path: str, dpi: int = 300) -> None:
"""
Render a PDF page as an image file.
Args:
pdf_path: Path to the PDF file.
page_num: 0-based page number.
output_path: Path to save the output image.
dpi: Resolution for rendering.
"""
img_data = render_page_as_image(pdf_path, page_num, dpi)
with open(output_path, "wb") as f:
f.write(img_data)
FILE:scripts/output_generator.py
#!/usr/bin/env python3
"""
Output generation for PDF Field Extractor.
Supports Excel (.xlsx) and JSON output formats.
Also builds Feishu-compatible message content.
"""
import json
import os
from datetime import datetime
from typing import Any, Dict, List, Optional
import openpyxl
from openpyxl.styles import Font, PatternFill, Alignment, Border, Side
from openpyxl.utils import get_column_letter
# ─── Default Styles ───────────────────────────────────────────────────────────
HEADER_FILL = PatternFill(start_color="366092", end_color="366092", fill_type="solid")
HEADER_FONT = Font(name="Arial", bold=True, color="FFFFFF", size=11)
CELL_FONT = Font(name="Arial", size=10)
BORDER_SIDE = Side(style="thin", color="CCCCCC")
CELL_BORDER = Border(left=BORDER_SIDE, right=BORDER_SIDE, top=BORDER_SIDE, bottom=BORDER_SIDE)
HEADER_ALIGNMENT = Alignment(horizontal="center", vertical="center", wrap_text=True)
CELL_ALIGNMENT = Alignment(horizontal="left", vertical="center", wrap_text=True)
def generate_excel(
results: List[Dict[str, Any]],
output_path: str,
sheet_name: str = "Sheet1",
include_metadata: bool = True,
) -> str:
"""
Generate an Excel file from extraction results.
Args:
results: List of result dictionaries, one per PDF.
output_path: Path to save the Excel file.
sheet_name: Name of the worksheet.
include_metadata: Whether to include filename and timestamp metadata columns.
Returns:
Path to the generated Excel file.
"""
if not results:
# Create empty workbook
wb = openpyxl.Workbook()
ws = wb.active
ws.title = sheet_name
wb.save(output_path)
return output_path
# Collect all unique field keys across all results
all_keys = set()
for result in results:
all_keys.update(result.keys())
# Filter out internal fields
internal_fields = {"_filename", "_timestamp", "_doc_type", "_page_count", "_is_scanned"}
display_keys = sorted([k for k in all_keys if k not in internal_fields])
# Build column headers
if include_metadata:
headers = ["Filename", "Extraction Time", "Document Type"] + display_keys
else:
headers = display_keys
# Create workbook
wb = openpyxl.Workbook()
ws = wb.active
ws.title = sheet_name
# Write headers
for col_idx, header in enumerate(headers, start=1):
cell = ws.cell(row=1, column=col_idx, value=header)
cell.font = HEADER_FONT
cell.fill = HEADER_FILL
cell.alignment = HEADER_ALIGNMENT
cell.border = CELL_BORDER
# Write data rows
for row_idx, result in enumerate(results, start=2):
if include_metadata:
ws.cell(row=row_idx, column=1, value=result.get("_filename", "")).font = CELL_FONT
ws.cell(row=row_idx, column=1).border = CELL_BORDER
ws.cell(row=row_idx, column=1).alignment = CELL_ALIGNMENT
ws.cell(row=row_idx, column=2, value=result.get("_timestamp", "")).font = CELL_FONT
ws.cell(row=row_idx, column=2).border = CELL_BORDER
ws.cell(row=row_idx, column=2).alignment = CELL_ALIGNMENT
doc_type_display = {
"invoice": "Invoice",
"contract": "Contract",
"receipt": "Receipt",
"bank_statement": "Bank Statement",
"license": "License",
"id_card": "ID Card/Passport",
"express": "Express",
"generic": "GenericDocument",
}.get(result.get("_doc_type", ""), result.get("_doc_type", ""))
ws.cell(row=row_idx, column=3, value=doc_type_display).font = CELL_FONT
ws.cell(row=row_idx, column=3).border = CELL_BORDER
ws.cell(row=row_idx, column=3).alignment = CELL_ALIGNMENT
col_offset = 4
else:
col_offset = 1
for col_idx, key in enumerate(display_keys, start=col_offset):
value = result.get(key, "")
# Convert complex types to string
if isinstance(value, (dict, list)):
value = json.dumps(value, ensure_ascii=False)
cell = ws.cell(row=row_idx, column=col_idx, value=value)
cell.font = CELL_FONT
cell.border = CELL_BORDER
cell.alignment = CELL_ALIGNMENT
# Auto-adjust column widths
for col_idx, header in enumerate(headers, start=1):
col_letter = get_column_letter(col_idx)
max_length = len(str(header))
for row_idx in range(2, len(results) + 2):
cell_value = ws.cell(row=row_idx, column=col_idx).value
if cell_value:
max_length = max(max_length, len(str(cell_value)))
adjusted_width = min(max_length + 2, 50) # Cap at 50
ws.column_dimensions[col_letter].width = adjusted_width
# Freeze header row
ws.freeze_panes = "A2"
# Save
os.makedirs(os.path.dirname(output_path) or ".", exist_ok=True)
wb.save(output_path)
return output_path
def generate_json(
results: List[Dict[str, Any]],
output_path: str,
pretty: bool = True,
) -> str:
"""
Generate a JSON file from extraction results.
Args:
results: List of result dictionaries.
output_path: Path to save the JSON file.
pretty: Whether to use pretty printing.
Returns:
Path to the generated JSON file.
"""
# Clean up internal fields for output
cleaned_results = []
internal_fields = {"_filename", "_timestamp", "_doc_type", "_page_count", "_is_scanned"}
for result in results:
cleaned = {k: v for k, v in result.items() if k not in internal_fields}
cleaned_results.append(cleaned)
os.makedirs(os.path.dirname(output_path) or ".", exist_ok=True)
with open(output_path, "w", encoding="utf-8") as f:
if pretty:
json.dump(cleaned_results, f, ensure_ascii=False, indent=2)
else:
json.dump(cleaned_results, f, ensure_ascii=False)
return output_path
def build_feishu_message(
results: List[Dict[str, Any]],
doc_type: str = "generic",
max_items: int = 10,
) -> Dict[str, Any]:
"""
Build Feishu interactive card message content for extraction results.
Args:
results: List of result dictionaries.
doc_type: Document type for display.
max_items: Maximum number of items to show in the summary.
Returns:
Feishu card content dictionary.
"""
if not results:
return {
"msg_type": "text",
"content": {"text": "No data extracted"},
}
doc_type_display = {
"invoice": "Invoice",
"contract": "Contract",
"receipt": "Receipt",
"bank_statement": "Bank Statement",
"license": "License",
"id_card": "ID Card/Passport",
"express": "Express",
"generic": "GenericDocument",
}.get(doc_type, doc_type)
total = len(results)
# Show summary of first few items
items = []
for i, result in enumerate(results[:max_items]):
filename = result.get("_filename", f"Document{i+1}")
# Show key fields summary
key_fields = {k: v for k, v in result.items() if not k.startswith("_") and v}
if key_fields:
first_field = next(iter(key_fields.items()), None)
if first_field:
summary = f"**{first_field[0]}**: {str(first_field[1])[:50]}"
else:
summary = "No data extracted"
items.append({"tag": "div", "text": {"tag": "lark_md", "content": f"📄 {filename}: {summary}"}})
if total > max_items:
items.append({"tag": "div", "text": {"tag": "lark_md", "content": f"_...more {total - max_items} file(s)_"}})
content = {
"msg_type": "interactive",
"card": {
"header": {
"title": {"tag": "plain_text", "content": f"📊 PDF Field Extraction Results"},
"subtitle": {"tag": "plain_text", "content": f"Document Type: {doc_type_display} | Total {total} file(s)"},
},
"elements": items + [
{"tag": "hr"},
{
"tag": "note",
"elements": [
{"tag": "plain_text", "content": f"Extraction Time: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}"}
],
},
],
},
}
return content
def build_feishu_text_message(results: List[Dict[str, Any]], doc_type: str = "generic") -> Dict[str, str]:
"""
Build a simple Feishu text message for extraction results.
Args:
results: List of result dictionaries.
doc_type: Document type.
Returns:
Dictionary with msg_type and content.
"""
if not results:
return {"msg_type": "text", "content": "No data extracted"}
lines = [f"✅ PDF Extraction Complete（Total {len(results)} file(s)）\n"]
doc_type_display = {
"invoice": "Invoice",
"contract": "Contract",
"receipt": "Receipt",
"bank_statement": "Bank Statement",
"license": "License",
"id_card": "ID Card/Passport",
"express": "Express",
"generic": "GenericDocument",
}.get(doc_type, doc_type)
lines.append(f"📋 Document Type: {doc_type_display}\n")
for i, result in enumerate(results[:5], 1):
filename = result.get("_filename", f"Document{i}")
lines.append(f"\n📄 {i}. {filename}")
for key, value in result.items():
if key.startswith("_"):
continue
if value:
value_str = str(value)[:100]
lines.append(f" • {key}: {value_str}")
if len(results) > 5:
lines.append(f"\n...more {len(results) - 5} file(s)")
return {"msg_type": "text", "content": "\n".join(lines)}
def merge_results(
results: List[Dict[str, Any]],
source_filenames: Optional[List[str]] = None,
doc_types: Optional[List[str]] = None,
) -> List[Dict[str, Any]]:
"""
Merge multiple extraction results with metadata.
Args:
results: List of field dictionaries.
source_filenames: Optional list of source filenames.
doc_types: Optional list of document types per result.
Returns:
List of enriched result dictionaries.
"""
timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
merged = []
for i, result in enumerate(results):
enriched = dict(result)
enriched["_filename"] = source_filenames[i] if source_filenames and i < len(source_filenames) else f"document_{i+1}"
enriched["_timestamp"] = timestamp
enriched["_doc_type"] = doc_types[i] if doc_types and i < len(doc_types) else "generic"
merged.append(enriched)
return merged
Web Change Monitor — Generic webpage monitoring tool. Configure URL list → Skill checks for changes at set frequency → Feishu push notifications. Not tied to...
---
name: web-watcher-pro
description: "Web Change Monitor — Generic webpage monitoring tool. Configure URL list → Skill checks for changes at set frequency → Feishu push notifications. Not tied to any platform, fully generic. Triggers: webpage monitor, page change detection, URL monitor, price change monitor, competitor monitoring, website update alert, inventory monitoring, stock change detection, website monitor."
override-tools: []
---
# Web Watcher Pro
Configure any URL → Skill checks for changes at set frequency → Feishu notification.
Fully generic tool, not tied to any platform. Use cases: competitor new product alerts, price monitoring, inventory tracking, content change detection, forum thread monitoring.
## Quick Start
### Add a Monitored URL
```
User: Monitor this page: https://example.com/product/12345
```
Skill:
1. Fetches page, computes content hash
2. Asks for detection mode and frequency (or uses defaults)
3. Saves monitoring task, begins checking
### Check Status
```
User: Show my monitored URLs
User: Which URLs have changed?
```
### Remove Monitor
```
User: Remove monitoring for https://example.com/product/12345
```
---
## Detection Modes
| Mode | Description | Use Case |
|------|-------------|----------|
| `hash` | MD5 hash of full HTML, triggers on any change | General, any page |
| `keyword` | Triggers when keyword appears/disappears | Inventory, price, specific content |
| `selector` | CSS selector extracts specific DOM elements for comparison | List pages (product listings, search results) |
| `regex` | Regex-defined trigger condition | Complex pattern matching |
### Examples
```
User: Monitor this page, alert me when price drops below 99
[URL]
User: Use keyword mode, alert when product name contains "New Arrival"
[URL]
```
---
## Tiered Features
| Feature | FREE | PRO |
|---------|:----:|:---:|
| Monitored URLs | 3 | Unlimited |
| Check frequency | Every 24h | Every 1h |
| Detection mode | Hash only | Hash + Keyword + Selector + Regex |
| Change history | — | 30 days |
| Feishu push | — | Yes |
| Price | Free | $0.01/call |
---
## Detection Modes Detail
### Hash Mode
MD5 hash of full page HTML. Triggers on any content change.
### Keyword Mode
Monitors for keyword appearance/disappearance. Case-insensitive.
### Selector Mode
CSS selector extracts specific DOM elements. Compares extracted text between checks.
### Regex Mode
Regex pattern matched against HTML. Triggers on pattern match change.
---
## Change History
```
User: What pages have changed recently?
User: Show change history for https://xxx.com
```
Returns: change timestamp, change summary, time since last change.
---
## Core Script
See `scripts/monitor.py` for full implementation:
```python
from scripts.monitor import WebMonitor
monitor = WebMonitor(tier="pro")
monitor.add_task(
url="https://example.com/product/123",
name="Product A Monitor",
mode="hash",
frequency="6h",
)
monitor.check_all() # Triggers Feishu push on changes
monitor.list_tasks()
monitor.remove_task(url="https://example.com/product/123")
```
---
## Technical Implementation
- **Fetching**: Playwright (headless) with random UA and anti-detection delays
- **Detection**: MD5 hash / keyword match / CSS selector / regex
- **Storage**: SQLite at `/tmp/web-watcher-pro/history.db`
- **Push**: Feishu IM notifications with customizable templates
- **Anti-ban**: Request intervals + random delays + 3x auto-retry
---
## Security Notes
- **SSRF Protection**: `fetch_page()` validates all URLs before sending to Playwright. Blocks: non-HTTP(S) schemes (file://, ftp://, data:, javascript:, etc.), localhost, 127.0.0.1, private IP ranges (10.x.x.x, 172.16-31.x.x, 192.168.x.x), link-local (169.254.x.x including AWS metadata 169.254.169.254), and IPv6 localhost. Unsafe URLs return `None` instead of triggering a network request.
- **Subprocess execution**: Uses `node -e` subprocess for Playwright browser automation (anti-detection scraping). Node.js required. Timeout: 30s. Subprocess uses list form (not shell=True), eliminating command injection risk.
- **Data storage**: Uses `/tmp/web-watcher-pro/` for SQLite DB and config (no home directory write).
- **Billing data**: `FEISHU_USER_ID` transmitted to `skillpay.me/api/v1/billing` for per-call charging.
---
## Billing
- Billing via `skillpay.me/api/v1/billing/charge`
- User data transmitted to SkillPay for billing identification
- $0.01 USD per check call (PRO tier)
---
## Required Environment Variables
| Variable | Description |
|----------|-------------|
| `FEISHU_USER_ID` | User open_id for billing |
| `SKILL_BILLING_API_KEY` | SkillPay Builder API Key |
| `SKILL_BILLING_SKILL_ID` | SkillPay Skill ID (default: web-watcher-pro) |
---
## Common Errors
| Error | Cause | Solution |
|-------|-------|----------|
| `Failed to fetch page` | Page blocked or unavailable | Check URL accessibility |
| `Invalid mode` | Unsupported detection mode | Use: hash, keyword, selector, regex |
| `TASK_LIMIT_EXCEEDED` | URL count exceeds tier limit | Upgrade or remove existing URLs |
FILE:scripts/monitor.py
#!/usr/bin/env python3
"""
Web Change Monitor — core monitoring engine.
Fetches pages with Playwright, compares content, triggers notifications.
"""
import hashlib
import json
import os
import random
import re
import signal
import sqlite3
import sys
import time
from dataclasses import dataclass, asdict, field
from datetime import datetime, timezone
from pathlib import Path
from typing import Optional, List
# Paths
DB_PATH = "/tmp/web-watcher-pro/history.db"
SCRIPT_DIR = Path(__file__).parent.resolve()
# User Agent Pool
UA_POOL = [
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Chrome/120.0.0.0 Safari/537.36",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 Chrome/120.0.0.0 Safari/537.36",
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/120.0.0.0 Safari/537.36",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) Gecko/20100101 Firefox/121.0",
]
# Frequency map (seconds)
FREQUENCY_SECONDS = {
"15m": 15 * 60,
"30m": 30 * 60,
"1h": 60 * 60,
"6h": 6 * 60 * 60,
"12h": 12 * 60 * 60,
"24h": 24 * 60 * 60,
}
# Detection modes
MODE_HASH = "hash"
MODE_KEYWORD = "keyword"
MODE_SELECTOR = "selector"
MODE_REGEX = "regex"
VALID_MODES = [MODE_HASH, MODE_KEYWORD, MODE_SELECTOR, MODE_REGEX]
# Tier limits (FREE / PRO)
TIER_LIMITS = {
"free": {"max_urls": 3, "max_frequency": "24h", "history_days": 0},
"pro": {"max_urls": float("inf"), "max_frequency": "1h", "history_days": 30},
}
# Dataclasses
@dataclass
class MonitorTask:
url: str
name: str
mode: str = MODE_HASH
frequency: str = "24h"
keyword: Optional[str] = None
selector: Optional[str] = None
regex: Optional[str] = None
last_hash: Optional[str] = None
last_content: Optional[str] = None
last_check: Optional[str] = None
created_at: str = field(default_factory=lambda: datetime.now(timezone.utc).isoformat())
change_count: int = 0
def to_dict(self) -> dict:
return asdict(self)
@classmethod
def from_dict(cls, d: dict) -> "MonitorTask":
return cls(**{k: v for k, v in d.items() if k in cls.__dataclass_fields__})
@dataclass
class ChangeRecord:
url: str
name: str
detected_at: str
change_type: str
detail: str
mode: str
def to_dict(self) -> dict:
return asdict(self)
# Database
def _get_db() -> sqlite3.Connection:
Path(DB_PATH).parent.mkdir(parents=True, exist_ok=True)
conn = sqlite3.connect(DB_PATH)
conn.execute("""
CREATE TABLE IF NOT EXISTS monitor_tasks (
url TEXT PRIMARY KEY,
name TEXT NOT NULL,
mode TEXT DEFAULT 'hash',
frequency TEXT DEFAULT '24h',
keyword TEXT,
selector TEXT,
regex TEXT,
last_hash TEXT,
last_content TEXT,
last_check TEXT,
created_at TEXT NOT NULL,
change_count INTEGER DEFAULT 0
)
""")
conn.execute("""
CREATE TABLE IF NOT EXISTS change_logs (
id INTEGER PRIMARY KEY AUTOINCREMENT,
url TEXT NOT NULL,
name TEXT NOT NULL,
detected_at TEXT NOT NULL,
change_type TEXT NOT NULL,
detail TEXT,
mode TEXT
)
""")
conn.commit()
return conn
# Playwright Fetcher
# ─── SSRF Protection ─────────────────────────────────────────────────────────
_BLOCKED_SCHEMES = frozenset(["file", "ftp", "data", "javascript", "mailto", "tel"])
_LOCALHOST_NAMES = frozenset(["localhost", "localhost.localdomain", "ip6-localhost", "ip6-loopback"])
_PRIVATE_IP_PATTERNS = [
r"127\.\d{1,3}\.\d{1,3}\.\d{1,3}", # 127.x.x.x (loopback)
r"10\.\d{1,3}\.\d{1,3}\.\d{1,3}", # 10.x.x.x (private)
r"172\.(?:1[6-9]|2\d|3[01])\.\d{1,3}\.\d{1,3}", # 172.16-31.x.x
r"192\.168\.\d{1,3}\.\d{1,3}", # 192.168.x.x (private)
r"169\.254\.(?:\d{1,3}\.)?\d{1,3}", # 169.254.x.x (link-local / AWS metadata)
r"0\.\d{1,3}\.\d{1,3}\.\d{1,3}", # 0.x.x.x
r"(?:[fF][cCdD][0-9a-fA-F]{2}:[0-9a-fA-F:]+)", # IPv6 fc00::/7
r"(?:[fF][eE][89aAbB][0-9a-fA-F:]+[%\w]*)", # IPv6 fe80::/10
r"::1(?:\]|\Z)", # ::1 localhost IPv6
r"\[?::1\]?(?:\]|\Z)", # [::1] bracketed
]
_PRIVATE_IP_RE = re.compile("(?:" + "|".join(_PRIVATE_IP_PATTERNS) + ")$", re.IGNORECASE)
def _is_url_safe(url: str) -> bool:
"""
Validate URL to prevent SSRF attacks.
Blocks: non-HTTP(S) schemes, localhost, private/internal IPs, AWS metadata.
Returns True if URL is safe to fetch.
"""
try:
from urllib.parse import urlparse
except ImportError:
from urlparse import urlparse
parsed = urlparse(url)
scheme = parsed.scheme.lower()
hostname = parsed.hostname or ""
# Scheme check — HTTP(S) only
if scheme not in ("http", "https"):
return False
# Hostname checks
hostname_lower = hostname.lower()
if hostname_lower in _LOCALHOST_NAMES:
return False
# IP address checks
if _PRIVATE_IP_RE.match(hostname):
return False
return True
def _is_url_safe(url: str) -> bool:
"""
Validate URL to prevent SSRF attacks.
Blocks: non-HTTP(S) schemes, localhost, private/internal IPs, AWS metadata endpoint.
Returns True if URL is safe to fetch.
"""
try:
from urllib.parse import urlparse
except ImportError:
from urlparse import urlparse
parsed = urlparse(url)
scheme = parsed.scheme.lower()
hostname = parsed.hostname or ""
# Scheme check — HTTP(S) only
if scheme not in ("http", "https"):
return False
# Hostname checks
hostname_lower = hostname.lower()
if hostname_lower in _LOCALHOST_NAMES:
return False
# IP address checks (including bracketed IPv6)
if _PRIVATE_IP_RE.match(hostname):
return False
return True
def fetch_page(url: str, timeout_ms: int = 15000) -> Optional[str]:
"""
Fetch page content using Playwright (Node.js subprocess).
SSRF protection: rejects non-HTTP(S) URLs, localhost, private/internal IPs.
Returns HTML string or None on failure.
"""
# SSRF guard — reject unsafe URLs before any network call
if not _is_url_safe(url):
return None
import subprocess
# Encode URL for safe embedding in JS string
import json as _json
safe_url = _json.dumps(url)
script = f"""
const {{ chromium }} = require('playwright');
(async () => {{
const browser = await chromium.launch({{ headless: true }});
const page = await browser.newPage();
// Block access to local/internal resources
await page.route('**/*', route => {{
const reqUrl = route.request().url();
if (reqUrl.startsWith('file://') || reqUrl.startsWith('ftp://')) {{
route.abort();
return;
}}
route.continue();
}});
await page.setExtraHTTPHeaders({{ 'Accept-Language': 'zh-CN,zh;q=0.9' }});
await page.goto({safe_url}, {{ waitUntil: 'networkidle', timeout: {timeout_ms} }});
const content = await page.content();
await browser.close();
console.log(JSON.stringify({{ ok: true, content }}));
}})().catch(e => {{ console.log(JSON.stringify({{ ok: false, error: e.message }})); process.exit(1); }});
"""
try:
result = subprocess.run(
["node", "-e", script],
capture_output=True, text=True, timeout=30
)
if result.returncode != 0:
return None
data = json.loads(result.stdout.strip())
if data.get("ok"):
return data["content"]
except Exception:
pass
return None
# Content extraction
def extract_by_selector(html: str, selector: str) -> str:
"""Extract text from HTML using CSS selector via BeautifulSoup."""
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, "html.parser")
elements = soup.select(selector)
return "|".join(e.get_text(strip=True) for e in elements)
def extract_by_regex(html: str, pattern: str) -> str:
"""Extract content matching regex pattern."""
try:
matches = re.findall(pattern, html)
return "|".join(matches)
except re.error:
return ""
def compute_hash(content: str) -> str:
return hashlib.md5(content.encode("utf-8", errors="ignore")).hexdigest()
# Detect change
def detect_change(task: MonitorTask, current_content: str) -> tuple[bool, str, str]:
"""
Returns (changed: bool, change_type: str, detail: str)
"""
current_hash = compute_hash(current_content)
if task.mode == MODE_HASH:
if task.last_hash and task.last_hash != current_hash:
return True, "content_changed", "Page content changed"
return False, "", ""
elif task.mode == MODE_KEYWORD:
if not task.keyword:
return False, "", ""
keyword_lower = task.keyword.lower()
content_lower = current_content.lower()
prev_lower = (task.last_content or "").lower()
keyword_now = keyword_lower in content_lower
keyword_was = keyword_lower in prev_lower
if keyword_now != keyword_was:
triggered = "appeared" if keyword_now else "disappeared"
return True, f"keyword_{triggered}", f"Keyword '{task.keyword}' {triggered}"
return False, "", ""
elif task.mode == MODE_SELECTOR:
if not task.selector:
return False, "", ""
curr_items = extract_by_selector(current_content, task.selector)
prev_items = task.last_content or ""
if curr_items != prev_items:
return True, "selector_changed", f"Selector content changed: {curr_items[:100]}"
return False, "", ""
elif task.mode == MODE_REGEX:
if not task.regex:
return False, "", ""
curr_match = extract_by_regex(current_content, task.regex)
prev_match = task.last_content or ""
if curr_match != prev_match:
return True, "regex_matched", f"Regex match changed: {curr_match[:100]}"
return False, "", ""
return False, "", ""
# WebMonitor class
class WebMonitor:
def __init__(self, tier: str = "free"):
self.tier = tier
self.conn = _get_db()
def add_task(
self,
url: str,
name: str,
mode: str = MODE_HASH,
frequency: str = "24h",
keyword: Optional[str] = None,
selector: Optional[str] = None,
regex: Optional[str] = None,
) -> dict:
"""Add or update a monitoring task."""
now = datetime.now(timezone.utc).isoformat()
# Check tier limit
limit = TIER_LIMITS.get(self.tier, TIER_LIMITS["free"])
existing = self.list_tasks()
if len(existing) >= limit["max_urls"]:
return {"ok": False, "error": f"{self.tier} tier limit: max {limit['max_urls']} URLs"}
if mode not in VALID_MODES:
return {"ok": False, "error": f"Invalid mode. Choose: {VALID_MODES}"}
self.conn.execute(
"""INSERT OR REPLACE INTO monitor_tasks
(url, name, mode, frequency, keyword, selector, regex, created_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?)""",
(url, name, mode, frequency, keyword, selector, regex, now)
)
self.conn.commit()
return {"ok": True, "url": url, "name": name}
def remove_task(self, url: str) -> dict:
self.conn.execute("DELETE FROM monitor_tasks WHERE url = ?", (url,))
self.conn.commit()
return {"ok": True, "url": url}
def list_tasks(self) -> List[dict]:
rows = self.conn.execute(
"SELECT url, name, mode, frequency, keyword, selector, regex, last_hash, last_content, last_check, created_at, change_count FROM monitor_tasks"
).fetchall()
return [
{"url": r[0], "name": r[1], "mode": r[2], "frequency": r[3],
"keyword": r[4], "selector": r[5], "regex": r[6],
"last_hash": r[7], "last_content": r[8], "last_check": r[9],
"created_at": r[10], "change_count": r[11]}
for r in rows
]
def get_task(self, url: str) -> Optional[dict]:
for t in self.list_tasks():
if t["url"] == url:
return t
return None
def check_task(self, url: str, dry_run: bool = False) -> dict:
"""Check a single task, return change result."""
task_data = self.get_task(url)
if not task_data:
return {"ok": False, "error": "Task not found"}
task = MonitorTask.from_dict(task_data)
interval = FREQUENCY_SECONDS.get(task.frequency, 86400)
# Check frequency
if task.last_check and not dry_run:
last_ts = datetime.fromisoformat(task.last_check.replace("Z", "+00:00"))
elapsed = (datetime.now(timezone.utc) - last_ts).total_seconds()
if elapsed < interval:
remaining = int(interval - elapsed)
return {"ok": True, "skipped": True, "reason": f"Check interval not reached, {remaining}s remaining"}
# Fetch page
html = fetch_page(task.url)
if not html:
return {"ok": False, "error": "Failed to fetch page"}
# Detect change
changed, change_type, detail = detect_change(task, html)
if dry_run:
return {
"ok": True, "changed": changed, "type": change_type, "detail": detail,
"html_length": len(html)
}
now = datetime.now(timezone.utc).isoformat()
new_hash = compute_hash(html)
if changed:
self.conn.execute(
"""INSERT INTO change_logs (url, name, detected_at, change_type, detail, mode)
VALUES (?, ?, ?, ?, ?, ?)""",
(task.url, task.name, now, change_type, detail, task.mode)
)
self.conn.execute(
"""UPDATE monitor_tasks SET last_hash=?, last_content=?, last_check=?, change_count=change_count+1 WHERE url=?""",
(new_hash, html, now, task.url)
)
self.conn.commit()
return {
"ok": True, "changed": True, "type": change_type, "detail": detail,
"task": {"url": task.url, "name": task.name}
}
else:
self.conn.execute(
"UPDATE monitor_tasks SET last_hash=?, last_check=? WHERE url=?",
(new_hash, now, task.url)
)
self.conn.commit()
return {"ok": True, "changed": False}
def check_all(self, on_change_callback=None) -> dict:
"""Check all tasks. Returns summary of changes."""
tasks = self.list_tasks()
changed_tasks = []
for task_data in tasks:
result = self.check_task(task_data["url"])
if result.get("changed"):
changed_tasks.append(result)
if on_change_callback:
on_change_callback(result)
return {
"ok": True,
"total": len(tasks),
"changed": len(changed_tasks),
"changes": changed_tasks
}
def get_change_logs(self, url: Optional[str] = None, limit: int = 50) -> List[dict]:
if url:
rows = self.conn.execute(
"SELECT url, name, detected_at, change_type, detail, mode FROM change_logs WHERE url=? ORDER BY detected_at DESC LIMIT ?",
(url, limit)
).fetchall()
else:
rows = self.conn.execute(
"SELECT url, name, detected_at, change_type, detail, mode FROM change_logs ORDER BY detected_at DESC LIMIT ?",
(limit,)
).fetchall()
return [
{"url": r[0], "name": r[1], "detected_at": r[2], "change_type": r[3], "detail": r[4], "mode": r[5]}
for r in rows
]
# Feishu notification
def build_change_message(task_name: str, url: str, change_type: str, detail: str) -> str:
"""Build Feishu notification text."""
emoji_map = {
"content_changed": "🔄",
"keyword_appeared": "🔍",
"keyword_disappeared": "🔍",
"selector_changed": "🎯",
"regex_matched": "⚙️",
}
emoji = emoji_map.get(change_type, "🔔")
lines = [
f"{emoji} **{task_name}** has changed",
f"URL: {url}",
f"Type: {change_type}",
f"Detail: {detail}",
]
return "\n".join(lines)
# CLI
def main():
if len(sys.argv) < 2:
print(json.dumps({"error": "Usage: python3 monitor.py <command> [args...]"}))
sys.exit(1)
cmd = sys.argv[1]
monitor = WebMonitor()
if cmd == "add":
url = sys.argv[2] if len(sys.argv) > 2 else ""
name = sys.argv[3] if len(sys.argv) > 3 else url
mode = sys.argv[4] if len(sys.argv) > 4 else MODE_HASH
freq = sys.argv[5] if len(sys.argv) > 5 else "24h"
result = monitor.add_task(url=url, name=name, mode=mode, frequency=freq)
print(json.dumps(result, ensure_ascii=False))
elif cmd == "remove":
url = sys.argv[2] if len(sys.argv) > 2 else ""
result = monitor.remove_task(url=url)
print(json.dumps(result, ensure_ascii=False))
elif cmd == "list":
tasks = monitor.list_tasks()
print(json.dumps({"ok": True, "tasks": tasks}, ensure_ascii=False))
elif cmd == "check":
url = sys.argv[2] if len(sys.argv) > 2 else ""
result = monitor.check_task(url)
print(json.dumps(result, ensure_ascii=False))
elif cmd == "check-all":
result = monitor.check_all()
print(json.dumps(result, ensure_ascii=False))
elif cmd == "logs":
url = sys.argv[2] if len(sys.argv) > 2 else None
limit = int(sys.argv[3]) if len(sys.argv) > 3 else 50
logs = monitor.get_change_logs(url=url, limit=limit)
print(json.dumps({"ok": True, "logs": logs}, ensure_ascii=False))
elif cmd == "dry-run":
url = sys.argv[2] if len(sys.argv) > 2 else ""
result = monitor.check_task(url, dry_run=True)
print(json.dumps(result, ensure_ascii=False))
else:
print(json.dumps({"error": f"Unknown command: {cmd}"}))
sys.exit(1)
if __name__ == "__main__":
main()
FILE:scripts/billing.py
"""
Billing integration for Web Change Monitor.
Pay-per-call: $0.01 USDT per check.
"""
import os
import time
from typing import Optional
BILLING_URL = "https://skillpay.me/api/v1/billing"
CACHE_TTL = 300
_cache: dict = {}
def _cache_get(key: str) -> Optional[dict]:
entry = _cache.get(key)
if entry is None:
return None
if time.time() - entry["_ts"] > CACHE_TTL:
del _cache[key]
return None
return entry
def _cache_set(key: str, data: dict) -> None:
_cache[key] = {**data, "_ts": time.time()}
def _get_headers() -> dict:
return {
"X-API-Key": os.environ.get("SKILL_BILLING_API_KEY", ""),
"Content-Type": "application/json",
}
def _get_skill_id() -> str:
return os.environ.get("SKILL_BILLING_SKILL_ID", "web-watcher-pro")
def _is_dev_mode() -> bool:
return os.environ.get("SKILL_BILLING_API_KEY", "").strip() == ""
def charge_user(user_id: str) -> dict:
"""
Charge user for one check call ($0.01 USDT).
Returns: {"ok": True, "balance": float} on success
{"ok": False, "balance": float, "payment_url": str} on insufficient balance
"""
if _is_dev_mode():
return {"ok": True, "balance": 999.0}
skill_id = _get_skill_id()
uid = user_id or os.environ.get("FEISHU_USER_ID", "") or "anonymous"
cache_key = f"balance:{uid}"
cached = _cache_get(cache_key)
if cached:
return cached
try:
import requests
resp = requests.post(
f"{BILLING_URL}/charge",
headers=_get_headers(),
json={
"user_id": uid,
"skill_id": skill_id,
"amount": 0.01,
},
timeout=10,
)
data = resp.json()
if data.get("success"):
result = {"ok": True, "balance": float(data.get("balance", 0.0))}
else:
result = {
"ok": False,
"balance": float(data.get("balance", 0.0)),
"payment_url": data.get("payment_url", f"https://skillpay.me/{skill_id}"),
}
_cache_set(cache_key, result)
return result
except Exception:
return {"ok": True, "balance": 999.0}Bank Statement Reconciler — Upload bank statements (CSV/Excel/PDF) + orders/invoices → AI auto-matching → Reconciliation results (matched/difference/unclaime...
---
name: bank-statement-reconcile
description: "Bank Statement Reconciler — Upload bank statements (CSV/Excel/PDF) + orders/invoices → AI auto-matching → Reconciliation results (matched/difference/unclaimed/unmatched). Supports Chinese banks (BOC/ICBC/CCB), Alipay/WeChat Pay, PayPal/Stripe, Amazon/Shopify/Temu. Trigger: bank reconciliation, statement matching, bank statement reconcile."
---
# Bank Statement Reconciler
AI-powered bank statement reconciliation — upload statements + orders → get matched/difference/unclaimed/unmatched results.
## AI Agent Full Flow
```python
from scripts import reconcile_bank_statements, TierConfig
result = reconcile_bank_statements(
statement_file="bank.csv",
order_file="orders.csv",
statement_type="auto",
order_type="auto",
match_mode="smart",
amount_tolerance=0.01,
date_range_days=3,
tier=TierConfig(is_pro=True),
)
# Result keys: matched, differences, unclaimed, unmatched_orders, summary, excel_path
```
## Supported Statement Formats
### Chinese Banks (CSV/Excel/PDF)
| Bank | Format | Key Columns |
|------|--------|-------------|
| BOC | CSV/Excel | Transaction Date, Amount, Counterparty, Balance, Summary |
| ICBC | CSV/Excel | Date, Amount, Counterparty Name, Balance, Summary |
| CCB | CSV/Excel | Transaction Time, Amount, Counterparty, Balance, Remark |
| ABC | CSV/Excel | Transaction Date, Amount, Counterparty Name, Balance, Usage |
### Payment Platforms
| Platform | Format | Key Columns |
|----------|--------|-------------|
| Alipay | CSV | Transaction Time, Counterparty, Amount, Status, Description |
| WeChat Pay | CSV | Transaction Time, Transaction Type, Amount, Counterparty, Remark |
| PayPal | CSV/JSON | Date, Amount, Item, Status, Counterparty |
| Stripe | CSV/JSON | Date, Amount, Description, Customer, Currency |
### E-commerce
| Platform | Format | Key Columns |
|----------|--------|-------------|
| Amazon | CSV/Excel | Order Date, Order ID, Order Status, Item Total, Payment |
| Shopify | CSV/Excel | Created, Name, Financial Status, Total, Source |
| Temu | CSV | Date, Order ID, Amount, Status, Payment Method |
## Matching Modes
### 1. Exact Matching
Same date + same amount. Best for real-time payments, bank transfers.
### 2. Fuzzy Matching
Date within ±N days + amount within ±X tolerance. Best for delayed settlements, batch payments.
### 3. Semantic Matching (PRO only)
AI-powered counterparty name similarity matching. Handles: "Alibaba" ↔ "Alibaba Cloud", "Zhang San" ↔ "Zhang San (Personal)".
## Tiered Features
| Feature | FREE | PRO |
|---------|:----:|:---:|
| Monthly statements | 50 | Unlimited |
| Bank accounts | 1 | Unlimited |
| Output format | Text | Excel + JSON |
| Alipay/WeChat | — | Yes |
| PayPal/Stripe | — | Yes |
| Semantic matching | — | Yes |
| Feishu card | — | Yes |
| Price | Free | $0.01/call |
## Excel Export Format
Exported Excel (`reconciliation_YYYYMMDD_HHMMSS.xlsx`) contains:
- **Sheet: Matched** — Matched transactions
- **Sheet: Differences** — Amount differences
- **Sheet: Unclaimed** — Money without order (unclaimed)
- **Sheet: Unmatched** — Order without payment (unmatched)
- **Sheet: Summary** — Summary statistics
## Feishu Card Output
PRO tier supports Feishu interactive cards with match rate, amounts, and action buttons.
## Billing
- Billing via `skillpay.me/api/v1/billing/charge`
- User data transmitted to SkillPay for billing identification
- $0.01 USD per reconciliation call (PRO tier)
## Required Environment Variables
| Variable | Description |
|----------|-------------|
| `FEISHU_USER_ID` | User ID for billing |
| `SKILL_BILLING_API_KEY` | SkillPay Builder API Key |
| `SKILL_BILLING_SKILL_ID` | SkillPay Skill ID (default: bank-statement-reconcile) |
| `OPENAI_API_KEY` | AI model API key (for semantic matching in PRO) |
## Common Errors
| Error | Cause | Solution |
|-------|-------|----------|
| `UNSUPPORTED_FORMAT` | File format not supported | Convert to CSV/Excel |
| `COLUMN_NOT_FOUND` | Required column missing | Check statement format |
| `AMOUNT_MISMATCH` | Amount parsing failed | Verify currency/decimal |
| `TIER_LIMIT_EXCEEDED` | Statement count exceeds tier | Upgrade or split files |
FILE:scripts/matcher.py
"""
Reconciliation Matcher
Implements exact, fuzzy, and semantic matching
"""
import re
from datetime import datetime, timedelta
from typing import List, Dict, Tuple, Optional, Set
from difflib import SequenceMatcher
import unicodedata
class ReconciliationMatcher:
"""Match bank transactions with orders/invoices."""
def __init__(
self,
match_mode: str = "smart",
amount_tolerance: float = 0.01,
date_range_days: int = 3,
tier=None,
):
"""
Initialize matcher.
Args:
match_mode: "exact", "fuzzy", or "smart"
amount_tolerance: Amount tolerance for fuzzy matching (CNY)
date_range_days: Date range for fuzzy matching
tier: TierConfig instance
"""
self.match_mode = match_mode
self.amount_tolerance = amount_tolerance
self.date_range_days = date_range_days
self.tier = tier
def match(
self,
transactions: List[Dict],
orders: List[Dict],
) -> Dict:
"""
Match transactions with orders.
Returns:
dict with keys: matched, differences, unclaimed, unmatched_orders, summary
"""
# Track matched items
matched_trans = []
matched_orders = []
differences = []
unclaimed = [] # Transactions without matching orders
unmatched_orders = [] # Orders without matching transactions
# Build lookup structures
trans_by_amount = self._index_by_amount(transactions)
order_by_amount = self._index_by_amount(orders)
# Track which items have been matched
trans_matched = [False] * len(transactions)
order_matched = [False] * len(orders)
# Phase 1: Exact matching (date + exact amount)
exact_matches = self._find_exact_matches(transactions, orders)
for trans_idx, order_idx, diff in exact_matches:
trans_matched[trans_idx] = True
order_matched[order_idx] = True
if diff == 0:
matched_trans.append({
**transactions[trans_idx],
"matched_order": orders[order_idx],
"match_type": "exact",
"difference": 0,
})
matched_orders.append(order_idx)
else:
differences.append({
**transactions[trans_idx],
"matched_order": orders[order_idx],
"match_type": "exact",
"difference": diff,
"trans_amount": transactions[trans_idx]["amount"],
"order_amount": orders[order_idx]["amount"],
})
# Phase 2: Fuzzy matching (date range + amount tolerance)
if self.match_mode in ("fuzzy", "smart"):
fuzzy_matches = self._find_fuzzy_matches(
transactions, orders, trans_matched, order_matched
)
for trans_idx, order_idx, diff in fuzzy_matches:
trans_matched[trans_idx] = True
order_matched[order_idx] = True
if abs(diff) <= self.amount_tolerance:
matched_trans.append({
**transactions[trans_idx],
"matched_order": orders[order_idx],
"match_type": "fuzzy",
"difference": diff,
})
matched_orders.append(order_idx)
else:
differences.append({
**transactions[trans_idx],
"matched_order": orders[order_idx],
"match_type": "fuzzy",
"difference": diff,
"trans_amount": transactions[trans_idx]["amount"],
"order_amount": orders[order_idx]["amount"],
})
# Phase 3: Semantic matching (counterparty name) - Professional tier
if self.match_mode == "smart" and self.tier and self.tier.is_pro:
semantic_matches = self._find_semantic_matches(
transactions, orders, trans_matched, order_matched
)
for trans_idx, order_idx, diff in semantic_matches:
trans_matched[trans_idx] = True
order_matched[order_idx] = True
if abs(diff) <= self.amount_tolerance:
matched_trans.append({
**transactions[trans_idx],
"matched_order": orders[order_idx],
"match_type": "semantic",
"difference": diff,
})
matched_orders.append(order_idx)
else:
differences.append({
**transactions[trans_idx],
"matched_order": orders[order_idx],
"match_type": "semantic",
"difference": diff,
"trans_amount": transactions[trans_idx]["amount"],
"order_amount": orders[order_idx]["amount"],
})
# Collect unmatched items
for i, trans in enumerate(transactions):
if not trans_matched[i]:
unclaimed.append({
**trans,
"status": "待处理",
})
for j, order in enumerate(orders):
if not order_matched[j]:
unmatched_orders.append({
**order,
"status": "待处理",
})
# Build summary
summary = self._build_summary(
matched_trans, differences, unclaimed, unmatched_orders
)
return {
"matched": matched_trans,
"differences": differences,
"unclaimed": unclaimed,
"unmatched_orders": unmatched_orders,
"summary": summary,
}
def _index_by_amount(self, items: List[Dict]) -> Dict[float, List[int]]:
"""Index items by rounded amount for quick lookup."""
index = {}
for i, item in enumerate(items):
if item.get("amount") is not None:
# Round to 2 decimal places
key = round(item["amount"], 2)
if key not in index:
index[key] = []
index[key].append(i)
return index
def _parse_date(self, date_str: Optional[str]) -> Optional[datetime]:
"""Parse date string to datetime."""
if not date_str:
return None
if isinstance(date_str, datetime):
return date_str
date_str = str(date_str).strip()
formats = [
"%Y-%m-%d",
"%Y/%m/%d",
"%Y%m%d",
"%Y年%m月%d日",
"%m/%d/%Y",
"%d/%m/%Y",
]
for fmt in formats:
try:
return datetime.strptime(date_str, fmt)
except ValueError:
continue
return None
def _date_distance(self, date1: str, date2: str) -> int:
"""Calculate days between two dates."""
d1 = self._parse_date(date1)
d2 = self._parse_date(date2)
if d1 is None or d2 is None:
return 999 # Unknown distance
delta = abs((d1 - d2).days)
return delta
def _find_exact_matches(
self,
transactions: List[Dict],
orders: List[Dict],
) -> List[Tuple[int, int, float]]:
"""Find exact matches (same date, same amount)."""
matches = []
for i, trans in enumerate(transactions):
if trans.get("amount") is None or trans.get("date") is None:
continue
trans_amount = round(trans["amount"], 2)
trans_date = trans["date"]
for j, order in enumerate(orders):
if order.get("amount") is None or order.get("date") is None:
continue
order_amount = round(order["amount"], 2)
order_date = order["date"]
# Exact amount match
if trans_amount != order_amount:
continue
# Date match (same day)
if self._date_distance(trans_date, order_date) == 0:
diff = trans_amount - order_amount
matches.append((i, j, diff))
return matches
def _find_fuzzy_matches(
self,
transactions: List[Dict],
orders: List[Dict],
trans_matched: List[bool],
order_matched: List[bool],
) -> List[Tuple[int, int, float]]:
"""Find fuzzy matches (date range + amount tolerance)."""
matches = []
for i, trans in enumerate(transactions):
if trans_matched[i]:
continue
if trans.get("amount") is None or trans.get("date") is None:
continue
trans_amount = round(trans["amount"], 2)
trans_date = trans["date"]
# Search for orders with similar amount
for tolerance in [0, 0.01, 0.1, 1, 10]:
for j, order in enumerate(orders):
if order_matched[j]:
continue
if order.get("amount") is None or order.get("date") is None:
continue
order_amount = round(order["amount"], 2)
order_date = order["date"]
# Amount within tolerance
if abs(trans_amount - order_amount) > tolerance + self.amount_tolerance:
continue
# Date within range
if self._date_distance(trans_date, order_date) > self.date_range_days:
continue
diff = trans_amount - order_amount
if abs(diff) <= self.amount_tolerance + tolerance:
matches.append((i, j, diff))
break
else:
continue
break
return matches
def _find_semantic_matches(
self,
transactions: List[Dict],
orders: List[Dict],
trans_matched: List[bool],
order_matched: List[bool],
) -> List[Tuple[int, int, float]]:
"""Find semantic matches (counterparty name similarity)."""
matches = []
for i, trans in enumerate(transactions):
if trans_matched[i]:
continue
if trans.get("amount") is None or trans.get("counterparty") is None:
continue
trans_counterparty = self._normalize_text(trans.get("counterparty", ""))
if not trans_counterparty:
continue
trans_amount = round(trans["amount"], 2)
trans_date = trans["date"]
best_match = None
best_score = 0
for j, order in enumerate(orders):
if order_matched[j]:
continue
if order.get("amount") is None:
continue
order_counterparty = self._normalize_text(order.get("counterparty", ""))
if not order_counterparty:
continue
# Calculate similarity
score = self._similarity(trans_counterparty, order_counterparty)
if score > 0.7 and score > best_score:
# Check amount
order_amount = round(order["amount"], 2)
if abs(trans_amount - order_amount) <= self.amount_tolerance:
# Check date
if trans_date and order.get("date"):
if self._date_distance(trans_date, order["date"]) <= self.date_range_days:
best_score = score
best_match = (i, j, trans_amount - order_amount)
else:
# No date constraint for semantic
best_score = score
best_match = (i, j, trans_amount - order_amount)
if best_match:
matches.append(best_match)
return matches
def _normalize_text(self, text: str) -> str:
"""Normalize text for comparison."""
if not text:
return ""
# Convert to lowercase
text = text.lower()
# Remove common business suffixes
suffixes = [
"有限公司", "co.,ltd", "co., ltd", "company", "ltd", "llc",
"inc.", "inc", "corporation", "corp",
]
for suffix in suffixes:
text = text.replace(suffix, "")
# Remove special characters
text = re.sub(r"[^\w\s]", "", text)
# Normalize unicode
text = unicodedata.normalize("NFKC", text)
return text.strip()
def _similarity(self, text1: str, text2: str) -> float:
"""Calculate similarity between two texts."""
if not text1 or not text2:
return 0.0
return SequenceMatcher(None, text1, text2).ratio()
def _build_summary(
self,
matched: List[Dict],
differences: List[Dict],
unclaimed: List[Dict],
unmatched_orders: List[Dict],
) -> Dict:
"""Build reconciliation summary."""
matched_amount = sum(
m.get("matched_order", {}).get("amount", 0) or 0
for m in matched
)
diff_amount = sum(d.get("difference", 0) or 0 for d in differences)
unclaimed_amount = sum(t.get("amount", 0) or 0 for t in unclaimed)
unmatched_amount = sum(o.get("amount", 0) or 0 for o in unmatched_orders)
total_transactions = len(matched) + len(differences) + len(unclaimed)
total_orders = len(matched) + len(differences) + len(unmatched_orders)
match_rate = 0.0
if total_transactions > 0:
match_rate = len(matched) / total_transactions * 100
recognition_rate = 0.0
if total_orders > 0:
recognition_rate = (len(matched) + len(differences)) / total_orders * 100
return {
"total_transactions": total_transactions,
"total_orders": total_orders,
"matched_count": len(matched),
"difference_count": len(differences),
"unclaimed_count": len(unclaimed),
"unmatched_count": len(unmatched_orders),
"match_rate": round(match_rate, 2),
"recognition_rate": round(recognition_rate, 2),
"matched_amount": round(matched_amount, 2),
"difference_amount": round(abs(diff_amount), 2),
"unclaimed_amount": round(unclaimed_amount, 2),
"unmatched_amount": round(unmatched_amount, 2),
}
FILE:scripts/billing.py
"""
Billing integration for Bank Statement Reconciler.
Implements per-call billing via SkillPay (skillpay.me).
"""
import os
import time
import hashlib
from typing import Optional
BILLING_URL = "https://skillpay.me/api/v1/billing"
CACHE_TTL = 300 # 5 minutes
_cache: dict = {}
def _cache_get(key: str) -> Optional[dict]:
if key in _cache:
if time.time() - _cache[key]["ts"] < CACHE_TTL:
return _cache[key]["data"]
return None
def _cache_set(key: str, data: dict) -> None:
_cache[key] = {"data": data, "ts": time.time()}
def _get_headers() -> dict:
api_key = os.environ.get("SKILL_BILLING_API_KEY", "")
return {"X-API-Key": api_key, "Content-Type": "application/json"}
def _get_skill_id() -> str:
return os.environ.get("SKILL_BILLING_SKILL_ID", "bank-statement-reconcile")
def _is_dev_mode() -> bool:
api_key = os.environ.get("SKILL_BILLING_API_KEY", "")
return not api_key
def charge_user(user_id: str) -> dict:
"""
Charge user for one reconciliation call.
Args:
user_id: User's Feishu open_id
Returns:
dict with keys: ok (bool), balance (float), payment_url (str, optional)
"""
if _is_dev_mode():
return {"ok": True, "balance": 999.0}
cache_key = f"balance:{user_id}"
cached = _cache_get(cache_key)
if cached:
return cached
try:
import requests
headers = _get_headers()
payload = {
"user_id": user_id,
"skill_id": _get_skill_id(),
"amount": 0.01,
}
resp = requests.post(
f"{BILLING_URL}/charge",
headers=headers,
json=payload,
timeout=10,
)
data = resp.json()
if data.get("success"):
result = {"ok": True, "balance": data.get("balance", 0.0)}
else:
result = {
"ok": False,
"balance": data.get("balance", 0.0),
"payment_url": data.get("payment_url", ""),
}
_cache_set(cache_key, result)
return result
except Exception:
return {"ok": True, "balance": 999.0}
FILE:scripts/exporter.py
"""
Reconciliation Results Exporter
Exports to Excel format
"""
import os
from datetime import datetime
from typing import Dict, List
from openpyxl.styles import Font, Alignment, PatternFill, Border, Side
from openpyxl.utils import get_column_letter
class ReconciliationExporter:
"""Export reconciliation results to Excel."""
def export(self, result: Dict, output_dir: str = "/tmp") -> str:
"""
Export reconciliation results to Excel.
Args:
result: Reconciliation result dict
output_dir: Output directory
Returns:
Path to exported Excel file
"""
try:
import openpyxl
from openpyxl.styles import Font, Alignment, PatternFill, Border, Side
from openpyxl.utils import get_column_letter
except ImportError:
# Fallback: create CSV
return self._export_csv(result, output_dir)
timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
filename = f"reconciliation_{timestamp}.xlsx"
filepath = os.path.join(output_dir, filename)
wb = openpyxl.Workbook()
# Summary sheet
self._write_summary(wb, result["summary"])
# Matched transactions
if result["matched"]:
self._write_matched(wb, result["matched"])
# Differences
if result["differences"]:
self._write_differences(wb, result["differences"])
# Unclaimed (money without order)
if result["unclaimed"]:
self._write_unclaimed(wb, result["unclaimed"])
# Unmatched orders (order without payment)
if result["unmatched_orders"]:
self._write_unmatched(wb, result["unmatched_orders"])
wb.save(filepath)
return filepath
def _write_summary(self, wb, summary: Dict):
"""Write summary sheet."""
ws = wb.active
ws.title = "汇总"
# Styles
label_font = Font(bold=True)
# Title
ws["A1"] = "对账汇总"
ws["A1"].font = Font(bold=True, size=16)
ws.merge_cells("A1:B1")
# Summary data
row = 3
summary_items = [
("总交易笔数", summary.get("total_transactions", 0)),
("总订单笔数", summary.get("total_orders", 0)),
("已匹配笔数", summary.get("matched_count", 0)),
("差异笔数", summary.get("difference_count", 0)),
("未认领笔数（有钱没订单）", summary.get("unclaimed_count", 0)),
("未核销笔数（有订单没收钱）", summary.get("unmatched_count", 0)),
("匹配率", f"{summary.get('match_rate', 0):.2f}%"),
("认账率", f"{summary.get('recognition_rate', 0):.2f}%"),
("已匹配金额", f"¥{summary.get('matched_amount', 0):,.2f}"),
("差异金额", f"¥{summary.get('difference_amount', 0):,.2f}"),
("未认领金额", f"¥{summary.get('unclaimed_amount', 0):,.2f}"),
("未核销金额", f"¥{summary.get('unmatched_amount', 0):,.2f}"),
]
for label, value in summary_items:
ws.cell(row=row, column=1, value=label).font = label_font
ws.cell(row=row, column=2, value=value)
row += 1
# Column widths
ws.column_dimensions["A"].width = 30
ws.column_dimensions["B"].width = 20
def _write_matched(self, wb, matched: List[Dict]):
"""Write matched transactions sheet."""
ws = wb.create_sheet("匹配结果")
headers = ["日期", "金额", "对方账户", "摘要", "订单日期", "订单金额", "订单对方", "匹配方式"]
# Write headers
header_fill = PatternFill("solid", fgColor="70AD47")
header_font = Font(bold=True, color="FFFFFF")
for col, header in enumerate(headers, 1):
cell = ws.cell(row=1, column=col, value=header)
cell.fill = header_fill
cell.font = header_font
# Write data
for row_idx, item in enumerate(matched, 2):
order = item.get("matched_order", {})
ws.cell(row=row_idx, column=1, value=item.get("date", ""))
ws.cell(row=row_idx, column=2, value=item.get("amount", 0))
ws.cell(row=row_idx, column=3, value=item.get("counterparty", ""))
ws.cell(row=row_idx, column=4, value=item.get("summary", ""))
ws.cell(row=row_idx, column=5, value=order.get("date", ""))
ws.cell(row=row_idx, column=6, value=order.get("amount", 0))
ws.cell(row=row_idx, column=7, value=order.get("counterparty", ""))
ws.cell(row=row_idx, column=8, value=item.get("match_type", ""))
# Column widths
for col, width in enumerate([15, 15, 25, 30, 15, 15, 25, 12], 1):
ws.column_dimensions[get_column_letter(col)].width = width
def _write_differences(self, wb, differences: List[Dict]):
"""Write differences sheet."""
ws = wb.create_sheet("差异")
headers = ["交易日期", "交易金额", "对方账户", "摘要",
"订单日期", "订单金额", "差异金额", "处理状态"]
header_fill = PatternFill("solid", fgColor="ED7D31")
header_font = Font(bold=True, color="FFFFFF")
for col, header in enumerate(headers, 1):
cell = ws.cell(row=1, column=col, value=header)
cell.fill = header_fill
cell.font = header_font
for row_idx, item in enumerate(differences, 2):
order = item.get("matched_order", {})
ws.cell(row=row_idx, column=1, value=item.get("date", ""))
ws.cell(row=row_idx, column=2, value=item.get("trans_amount", item.get("amount", 0)))
ws.cell(row=row_idx, column=3, value=item.get("counterparty", ""))
ws.cell(row=row_idx, column=4, value=item.get("summary", ""))
ws.cell(row=row_idx, column=5, value=order.get("date", ""))
ws.cell(row=row_idx, column=6, value=order.get("order_amount", order.get("amount", 0)))
ws.cell(row=row_idx, column=7, value=item.get("difference", 0))
ws.cell(row=row_idx, column=8, value=item.get("status", "待处理"))
for col, width in enumerate([15, 15, 25, 30, 15, 15, 15, 12], 1):
ws.column_dimensions[get_column_letter(col)].width = width
def _write_unclaimed(self, wb, unclaimed: List[Dict]):
"""Write unclaimed transactions sheet."""
ws = wb.create_sheet("未认领")
headers = ["日期", "金额", "对方账户", "摘要", "余额", "处理状态"]
header_fill = PatternFill("solid", fgColor="FFC000")
header_font = Font(bold=True, color="000000")
for col, header in enumerate(headers, 1):
cell = ws.cell(row=1, column=col, value=header)
cell.fill = header_fill
cell.font = header_font
for row_idx, item in enumerate(unclaimed, 2):
ws.cell(row=row_idx, column=1, value=item.get("date", ""))
ws.cell(row=row_idx, column=2, value=item.get("amount", 0))
ws.cell(row=row_idx, column=3, value=item.get("counterparty", ""))
ws.cell(row=row_idx, column=4, value=item.get("summary", ""))
ws.cell(row=row_idx, column=5, value=item.get("balance", ""))
ws.cell(row=row_idx, column=6, value=item.get("status", "待处理"))
for col, width in enumerate([15, 15, 25, 30, 15, 12], 1):
ws.column_dimensions[get_column_letter(col)].width = width
def _write_unmatched(self, wb, unmatched: List[Dict]):
"""Write unmatched orders sheet."""
ws = wb.create_sheet("未核销")
headers = ["订单日期", "订单金额", "客户", "订单号", "商品", "处理状态"]
header_fill = PatternFill("solid", fgColor="9E480E")
header_font = Font(bold=True, color="FFFFFF")
for col, header in enumerate(headers, 1):
cell = ws.cell(row=1, column=col, value=header)
cell.fill = header_fill
cell.font = header_font
for row_idx, item in enumerate(unmatched, 2):
ws.cell(row=row_idx, column=1, value=item.get("date", ""))
ws.cell(row=row_idx, column=2, value=item.get("amount", 0))
ws.cell(row=row_idx, column=3, value=item.get("counterparty", ""))
ws.cell(row=row_idx, column=4, value=item.get("order_no", ""))
ws.cell(row=row_idx, column=5, value=item.get("summary", ""))
ws.cell(row=row_idx, column=6, value=item.get("status", "待处理"))
for col, width in enumerate([15, 15, 25, 20, 30, 12], 1):
ws.column_dimensions[get_column_letter(col)].width = width
def _export_csv(self, result: Dict, output_dir: str) -> str:
"""Fallback CSV export."""
import csv
timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
# Export matched
if result["matched"]:
filepath = os.path.join(output_dir, f"reconciliation_matched_{timestamp}.csv")
with open(filepath, "w", newline="", encoding="utf-8-sig") as f:
writer = csv.writer(f)
writer.writerow(["日期", "金额", "对方账户", "摘要", "订单日期", "订单金额"])
for item in result["matched"]:
order = item.get("matched_order", {})
writer.writerow([
item.get("date", ""),
item.get("amount", 0),
item.get("counterparty", ""),
item.get("summary", ""),
order.get("date", ""),
order.get("amount", 0),
])
return filepath
return ""
FILE:scripts/__init__.py
"""
Bank Statement Reconciler - Core Module
"""
from .parser import StatementParser, OrderParser, detect_format
from .matcher import ReconciliationMatcher
from .exporter import ReconciliationExporter
from .feishu_card import build_feishu_card
from .tier_config import TierConfig
from .billing import charge_user
__all__ = [
"StatementParser",
"OrderParser",
"ReconciliationMatcher",
"ReconciliationExporter",
"build_feishu_card",
"TierConfig",
"charge_user",
"detect_format",
"reconcile_bank_statements",
]
def reconcile_bank_statements(
statement_file=None,
statement_text=None,
order_file=None,
order_text=None,
statement_type="auto",
order_type="auto",
match_mode="smart",
amount_tolerance=0.01,
date_range_days=3,
tier=None,
user_id=None,
):
"""
Main entry point for bank statement reconciliation.
Args:
statement_file: Path to bank statement file
statement_text: Raw bank statement text (alternative to file)
order_file: Path to orders/invoices file
order_text: Raw order text (alternative to file)
statement_type: "auto", "boc", "icbc", "ccb", "alipay", "wechat", "paypal", "stripe", "amazon", "shopify", "temu"
order_type: "auto", "invoice", "order"
match_mode: "exact", "fuzzy", "smart"
amount_tolerance: Amount tolerance for fuzzy matching (CNY)
date_range_days: Date range for fuzzy matching
tier: TierConfig instance
Returns:
dict with keys: matched, differences, unclaimed, unmatched_orders, summary, excel_path
"""
if tier is None:
tier = TierConfig()
# Billing (PRO tier)
if user_id and tier.is_pro:
bill = charge_user(user_id)
if not bill.get("ok"):
return {"error": "Insufficient balance", "payment_url": bill.get("payment_url", "")}
# Parse statements
stmt_parser = StatementParser()
if statement_file:
transactions = stmt_parser.parse_file(statement_file, statement_type)
else:
transactions = stmt_parser.parse_text(statement_text, statement_type)
# Parse orders
ord_parser = OrderParser()
if order_file:
orders = ord_parser.parse_file(order_file, order_type)
else:
orders = ord_parser.parse_text(order_text, order_type)
# Match
matcher = ReconciliationMatcher(
match_mode=match_mode,
amount_tolerance=amount_tolerance,
date_range_days=date_range_days,
tier=tier,
)
result = matcher.match(transactions, orders)
# Export Excel if tier supports
excel_path = None
if tier.can_export_excel() and (result["matched"] or result["differences"] or result["unclaimed"] or result["unmatched_orders"]):
exporter = ReconciliationExporter()
excel_path = exporter.export(result)
result["excel_path"] = excel_path
return result
FILE:scripts/feishu_card.py
"""
Feishu Card Builder for Reconciliation Results
"""
from typing import Dict, List
def build_feishu_card(result: Dict) -> Dict:
"""
Build Feishu interactive card for reconciliation results.
Args:
result: Reconciliation result dict
Returns:
Feishu card content dict
"""
summary = result.get("summary", {})
matched = result.get("matched", [])
differences = result.get("differences", [])
unclaimed = result.get("unclaimed", [])
unmatched_orders = result.get("unmatched_orders", [])
# Determine color based on match rate
match_rate = summary.get("match_rate", 0)
if match_rate >= 90:
template = "green"
elif match_rate >= 70:
template = "yellow"
else:
template = "red"
card = {
"msg_type": "interactive",
"card": {
"header": {
"title": {
"tag": "plain_text",
"content": "📊 银行流水对账结果"
},
"template": template
},
"elements": [
{
"tag": "div",
"text": {
"tag": "lark_md",
"content": f"**对账时间**: {summary.get('reconcile_time', 'N/A')}"
}
},
{"tag": "hr"},
{
"tag": "div",
"text": {
"tag": "lark_md",
"content": "### 📈 匹配概况"
}
},
{
"tag": "column_set",
"flex_mode": "BetweenBaseline",
"horizontal_spacing": "large",
"elements": [
{
"tag": "column",
"width": " stretched",
"elements": [
{
"tag": "div",
"text": {
"tag": "lark_md",
"content": f"**匹配率**\n**{match_rate:.1f}%**"
}
}
]
},
{
"tag": "column",
"width": "stretched",
"elements": [
{
"tag": "div",
"text": {
"tag": "lark_md",
"content": f"**已匹配**\n**{summary.get('matched_count', 0)} 笔**"
}
}
]
},
{
"tag": "column",
"width": "stretched",
"elements": [
{
"tag": "div",
"text": {
"tag": "lark_md",
"content": f"**差异**\n**{summary.get('difference_count', 0)} 笔**"
}
}
]
}
]
},
{"tag": "hr"},
{
"tag": "div",
"text": {
"tag": "lark_md",
"content": "### 💰 金额汇总"
}
},
{
"tag": "column_set",
"flex_mode": "BetweenBaseline",
"horizontal_spacing": "large",
"elements": [
{
"tag": "column",
"width": "stretched",
"elements": [
{
"tag": "div",
"text": {
"tag": "lark_md",
"content": f"已匹配金额\n**¥{summary.get('matched_amount', 0):,.2f}**"
}
}
]
},
{
"tag": "column",
"width": "stretched",
"elements": [
{
"tag": "div",
"text": {
"tag": "lark_md",
"content": f"差异金额\n**¥{summary.get('difference_amount', 0):,.2f}**"
}
}
]
},
{
"tag": "column",
"width": "stretched",
"elements": [
{
"tag": "div",
"text": {
"tag": "lark_md",
"content": f"未认领\n**¥{summary.get('unclaimed_amount', 0):,.2f}**"
}
}
]
}
]
},
{"tag": "hr"},
]
}
}
# Add details if there are issues
if unclaimed:
card["card"]["elements"].append({
"tag": "div",
"text": {
"tag": "lark_md",
"content": f"⚠️ **未认领**: {len(unclaimed)} 笔（有钱没订单）"
}
})
if unmatched_orders:
card["card"]["elements"].append({
"tag": "div",
"text": {
"tag": "lark_md",
"content": f"⚠️ **未核销**: {len(unmatched_orders)} 笔（有订单没收钱）"
}
})
# Add Excel download if available
if result.get("excel_path"):
card["card"]["elements"].append({"tag": "hr"})
card["card"]["elements"].append({
"tag": "note",
"elements": [
{
"tag": "plain_text",
"content": f"📎 详细报告已导出: {result['excel_path']}"
}
]
})
# Add action buttons
if unclaimed or unmatched_orders:
card["card"]["elements"].append({"tag": "hr"})
card["card"]["elements"].append({
"tag": "action",
"actions": [
{
"tag": "button",
"text": {
"tag": "plain_text",
"content": "标记已处理"
},
"type": "primary"
},
{
"tag": "button",
"text": {
"tag": "plain_text",
"content": "标记待追款"
},
"type": "warning"
},
{
"tag": "button",
"text": {
"tag": "plain_text",
"content": "标记坏账"
},
"type": "danger"
}
]
})
return card
def build_feishu_simple_message(result: Dict) -> str:
"""
Build simple Feishu text message for reconciliation results.
Returns:
Markdown formatted text
"""
summary = result.get("summary", {})
match_rate = summary.get("match_rate", 0)
emoji = "✅" if match_rate >= 90 else "⚠️" if match_rate >= 70 else "❌"
lines = [
f"📊 **银行流水对账结果**",
"",
f"📊 **匹配率**: {match_rate:.1f}%",
f"✅ **已匹配**: {summary.get('matched_count', 0)} 笔",
f"⚠️ **差异**: {summary.get('difference_count', 0)} 笔",
f"💰 **差异金额**: ¥{summary.get('difference_amount', 0):,.2f}",
f"❗ **未认领**: {summary.get('unclaimed_count', 0)} 笔",
f"❗ **未核销**: {summary.get('unmatched_count', 0)} 笔",
"",
]
if result.get("excel_path"):
lines.append(f"📎 详细报告: {result['excel_path']}")
return "\n".join(lines)
FILE:scripts/parser.py
"""
Bank Statement and Order Parser
Supports CSV, Excel, PDF, JSON formats
"""
import re
import json
import csv
import io
from datetime import datetime
from typing import List, Dict, Optional, Tuple
# Column mapping for different bank formats
BANK_COLUMN_MAPPINGS = {
# Chinese Bank (BOC)
"boc": {
"date": ["交易日期", "日期", "交易时间", "记账日期"],
"amount": ["交易金额", "金额", "发生额", "支出", "收入"],
"counterparty": ["对方账户", "对方户名", "交易对方", "对方"],
"balance": ["余额", "账户余额", "可用余额"],
"summary": ["摘要", "备注", "说明", "用途"],
"type": ["收支", "交易类型", "类型"],
},
# ICBC
"icbc": {
"date": ["日期", "交易日期", "记账日期"],
"amount": ["金额", "交易金额", "发生额"],
"counterparty": ["对方户名", "对方账户", "收款人", "付款人"],
"balance": ["余额", "账户余额"],
"summary": ["摘要", "备注", "用途"],
"type": ["交易类型", "收支"],
},
# CCB
"ccb": {
"date": ["交易时间", "日期", "交易日期"],
"amount": ["交易金额", "金额", "支出金额", "收入金额"],
"counterparty": ["对方账户", "对方户名", "收款人", "付款人"],
"balance": ["余额"],
"summary": ["备注", "摘要", "用途"],
"type": ["交易类型"],
},
# ABC (Agricultural Bank)
"abc": {
"date": ["交易日期", "日期"],
"amount": ["金额", "交易金额", "支出金额", "收入金额"],
"counterparty": ["对方姓名", "对方账户", "收款人"],
"balance": ["余额"],
"summary": ["用途", "摘要", "备注"],
"type": ["交易类型", "收支"],
},
# Alipay
"alipay": {
"date": ["交易时间", "日期时间", "创建时间"],
"amount": ["金额", "交易金额", "收入", "支出"],
"counterparty": ["对方", "交易对方", "收款人", "付款人"],
"balance": ["余额"],
"summary": ["说明", "商品说明", "备注"],
"type": ["状态", "交易类型"],
},
# WeChat Pay
"wechat": {
"date": ["交易时间", "日期时间"],
"amount": ["交易金额", "金额", "收支金额"],
"counterparty": ["交易对方", "对手方", "商户"],
"balance": [],
"summary": ["备注", "备注信息"],
"type": ["交易类型", "状态"],
},
# PayPal
"paypal": {
"date": ["Date", "Transaction Date", "日期"],
"amount": ["Amount", "Net", "交易金额"],
"counterparty": ["Name", "Counterparty", "From", "To", "交易对方"],
"balance": ["Balance"],
"summary": ["Subject", "Item", "说明", "备注"],
"type": ["Status", "Type", "状态", "类型"],
},
# Stripe
"stripe": {
"date": ["Created", "Date", "结算日期"],
"amount": ["Amount", "Gross", "金额"],
"counterparty": ["Description", "Customer", "商户", "描述"],
"balance": [],
"summary": ["Note", "备注"],
"type": ["Status", "状态", "Type"],
},
# Amazon
"amazon": {
"date": ["Order Date", "Date", "日期", "order date"],
"amount": ["Item Total", "Order Total", "Amount", "金额", "order total"],
"counterparty": ["Buyer", "Customer", "买家"],
"balance": [],
"summary": ["Product Name", "Item", "商品"],
"type": ["Order Status", "Status", "状态"],
},
# Shopify
"shopify": {
"date": ["Created", "Date", "日期", "created at"],
"amount": ["Total", "Order Total", "Amount", "金额"],
"counterparty": ["Name", "Customer", "Customer Name", "买家"],
"balance": [],
"summary": ["Lineitem name", "Product", "商品"],
"type": ["Financial Status", "Fulfillment Status", "状态"],
},
# Temu
"temu": {
"date": ["Date", "Order Date", "日期", "创建时间"],
"amount": ["Amount", "Order Amount", "金额", "实付金额"],
"counterparty": ["Supplier", "Merchant", "商户", "供应商"],
"balance": [],
"summary": ["Product", "Item", "商品", "备注"],
"type": ["Status", "Order Status", "状态"],
},
}
def detect_format(text: str, filename: str = "") -> str:
"""Auto-detect the statement format from content and filename."""
# Check filename hints
filename_lower = filename.lower()
if "alipay" in filename_lower or "支付宝" in filename_lower:
return "alipay"
if "wechat" in filename_lower or "微信" in filename_lower:
return "wechat"
if "paypal" in filename_lower:
return "paypal"
if "stripe" in filename_lower:
return "stripe"
if "amazon" in filename_lower or "亚马逊" in filename_lower:
return "amazon"
if "shopify" in filename_lower:
return "shopify"
if "temu" in filename_lower:
return "temu"
if "icbc" in filename_lower or "工商银行" in filename_lower:
return "icbc"
if "ccb" in filename_lower or "建设银行" in filename_lower:
return "ccb"
if "boc" in filename_lower or "中国银行" in filename_lower:
return "boc"
# Check content patterns
text_lower = text.lower()
if "交易时间" in text and "交易对方" in text:
if "支付宝" in text:
return "alipay"
if "微信" in text:
return "wechat"
if "对方户名" in text or "对方账户" in text:
if "工商银行" in text or "icbc" in text_lower:
return "icbc"
if "建设银行" in text or "ccb" in text_lower:
return "ccb"
if "中国银行" in text or "boc" in text_lower:
return "boc"
if "农业银行" in text:
return "abc"
return "boc" # Default for Chinese bank format
if "transaction date" in text_lower and "amount" in text_lower:
if "counterparty" in text_lower or "name" in text_lower:
return "paypal"
if "stripe" in text_lower:
return "stripe"
if "order date" in text_lower or "order total" in text_lower:
if "amazon" in text_lower:
return "amazon"
if "shopify" in text_lower:
return "shopify"
if "temu" in text_lower:
return "temu"
# Default
return "boc"
class StatementParser:
"""Parser for bank statements."""
def parse_file(self, filepath: str, format_type: str = "auto") -> List[Dict]:
"""Parse a bank statement file."""
filepath = filepath.strip()
if filepath.endswith(".csv"):
return self._parse_csv(filepath, format_type)
elif filepath.endswith((".xlsx", ".xls")):
return self._parse_excel(filepath, format_type)
elif filepath.endswith(".pdf"):
return self._parse_pdf(filepath, format_type)
elif filepath.endswith(".json"):
return self._parse_json(filepath, format_type)
else:
# Try CSV first
try:
return self._parse_csv(filepath, format_type)
except Exception:
raise ValueError(f"Unsupported file format: {filepath}")
def parse_text(self, text: str, format_type: str = "auto") -> List[Dict]:
"""Parse bank statement from text content."""
if format_type == "auto":
format_type = detect_format(text)
lines = text.strip().split("\n")
return self._parse_lines(lines, format_type)
def _parse_csv(self, filepath: str, format_type: str) -> List[Dict]:
"""Parse CSV file."""
with open(filepath, "r", encoding="utf-8-sig") as f:
reader = csv.DictReader(f)
rows = list(reader)
if not rows:
return []
return self._map_columns(rows, format_type)
def _parse_excel(self, filepath: str, format_type: str) -> List[Dict]:
"""Parse Excel file."""
import openpyxl
wb = openpyxl.load_workbook(filepath, data_only=True)
ws = wb.active
# Get headers
headers = [cell.value for cell in ws[1]]
if not headers or all(h is None for h in headers):
# Try second row
headers = [cell.value for cell in ws[2]]
# Get data rows
rows = []
for row_idx in range(2 if headers else 1, ws.max_row + 1):
row = {}
for col_idx, header in enumerate(headers, 1):
if header:
row[header] = ws.cell(row=row_idx, column=col_idx).value
if any(v is not None for v in row.values()):
rows.append(row)
if not rows:
return []
return self._map_columns(rows, format_type)
def _parse_pdf(self, filepath: str, format_type: str) -> List[Dict]:
"""Parse PDF file using doc_parse."""
import subprocess
result = subprocess.run(
["miaoda-studio-cli", "doc-parse", "--file", filepath, "--output", "text"],
capture_output=True, text=True, timeout=60
)
if result.returncode != 0:
raise ValueError(f"PDF parsing failed: {result.stderr}")
text = result.stdout
lines = text.strip().split("\n")
return self._parse_lines(lines, format_type)
def _parse_json(self, filepath: str, format_type: str) -> List[Dict]:
"""Parse JSON file."""
with open(filepath, "r", encoding="utf-8") as f:
data = json.load(f)
if isinstance(data, list):
items = data
elif isinstance(data, dict):
if "transactions" in data:
items = data["transactions"]
elif "data" in data:
items = data["data"]
elif "records" in data:
items = data["records"]
else:
items = [data]
else:
raise ValueError("Unknown JSON structure")
return self._map_columns(items, format_type)
def _parse_lines(self, lines: List[str], format_type: str) -> List[Dict]:
"""Parse text lines into transaction records."""
transactions = []
for line in lines:
line = line.strip()
if not line:
continue
# Try to extract date, amount, counterparty from line
# This is a fallback for parsed PDF text
trans = self._extract_from_line(line, format_type)
if trans:
transactions.append(trans)
return transactions
def _extract_from_line(self, line: str, format_type: str) -> Optional[Dict]:
"""Extract transaction info from a text line."""
# Date patterns
date_patterns = [
r"(\d{4}[-/]\d{1,2}[-/]\d{1,2})",
r"(\d{4}年\d{1,2}月\d{1,2}日)",
r"(\d{1,2}[-/]\d{1,2}[-/]\d{2,4})",
]
# Amount patterns
amount_patterns = [
r"[¥$]?\s*([-+]?\d{1,3}(?:,\d{3})*(?:\.\d{2})?)",
r"([-+]?\d+\.\d{2})",
]
date = None
for pattern in date_patterns:
match = re.search(pattern, line)
if match:
date = match.group(1)
break
amount = None
for pattern in amount_patterns:
match = re.search(pattern, line)
if match:
amount_str = match.group(1).replace(",", "")
try:
amount = float(amount_str)
break
except ValueError:
continue
if date or amount:
return {
"date": date,
"amount": amount,
"counterparty": line[:50] if len(line) > 50 else line,
"raw": line,
"summary": "",
}
return None
def _map_columns(self, rows: List[Dict], format_type: str) -> List[Dict]:
"""Map column names to standard fields."""
if format_type == "auto":
# Detect from first row
if rows:
sample = rows[0]
format_type = detect_format(str(sample))
mapping = BANK_COLUMN_MAPPINGS.get(format_type, BANK_COLUMN_MAPPINGS["boc"])
transactions = []
for row in rows:
trans = {
"date": None,
"amount": None,
"counterparty": None,
"balance": None,
"summary": None,
"type": None,
"raw": str(row),
}
# Map each field
for field, possible_names in mapping.items():
for name in possible_names:
if name in row and row[name] is not None:
value = row[name]
if field == "date":
trans["date"] = self._parse_date(value)
elif field == "amount":
trans["amount"] = self._parse_amount(value)
else:
trans[field] = str(value) if value else ""
break
# Only add if has date or amount
if trans["date"] or trans["amount"]:
transactions.append(trans)
return transactions
def _parse_date(self, value) -> Optional[str]:
"""Parse date value."""
if value is None:
return None
if isinstance(value, datetime):
return value.strftime("%Y-%m-%d")
value = str(value).strip()
# Try common formats
formats = [
"%Y-%m-%d",
"%Y/%m/%d",
"%Y%m%d",
"%Y年%m月%d日",
"%m/%d/%Y",
"%d/%m/%Y",
"%Y-%m-%d %H:%M:%S",
"%Y/%m/%d %H:%M:%S",
]
for fmt in formats:
try:
dt = datetime.strptime(value, fmt)
return dt.strftime("%Y-%m-%d")
except ValueError:
continue
return value
def _parse_amount(self, value) -> Optional[float]:
"""Parse amount value."""
if value is None:
return None
if isinstance(value, (int, float)):
return float(value)
value = str(value).strip()
# Remove currency symbols and commas
value = re.sub(r"[¥$,，]", "", value)
# Handle parentheses as negative
if value.startswith("(") and value.endswith(")"):
value = "-" + value[1:-1]
# Remove parentheses
value = re.sub(r"[()]", "", value)
try:
return float(value)
except ValueError:
return None
class OrderParser:
"""Parser for orders and invoices."""
ORDER_COLUMN_MAPPINGS = {
"invoice": {
"date": ["开票日期", "发票日期", "日期", "invoice date"],
"amount": ["金额", "发票金额", "税额", "total", "amount"],
"counterparty": ["购买方", "客户", "buyer", "customer"],
"invoice_no": ["发票号", "invoice number", "invoice_no"],
"summary": ["商品名称", "项目", "description", "品名"],
},
"order": {
"date": ["订单日期", "下单日期", "日期", "order date", "created"],
"amount": ["订单金额", "金额", "total", "amount", "order total"],
"counterparty": ["客户", "买家", "customer", "buyer"],
"order_no": ["订单号", "order id", "order_no", "name"],
"summary": ["商品", "商品名称", "product", "item"],
},
"auto": {
"date": ["日期", "date", "order date", "invoice date", "订单日期", "开票日期"],
"amount": ["金额", "amount", "total", "order total", "invoice amount", "订单金额", "发票金额"],
"counterparty": ["客户", "customer", "buyer", "买家", "购买方"],
"order_no": ["订单号", "order id", "invoice number", "订单ID", "invoice_no"],
"summary": ["商品", "product", "description", "商品名称", "项目"],
},
}
def parse_file(self, filepath: str, order_type: str = "auto") -> List[Dict]:
"""Parse orders/invoices file."""
filepath = filepath.strip()
if filepath.endswith(".csv"):
return self._parse_csv(filepath, order_type)
elif filepath.endswith((".xlsx", ".xls")):
return self._parse_excel(filepath, order_type)
elif filepath.endswith(".pdf"):
return self._parse_pdf(filepath, order_type)
elif filepath.endswith(".json"):
return self._parse_json(filepath, order_type)
else:
try:
return self._parse_csv(filepath, order_type)
except Exception:
raise ValueError(f"Unsupported order file format: {filepath}")
def parse_text(self, text: str, order_type: str = "auto") -> List[Dict]:
"""Parse orders from text content."""
lines = text.strip().split("\n")
return self._parse_lines(lines, order_type)
def _parse_csv(self, filepath: str, order_type: str) -> List[Dict]:
"""Parse CSV file."""
with open(filepath, "r", encoding="utf-8-sig") as f:
reader = csv.DictReader(f)
rows = list(reader)
if not rows:
return []
return self._map_columns(rows, order_type)
def _parse_excel(self, filepath: str, order_type: str) -> List[Dict]:
"""Parse Excel file."""
import openpyxl
wb = openpyxl.load_workbook(filepath, data_only=True)
ws = wb.active
headers = [cell.value for cell in ws[1]]
if not headers or all(h is None for h in headers):
headers = [cell.value for cell in ws[2]]
rows = []
for row_idx in range(2 if headers else 1, ws.max_row + 1):
row = {}
for col_idx, header in enumerate(headers, 1):
if header:
row[header] = ws.cell(row=row_idx, column=col_idx).value
if any(v is not None for v in row.values()):
rows.append(row)
if not rows:
return []
return self._map_columns(rows, order_type)
def _parse_pdf(self, filepath: str, order_type: str) -> List[Dict]:
"""Parse PDF file."""
import subprocess
result = subprocess.run(
["miaoda-studio-cli", "doc-parse", "--file", filepath, "--output", "text"],
capture_output=True, text=True, timeout=60
)
if result.returncode != 0:
raise ValueError(f"PDF parsing failed: {result.stderr}")
text = result.stdout
lines = text.strip().split("\n")
return self._parse_lines(lines, order_type)
def _parse_json(self, filepath: str, order_type: str) -> List[Dict]:
"""Parse JSON file."""
with open(filepath, "r", encoding="utf-8") as f:
data = json.load(f)
if isinstance(data, list):
items = data
elif isinstance(data, dict):
if "orders" in data:
items = data["orders"]
elif "invoices" in data:
items = data["invoices"]
elif "data" in data:
items = data["data"]
else:
items = [data]
else:
raise ValueError("Unknown JSON structure")
return self._map_columns(items, order_type)
def _parse_lines(self, lines: List[str], order_type: str) -> List[Dict]:
"""Parse text lines into order records."""
orders = []
for line in lines:
line = line.strip()
if not line:
continue
order = self._extract_from_line(line, order_type)
if order:
orders.append(order)
return orders
def _extract_from_line(self, line: str, order_type: str) -> Optional[Dict]:
"""Extract order info from text line."""
date_patterns = [
r"(\d{4}[-/]\d{1,2}[-/]\d{1,2})",
r"(\d{4}年\d{1,2}月\d{1,2}日)",
]
amount_patterns = [
r"[¥$]?\s*([-+]?\d{1,3}(?:,\d{3})+(?:\.\d{2})?)",
r"([-+]?\d+\.\d{2})",
]
date = None
for pattern in date_patterns:
match = re.search(pattern, line)
if match:
date = match.group(1)
break
amount = None
for pattern in amount_patterns:
match = re.search(pattern, line)
if match:
amount_str = match.group(1).replace(",", "")
try:
amount = float(amount_str)
break
except ValueError:
continue
if date or amount:
return {
"date": date,
"amount": amount,
"counterparty": line[:50] if len(line) > 50 else line,
"order_no": "",
"raw": line,
"summary": "",
}
return None
def _map_columns(self, rows: List[Dict], order_type: str) -> List[Dict]:
"""Map column names to standard fields."""
mapping = self.ORDER_COLUMN_MAPPINGS.get(order_type, self.ORDER_COLUMN_MAPPINGS["auto"])
orders = []
for row in rows:
order = {
"date": None,
"amount": None,
"counterparty": None,
"order_no": None,
"summary": None,
"raw": str(row),
}
for field, possible_names in mapping.items():
for name in possible_names:
if name in row and row[name] is not None:
value = row[name]
if field == "date":
order["date"] = self._parse_date(value)
elif field == "amount":
order["amount"] = self._parse_amount(value)
else:
order[field] = str(value) if value else ""
break
if order["date"] or order["amount"]:
orders.append(order)
return orders
def _parse_date(self, value) -> Optional[str]:
"""Parse date value."""
if value is None:
return None
if isinstance(value, datetime):
return value.strftime("%Y-%m-%d")
value = str(value).strip()
formats = [
"%Y-%m-%d",
"%Y/%m/%d",
"%Y%m%d",
"%Y年%m月%d日",
"%m/%d/%Y",
"%Y-%m-%d %H:%M:%S",
]
for fmt in formats:
try:
dt = datetime.strptime(value, fmt)
return dt.strftime("%Y-%m-%d")
except ValueError:
continue
return value
def _parse_amount(self, value) -> Optional[float]:
"""Parse amount value."""
if value is None:
return None
if isinstance(value, (int, float)):
return float(value)
value = str(value).strip()
value = re.sub(r"[¥$,，]", "", value)
if value.startswith("(") and value.endswith(")"):
value = "-" + value[1:-1]
value = re.sub(r"[()]", "", value)
try:
return float(value)
except ValueError:
return None
FILE:scripts/tier_config.py
"""
Tier Configuration for Bank Statement Reconciler
"""
from typing import Optional
# Token prefixes for each tier
TOKEN_PREFIXES = {
"FREE": "BANK-FREE",
"BASIC": "BANK-BSC",
"STANDARD": "BANK-STD",
"PROFESSIONAL": "BANK-PRO",
"ENTERPRISE": "BANK-ENT",
}
class TierConfig:
"""
Tier configuration for Bank Statement Reconciler.
Determines feature access based on subscription tier.
"""
def __init__(
self,
token: str = None,
plan_id: str = None,
tier_name: str = None,
is_pro: bool = False,
):
self.token = token
self.plan_id = plan_id
self.tier_name = tier_name or self._detect_tier()
self.is_pro = is_pro or self._is_pro_tier()
def _detect_tier(self) -> str:
if not self.token:
return "FREE"
token_upper = self.token.upper()
for tier_name, prefix in TOKEN_PREFIXES.items():
if token_upper.startswith(prefix):
return tier_name
return "FREE"
def _is_pro_tier(self) -> bool:
return self.tier_name in ["PROFESSIONAL", "ENTERPRISE"]
def get_limits(self) -> dict:
limits = {
"FREE": {
"monthly_statements": 50,
"bank_accounts": 1,
"output_formats": ["text"],
"alipay_wechat": False,
"paypal_stripe": False,
"semantic_matching": False,
"feishu_card": False,
},
"BASIC": {
"monthly_statements": 500,
"bank_accounts": 3,
"output_formats": ["text", "excel"],
"alipay_wechat": False,
"paypal_stripe": False,
"semantic_matching": False,
"feishu_card": False,
},
"STANDARD": {
"monthly_statements": 5000,
"bank_accounts": -1,
"output_formats": ["text", "excel"],
"alipay_wechat": True,
"paypal_stripe": False,
"semantic_matching": False,
"feishu_card": True,
},
"PROFESSIONAL": {
"monthly_statements": -1,
"bank_accounts": -1,
"output_formats": ["text", "excel", "json"],
"alipay_wechat": True,
"paypal_stripe": True,
"semantic_matching": True,
"feishu_card": True,
},
"ENTERPRISE": {
"monthly_statements": -1,
"bank_accounts": -1,
"output_formats": ["text", "excel", "json", "api"],
"alipay_wechat": True,
"paypal_stripe": True,
"semantic_matching": True,
"feishu_card": True,
},
}
return limits.get(self.tier_name, limits["FREE"])
def can_export_excel(self) -> bool:
return "excel" in self.get_limits().get("output_formats", [])
def can_use_semantic(self) -> bool:
return self.get_limits().get("semantic_matching", False)
def can_push_feishu(self) -> bool:
return self.get_limits().get("feishu_card", False)
def supports_platform(self, platform: str) -> bool:
limits = self.get_limits()
if platform in ["alipay", "wechat"]:
return limits.get("alipay_wechat", False)
elif platform in ["paypal", "stripe"]:
return limits.get("paypal_stripe", False)
elif platform in ["boc", "icbc", "ccb", "abc", "amazon", "shopify", "temu"]:
return True
return False
def check_limit(self, statement_count: int) -> bool:
limit = self.get_limits().get("monthly_statements", 50)
if limit == -1:
return True
return statement_count <= limit
FILE:scripts/test_reconciler.py
"""
Tests for Bank Statement Reconciler
"""
import pytest
import os
import csv
import tempfile
from datetime import datetime
# Import the modules
import sys
sys.path.insert(0, os.path.join(os.path.dirname(__file__)))
from parser import StatementParser, OrderParser, detect_format
from matcher import ReconciliationMatcher
from exporter import ReconciliationExporter
from feishu_card import build_feishu_card, build_feishu_simple_message
from tier_config import TierConfig, validate_token_for_tier
class TestDetectFormat:
"""Test format detection."""
def test_detect_alipay(self):
text = "交易时间,交易对方,金额,状态\n2024-01-01,淘宝,100.00,成功"
assert detect_format(text, "alipay.csv") == "alipay"
def test_detect_paypal(self):
text = "Date,Amount,Name\n2024-01-01,100.00,John"
assert detect_format(text, "paypal.csv") == "paypal"
def test_detect_icbc(self):
text = "日期,金额,对方户名\n2024-01-01,100.00,张三"
assert detect_format(text, "icbc.csv") == "icbc"
class TestStatementParser:
"""Test bank statement parsing."""
def test_parse_csv_boc(self):
"""Test parsing BOC format CSV."""
parser = StatementParser()
# Create temp CSV
with tempfile.NamedTemporaryFile(mode='w', suffix='.csv', delete=False, encoding='utf-8') as f:
f.write("交易日期,交易金额,对方账户,余额,摘要\n")
f.write("2024-01-01,1000.00,张三,5000.00,工资\n")
f.write("2024-01-02,-200.00,李四,4800.00,购物\n")
f.write("2024-01-03,500.00,王五,5300.00,退款\n")
temp_path = f.name
try:
transactions = parser.parse_file(temp_path, "boc")
assert len(transactions) == 3
assert transactions[0]["amount"] == 1000.00
assert transactions[0]["counterparty"] == "张三"
assert transactions[1]["amount"] == -200.00
assert transactions[2]["amount"] == 500.00
finally:
os.unlink(temp_path)
def test_parse_csv_paypal(self):
"""Test parsing PayPal format CSV."""
parser = StatementParser()
with tempfile.NamedTemporaryFile(mode='w', suffix='.csv', delete=False, encoding='utf-8') as f:
f.write("Date,Amount,Name,Currency\n")
f.write("2024-01-01,100.00,John Smith,USD\n")
f.write("2024-01-02,-50.00,Jane Doe,USD\n")
temp_path = f.name
try:
transactions = parser.parse_file(temp_path, "paypal")
assert len(transactions) == 2
assert transactions[0]["amount"] == 100.00
assert transactions[0]["counterparty"] == "John Smith"
finally:
os.unlink(temp_path)
def test_parse_text(self):
"""Test parsing text content."""
parser = StatementParser()
text = """
2024-01-01 交易金额: ¥1000.00 对方: 张三 余额: 5000
2024-01-02 交易金额: ¥-200.00 对方: 李四 余额: 4800
"""
transactions = parser.parse_text(text, "boc")
# Should extract at least some transactions
assert len(transactions) >= 0 # Text parsing is fallback
def test_parse_amount_formats(self):
"""Test various amount formats."""
parser = StatementParser()
amounts = ["1000", "1,000.00", "¥1000.00", "$1000.00", "(100.00)", "-100"]
for amount_str in amounts:
result = parser._parse_amount(amount_str)
assert result is not None, f"Failed to parse: {amount_str}"
# Positive with parentheses
result = parser._parse_amount("(100.00)")
assert result == -100.00
# Invalid
result = parser._parse_amount("invalid")
assert result is None
def test_parse_date_formats(self):
"""Test various date formats."""
parser = StatementParser()
dates = [
("2024-01-15", "2024-01-15"),
("2024/01/15", "2024-01-15"),
("2024年1月15日", "2024-01-15"),
("01/15/2024", "2024-01-15"),
]
for input_date, expected in dates:
result = parser._parse_date(input_date)
assert result == expected, f"Failed: {input_date} -> {result}"
class TestOrderParser:
"""Test order/invoice parsing."""
def test_parse_csv_order(self):
"""Test parsing order CSV."""
parser = OrderParser()
with tempfile.NamedTemporaryFile(mode='w', suffix='.csv', delete=False, encoding='utf-8') as f:
f.write("订单日期,订单金额,客户,订单号\n")
f.write("2024-01-01,1000.00,张三,ORDER001\n")
f.write("2024-01-02,500.00,李四,ORDER002\n")
temp_path = f.name
try:
orders = parser.parse_file(temp_path, "order")
assert len(orders) == 2
assert orders[0]["amount"] == 1000.00
assert orders[0]["counterparty"] == "张三"
assert orders[0]["order_no"] == "ORDER001"
finally:
os.unlink(temp_path)
def test_parse_csv_invoice(self):
"""Test parsing invoice CSV."""
parser = OrderParser()
with tempfile.NamedTemporaryFile(mode='w', suffix='.csv', delete=False, encoding='utf-8') as f:
f.write("发票日期,金额,购买方,发票号\n")
f.write("2024-01-01,1000.00,XX公司,INV001\n")
temp_path = f.name
try:
orders = parser.parse_file(temp_path, "invoice")
assert len(orders) == 1
assert orders[0]["amount"] == 1000.00
finally:
os.unlink(temp_path)
class TestReconciliationMatcher:
"""Test reconciliation matching."""
def test_exact_match(self):
"""Test exact matching."""
matcher = ReconciliationMatcher(match_mode="exact")
transactions = [
{"date": "2024-01-01", "amount": 1000.00, "counterparty": "张三", "raw": ""},
{"date": "2024-01-02", "amount": 500.00, "counterparty": "李四", "raw": ""},
]
orders = [
{"date": "2024-01-01", "amount": 1000.00, "counterparty": "张三", "raw": ""},
{"date": "2024-01-03", "amount": 300.00, "counterparty": "王五", "raw": ""},
]
result = matcher.match(transactions, orders)
assert result["summary"]["matched_count"] == 1
assert result["summary"]["unclaimed_count"] == 1
assert result["summary"]["unmatched_count"] == 1
def test_fuzzy_match(self):
"""Test fuzzy matching."""
matcher = ReconciliationMatcher(
match_mode="fuzzy",
amount_tolerance=0.1,
date_range_days=3
)
transactions = [
{"date": "2024-01-01", "amount": 1000.00, "counterparty": "张三", "raw": ""},
]
orders = [
{"date": "2024-01-03", "amount": 1000.05, "counterparty": "张三", "raw": ""}, # 3 days diff, 0.05 amount diff
]
result = matcher.match(transactions, orders)
assert result["summary"]["matched_count"] == 1
def test_no_match_when_amount_differs(self):
"""Test that different amounts don't match."""
matcher = ReconciliationMatcher(match_mode="exact")
transactions = [
{"date": "2024-01-01", "amount": 1000.00, "counterparty": "张三", "raw": ""},
]
orders = [
{"date": "2024-01-01", "amount": 999.00, "counterparty": "张三", "raw": ""}, # Different amount
]
result = matcher.match(transactions, orders)
assert result["summary"]["matched_count"] == 0
assert result["summary"]["unclaimed_count"] == 1
assert result["summary"]["unmatched_count"] == 1
def test_difference_detection(self):
"""Test that differences are detected with fuzzy matching."""
matcher = ReconciliationMatcher(
match_mode="fuzzy",
amount_tolerance=10.0,
date_range_days=0
)
transactions = [
{"date": "2024-01-01", "amount": 1000.00, "counterparty": "张三", "raw": ""},
]
orders = [
{"date": "2024-01-01", "amount": 990.00, "counterparty": "张三", "raw": ""}, # Different amount but within tolerance
]
result = matcher.match(transactions, orders)
# In fuzzy mode, small differences are captured as matched with difference recorded
# or as differences depending on tolerance settings
assert result["summary"]["difference_count"] + result["summary"]["matched_count"] == 1
def test_summary_calculation(self):
"""Test summary calculations."""
matcher = ReconciliationMatcher(match_mode="exact")
transactions = [
{"date": "2024-01-01", "amount": 1000.00, "counterparty": "张三", "raw": ""},
{"date": "2024-01-02", "amount": 500.00, "counterparty": "李四", "raw": ""},
{"date": "2024-01-03", "amount": 300.00, "counterparty": "王五", "raw": ""},
]
orders = [
{"date": "2024-01-01", "amount": 1000.00, "counterparty": "张三", "raw": ""},
]
result = matcher.match(transactions, orders)
summary = result["summary"]
assert summary["total_transactions"] == 3
assert summary["total_orders"] == 1
assert summary["matched_count"] == 1
assert summary["unclaimed_count"] == 2
assert summary["unmatched_count"] == 0
assert summary["matched_amount"] == 1000.00
assert summary["match_rate"] == pytest.approx(33.33, rel=0.1)
class TestTierConfig:
"""Test tier configuration."""
def test_free_tier_defaults(self):
"""Test Free tier defaults."""
tier = TierConfig()
limits = tier.get_limits()
assert limits["monthly_statements"] == 50
assert limits["bank_accounts"] == 1
assert "text" in limits["output_formats"]
assert "excel" not in limits["output_formats"]
assert tier.can_export_excel() is False
assert tier.can_push_feishu() is False
def test_basic_tier(self):
"""Test Basic tier."""
tier = TierConfig(token="BANK-BSC-xxxxx")
assert tier.tier_name == "BASIC"
assert tier.can_export_excel() is True
assert tier.can_push_feishu() is False
def test_standard_tier(self):
"""Test Standard tier."""
tier = TierConfig(token="BANK-STD-xxxxx")
assert tier.tier_name == "STANDARD"
assert tier.can_export_excel() is True
assert tier.can_push_feishu() is True
assert tier.supports_platform("alipay") is True
assert tier.supports_platform("paypal") is False
def test_professional_tier(self):
"""Test Professional tier."""
tier = TierConfig(token="BANK-PRO-xxxxx")
assert tier.tier_name == "PROFESSIONAL"
assert tier.is_pro is True
assert tier.can_use_semantic() is True
assert tier.supports_platform("paypal") is True
def test_enterprise_tier(self):
"""Test Enterprise tier."""
tier = TierConfig(token="BANK-ENT-xxxxx")
assert tier.tier_name == "ENTERPRISE"
assert tier.is_pro is True
assert tier.get_limits()["custom_rules"] is True
def test_validate_token(self):
"""Test token validation."""
assert validate_token_for_tier("BANK-PRO-123", "PROFESSIONAL") is True
assert validate_token_for_tier("BANK-FREE-123", "PROFESSIONAL") is False
assert validate_token_for_tier(None, "FREE") is True
assert validate_token_for_tier("", "FREE") is True
class TestReconciliationExporter:
"""Test Excel export."""
def test_export_creates_file(self):
"""Test that export creates an Excel file."""
exporter = ReconciliationExporter()
result = {
"matched": [
{
"date": "2024-01-01",
"amount": 1000.00,
"counterparty": "张三",
"summary": "测试",
"matched_order": {"date": "2024-01-01", "amount": 1000.00},
"match_type": "exact",
}
],
"differences": [],
"unclaimed": [],
"unmatched_orders": [],
"summary": {
"total_transactions": 1,
"total_orders": 1,
"matched_count": 1,
"difference_count": 0,
"unclaimed_count": 0,
"unmatched_count": 0,
"match_rate": 100.0,
"recognition_rate": 100.0,
"matched_amount": 1000.00,
"difference_amount": 0,
"unclaimed_amount": 0,
"unmatched_amount": 0,
}
}
with tempfile.TemporaryDirectory() as tmpdir:
filepath = exporter.export(result, tmpdir)
assert filepath is not None
assert os.path.exists(filepath)
assert filepath.endswith(".xlsx")
class TestFeishuCard:
"""Test Feishu card generation."""
def test_build_feishu_card(self):
"""Test building Feishu card."""
result = {
"matched": [],
"differences": [],
"unclaimed": [],
"unmatched_orders": [],
"summary": {
"matched_count": 10,
"difference_count": 2,
"unclaimed_count": 1,
"unmatched_count": 3,
"match_rate": 76.92,
"matched_amount": 10000.00,
"difference_amount": 50.00,
"unclaimed_amount": 200.00,
"unmatched_amount": 300.00,
}
}
card = build_feishu_card(result)
assert card["msg_type"] == "interactive"
assert "card" in card
assert card["card"]["header"]["title"]["content"] == "📊 银行流水对账结果"
def test_build_feishu_simple_message(self):
"""Test building simple Feishu message."""
result = {
"summary": {
"match_rate": 95.0,
"matched_count": 19,
"difference_count": 1,
"unclaimed_count": 0,
"unmatched_count": 0,
"difference_amount": 10.00,
}
}
message = build_feishu_simple_message(result)
assert "📊 **银行流水对账结果**" in message
assert "95.0%" in message
assert "✅ **已匹配**: 19 笔" in message
class TestIntegration:
"""Integration tests."""
def test_full_reconciliation_flow(self):
"""Test complete reconciliation flow."""
from scripts import reconcile_bank_statements
# Create temp files
with tempfile.NamedTemporaryFile(mode='w', suffix='.csv', delete=False, encoding='utf-8') as f:
f.write("交易日期,交易金额,对方账户,余额,摘要\n")
f.write("2024-01-01,1000.00,张三,5000.00,工资\n")
f.write("2024-01-02,500.00,李四,4500.00,退款\n")
f.write("2024-01-03,200.00,王五,4300.00,购物\n")
f.write("2024-01-04,300.00,赵六,4000.00,转账\n")
stmt_path = f.name
with tempfile.NamedTemporaryFile(mode='w', suffix='.csv', delete=False, encoding='utf-8') as f:
f.write("订单日期,订单金额,客户,订单号\n")
f.write("2024-01-01,1000.00,张三,ORDER001\n")
f.write("2024-01-02,490.00,李四,ORDER002\n") # Difference
f.write("2024-01-05,300.00,赵六,ORDER003\n") # Unmatched order
order_path = f.name
try:
result = reconcile_bank_statements(
statement_file=stmt_path,
order_file=order_path,
match_mode="exact",
)
assert result["summary"]["total_transactions"] == 4
assert result["summary"]["total_orders"] == 3
assert result["summary"]["matched_count"] >= 1
assert result["summary"]["difference_count"] >= 0
finally:
os.unlink(stmt_path)
os.unlink(order_path)
def test_tier_config_integration(self):
"""Test tier config with full reconciliation."""
from scripts import reconcile_bank_statements
# Create temp files
with tempfile.NamedTemporaryFile(mode='w', suffix='.csv', delete=False, encoding='utf-8') as f:
f.write("交易日期,交易金额,对方账户\n")
f.write("2024-01-01,1000.00,张三\n")
stmt_path = f.name
with tempfile.NamedTemporaryFile(mode='w', suffix='.csv', delete=False, encoding='utf-8') as f:
f.write("订单日期,订单金额,客户\n")
f.write("2024-01-01,1000.00,张三\n")
order_path = f.name
try:
# Free tier - no Excel
tier_free = TierConfig(token="BANK-FREE-xxxxx")
result = reconcile_bank_statements(
statement_file=stmt_path,
order_file=order_path,
tier=tier_free,
)
assert result.get("excel_path") is None
# Professional tier - Excel enabled
tier_pro = TierConfig(token="BANK-PRO-xxxxx")
result = reconcile_bank_statements(
statement_file=stmt_path,
order_file=order_path,
tier=tier_pro,
)
# Excel path should be set
assert result.get("excel_path") is not None
finally:
os.unlink(stmt_path)
os.unlink(order_path)
if __name__ == "__main__":
pytest.main([__file__, "-v"])
Doc Format Converter — Convert between CSV, Excel, JSON, PDF, Markdown, DOCX, HTML, and images with one click. Batch processing supported. Triggers: format c...
---
name: doc-format-converter
description: "Doc Format Converter — Convert between CSV, Excel, JSON, PDF, Markdown, DOCX, HTML, and images with one click. Batch processing supported. Triggers: format conversion, batch convert, file converter, CSV to Excel, Excel to JSON, PDF to image, convert files"
override-tools: []
---
# Doc Format Converter
Batch file format conversion: CSV ↔ Excel ↔ JSON ↔ PDF / Markdown / DOCX / HTML / PNG / TXT — one-click batch conversion.
---
## Features Overview
| Feature | Description |
|---------|-------------|
| Multi-format | CSV, Excel, JSON, PDF, Markdown, DOCX, HTML, PNG/JPG, TXT |
| Batch Processing | Convert multiple files in one execution |
| AI Custom Conversion | Rename fields, extract tables, restructure data (PRO) |
| Feishu Integration | Conversion complete → Feishu card notification |
---
## Supported Conversions
| Source | Targets |
|--------|---------|
| CSV | Excel (.xlsx), JSON |
| Excel (.xlsx/.xls) | JSON, CSV, PNG (table image) |
| JSON | Excel (multi-sheet) |
| Markdown | DOCX, HTML |
| DOCX | Markdown |
| HTML | Markdown |
| PDF | PNG/JPG (image) |
| Image (PNG/JPG) | PDF |
| TXT | CSV |
---
## Tiered Features
| Feature | FREE | PRO |
|---------|:----:|:---:|
| Total files | 10 (lifetime) | Unlimited |
| Batch size | 5 files | Unlimited |
| Formats | Basic | All |
| AI custom conversion | — | Yes |
| PDF OCR | — | Yes |
| Feishu result push | — | Yes |
---
## Technical Implementation
| Category | Technology |
|----------|------------|
| Spreadsheet | pandas (CSV/Excel/JSON) |
| PDF | PyMuPDF + pdfplumber |
| Document | pandoc + python-docx |
| Image | Pillow (PIL) |
| Encoding | UTF-8/GBK/ISO auto-detect |
---
## Usage
### Standard Conversion
```
Convert these files:
[upload file list]
Target format: XLSX
```
### AI Custom Conversion (PRO)
```
Rename fields in this CSV to English, then convert to JSON
Extract tables from this DOCX into Excel, convert other content to Markdown
```
---
## Core Script
See `scripts/converter.py` for full implementation:
```python
from scripts.converter import batch_convert, check_quota_free
# Free tier: check before converting
can_proceed, msg = check_quota_free(remaining=7)
if not can_proceed:
print(msg)
sys.exit(1)
# Batch conversion
results = batch_convert(
file_paths=["data.csv", "report.xlsx"],
target_format="json",
output_dir="/tmp/output",
plan="FREE",
)
print(results)
```
---
## Billing
- **Pay-per-call**: $0.0100 USDT per execution via SkillPay.me
- **Balance insufficient**: Payment URL returned — user tops up at `https://skillpay.me/doc-format-converter`
- **External data flow**: `FEISHU_USER_ID` transmitted to `skillpay.me/api/v1/billing` for billing identification
- **Billing model**: Each batch conversion run = 1 call = $0.0100 USDT
---
## Required Environment Variables
| Variable | Description |
|----------|-------------|
| `FEISHU_USER_ID` | User open_id for billing (passed by Feishu runtime) |
| `SKILL_BILLING_API_KEY` | SkillPay Builder API Key |
| `SKILL_BILLING_SKILL_ID` | SkillPay Skill ID (defaults to `doc-format-converter`) |
---
## Error Handling
| Error Type | Handling |
|------------|----------|
| Encoding error | Auto-try UTF-8 → GBK → ISO-8859-1 |
| Unsupported format | Friendly error + suggested formats |
| Corrupted file | Skip and report, continue to next |
| Quota exceeded | Card prompts upgrade option |
FILE:scripts/billing.py
#!/usr/bin/env python3
"""
Billing integration for doc-format-converter via SkillPay.me.
Pay-per-call: $0.01 USDT per execution.
Balance insufficient -> payment_url returned (user tops up at skillpay.me/{slug}).
Required environment variables:
SKILL_BILLING_API_KEY - SkillPay Builder API Key
SKILL_BILLING_SKILL_ID - SkillPay Skill ID (slug: doc-format-converter)
FEISHU_USER_ID - User open_id for billing
Billing API docs: https://skillpay.me/api/v1/billing
"""
import os
import time
import requests
BILLING_API_URL = "https://skillpay.me/api/v1/billing"
CALL_PRICE = 0.0100 # USDT per execution
_CACHE_TTL = 300
_cache: dict = {}
def _cache_get(key: str) -> dict | None:
entry = _cache.get(key)
if entry is None:
return None
if time.time() - entry["_ts"] > _CACHE_TTL:
del _cache[key]
return None
return entry
def _cache_set(key: str, data: dict) -> None:
_cache[key] = {**data, "_ts": time.time()}
def _get_headers() -> dict:
return {
"X-API-Key": os.environ.get("SKILL_BILLING_API_KEY", ""),
"Content-Type": "application/json",
}
def _get_skill_id() -> str:
return os.environ.get("SKILL_BILLING_SKILL_ID", "doc-format-converter")
def _is_dev_mode() -> bool:
return os.environ.get("SKILL_BILLING_API_KEY", "").strip() == ""
def check_balance(user_id: str) -> dict:
"""Returns current user balance in USDT."""
if _is_dev_mode():
return {"balance": 999.0, "ok": True}
cache_key = f"balance_{user_id}"
cached = _cache_get(cache_key)
if cached:
return {"balance": cached["balance"], "ok": True}
try:
resp = requests.get(
f"{BILLING_API_URL}/balance",
headers=_get_headers(),
params={"user_id": user_id, "skill_id": _get_skill_id()},
timeout=10,
)
data = resp.json()
balance = float(data.get("balance", 0.0))
_cache_set(cache_key, {"balance": balance})
return {"balance": balance, "ok": True}
except Exception:
return {"balance": 999.0, "ok": True}
def charge_user(user_id: str) -> dict:
"""
Charge user for one execution ($0.01 USDT).
Returns: {"ok": True, "balance": float} on success
{"ok": False, "balance": float, "payment_url": str} on insufficient balance
"""
if _is_dev_mode():
return {"ok": True, "balance": 999.0}
skill_id = _get_skill_id()
uid = user_id or os.environ.get("FEISHU_USER_ID", "") or "anonymous"
try:
resp = requests.post(
f"{BILLING_API_URL}/charge",
headers=_get_headers(),
json={
"user_id": uid,
"skill_id": skill_id,
"amount": CALL_PRICE,
},
timeout=10,
)
data = resp.json()
if data.get("success"):
return {"ok": True, "balance": float(data.get("balance", 0.0))}
return {
"ok": False,
"balance": float(data.get("balance", 0.0)),
"payment_url": data.get("payment_url", f"https://skillpay.me/{skill_id}"),
}
except Exception:
return {"ok": True, "balance": 999.0}
FILE:scripts/requirements.txt
pandas>=2.0.0
openpyxl>=3.0.0
python-docx>=1.0.0
pytesseract>=0.3.10
pdf2image>=1.16.0
Pillow>=10.0.0
requests>=2.28.0
FILE:scripts/converter.py
#!/usr/bin/env python3
"""
doc-format-converter 核心转换引擎
支持：CSV↔Excel↔JSON↔PDF/Markdown/DOCX/HTML/PNG/TXT
"""
import os
import sys
import json
import logging
from pathlib import Path
from billing import charge_user
from typing import Optional, List, Dict, Any, Tuple
# ─── 依赖检测 ────────────────────────────────────────────────
_deps = {}
try:
import pandas as pd
_deps["pandas"] = True
except ImportError:
_deps["pandas"] = False
try:
import fitz # PyMuPDF
_deps["pymupdf"] = True
except ImportError:
_deps["pymupdf"] = False
try:
import pdfplumber
_deps["pdfplumber"] = True
except ImportError:
_deps["pdfplumber"] = False
try:
import docx
_deps["python-docx"] = True
except ImportError:
_deps["python-docx"] = False
try:
from PIL import Image
_deps["pillow"] = True
except ImportError:
_deps["pillow"] = False
try:
import pandoc
_deps["pandoc"] = True
except ImportError:
_deps["pandoc"] = False
logging.basicConfig(level=logging.INFO, format="%(levelname)s | %(message)s")
log = logging.getLogger("converter")
# ─── 编码检测 ────────────────────────────────────────────────
def detect_encoding(file_path: str) -> str:
"""自动检测文件编码，依次尝试 UTF-8 → GBK → ISO-8859-1"""
for enc in ["utf-8", "gbk", "gb2312", "iso-8859-1"]:
try:
with open(file_path, "r", encoding=enc) as f:
f.read()
return enc
except (UnicodeDecodeError, UnicodeError):
continue
return "utf-8"
# ─── 表格转换 ────────────────────────────────────────────────
def csv_to_excel(csv_path: str, out_path: str) -> str:
enc = detect_encoding(csv_path)
df = pd.read_csv(csv_path, encoding=enc)
df.to_excel(out_path, index=False, engine="openpyxl")
return out_path
def csv_to_json(csv_path: str, out_path: str) -> str:
enc = detect_encoding(csv_path)
df = pd.read_csv(csv_path, encoding=enc)
df.to_json(out_path, orient="records", force_ascii=False, indent=2)
return out_path
def excel_to_csv(xlsx_path: str, out_path: str) -> str:
df = pd.read_excel(xlsx_path, engine="openpyxl")
df.to_csv(out_path, index=False, encoding="utf-8-sig")
return out_path
def excel_to_json(xlsx_path: str, out_path: str) -> str:
xl = pd.ExcelFile(xlsx_path, engine="openpyxl")
sheets = {sheet: xl.parse(sheet).to_dict(orient="records") for sheet in xl.sheet_names}
with open(out_path, "w", encoding="utf-8") as f:
json.dump(sheets, f, ensure_ascii=False, indent=2)
return out_path
def excel_to_png(xlsx_path: str, out_path: str) -> str:
"""将 Excel 表格渲染为 PNG 图片"""
df = pd.read_excel(xlsx_path, engine="openpyxl")
# 简单表格渲染为图片
from PIL import Image, ImageDraw, ImageFont
col_widths = [max(len(str(v)) for v in df[col].astype(str)) + 2 for col in df.columns]
row_height = 30
header_height = 35
total_width = sum(col_widths) * 9 + 20
total_height = (len(df) + 1) * row_height + header_height + 20
img = Image.new("RGB", (total_width, total_height), color="white")
draw = ImageDraw.Draw(img)
try:
font = ImageFont.truetype("/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf", 12)
header_font = ImageFont.truetype("/usr/share/fonts/truetype/dejavu/DejaVuSans-Bold.ttf", 13)
except Exception:
font = ImageFont.load_default()
header_font = font
x = 10
for i, col in enumerate(df.columns):
draw.text((x, 10), str(col), fill="1a3c5e", font=header_font)
x += col_widths[i] * 9
y = header_height
for _, row in df.iterrows():
x = 10
for i, val in enumerate(row):
draw.text((x, y), str(val)[:50], fill="black", font=font)
x += col_widths[i] * 9
y += row_height
img.save(out_path)
return out_path
def json_to_excel(json_path: str, out_path: str) -> str:
with open(json_path, "r", encoding="utf-8") as f:
data = json.load(f)
if isinstance(data, dict):
# 多 sheet 场景
with pd.ExcelWriter(out_path, engine="openpyxl") as writer:
for sheet_name, records in data.items():
if isinstance(records, list):
df = pd.DataFrame(records)
else:
df = pd.DataFrame([records])
safe_name = str(sheet_name)[:31]
df.to_excel(writer, sheet_name=safe_name, index=False)
else:
df = pd.DataFrame(data)
df.to_excel(out_path, index=False, engine="openpyxl")
return out_path
def txt_to_csv(txt_path: str, out_path: str) -> str:
enc = detect_encoding(txt_path)
with open(txt_path, "r", encoding=enc) as f:
lines = f.readlines()
# 简单按逗号分割，无表头
rows = [line.strip().split(",") for line in lines if line.strip()]
if rows:
df = pd.DataFrame(rows)
df.to_csv(out_path, index=False, header=False, encoding="utf-8-sig")
return out_path
# ─── 文档转换 ────────────────────────────────────────────────
def markdown_to_docx(md_path: str, out_path: str) -> str:
try:
import pandoc
doc = pandoc.read(open(md_path, encoding="utf-8").read())
pandoc.write(doc, out_path, format="docx")
except Exception:
# fallback: python-docx 直接创建简单 docx
from docx import Document
with open(md_path, "r", encoding="utf-8") as f:
content = f.read()
doc = Document()
for para in content.split("\n"):
if para.strip():
doc.add_paragraph(para.strip())
doc.save(out_path)
return out_path
def markdown_to_html(md_path: str, out_path: str) -> str:
try:
import pandoc
doc = pandoc.read(open(md_path, encoding="utf-8").read())
pandoc.write(doc, out_path, format="html5")
except Exception:
with open(md_path, "r", encoding="utf-8") as f:
content = f.read().replace("&", "&").replace("<", "<").replace(">", ">")
html = f"<!DOCTYPE html><html><head><meta charset='utf-8'><title>Converted</title></head><body><pre>{content}</pre></body></html>"
with open(out_path, "w", encoding="utf-8") as f:
f.write(html)
return out_path
def docx_to_markdown(docx_path: str, out_path: str) -> str:
try:
import pandoc
doc = pandoc.read(open(docx_path, "rb").read(), format="docx")
pandoc.write(doc, out_path, format="markdown")
except Exception:
from docx import Document
doc = Document(docx_path)
lines = []
for para in doc.paragraphs:
if para.text.strip():
lines.append(para.text.strip() + "\n")
with open(out_path, "w", encoding="utf-8") as f:
f.write("\n".join(lines))
return out_path
def html_to_markdown(html_path: str, out_path: str) -> str:
try:
import pandoc
doc = pandoc.read(open(html_path, encoding="utf-8").read(), format="html")
pandoc.write(doc, out_path, format="markdown")
except Exception:
with open(html_path, "r", encoding="utf-8") as f:
content = f.read()
import re
text = re.sub(r"<[^>]+>", "", content)
text = re.sub(r"\s+", " ", text).strip()
with open(out_path, "w", encoding="utf-8") as f:
f.write(text)
return out_path
# ─── PDF / 图片转换 ──────────────────────────────────────────
def pdf_to_png(pdf_path: str, out_dir: str, dpi: int = 150) -> List[str]:
if not _deps["pymupdf"]:
raise ImportError("PyMuPDF (fitz) 未安装，无法处理 PDF")
import fitz
doc = fitz.open(pdf_path)
os.makedirs(out_dir, exist_ok=True)
paths = []
for i, page in enumerate(doc):
mat = fitz.Matrix(dpi / 72, dpi / 72)
pix = page.get_pixmap(matrix=mat)
out = os.path.join(out_dir, f"page_{i+1:03d}.png")
pix.save(out)
paths.append(out)
return paths
def pdf_to_jpg(pdf_path: str, out_dir: str, dpi: int = 150) -> List[str]:
if not _deps["pymupdf"]:
raise ImportError("PyMuPDF (fitz) 未安装，无法处理 PDF")
import fitz
doc = fitz.open(pdf_path)
os.makedirs(out_dir, exist_ok=True)
paths = []
for i, page in enumerate(doc):
mat = fitz.Matrix(dpi / 72, dpi / 72)
pix = page.get_pixmap(matrix=mat)
out = os.path.join(out_dir, f"page_{i+1:03d}.jpg")
pix.save(out)
paths.append(out)
return paths
def image_to_pdf(image_path: str, out_path: str) -> str:
if not _deps["pillow"]:
raise ImportError("Pillow 未安装，无法处理图片")
from PIL import Image
img = Image.open(image_path)
if img.mode != "RGB":
img = img.convert("RGB")
img.save(out_path, "PDF", resolution=150)
return out_path
# ─── 批量处理 ────────────────────────────────────────────────
def batch_convert(
file_paths: List[str],
target_format: str,
output_dir: str,
ai_instruction: Optional[str] = None,
) -> Dict[str, Any]:
"""
批量转换主入口
plan: FREE | STD | PRO | MAX
"""
os.makedirs(output_dir, exist_ok=True)
results = {"success": [], "failed": [], "skipped": []}
converted_count = 0
for fp in file_paths:
fname = Path(fp).name
stem = Path(fp).stem
ext = target_format.lower().strip(".")
try:
out_path = os.path.join(output_dir, f"{stem}.{ext}")
src_ext = Path(fp).suffix.lower().strip(".")
# Sanitize target extension to prevent path traversal
import re as _re
safe_ext = _re.sub(r"[^a-zA-Z0-9]", "", ext)
if not safe_ext:
results["skipped"].append({"file": fname, "reason": f"Unsupported format: {ext}"})
continue
ext = safe_ext
# ── Single file conversion router ───────────────────
if src_ext == "csv":
if ext in ("xlsx", "xls"):
csv_to_excel(fp, out_path)
elif ext == "json":
csv_to_json(fp, out_path)
else:
results["skipped"].append({"file": fname, "reason": f"CSV不支持转为 {target_format}"})
continue
elif src_ext in ("xlsx", "xls"):
if ext == "json":
excel_to_json(fp, out_path)
elif ext == "csv":
excel_to_csv(fp, out_path)
elif ext in ("png", "jpg", "jpeg"):
excel_to_png(fp, out_path)
else:
results["skipped"].append({"file": fname, "reason": f"Excel不支持转为 {target_format}"})
continue
elif src_ext == "json":
if ext in ("xlsx", "xls"):
json_to_excel(fp, out_path)
else:
results["skipped"].append({"file": fname, "reason": f"JSON不支持转为 {target_format}"})
continue
elif src_ext == "md":
if ext == "docx":
markdown_to_docx(fp, out_path)
elif ext == "html":
markdown_to_html(fp, out_path)
else:
results["skipped"].append({"file": fname, "reason": f"Markdown不支持转为 {target_format}"})
continue
elif src_ext == "docx":
if ext == "md":
docx_to_markdown(fp, out_path)
else:
results["skipped"].append({"file": fname, "reason": f"DOCX不支持转为 {target_format}"})
continue
elif src_ext == "html":
if ext == "md":
html_to_markdown(fp, out_path)
else:
results["skipped"].append({"file": fname, "reason": f"HTML不支持转为 {target_format}"})
continue
elif src_ext == "pdf":
if ext == "png":
imgs = pdf_to_png(fp, output_dir)
results["success"].append({"file": fname, "outputs": imgs})
converted_count += 1
out_path = imgs[0] if imgs else output_dir
results["out_path"] = out_path
elif ext in ("jpg", "jpeg"):
imgs = pdf_to_jpg(fp, output_dir)
results["success"].append({"file": fname, "outputs": imgs})
converted_count += 1
out_path = imgs[0] if imgs else output_dir
results["out_path"] = out_path
else:
results["skipped"].append({"file": fname, "reason": f"PDF不支持转为 {target_format}"})
continue
elif src_ext in ("png", "jpg", "jpeg"):
if ext == "pdf":
image_to_pdf(fp, out_path)
else:
results["skipped"].append({"file": fname, "reason": f"图片不支持转为 {target_format}"})
continue
elif src_ext == "txt":
if ext == "csv":
txt_to_csv(fp, out_path)
else:
results["skipped"].append({"file": fname, "reason": f"TXT不支持转为 {target_format}"})
continue
else:
results["skipped"].append({"file": fname, "reason": f"不支持的源格式: {src_ext}"})
continue
results["success"].append({"file": fname, "output": out_path})
converted_count += 1
except Exception as e:
log.error(f"转换失败 {fname}: {e}")
results["failed"].append({"file": fname, "error": str(e)})
results["converted_count"] = converted_count
return results
# ─── 套餐额度检查 ─────────────────────────────────────────────
def check_quota_free(remaining: int) -> Tuple[bool, str]:
"""Check if free tier allows this run. Returns (can_proceed, message)."""
if remaining <= 0:
return False, "Free tier limit reached (10 files total). Upgrade to PRO at https://skillpay.me/doc-format-converter"
return True, f"Free tier: {remaining} files remaining"
# ─── 飞书卡片构建 ─────────────────────────────────────────────
def build_result_card(results: Dict[str, Any], plan: str) -> Dict[str, Any]:
succ = len(results["success"])
fail = len(results["failed"])
skip = len(results["skipped"])
succ_files = [r["file"] for r in results["success"]]
fail_files = [f"{r['file']}: {r['error']}" for r in results["failed"]]
title = "Conversion Complete" if fail == 0 else "Partially Completed"
color = "green" if fail == 0 else "orange"
card = {
"msg_type": "interactive",
"card": {
"header": {
"title": {"tag": "plain_text", "text": f"Batch Format Converter | {title}"},
"template": color
},
"elements": [
{"tag": "markdown", "content": f"**Success:** {succ} | **Failed:** {fail} | **Skipped:** {skip}"},
{"tag": "hr"},
]
}
}
if succ_files:
file_list = "\n".join([f"- {f}" for f in succ_files])
card["card"]["elements"].append(
{"tag": "markdown", "content": f"**Successful files:**\n{file_list}"}
)
if fail_files:
err_list = "\n".join([f"- {f}" for f in fail_files])
card["card"]["elements"].append(
{"tag": "markdown", "content": f"**Failed files:**\n{err_list}"}
)
if plan == "FREE":
card["card"]["elements"].append(
{"tag": "note", "elements": [
{"tag": "plain_text", "text": "Free tier: 10 files total. Upgrade to PRO at https://skillpay.me/doc-format-converter"}
]}
)
return card
# ─── CLI / 入口 ──────────────────────────────────────────────
def main():
import argparse
import os
parser = argparse.ArgumentParser(description="doc-format-converter — Batch file format conversion")
parser.add_argument("--files", nargs="+", required=True, help="Source file paths")
parser.add_argument("--format", required=True, help="Target format (e.g., xlsx, json, pdf)")
parser.add_argument("--output-dir", default="/tmp/converter_output", help="Output directory")
parser.add_argument("--ai-instruction", default=None, help="AI custom instruction (PRO)")
args = parser.parse_args()
# ── Billing: charge per execution ─────────────────────────
user_id = os.environ.get("FEISHU_USER_ID", "")
bill = charge_user(user_id)
if not bill.get("ok"):
payment_url = bill.get("payment_url", "https://skillpay.me/doc-format-converter")
print(f"[ERROR] Insufficient balance. Top up at: {payment_url}", file=sys.stderr)
sys.exit(1)
# Free tier local quota check
remaining = 10 - args.total_count
can_proceed, msg = check_quota_free(remaining)
print(f"[quota] {msg}")
if not can_proceed:
print(f"[ERROR] {msg}", file=sys.stderr)
sys.exit(1)
print(f"[start] Converting {len(args.files)} files to {args.format}")
results = batch_convert(
args.files,
args.format,
args.output_dir,
ai_instruction=args.ai_instruction,
)
print(f"[done] Success {len(results['success'])}, failed {len(results['failed'])}, skipped {len(results['skipped'])}")
print("[results]", json.dumps(results, ensure_ascii=False))
if __name__ == "__main__":
main()
FILE:scripts/__init__.py
# doc-format-converter test suite
AI Report Builder — Upload CSV/Excel, AI analyzes data and generates professional reports (charts + narrative). Supports monthly/financial/sales reports. Tri...
---
name: ai-report-builder
description: "AI Report Builder — Upload CSV/Excel, AI analyzes data and generates professional reports (charts + narrative). Supports monthly/financial/sales reports. Triggers: auto report, generate report, data report, monthly report, financial report, sales report, data analysis."
triggers:
- auto report
- generate report
- data report
- monthly report
- financial report
- sales report
- data analysis
allowed-tools: Bash(python3)
---
# AI Report Builder
Upload data (CSV/Excel) → AI analyzes → generates professional reports (charts + narrative + formatting).
---
## Quick Start
```bash
python3 scripts/generator.py --input data.csv --output report.xlsx --template monthly_operation
python3 scripts/generator.py --input sales.xlsx --output monthly.xlsx --template sales
```
---
## Tiered Features
| Feature | FREE | PRO |
|----------------------|:-----------------:|:-----------------:|
| Total uses | 5 (lifetime) | Unlimited |
| Chart types | Line only | Line only |
| AI narrative analysis | — | Yes |
| Multi-sheet Excel | — | Yes |
| PDF export | — | — |
| Price | Free | $0.01/report |
---
## Core Features
- **Multi-format support**: CSV, Excel (.xlsx/.xls)
- **AI-powered analysis**: OpenAI-compatible API
- **Chart generation**: Line, bar, pie, scatter, histogram
- **Multi-sheet Excel reports**: Professional formatting
- **Template system**: Monthly, financial, sales, data comparison, custom
---
## Usage
```bash
python3 scripts/generator.py \
--input data.csv \
--output report.xlsx \
--template monthly_operation \
--ai-provider openai \
--ai-model gpt-4o-mini \
--no-ai
```
**Arguments:**
- `--input/-i`: Data file path (CSV/Excel) — required
- `--output/-o`: Output report path (default: report.xlsx)
- `--template/-t`: Template type (monthly_operation/financial/sales/data_comparison/custom)
- `--ai-provider`: AI provider (openai/deepseek)
- `--ai-model`: AI model name
- `--no-ai`: Skip AI analysis (charts only)
- `--sheet`: Excel sheet name
---
## Supported Templates
| Template | Description |
|----------|-------------|
| `monthly_operation` | Monthly operational report |
| `financial` | Financial analysis report |
| `sales` | Sales performance report |
| `data_comparison` | Period-over-period comparison |
| `custom` | Custom format |
---
## Directory Structure
```
ai-report-builder/
├── SKILL.md
├── requirements.txt
├── scripts/
│ ├── generator.py # CLI entry point
│ └── __init__.py
└── core/
├── parser.py # Data parsing (pandas)
├── charts.py # Chart generation (matplotlib)
├── ai_analyzer.py # AI analysis (OpenAI-compatible)
├── report_builder.py # Excel multi-sheet builder
├── quota.py # Quota management
└── templates.py # Template system
```
---
## Billing
- **Pay-per-call**: $0.0100 USDT per execution via SkillPay.me
- **Balance insufficient**: Payment URL returned — user tops up at `https://skillpay.me/ai-report-builder`
- **External data flow**: `FEISHU_USER_ID` transmitted to `skillpay.me/api/v1/billing` for billing identification only; not stored or shared with any third party
- **Billing model**: Each report generation = 1 call = $0.0100 USDT
- **Privacy**: FEISHU_USER_ID is used solely to identify the billing account; no personal data is retained or shared beyond the payment processor
---
## Required Environment Variables
| Variable | Description |
|----------|-------------|
| `SKILL_BILLING_API_KEY` | SkillPay Builder API Key |
| `SKILL_BILLING_SKILL_ID` | SkillPay Skill ID (`ai-report-builder`) |
| `FEISHU_USER_ID` | User open_id for billing (passed by Feishu runtime) |
---
## License
MIT
FILE:requirements.txt
pandas>=1.3.0
openpyxl>=3.0.0
xlsxwriter>=3.0.0
matplotlib>=3.4.0
Pillow>=8.0.0
requests>=2.25.0
numpy>=1.20.0
FILE:scripts/__init__.py
FILE:scripts/generator.py
#!/usr/bin/env python3
"""AI Report Builder — Main CLI entry point. Per-call billing via SkillPay ($0.0100 per report)."""
import argparse
import os
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).parent.parent))
from core import parser, charts, ai_analyzer, report_builder, templates
from core.quota import check_quota, increment
from core.billing import charge_user
def main():
parser_cli = argparse.ArgumentParser(
description="AI Report Builder — Generate professional reports from CSV/Excel data",
formatter_class=argparse.RawDescriptionHelpFormatter,
)
parser_cli.add_argument("--input", "-i", required=True, help="Input data file path (CSV/Excel)")
parser_cli.add_argument("--output", "-o", default="report.xlsx", help="Output report path (default: report.xlsx)")
parser_cli.add_argument("--template", "-t", default="custom",
choices=["monthly_operation", "financial", "sales", "data_comparison", "custom"],
help="Report template (default: custom)")
parser_cli.add_argument("--api-key", help="AI API key (optional, skip AI analysis if not provided)")
parser_cli.add_argument("--api-base", default="https://api.openai.com/v1",
help="AI API base URL (default: https://api.openai.com/v1)")
parser_cli.add_argument("--no-ai", action="store_true", help="Skip AI analysis (charts only)")
parser_cli.add_argument("--sheet", default=None, help="Excel sheet name (Excel files only)")
args = parser_cli.parse_args()
user_id = os.environ.get("FEISHU_USER_ID", "")
# ── Step 1: Quota check (FREE tier: 5 lifetime uses) ─────────────────
# Done BEFORE billing — prevents charging users who have exhausted free quota.
quota_result = check_quota(1)
if not quota_result["allowed"]:
print(f"[ERROR] Free tier limit reached. Upgrade to PRO: $0.01 per report at https://skillpay.me/ai-report-builder", file=sys.stderr)
sys.exit(1)
# ── Step 2: Billing check (PRO — pay per call) ──────────────────────
# Only PRO users reach this line (FREE users already exited above).
bill = charge_user(user_id)
if not bill.get("ok"):
payment_url = bill.get("payment_url", "https://skillpay.me/ai-report-builder")
print(f"[ERROR] Insufficient balance. Top up at: {payment_url}", file=sys.stderr)
sys.exit(1)
# ── Step 3: Validate input file ─────────────────────────────────────
input_path = Path(args.input)
if not input_path.exists():
print(f"[ERROR] Input file not found: {args.input}", file=sys.stderr)
sys.exit(1)
# ── Step 4: Load data ───────────────────────────────────────────────
print(f"Loading data: {args.input}")
if input_path.suffix.lower() == ".csv":
df = parser.load_csv(str(input_path))
elif input_path.suffix.lower() in (".xlsx", ".xls"):
df = parser.load_excel(str(input_path), sheet=args.sheet)
else:
print(f"[ERROR] Unsupported file format: {input_path.suffix}", file=sys.stderr)
sys.exit(1)
print(f"Data loaded: {len(df)} rows, {len(df.columns)} columns")
# ── Step 5: Generate stats ──────────────────────────────────────────
stats = parser.get_stats(df)
print("Stats computed")
# ── Step 6: Generate charts ─────────────────────────────────────────
chart_paths = []
if not df.select_dtypes(include=["number"]).columns.empty:
numeric_col = df.select_dtypes(include=["number"]).columns[0]
try:
chart_path = charts.generate_chart(df, numeric_col, "line")
chart_paths.append(chart_path)
print(f"Chart generated: {chart_path}")
except Exception as e:
print(f"Chart skipped: {e}")
# ── Step 7: AI analysis ─────────────────────────────────────────────
ai_analysis = ""
if not args.no_ai and args.api_key:
print(f"Running AI analysis ({quota_result['remaining'] - 1} free uses remaining)...")
ai_analysis = ai_analyzer.analyze_data(df, args.api_key, args.api_base)
else:
print("AI analysis skipped")
# ── Step 8: Build report ─────────────────────────────────────────────
template = templates.get_template(args.template)
report_data = {
"title": f"{template['name']} — {input_path.stem}",
"stats": stats,
"charts": chart_paths,
"ai_analysis": ai_analysis,
"raw_data": df,
"template": template["name"],
}
print(f"Building report: {args.output}")
output_path = report_builder.build_excel_report(report_data, args.output)
print(f"Report complete: {output_path}")
# ── Step 9: Increment quota (always, after successful execution) ─────
# This runs for both FREE (counts toward 5-use limit) and PRO (no-op for PRO).
increment()
if __name__ == "__main__":
main()
FILE:scripts/quick_report.sh
#!/bin/bash
# 快速报表生成脚本 - shell 包装
SCRIPT_DIR="$(cd "$(dirname "BASH_SOURCE[0]")" && pwd)"
PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
python3 "$PROJECT_ROOT/scripts/generator.py" "$@"
FILE:core/report_builder.py
"""Excel report builder — multi-sheet professional reports."""
import openpyxl
from openpyxl.styles import Font, Alignment, PatternFill, Border, Side
from openpyxl.utils import get_column_letter
import pandas as pd
from pathlib import Path
from datetime import datetime
def build_excel_report(data: dict, output_path: str) -> str:
"""Build multi-sheet Excel report.
Args:
data: Report data dict with keys: title, stats, charts, ai_analysis, raw_data, template
output_path: Output file path
Returns:
str: Path to generated report file
"""
wb = openpyxl.Workbook()
wb.remove(wb.active)
_build_cover_sheet(wb, data)
_build_overview_sheet(wb, data)
_build_charts_sheet(wb, data)
_build_ai_analysis_sheet(wb, data)
_build_raw_data_sheet(wb, data)
output_path = Path(output_path)
output_path.parent.mkdir(parents=True, exist_ok=True)
wb.save(str(output_path))
return str(output_path)
def _build_cover_sheet(wb: openpyxl.Workbook, data: dict):
"""Build cover sheet."""
ws = wb.create_sheet("Cover", 0)
title = data.get('title', 'Auto Report')
template = data.get('template', 'Default Template')
title_font = Font(name='Arial', size=24, bold=True, color='FFFFFF')
subtitle_font = Font(name='Arial', size=14, color='CCCCCC')
fill = PatternFill(start_color='1F4E79', end_color='1F4E79', fill_type='solid')
for row in range(1, 10):
for col in range(1, 8):
cell = ws.cell(row=row, column=col)
cell.fill = fill
ws.merge_cells('A2:G7')
title_cell = ws.cell(row=4, column=1)
title_cell.value = title
title_cell.font = title_font
title_cell.alignment = Alignment(horizontal='center', vertical='center')
ws.merge_cells('A8:G8')
date_cell = ws.cell(row=8, column=1)
date_cell.value = f"Generated: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}"
date_cell.font = subtitle_font
date_cell.alignment = Alignment(horizontal='center')
ws.merge_cells('A9:G9')
template_cell = ws.cell(row=9, column=1)
template_cell.value = f"Template: {template}"
template_cell.font = subtitle_font
template_cell.alignment = Alignment(horizontal='center')
ws.column_dimensions['A'].width = 3
ws.column_dimensions['G'].width = 3
def _build_overview_sheet(wb: openpyxl.Workbook, data: dict):
"""Build data overview sheet."""
ws = wb.create_sheet("Overview", 1)
header_font = Font(name='Arial', size=12, bold=True, color='FFFFFF')
header_fill = PatternFill(start_color='4472C4', end_color='4472C4', fill_type='solid')
cell_font = Font(name='Arial', size=11)
border = Border(
left=Side(style='thin'),
right=Side(style='thin'),
top=Side(style='thin'),
bottom=Side(style='thin')
)
stats = data.get('stats', {})
if isinstance(stats, dict):
overview_data = [
["Metric", "Value"],
["Row Count", stats.get('row_count', 'N/A')],
["Column Count", stats.get('col_count', 'N/A')],
["Total Missing", sum(stats.get('missing_values', {}).values()) if isinstance(stats.get('missing_values'), dict) else 'N/A'],
]
for i, row_data in enumerate(overview_data, start=1):
for j, value in enumerate(row_data, start=1):
cell = ws.cell(row=i, column=j, value=value)
cell.font = header_font if i == 1 else cell_font
cell.fill = header_fill if i == 1 else PatternFill(fill_type=None)
cell.border = border
cell.alignment = Alignment(horizontal='center', vertical='center')
ws.column_dimensions['A'].width = 20
ws.column_dimensions['B'].width = 20
def _build_charts_sheet(wb: openpyxl.Workbook, data: dict):
"""Build charts sheet."""
ws = wb.create_sheet("Charts", 2)
charts = data.get('charts', [])
if charts:
for i, chart_path in enumerate(charts[:5], start=1):
from openpyxl.drawing.image import Image as XLImage
img = XLImage(chart_path)
img.width = 400
img.height = 240
ws.add_image(img, f'A{i*10}')
else:
ws.cell(row=1, column=1, value="No chart data available")
ws.column_dimensions['A'].width = 30
def _build_ai_analysis_sheet(wb: openpyxl.Workbook, data: dict):
"""Build AI analysis sheet."""
ws = wb.create_sheet("AI Analysis", 3)
ai_analysis = data.get('ai_analysis', 'No AI analysis available')
header_font = Font(name='Arial', size=12, bold=True)
content_font = Font(name='Arial', size=11)
ws.cell(row=1, column=1, value="AI Data Analysis Report").font = header_font
ws.merge_cells('A1:D1')
ws.cell(row=3, column=1, value="Analysis:").font = header_font
ws.merge_cells('A3:D3')
ws.cell(row=4, column=1, value=ai_analysis).font = content_font
ws.merge_cells('A4:D10')
ws.cell(row=4, column=1).alignment = Alignment(wrap_text=True, vertical='top')
ws.column_dimensions['A'].width = 80
def _build_raw_data_sheet(wb: openpyxl.Workbook, data: dict):
"""Build raw data sheet."""
ws = wb.create_sheet("Raw Data", 4)
raw_data = data.get('raw_data')
if raw_data is not None and isinstance(raw_data, pd.DataFrame):
df = raw_data
header_font = Font(name='Arial', size=11, bold=True, color='FFFFFF')
header_fill = PatternFill(start_color='4472C4', end_color='4472C4', fill_type='solid')
for col_idx, col_name in enumerate(df.columns, start=1):
cell = ws.cell(row=1, column=col_idx, value=str(col_name))
cell.font = header_font
cell.fill = header_fill
cell.alignment = Alignment(horizontal='center', vertical='center')
for row_idx, row in enumerate(df.itertuples(index=False), start=2):
for col_idx, value in enumerate(row, start=1):
cell = ws.cell(row=row_idx, column=col_idx, value=value)
cell.alignment = Alignment(horizontal='left', vertical='center')
for col_idx in range(1, len(df.columns) + 1):
ws.column_dimensions[get_column_letter(col_idx)].width = 18
FILE:core/billing.py
"""
Billing integration for ai-report-builder via SkillPay.me.
Pay-per-call: $0.01 USDT per execution.
Balance insufficient -> payment_url returned (user tops up at skillpay.me/{slug}).
Required environment variables:
SKILL_BILLING_API_KEY - SkillPay Builder API Key
SKILL_BILLING_SKILL_ID - SkillPay Skill ID (slug: ai-report-builder)
FEISHU_USER_ID - User open_id for billing
Billing API docs: https://skillpay.me/api/v1/billing
"""
import os
import requests
# Constants
BILLING_API_URL = "https://skillpay.me/api/v1/billing"
CALL_PRICE = 0.0100 # USDT per execution
def _get_headers() -> dict:
return {
"X-API-Key": os.environ.get("SKILL_BILLING_API_KEY", ""),
"Content-Type": "application/json",
}
def _get_skill_id() -> str:
return os.environ.get("SKILL_BILLING_SKILL_ID", "ai-report-builder")
def _is_dev_mode() -> bool:
key = os.environ.get("SKILL_BILLING_API_KEY", "").strip()
return key == ""
def charge_user(user_id: str) -> dict:
if _is_dev_mode():
return {"ok": True, "balance": 999.0}
try:
resp = requests.post(
f"{BILLING_API_URL}/charge",
headers=_get_headers(),
json={
"user_id": user_id,
"skill_id": _get_skill_id(),
"amount": CALL_PRICE,
},
timeout=10,
)
data = resp.json()
if data.get("success"):
return {
"ok": True,
"balance": float(data.get("balance", 0.0)),
}
return {
"ok": False,
"balance": float(data.get("balance", 0.0)),
"payment_url": data.get("payment_url", ""),
}
except Exception:
return {"ok": True, "balance": 999.0}
FILE:core/charts.py
"""Chart generation module — supports line, bar, pie, scatter, and histogram."""
import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt
from matplotlib import font_manager
import pandas as pd
import numpy as np
from pathlib import Path
import uuid
# Font configuration
plt.rcParams['font.sans-serif'] = ['DejaVu Sans', 'Arial', 'Helvetica']
plt.rcParams['axes.unicode_minus'] = False
CHART_COLORS = ['#4A90E2', '#50C878', '#FF6B6B', '#FFD93D', '#6BCB77', '#4D96FF', '#FF922B']
def _ensure_font():
"""Ensure a usable font is available."""
fonts = font_manager.findSystemFonts()
if not fonts:
matplotlib.rcParams['font.family'] = 'DejaVu Sans'
else:
matplotlib.rcParams['font.family'] = 'sans-serif'
def generate_chart(df: pd.DataFrame, column: str, chart_type: str, output_dir: str = None) -> str:
"""Generate a chart and save as an image.
Args:
df: DataFrame
column: Column name for chart data
chart_type: Chart type ('line', 'bar', 'pie', 'scatter')
output_dir: Output directory (default: /tmp/auto_report_charts)
Returns:
str: Path to saved image
"""
_ensure_font()
if output_dir is None:
output_dir = Path('/tmp/auto_report_charts')
else:
output_dir = Path(output_dir)
output_dir.mkdir(parents=True, exist_ok=True)
if column not in df.columns:
raise ValueError(f"Column '{column}' not found in DataFrame")
fig, ax = plt.subplots(figsize=(10, 6))
if chart_type == 'line':
_plot_line(df, column, ax)
elif chart_type == 'bar':
_plot_bar(df, column, ax)
elif chart_type == 'pie':
_plot_pie(df, column, ax)
elif chart_type == 'scatter':
_plot_scatter(df, column, ax)
else:
raise ValueError(f"Unsupported chart type: {chart_type}. Supported: line, bar, pie, scatter")
plt.tight_layout()
filename = f"{chart_type}_{column}_{uuid.uuid4().hex[:8]}.png"
filepath = output_dir / filename
plt.savefig(filepath, dpi=150, bbox_inches='tight')
plt.close()
return str(filepath)
def _plot_line(df: pd.DataFrame, column: str, ax):
"""Plot a line chart."""
numeric_data = df[column].dropna()
ax.plot(numeric_data.reset_index(drop=True), color=CHART_COLORS[0], linewidth=2, marker='o', markersize=4)
ax.set_title(f'{column} Trend', fontsize=14, fontweight='bold')
ax.set_xlabel('Index', fontsize=11)
ax.set_ylabel(column, fontsize=11)
ax.grid(True, alpha=0.3)
def _plot_bar(df: pd.DataFrame, column: str, ax):
"""Plot a bar chart."""
value_counts = df[column].value_counts().head(20)
colors = CHART_COLORS[:len(value_counts)]
bars = ax.bar(range(len(value_counts)), value_counts.values, color=colors)
ax.set_xticks(range(len(value_counts)))
ax.set_xticklabels(value_counts.index, rotation=45, ha='right', fontsize=9)
ax.set_title(f'{column} Distribution', fontsize=14, fontweight='bold')
ax.set_ylabel('Count', fontsize=11)
ax.grid(axis='y', alpha=0.3)
def _plot_pie(df: pd.DataFrame, column: str, ax):
"""Plot a pie chart."""
value_counts = df[column].value_counts().head(8)
colors = CHART_COLORS[:len(value_counts)]
wedges, texts, autotexts = ax.pie(
value_counts.values,
labels=value_counts.index,
colors=colors,
autopct='%1.1f%%',
startangle=90
)
ax.set_title(f'{column} Ratio', fontsize=14, fontweight='bold')
def _plot_scatter(df: pd.DataFrame, column: str, ax):
"""Plot a scatter chart."""
numeric_cols = df.select_dtypes(include=[np.number]).columns.tolist()
if len(numeric_cols) < 2:
raise ValueError("Scatter chart requires at least two numeric columns")
other_col = numeric_cols[0] if numeric_cols[0] != column else numeric_cols[1]
ax.scatter(df[other_col], df[column], alpha=0.6, color=CHART_COLORS[0], s=30)
ax.set_title(f'{column} vs {other_col}', fontsize=14, fontweight='bold')
ax.set_xlabel(other_col, fontsize=11)
ax.set_ylabel(column, fontsize=11)
ax.grid(True, alpha=0.3)
def generate_histogram(df: pd.DataFrame, column: str, bins: int = 30, output_dir: str = None) -> str:
"""Generate a histogram."""
_ensure_font()
if output_dir is None:
output_dir = Path('/tmp/auto_report_charts')
else:
output_dir = Path(output_dir)
output_dir.mkdir(parents=True, exist_ok=True)
series = df[column].dropna()
fig, ax = plt.subplots(figsize=(10, 6))
ax.hist(series, bins=bins, color=CHART_COLORS[0], edgecolor='white', alpha=0.8)
ax.set_title(f'{column} Histogram', fontsize=14, fontweight='bold')
ax.set_xlabel(column, fontsize=11)
ax.set_ylabel('Frequency', fontsize=11)
ax.grid(axis='y', alpha=0.3)
plt.tight_layout()
filename = f"histogram_{column}_{uuid.uuid4().hex[:8]}.png"
filepath = output_dir / filename
plt.savefig(filepath, dpi=150, bbox_inches='tight')
plt.close()
return str(filepath)
FILE:core/quota.py
"""Quota management module — FREE tier: 5 free uses (local counter, lifetime). PRO users skip quota check and pay per call via SkillPay ($0.01/report)."""
import json
from pathlib import Path
# Free tier limit: 5 total uses
FREE_LIMIT = 5
# Quota file path
QUOTA_FILE = Path("/tmp") / "ai_report_builder" / "quota.json"
def get_quota() -> dict:
"""Load quota data, initialize if missing."""
if not QUOTA_FILE.exists():
return {"count": 0}
try:
with open(QUOTA_FILE, "r", encoding="utf-8") as f:
return json.load(f)
except Exception:
return {"count": 0}
def save_quota(data: dict) -> None:
"""Save quota data to file."""
QUOTA_FILE.parent.mkdir(parents=True, exist_ok=True)
with open(QUOTA_FILE, "w", encoding="utf-8") as f:
json.dump(data, f, ensure_ascii=False, indent=2)
def check_quota(count: int = 1) -> dict:
"""
Check if free tier quota allows this run.
Returns (allowed: bool, message: str).
"""
data = get_quota()
current = data.get("count", 0)
remaining = FREE_LIMIT - current
if remaining >= count:
return {
"allowed": True,
"message": f"Free tier: {remaining} uses remaining",
"remaining": remaining,
"limit": FREE_LIMIT,
}
else:
return {
"allowed": False,
"message": f"Free tier limit reached ({FREE_LIMIT} uses). Upgrade to PRO for unlimited — $0.01 per report at https://skillpay.me/ai-report-builder",
"remaining": 0,
"limit": FREE_LIMIT,
}
def increment() -> int:
"""Increment usage count by 1. Returns new count."""
data = get_quota()
data["count"] = data.get("count", 0) + 1
save_quota(data)
return data["count"]
def reset_quota() -> None:
"""Reset quota to zero (for testing/admin)."""
save_quota({"count": 0})
FILE:core/__init__.py
FILE:core/ai_analyzer.py
"""AI分析模块 - 使用 OpenAI 兼容接口进行数据解读"""
import json
import requests
import pandas as pd
from typing import Optional
def analyze_data(df: pd.DataFrame, api_key: str, api_base: str = "https://api.openai.com/v1") -> str:
"""使用 AI 分析数据并生成统计解读
Args:
df: 数据框
api_key: API 密钥
api_base: API 基础 URL
Returns:
str: AI 生成的统计分析解读
"""
summary = _prepare_data_summary(df)
prompt = f"""你是一位数据分析专家，请对以下数据进行统计分析并给出简洁的解读：
数据概况：
{summary}
请提供：
1. 数据整体质量评估
2. 主要发现和特点
3. 潜在的数据问题或异常
请用中文回复，保持简洁，控制在200字以内。"""
return _call_llm(prompt, api_key, api_base)
def generate_insights(df: pd.DataFrame, chart_descriptions: list, api_key: str, api_base: str = "https://api.openai.com/v1") -> str:
"""基于图表描述生成 AI 文字分析
Args:
df: 数据框
chart_descriptions: 图表描述列表
api_key: API 密钥
api_base: API 基础 URL
Returns:
str: AI 生成的分析报告
"""
summary = _prepare_data_summary(df)
charts_text = "\n".join([f"- {desc}" for desc in chart_descriptions])
prompt = f"""你是一位数据分析专家，请基于以下数据摘要和图表信息，撰写一份数据分析报告：
数据摘要：
{summary}
图表信息：
{charts_text}
请提供：
1. 关键发现（3-5条）
2. 数据趋势分析
3. 业务建议或洞察
请用中文回复，结构清晰，总字数控制在300字以内。"""
return _call_llm(prompt, api_key, api_base)
def _prepare_data_summary(df: pd.DataFrame) -> str:
"""准备数据摘要文本"""
row_count = len(df)
col_count = len(df.columns)
numeric_cols = df.select_dtypes(include=['number']).columns.tolist()
missing = df.isnull().sum().sum()
lines = [
f"行数: {row_count}, 列数: {col_count}",
f"数值列: {', '.join(numeric_cols[:10])}" + ("..." if len(numeric_cols) > 10 else ""),
f"缺失值总数: {missing}",
]
if numeric_cols:
desc = df[numeric_cols].describe().to_string()
lines.append(f"\n数值列统计：\n{desc}")
return "\n".join(lines)
def _call_llm(prompt: str, api_key: str, api_base: str, model: str = "gpt-3.5-turbo") -> str:
"""调用 LLM API"""
headers = {
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json",
}
payload = {
"model": model,
"messages": [{"role": "user", "content": prompt}],
"temperature": 0.7,
"max_tokens": 500,
}
try:
response = requests.post(
f"{api_base.rstrip('/')}/chat/completions",
headers=headers,
json=payload,
timeout=30,
)
response.raise_for_status()
result = response.json()
return result["choices"][0]["message"]["content"].strip()
except requests.exceptions.Timeout:
return "[AI 分析超时，请稍后重试]"
except requests.exceptions.RequestException as e:
return f"[AI 分析请求失败: {str(e)}]"
except (KeyError, IndexError):
return "[AI 分析响应格式错误]"
FILE:core/parser.py
"""数据解析模块 - 支持 CSV/Excel 文件加载与统计分析"""
import pandas as pd
import numpy as np
from pathlib import Path
def load_csv(path: str) -> pd.DataFrame:
"""加载 CSV 文件
Args:
path: CSV 文件路径
Returns:
pd.DataFrame: 加载的数据框
"""
try:
df = pd.read_csv(path, encoding='utf-8-sig')
return df
except UnicodeDecodeError:
df = pd.read_csv(path, encoding='gbk')
return df
except Exception as e:
raise ValueError(f"无法加载 CSV 文件 {path}: {str(e)}")
def load_excel(path: str, sheet: str = None) -> pd.DataFrame:
"""加载 Excel 文件
Args:
path: Excel 文件路径
sheet: 工作表名称或索引，默认为 None（读取第一个工作表）
Returns:
pd.DataFrame: 加载的数据框
"""
try:
if sheet is None:
df = pd.read_excel(path, engine='openpyxl')
else:
df = pd.read_excel(path, sheet_name=sheet, engine='openpyxl')
return df
except Exception as e:
raise ValueError(f"无法加载 Excel 文件 {path}: {str(e)}")
def get_stats(df: pd.DataFrame) -> dict:
"""获取数据框的统计摘要
Args:
df: 数据框
Returns:
dict: 包含以下键的统计字典
- row_count: 行数
- col_count: 列数
- means: 各列均值
- medians: 各列中位数
- mins: 各列最小值
- maxs: 各列最大值
- missing_values: 各列缺失值数量
"""
numeric_df = df.select_dtypes(include=[np.number])
stats = {
'row_count': len(df),
'col_count': len(df.columns),
'means': numeric_df.mean().to_dict(),
'medians': numeric_df.median().to_dict(),
'mins': numeric_df.min().to_dict(),
'maxs': numeric_df.max().to_dict(),
'missing_values': df.isnull().sum().to_dict(),
'column_types': df.dtypes.astype(str).to_dict(),
}
return stats
def detect_anomalies(df: pd.DataFrame, column: str) -> dict:
"""检测指定列的异常值（基于 IQR 方法）
Args:
df: 数据框
column: 列名
Returns:
dict: 异常值检测结果
- count: 异常值数量
- indices: 异常值所在行索引
- lower_bound: 下界
- upper_bound: 上界
- method: 检测方法
"""
if column not in df.columns:
raise ValueError(f"列 '{column}' 不存在于数据框中")
series = df[column].dropna()
if not pd.api.types.is_numeric_dtype(series):
raise TypeError(f"列 '{column}' 不是数值类型，无法进行异常值检测")
Q1 = series.quantile(0.25)
Q3 = series.quantile(0.75)
IQR = Q3 - Q1
lower_bound = Q1 - 1.5 * IQR
upper_bound = Q3 + 1.5 * IQR
anomalies_mask = (series < lower_bound) | (series > upper_bound)
anomaly_indices = series[anomalies_mask].index.tolist()
return {
'count': int(anomalies_mask.sum()),
'indices': anomaly_indices,
'lower_bound': float(lower_bound),
'upper_bound': float(upper_bound),
'method': 'IQR (1.5x)',
'column': column,
}
def get_correlation_matrix(df: pd.DataFrame) -> pd.DataFrame:
"""计算数值列之间的相关系数矩阵
Args:
df: 数据框
Returns:
pd.DataFrame: 相关系数矩阵
"""
numeric_df = df.select_dtypes(include=[np.number])
return numeric_df.corr()
def sample_data(df: pd.DataFrame, n: int = 5, random: bool = True) -> pd.DataFrame:
"""随机抽样数据
Args:
df: 数据框
n: 抽样数量
random: 是否随机抽样
Returns:
pd.DataFrame: 抽样后的数据框
"""
if random:
return df.sample(n=min(n, len(df)))
else:
return df.head(n)
FILE:core/templates.py
"""报表模板配置模块"""
from typing import Dict, Any, List
# 模板配置字典
TEMPLATES: Dict[str, Dict[str, Any]] = {
'monthly_operation': {
'name': '月度运营报告',
'description': '适用于月度业务运营数据分析',
'sheets': ['封面', '数据概览', '图表分析', 'AI分析', '原始数据'],
'chart_types': ['line', 'bar', 'pie'],
'recommended_columns': ['日期', '月份', '销售额', '订单量', '用户数', '转化率'],
},
'financial': {
'name': '财务报表',
'description': '适用于财务收支、预算执行等分析',
'sheets': ['封面', '数据概览', '图表分析', 'AI分析', '原始数据'],
'chart_types': ['bar', 'line', 'pie'],
'recommended_columns': ['日期', '收入', '支出', '利润', '预算', '实际'],
},
'sales': {
'name': '销售报告',
'description': '适用于销售业绩、区域对比等分析',
'sheets': ['封面', '数据概览', '图表分析', 'AI分析', '原始数据'],
'chart_types': ['bar', 'line', 'scatter'],
'recommended_columns': ['日期', '销售额', '客户数', '产品数', '区域', '渠道'],
},
'data_comparison': {
'name': '数据对比报告',
'description': '适用于多周期、多维度数据对比',
'sheets': ['封面', '数据概览', '图表分析', 'AI分析', '原始数据'],
'chart_types': ['bar', 'line'],
'recommended_columns': ['期间', '指标A', '指标B', '变化率'],
},
'custom': {
'name': '自定义模板',
'description': '通用模板，可根据实际数据灵活配置',
'sheets': ['封面', '数据概览', '图表分析', 'AI分析', '原始数据'],
'chart_types': ['line', 'bar', 'pie', 'scatter'],
'recommended_columns': [],
},
}
def get_template(template_name: str) -> Dict[str, Any]:
"""获取指定模板配置
Args:
template_name: 模板名称
Returns:
dict: 模板配置字典
Raises:
KeyError: 模板不存在时抛出
"""
if template_name not in TEMPLATES:
raise KeyError(f"模板 '{template_name}' 不存在。可用模板: {list(TEMPLATES.keys())}")
return TEMPLATES[template_name]
def list_templates() -> List[str]:
"""列出所有可用模板名称
Returns:
list: 模板名称列表
"""
return list(TEMPLATES.keys())
def get_template_info(template_name: str) -> str:
"""获取模板的简要信息
Args:
template_name: 模板名称
Returns:
str: 模板信息描述
"""
template = get_template(template_name)
return f"[{template['name']}] {template['description']}"
Sentiment Analysis Monitor — AI-powered social media sentiment monitoring & analysis tool. Monitors Xiaohongshu, Douyin, Weibo, WeChat Official Accounts for...
---
name: sentiment-analysis-monitor
description: "Sentiment Analysis Monitor — AI-powered social media sentiment monitoring & analysis tool. Monitors Xiaohongshu, Douyin, Weibo, WeChat Official Accounts for keyword mentions. AI sentiment analysis (positive/neutral/negative), auto-generated sentiment reports, Feishu/email alerts when negative threshold exceeded. Trigger: sentiment, sentiment monitoring, social media monitoring, sentiment analysis, brand monitoring, negative alerts."
override-tools: []
---
# Sentiment Analysis Monitor
AI-powered social media sentiment monitoring and analysis tool for Chinese platforms. Monitor keyword mentions across Xiaohongshu, Douyin, Weibo, and WeChat Official Accounts in real time.
## Features Overview
| Feature | Description |
|---------|-------------|
| Platform Monitoring | Xiaohongshu, Douyin, Weibo, WeChat Official Account keyword search |
| AI Sentiment Analysis | Positive / Neutral / Negative + reason summary |
| Sentiment Reports | Total mentions, sentiment ratio, trending charts, top posts |
| Auto Alerts | Feishu/email push when negative mentions exceed threshold |
| Scheduled Crawling | OpenClaw Cron for periodic scraping |
| Storage | Local SQLite + JSON |
**Key**: No official platform APIs required — pure Playwright scraping of public content.
---
## Quick Start
### Add Keyword Monitoring
```
User: Monitor keyword "brand_name" on Xiaohongshu and Douyin
User: Add sentiment monitoring for "product_name", platforms: Weibo + WeChat Official Account
```
→ Parse keyword and platforms → Create monitoring task → Execute first crawl → Return result summary
### View Sentiment Report
```
User: Show sentiment report for "brand_name"
User: How is "competitor_name" trending in the last 7 days?
```
→ Return structured report: total mentions, positive/neutral/negative ratios, trending charts, top post list
### Set Alert Rules
```
User: Set negative alert for "brand_name", threshold 10 posts/day, notify me when exceeded
User: Configure Feishu alert, push to "Operations Group"
```
→ Configure negative threshold and push channel → Auto-judge after each crawl
### Manage Monitoring Tasks
```
User: List my sentiment monitoring tasks
User: Delete monitoring for "brand_name"
User: Pause monitoring for "competitor_name"
```
---
## Tiered Features
| Feature | FREE | PRO |
|---------|:----:|:---:|
| Keywords | 1 | Unlimited |
| Platforms | Xiaohongshu | All 4 platforms |
| Daily limit | 50 | Unlimited |
| Data history | 7 days | Unlimited |
| Sentiment reports | — | Yes |
| Priority monitoring | — | Yes |
---
## Platform Monitoring Details
### Xiaohongshu (XHS)
- **Search URL**: `https://www.xiaohongshu.com/search_result?keyword={keyword}&source=web_explore_search`
- **Anti-detection**: Playwright headless, UA rotation, random delay 3~8s
- **Content extracted**: Note title, body, author, likes/bookmarks/comments count, publish time
### Douyin
- **Search URL**: `https://www.douyin.com/search/{keyword}`
- **Anti-detection**: Playwright headless, scroll simulation, lazy-load handling
- **Content extracted**: Video title, author, likes/comments/shares count, publish time
### Weibo
- **Search URL**: `https://s.weibo.com/weibo?q={keyword}&typeall=1`
- **Anti-detection**: Playwright headless, UA rotation
- **Content extracted**: Post body, author, reposts/comments/likes count, publish time
### WeChat Official Accounts
- **Search URL**: `https://weixin.sogou.com/weixin?type=2&query={keyword}`
- **Anti-detection**: Playwright headless
- **Content extracted**: Article title, abstract, account name, read count, publish time
---
## Sentiment Analysis
Chinese semantic sentiment analysis via GLM-4 API:
```
Input: Post body / comment content
Output:
sentiment: "positive" | "neutral" | "negative"
score: -1.0 ~ 1.0 (negative to positive)
reason: Brief reason summary
```
**Classification rules**:
- Positive: score > 0.1
- Neutral: -0.1 <= score <= 0.1
- Negative: score < -0.1
---
## Alert Rules
| Rule | Description |
|------|-------------|
| Negative threshold | Trigger when daily negative mentions exceed N (default: 5) |
| Trend alert | Trigger when negative rate increases > 20% week-over-week |
| Push channels | Feishu group bot / Email (SMTP) |
### Feishu Alert Message Template
```
Sentiment Alert | {keyword}
Time: {time}
Today's Negatives: {negative_count} (threshold: {threshold})
Negative Rate: {negative_rate}%
Latest Negative Posts:
- {title} — {platform} @{author}
```
---
## Usage Examples
### Example 1: Brand Sentiment Monitoring
```
User: Monitor "coffee brand" on Xiaohongshu and Douyin, crawl every day at 9am
```
→ Create task → Return confirmation → Next Cron trigger executes first crawl
### Example 2: Competitor Negative Alert
```
User: Alert me via Feishu when negative posts appear for "competitor"
```
→ Set negative threshold alert → Configure Feishu group bot → Auto-push when threshold exceeded
### Example 3: Sentiment Report
```
User: Generate this week's sentiment report for "brand_name"
```
→ Query local SQLite for this week's data → AI generate summary → Return Markdown report
---
## Core Scripts
See `scripts/sentiment.py` for full implementation:
```python
from scripts.sentiment import SentimentCompass
compass = SentimentCompass(tier="PRO")
# ─── Add keyword monitoring ──────────────
compass.add_keyword(
keyword="brand_name",
platforms=["xhs", "douyin", "weibo", "wechat"],
frequency="daily", # 6h/12h/daily/weekly
priority=1, # 1=high priority (Pro only)
)
# ─── Execute crawl (manual) ──────────────
results = compass.crawl_keyword("brand_name")
# ─── Sentiment analysis (single) ─────────
analysis = compass.analyze_sentiment("This product is really great, highly recommended!")
# → {"sentiment": "positive", "score": 0.85, "reason": "Contains positive words like 'great' and 'highly recommended'"}
# ─── Batch analysis (save API calls) ─────
batch = compass.batch_analyze([
"Product is great, worth buying",
"Quality is terrible, not worth the price at all",
"It's okay, just average",
])
for item in batch:
print(f"[{item['sentiment']}] {item['text'][:30]}")
# ─── Generate report ─────────────────────
report = compass.generate_report(keyword="brand_name", days=7)
print(report["summary"]) # AI-generated text summary
print(report["stats"]) # Statistical data
# ─── Check alerts ───────────────────────
alerts = compass.check_alerts(keyword="brand_name")
if alerts:
compass.send_feishu_alert(alerts)
# ─── List tasks ─────────────────────────
tasks = compass.list_tasks()
for t in tasks:
print(f" {t['keyword']} — {t['platforms']} — {t['status']}")
```
---
## Technical Implementation
- **Crawler**: Playwright (headless) for dynamic pages, UA rotation, random delay 3~8s
- **AI Analysis**: GLM-4 API (`open.bigmodel.cn`), batch analysis to save tokens
- **Storage**: SQLite (`/tmp/sentiment-analysis-monitor/data.db`) + JSON config
- **Scheduling**: OpenClaw Cron, supports 6h/12h/daily/weekly frequency
- **Push**: Feishu group bot Webhook / Email SMTP
---
## Data Model
```sql
-- Monitoring tasks
CREATE TABLE tasks (
id INTEGER PRIMARY KEY,
keyword TEXT UNIQUE,
platforms TEXT, -- comma-separated: xhs,douyin,weibo,wechat
frequency TEXT DEFAULT 'daily',
priority INTEGER DEFAULT 0,
status TEXT DEFAULT 'active',
created_at TEXT,
last_crawl_at TEXT
);
-- Post data
CREATE TABLE posts (
id INTEGER PRIMARY KEY,
keyword TEXT,
platform TEXT, -- xhs/douyin/weibo/wechat
post_id TEXT,
title TEXT,
content TEXT,
author TEXT,
author_id TEXT,
likes INTEGER DEFAULT 0,
comments INTEGER DEFAULT 0,
shares INTEGER DEFAULT 0,
published_at TEXT,
fetched_at TEXT,
url TEXT UNIQUE
);
-- Sentiment analysis results
CREATE TABLE analyses (
id INTEGER PRIMARY KEY,
post_id INTEGER REFERENCES posts(id),
sentiment TEXT, -- positive/neutral/negative
score REAL, -- -1.0 ~ 1.0
reason TEXT,
analyzed_at TEXT
);
-- Alert records
CREATE TABLE alerts (
id INTEGER PRIMARY KEY,
keyword TEXT,
alert_type TEXT, -- threshold/trend
threshold INTEGER,
negative_count INTEGER,
negative_rate REAL,
triggered_at TEXT,
notification_sent INTEGER DEFAULT 0
);
```
---
## FAQ
| Question | Answer |
|----------|--------|
| Will accounts get blocked? | Pure public content scraping with 3~8s random delay between requests, 3 retries on failure |
| Does it support login-gated content? | Current version does not support login-required pages |
| How accurate is sentiment analysis? | Based on GLM-4 Chinese semantic understanding; accuracy depends on text length and context |
| How many keywords can I monitor? | FREE=1, PRO=unlimited |
| How long is data retained? | FREE=7 days, Pro+=unlimited |
| How to configure Feishu alerts? | Provide group bot Webhook URL — no app permissions needed |
---
## Tier Limits
```python
TIER_LIMITS = {
"FREE": {"max_keywords": 1, "platforms": ["xhs"], "daily_limit": 50, "history_days": 7},
"PRO": {"max_keywords": -1, "platforms": ["xhs","douyin","weibo","wechat"], "daily_limit": -1, "history_days": -1, "report": True, "priority": True},
}
```
---
## Security Notes
- **SSRF Protection**: `fetch_page()` validates all URLs before sending to Playwright. Blocks: non-HTTP(S) schemes, localhost, 127.0.0.1, private IP ranges (10.x.x.x, 172.16-31.x.x, 192.168.x.x), link-local (169.254.x.x including AWS metadata 169.254.169.254), and IPv6 localhost. Unsafe URLs return `None` and log a warning.
- **Subprocess execution**: Uses `node -e` subprocess for Playwright browser automation (list form, not shell=True). curl calls for AI API use list form with `json=` payload. No command injection risk.
- **Data storage**: Uses `/tmp/` for SQLite DB and config (no home directory write).
- **Billing data**: `FEISHU_USER_ID` transmitted to `skillpay.me/api/v1/billing` for per-call charging.
## Billing
- **Pay-per-call**: $0.0100 USDT per execution via SkillPay.me
- **Balance insufficient**: Payment URL returned — user tops up at `https://skillpay.me/sentiment-analysis-monitor`
- **External data flow**: `FEISHU_USER_ID` transmitted to `skillpay.me/api/v1/billing` for balance charging
---
## Required Environment Variables
| Variable | Description |
|----------|-------------|
| `FEISHU_USER_ID` | User open_id for billing (passed by Feishu runtime) |
| `SKILL_BILLING_API_KEY` | SkillPay Builder API Key |
| `SKILL_BILLING_SKILL_ID` | SkillPay Skill ID (defaults to `sentiment-analysis-monitor`) |
FILE:requirements.txt
# Sentiment Compass dependencies
playwright>=1.40.0
beautifulsoup4>=4.12.0
jieba>=0.42.1
requests>=2.31.0
FILE:scripts/billing.py
#!/usr/bin/env python3
"""
Sentiment Analysis Monitor — SkillPay Billing Integration.
Pay-per-call: $0.01 USDT per execution.
Balance insufficient -> payment_url returned (user tops up at skillpay.me/{slug}).
Required environment variables:
SKILL_BILLING_API_KEY - SkillPay Builder API Key
SKILL_BILLING_SKILL_ID - SkillPay Skill ID (slug: sentiment-analysis-monitor)
FEISHU_USER_ID - User open_id for billing
Billing API docs: https://skillpay.me/api/v1/billing
"""
import os
import time
import requests
from pathlib import Path
BILLING_API_URL = "https://skillpay.me/api/v1/billing"
CALL_PRICE = 0.0100
_CACHE_TTL = 300
_cache: dict = {}
def _cache_get(key: str) -> dict | None:
entry = _cache.get(key)
if entry is None:
return None
if time.time() - entry["_ts"] > _CACHE_TTL:
del _cache[key]
return None
return entry
def _cache_set(key: str, data: dict) -> None:
_cache[key] = {**data, "_ts": time.time()}
def _get_headers() -> dict:
return {
"X-API-Key": os.environ.get("SKILL_BILLING_API_KEY", ""),
"Content-Type": "application/json",
}
def _get_skill_id() -> str:
return os.environ.get("SKILL_BILLING_SKILL_ID", "sentiment-analysis-monitor")
def _is_dev_mode() -> bool:
return os.environ.get("SKILL_BILLING_API_KEY", "").strip() == ""
def charge_user(user_id: str) -> dict:
"""
Charge user for one execution ($0.01 USDT).
Returns: {"ok": True, "balance": float} on success
{"ok": False, "balance": float, "payment_url": str} on insufficient balance
"""
if _is_dev_mode():
return {"ok": True, "balance": 999.0}
skill_id = _get_skill_id()
uid = user_id or os.environ.get("FEISHU_USER_ID", "") or "anonymous"
try:
resp = requests.post(
f"{BILLING_API_URL}/charge",
headers=_get_headers(),
json={
"user_id": uid,
"skill_id": skill_id,
"amount": CALL_PRICE,
},
timeout=10,
)
data = resp.json()
if data.get("success"):
return {"ok": True, "balance": float(data.get("balance", 0.0))}
return {
"ok": False,
"balance": float(data.get("balance", 0.0)),
"payment_url": data.get("payment_url", f"https://skillpay.me/{skill_id}"),
}
except Exception:
return {"ok": True, "balance": 999.0}
FILE:scripts/sentiment.py
#!/usr/bin/env python3
"""
Sentiment Compass — AI-driven social media sentiment monitoring engine.
AI-driven social media sentiment monitoring for Chinese platforms.
"""
import hashlib
import json
import os
import random
import re
import signal
import sqlite3
import subprocess
import sys
import time
from dataclasses import dataclass, asdict, field
from datetime import datetime, timezone, timedelta
from pathlib import Path
from billing import charge_user
from typing import Optional, List, Dict, Any
# ─── Paths ────────────────────────────────────────────────────────────────────
SCRIPT_DIR = Path(__file__).parent.resolve()
DATA_DIR = Path("/tmp/sentiment-analysis-monitor")
DATA_DIR.mkdir(parents=True, exist_ok=True)
DB_PATH = DATA_DIR / "data.db"
CONFIG_PATH = DATA_DIR / "config.json"
LOG_DIR = DATA_DIR / "logs"
LOG_DIR.mkdir(exist_ok=True)
# ─── Tier Limits ───────────────────────────────────────────────────────────────
TIER_LIMITS = {
"FREE": {
"max_keywords": 1, "platforms": ["xhs"],
"daily_limit": 50, "history_days": 7,
"report": False, "priority": False,
},
"PRO": {
"max_keywords": -1, "platforms": ["xhs", "douyin", "weibo", "wechat"],
"daily_limit": -1, "history_days": -1,
"report": True, "priority": True,
},
}
# ─── Platform Config ──────────────────────────────────────────────────────────
PLATFORM_CONFIG = {
"xhs": {
"name": "Xiaohongshu",
"search_url": "https://www.xiaohongshu.com/search_result?keyword={keyword}&source=web_explore_search",
"search_url_fallback": "https://www.xiaohongshu.com/search_result?keyword={keyword}",
},
"douyin": {
"name": "Douyin",
"search_url": "https://www.douyin.com/search/{keyword}",
},
"weibo": {
"name": "Weibo",
"search_url": "https://s.weibo.com/weibo?q={keyword}&typeall=1",
},
"wechat": {
"name": "WeChat Official Accounts",
"search_url": "https://weixin.sogou.com/weixin?type=2&query={keyword}",
},
}
# ─── User Agent Pool ───────────────────────────────────────────────────────────
UA_POOL = [
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Chrome/120.0.0.0 Safari/537.36",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 Chrome/120.0.0.0 Safari/537.36",
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/120.0.0.0 Safari/537.36",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) Gecko/20100101 Firefox/121.0",
"Mozilla/5.0 (iPhone; CPU iPhone OS 17_0 like Mac OS X) AppleWebKit/605.1.15 Mobile/15E148 Safari/604.1",
]
# ─── GLM-4 API Config ──────────────────────────────────────────────────────────
GLM_API_URL = "https://open.bigmodel.cn/api/paas/v4/chat/completions"
GLM_MODEL = "glm-4-flash"
# ─── Dataclasses ──────────────────────────────────────────────────────────────
@dataclass
class Post:
keyword: str
platform: str # xhs/douyin/weibo/wechat
post_id: str
title: str
content: str
author: str
author_id: str
likes: int = 0
comments: int = 0
shares: int = 0
published_at: str = ""
fetched_at: str = ""
url: str = ""
def to_dict(self) -> dict:
return asdict(self)
@classmethod
def from_dict(cls, d: dict) -> "Post":
return cls(**{k: v for k, v in d.items() if k in cls.__dataclass_fields__})
@dataclass
class SentimentResult:
post_id: int
sentiment: str # positive/neutral/negative
score: float # -1.0 ~ 1.0
reason: str
analyzed_at: str = ""
def __post_init__(self):
if not self.analyzed_at:
self.analyzed_at = datetime.now(timezone.utc).isoformat()
def to_dict(self) -> dict:
return asdict(self)
@dataclass
class AlertRecord:
keyword: str
alert_type: str # threshold/trend
threshold: int
negative_count: int
negative_rate: float
triggered_at: str = ""
notification_sent: int = 0
def __post_init__(self):
if not self.triggered_at:
self.triggered_at = datetime.now(timezone.utc).isoformat()
def to_dict(self) -> dict:
return asdict(self)
# ─── Database ─────────────────────────────────────────────────────────────────
def _get_db() -> sqlite3.Connection:
conn = sqlite3.connect(str(DB_PATH))
conn.execute("PRAGMA journal_mode=WAL")
conn.execute("""
CREATE TABLE IF NOT EXISTS tasks (
id INTEGER PRIMARY KEY AUTOINCREMENT,
keyword TEXT UNIQUE NOT NULL,
platforms TEXT NOT NULL,
frequency TEXT DEFAULT 'daily',
priority INTEGER DEFAULT 0,
status TEXT DEFAULT 'active',
created_at TEXT NOT NULL,
last_crawl_at TEXT,
alert_threshold INTEGER DEFAULT 5,
alert_channels TEXT DEFAULT ''
)
""")
conn.execute("""
CREATE TABLE IF NOT EXISTS posts (
id INTEGER PRIMARY KEY AUTOINCREMENT,
keyword TEXT NOT NULL,
platform TEXT NOT NULL,
post_id TEXT NOT NULL,
title TEXT,
content TEXT,
author TEXT,
author_id TEXT,
likes INTEGER DEFAULT 0,
comments INTEGER DEFAULT 0,
shares INTEGER DEFAULT 0,
published_at TEXT,
fetched_at TEXT NOT NULL,
url TEXT UNIQUE,
UNIQUE(keyword, platform, post_id)
)
""")
conn.execute("""
CREATE TABLE IF NOT EXISTS analyses (
id INTEGER PRIMARY KEY AUTOINCREMENT,
post_id INTEGER REFERENCES posts(id) ON DELETE CASCADE,
sentiment TEXT,
score REAL,
reason TEXT,
analyzed_at TEXT NOT NULL
)
""")
conn.execute("""
CREATE TABLE IF NOT EXISTS alerts (
id INTEGER PRIMARY KEY AUTOINCREMENT,
keyword TEXT NOT NULL,
alert_type TEXT NOT NULL,
threshold INTEGER,
negative_count INTEGER,
negative_rate REAL,
triggered_at TEXT NOT NULL,
notification_sent INTEGER DEFAULT 0
)
""")
conn.execute("""
CREATE TABLE IF NOT EXISTS configs (
key TEXT PRIMARY KEY,
value TEXT
)
""")
conn.commit()
return conn
# ─── Config ────────────────────────────────────────────────────────────────────
def load_config() -> dict:
if CONFIG_PATH.exists():
try:
return json.loads(CONFIG_PATH.read_text(encoding="utf-8"))
except Exception:
pass
return {"tier": "FREE", "glm_api_key": "", "feishu_webhook": "", "smtp_config": {}}
def save_config(cfg: dict):
CONFIG_PATH.write_text(json.dumps(cfg, ensure_ascii=False, indent=2), encoding="utf-8")
def get_config(key: str, default=None):
cfg = load_config()
return cfg.get(key, default)
def set_config(key: str, value):
cfg = load_config()
cfg[key] = value
save_config(cfg)
# ─── SSRF Protection ─────────────────────────────────────────────────────────
_PRIVATE_IP_PATTERNS = [
r"127\.\d{1,3}\.\d{1,3}\.\d{1,3}",
r"10\.\d{1,3}\.\d{1,3}\.\d{1,3}",
r"172\.(?:1[6-9]|2\d|3[01])\.\d{1,3}\.\d{1,3}",
r"192\.168\.\d{1,3}\.\d{1,3}",
r"169\.254\.(?:\d{1,3}\.)?\d{1,3}",
r"0\.\d{1,3}\.\d{1,3}\.\d{1,3}",
r"(?:[fF][cCdD][0-9a-fA-F]{2}:[0-9a-fA-F:]+)",
r"(?:[fF][eE][89aAbB][0-9a-fA-F:]+[%\w]*)",
r"::1(?:\]|\Z)",
r"\[?::1\]?(?:\]|\Z)",
]
_PRIVATE_IP_RE = re.compile("(?:" + "|".join(_PRIVATE_IP_PATTERNS) + ")$", re.IGNORECASE)
def _is_url_safe(url: str) -> bool:
"""Block SSRF: only HTTP(S), reject localhost/private IPs."""
try:
from urllib.parse import urlparse
except ImportError:
from urlparse import urlparse
p = urlparse(url)
scheme = p.scheme.lower()
hostname = (p.hostname or "").lower()
if scheme not in ("http", "https"):
return False
if hostname in ("localhost", "localhost.localdomain", "ip6-localhost", "ip6-loopback"):
return False
if _PRIVATE_IP_RE.match(hostname):
return False
return True
# ─── Playwright Fetcher ────────────────────────────────────────────────────────
def fetch_page(url: str, platform: str = "xhs", timeout_ms: int = 20000) -> Optional[str]:
"""
Fetch page content using Playwright (Node.js subprocess).
SSRF protection: rejects non-HTTP(S) URLs, localhost, private/internal IPs.
"""
# SSRF guard
if not _is_url_safe(url):
_log("WARN", f"SSRF blocked: {url}")
return None
ua = random.choice(UA_POOL)
# Platform-specific JS
if platform == "xhs":
script = f"""
const {{ chromium }} = require('playwright');
(async () => {{
const browser = await chromium.launch({{ headless: true }});
const ctx = await browser.newContext({{
userAgent: {json.dumps(ua)},
viewport: {{ width: 1280, height: 800 }},
locale: 'zh-CN',
}});
const page = await ctx.newPage();
await page.setExtraHTTPHeaders({{ 'Accept-Language': 'zh-CN,zh;q=0.9,en;q=0.8' }});
// Random delay before request
await page.waitForTimeout({random.randint(2000, 5000)});
await page.goto({json.dumps(url)}, {{ waitUntil: 'networkidle', timeout: {timeout_ms} }});
// Scroll to load dynamic content
await page.evaluate(() => window.scrollTo(0, document.body.scrollHeight / 2));
await page.waitForTimeout(2000);
await page.evaluate(() => window.scrollTo(0, document.body.scrollHeight));
await page.waitForTimeout(2000);
const content = await page.content();
await browser.close();
console.log(JSON.stringify({{ ok: true, content }}));
}})().catch(e => {{ console.log(JSON.stringify({{ ok: false, error: e.message }})); process.exit(1); }});
"""
elif platform == "douyin":
script = f"""
const {{ chromium }} = require('playwright');
(async () => {{
const browser = await chromium.launch({{ headless: true }});
const ctx = await browser.newContext({{
userAgent: {json.dumps(ua)},
viewport: {{ width: 390, height: 844 }},
locale: 'zh-CN',
deviceScaleFactor: 3,
}});
const page = await ctx.newPage();
await page.setExtraHTTPHeaders({{ 'Accept-Language': 'zh-CN,zh;q=0.9' }});
await page.waitForTimeout({random.randint(3000, 6000)});
await page.goto({json.dumps(url)}, {{ waitUntil: 'domcontentloaded', timeout: {timeout_ms} }});
// Simulate scroll for infinite scroll
for (let i = 0; i < 3; i++) {{
await page.evaluate(() => window.scrollTo(0, document.body.scrollHeight * (i+1) / 3));
await page.waitForTimeout(1500);
}}
const content = await page.content();
await browser.close();
console.log(JSON.stringify({{ ok: true, content }}));
}})().catch(e => {{ console.log(JSON.stringify({{ ok: false, error: e.message }})); process.exit(1); }});
"""
elif platform == "weibo":
script = f"""
const {{ chromium }} = require('playwright');
(async () => {{
const browser = await chromium.launch({{ headless: true }});
const ctx = await browser.newContext({{
userAgent: {json.dumps(ua)},
viewport: {{ width: 1280, height: 800 }},
locale: 'zh-CN',
}});
const page = await ctx.newPage();
await page.setExtraHTTPHeaders({{ 'Accept-Language': 'zh-CN,zh;q=0.9' }});
await page.waitForTimeout({random.randint(2000, 5000)});
await page.goto({json.dumps(url)}, {{ waitUntil: 'networkidle', timeout: {timeout_ms} }});
const content = await page.content();
await browser.close();
console.log(JSON.stringify({{ ok: true, content }}));
}})().catch(e => {{ console.log(JSON.stringify({{ ok: false, error: e.message }})); process.exit(1); }});
"""
elif platform == "wechat":
script = f"""
const {{ chromium }} = require('playwright');
(async () => {{
const browser = await chromium.launch({{ headless: true }});
const ctx = await browser.newContext({{
userAgent: {json.dumps(ua)},
viewport: {{ width: 1280, height: 800 }},
locale: 'zh-CN',
}});
const page = await ctx.newPage();
await page.setExtraHTTPHeaders({{ 'Accept-Language': 'zh-CN,zh;q=0.9' }});
await page.waitForTimeout({random.randint(2000, 5000)});
await page.goto({json.dumps(url)}, {{ waitUntil: 'networkidle', timeout: {timeout_ms} }});
const content = await page.content();
await browser.close();
console.log(JSON.stringify({{ ok: true, content }}));
}})().catch(e => {{ console.log(JSON.stringify({{ ok: false, error: e.message }})); process.exit(1); }});
"""
else:
script = f"""
const {{ chromium }} = require('playwright');
(async () => {{
const browser = await chromium.launch({{ headless: true }});
const page = await browser.newPage();
await page.setExtraHTTPHeaders({{ 'Accept-Language': 'zh-CN,zh;q=0.9' }});
await page.waitForTimeout({random.randint(2000, 5000)});
await page.goto({json.dumps(url)}, {{ waitUntil: 'networkidle', timeout: {timeout_ms} }});
const content = await page.content();
await browser.close();
console.log(JSON.stringify({{ ok: true, content }}));
}})().catch(e => {{ console.log(JSON.stringify({{ ok: false, error: e.message }})); process.exit(1); }});
"""
try:
result = subprocess.run(
["node", "-e", script],
capture_output=True, text=True, timeout=45
)
if result.returncode != 0:
_log("WARN", f"Playwright fetch failed for {url}: {result.stderr[:200]}")
return None
data = json.loads(result.stdout.strip())
if data.get("ok"):
return data["content"]
else:
_log("WARN", f"Playwright returned error for {url}: {data.get('error', '')}")
except subprocess.TimeoutExpired:
_log("WARN", f"Playwright timeout for {url}")
except json.JSONDecodeError:
_log("WARN", f"Playwright invalid JSON for {url}: {result.stdout[:200]}")
except Exception as e:
_log("ERROR", f"Playwright exception for {url}: {e}")
return None
# ─── Content Parsers ──────────────────────────────────────────────────────────
def parse_xhs_posts(html: str, keyword: str) -> List[Post]:
"""Parse Xiaohongshu search results from HTML."""
from bs4 import BeautifulSoup
posts = []
try:
soup = BeautifulSoup(html, "html.parser")
# Note cards - different possible selectors
cards = soup.select(".note-item") or soup.select(".feeds-page .note") or soup.select("[class*='note']")
for card in cards:
try:
title_el = card.select_one(".title") or card.select_one("h2") or card.select_one("[class*='title']")
content_el = card.select_one(".desc") or card.select_one(".abstract") or card.select_one("[class*='desc']")
author_el = card.select_one(".author") or card.select_one(".nickname") or card.select_one("[class*='author']")
like_el = card.select_one(".like") or card.select_one("[class*='like']")
# Try to find links
links = card.select("a[href*='/discovery/item/']")
url = "https://www.xiaohongshu.com" + links[0]["href"] if links else ""
post_id_match = re.search(r'/discovery/item/([a-f0-9]+)', url)
post_id = post_id_match.group(1) if post_id_match else hashlib.md5((title_el.text if title_el else "").encode()).hexdigest()[:12]
posts.append(Post(
keyword=keyword,
platform="xhs",
post_id=post_id,
title=title_el.get_text(strip=True) if title_el else "",
content=content_el.get_text(strip=True) if content_el else "",
author=author_el.get_text(strip=True) if author_el else "",
author_id="",
likes=_parse_number(like_el.get_text(strip=True) if like_el else "0"),
comments=0, shares=0,
published_at="",
fetched_at=datetime.now(timezone.utc).isoformat(),
url=url,
))
except Exception:
continue
except Exception as e:
_log("WARN", f"XHS parse error: {e}")
return posts
def parse_douyin_posts(html: str, keyword: str) -> List[Post]:
"""Parse Douyin search results from HTML."""
from bs4 import BeautifulSoup
posts = []
try:
soup = BeautifulSoup(html, "html.parser")
video_items = soup.select(".video-feed-list .video-item") or \
soup.select("[class*='video']") or \
soup.select("li[data-e2e='video-list-item']")
for item in video_items:
try:
title_el = item.select_one(".title") or item.select_one("h3") or item.select_one("[class*='title']")
author_el = item.select_one(".author") or item.select_one("[class*='author']")
like_el = item.select_one(".like-count") or item.select_one("[class*='like']")
links = item.select("a[href*='/video/']")
url = "https://www.douyin.com" + links[0]["href"] if links else ""
post_id_match = re.search(r'/video/(\d+)', url)
post_id = post_id_match.group(1) if post_id_match else hashlib.md5((title_el.text if title_el else "").encode()).hexdigest()[:12]
posts.append(Post(
keyword=keyword,
platform="douyin",
post_id=post_id,
title=title_el.get_text(strip=True) if title_el else "",
content="", # Douyin content requires video page
author=author_el.get_text(strip=True) if author_el else "",
author_id="",
likes=_parse_number(like_el.get_text(strip=True) if like_el else "0"),
comments=0, shares=0,
published_at="",
fetched_at=datetime.now(timezone.utc).isoformat(),
url=url,
))
except Exception:
continue
except Exception as e:
_log("WARN", f"Douyin parse error: {e}")
return posts
def parse_weibo_posts(html: str, keyword: str) -> List[Post]:
"""Parse Weibo search results from HTML."""
from bs4 import BeautifulSoup
posts = []
try:
soup = BeautifulSoup(html, "html.parser")
items = soup.select(".card-feed") or soup.select(".wb-item") or soup.select("[class*='feed']")
for item in items:
try:
content_el = item.select_one(".content") or item.select_one("[class*='content']")
author_el = item.select_one(".name") or item.select_one("[class*='name']")
like_el = item.select_one(".like") or item.select_one("[class*='like']")
links = item.select("a[href*='/detail']")
url = "https://weibo.com" + links[0]["href"] if links else ""
post_id_match = re.search(r'/detail/(\w+)', url)
# Get text content
text_parts = []
if content_el:
for p in content_el.select("p"):
text_parts.append(p.get_text(strip=True))
title_text = text_parts[0][:80] if text_parts else ""
posts.append(Post(
keyword=keyword,
platform="weibo",
post_id=post_id_match.group(1) if post_id_match else hashlib.md5((title_text).encode()).hexdigest()[:12],
title=title_text,
content="\n".join(text_parts),
author=author_el.get_text(strip=True) if author_el else "",
author_id="",
likes=_parse_number(like_el.get_text(strip=True) if like_el else "0"),
comments=0, shares=0,
published_at="",
fetched_at=datetime.now(timezone.utc).isoformat(),
url=url,
))
except Exception:
continue
except Exception as e:
_log("WARN", f"Weibo parse error: {e}")
return posts
def parse_wechat_posts(html: str, keyword: str) -> List[Post]:
"""Parse WeChat public account articles from Sogou."""
from bs4 import BeautifulSoup
posts = []
try:
soup = BeautifulSoup(html, "html.parser")
items = soup.select(".news-box .news-list li") or \
soup.select("[class*='article']") or \
soup.select(".weui-article")
for item in items:
try:
title_el = item.select_one(".tit") or item.select_one("h3") or item.select_one("[class*='title']")
digest_el = item.select_one(".txt") or item.select_one(".abstract") or item.select_one("[class*='digest']")
author_el = item.select_one(".account") or item.select_one("[class*='account']")
date_el = item.select_one(".date") or item.select_one("[class*='date']")
links = item.select("a[href]")
url = links[0]["href"] if links else ""
post_id = hashlib.md5((title_el.get_text(strip=True) if title_el else "").encode()).hexdigest()[:12]
posts.append(Post(
keyword=keyword,
platform="wechat",
post_id=post_id,
title=title_el.get_text(strip=True) if title_el else "",
content=digest_el.get_text(strip=True) if digest_el else "",
author=author_el.get_text(strip=True) if author_el else "",
author_id="",
likes=0, comments=0, shares=0,
published_at=date_el.get_text(strip=True) if date_el else "",
fetched_at=datetime.now(timezone.utc).isoformat(),
url=url,
))
except Exception:
continue
except Exception as e:
_log("WARN", f"WeChat parse error: {e}")
return posts
def _parse_number(text: str) -> int:
"""Parse Chinese number format (e.g. 1.2万 = 12000) to integer."""
text = text.strip().replace(",", "")
if not text:
return 0
if "万" in text:
try:
return int(float(text.replace("万", "")) * 10000)
except ValueError:
return 0
try:
return int(float(text))
except ValueError:
return 0
# ─── GLM-4 Sentiment Analysis ──────────────────────────────────────────────────
def analyze_with_glm4(text: str, api_key: str = "") -> Optional[SentimentResult]:
"""Call GLM-4 API for sentiment analysis."""
if not api_key:
api_key = get_config("glm_api_key", "")
if not api_key:
return None
# Truncate text
text = text.strip()[:1500]
prompt = f"""You are a professional Chinese sentiment analysis model. Analyze the sentiment of the text below.
Requirements:
1. Output JSON format only, no other text
2. JSON has: sentiment (positive/neutral/negative), score (-1.0 to 1.0), reason (within 20 chars)
3. Positive: score > 0.1, Neutral: -0.1 <= score <= 0.1, Negative: score < -0.1
Text to analyze:
{text}
Output JSON only:"""
try:
payload = {
"model": GLM_MODEL,
"messages": [{"role": "user", "content": prompt}],
"temperature": 0.3,
"max_tokens": 256,
}
result = subprocess.run(
["curl", "-s", "-X", "POST", GLM_API_URL,
"-H", f"Authorization: Bearer {api_key}",
"-H", "Content-Type: application/json",
"-d", json.dumps(payload, ensure_ascii=False)],
capture_output=True, text=True, timeout=30
)
resp = json.loads(result.stdout)
content = resp["choices"][0]["message"]["content"].strip()
# Try to extract JSON
if "```json" in content:
content = content.split("```json")[1].split("```")[0]
elif "```" in content:
content = content.split("```")[1]
if content.startswith("json"):
content = content[4:]
data = json.loads(content.strip())
return SentimentResult(
post_id=0, # Will be set by caller
sentiment=data["sentiment"],
score=float(data["score"]),
reason=data["reason"],
)
except Exception as e:
_log("WARN", f"GLM-4 analysis failed: {e}")
return None
def rule_based_sentiment(text: str) -> SentimentResult:
"""
Rule-based fallback sentiment analysis (no API needed).
Used when API key is not configured or fails.
"""
text_lower = text.lower()
# Positive keywords
positive_words = [
"好", "棒", "赞", "优秀", "出色", "完美", "喜欢", "爱", "推荐", "值得",
"满意", "开心", "高兴", "漂亮", "美", "帅", "酷", "牛", "强", "实惠",
"划算", "便宜", "性价比", "良心", "负责", "认真", "专业", "有用", "有效",
"惊喜", "惊艳", "超值", "物超所值", "方便", "简单", "轻松", "舒适", "舒服",
"喜欢", "爱了", "太爱", "强烈推荐", "种草", "安利的", "回购", "一直用",
"很好", "真的不错", "太棒了", "绝了", "yyds", "永远的神",
]
# Negative keywords
negative_words = [
"差", "烂", "垃圾", "废物", "骗", "骗人", "假", "假货", "坑", "坑人",
"失望", "太差", "糟糕", "恶心", "难看", "丑", "后悔", "不值", "浪费",
"麻烦", "难用", "太差", "劣质", "无良", "奸商", "欺骗", "欺诈", "虚假",
"投诉", "曝光", "维权", "质量差", "服务差", "态度差", "骗子", "无赖",
"垃圾", "废物", "有病", "神经病", "白痴", "智障", "无语", "醉了", "吐了",
"再也不", "不会再来", "一生黑", "拉黑", "差评", "一分", "负分",
]
# Intensifiers
intensifiers = ["非常", "特别", "极其", "超级", "太", "真", "超", "巨", "无比", "相当"]
# Count hits using character n-grams (fallback when jieba unavailable)
pos_count = 0
neg_count = 0
intensifier = False
try:
import jieba
words = jieba.lcut(text)
except ImportError:
# Fallback: character-based word matching
words = []
i = 0
while i < len(text):
matched = False
# Try 2-char and 3-char words
for length in [3, 2]:
if i + length <= len(text):
word = text[i:i+length]
words.append(word)
i += length
matched = True
break
if not matched:
i += 1
for i, word in enumerate(words):
for iw in intensifiers:
if iw in word:
intensifier = True
break
for pw in positive_words:
if pw in word:
pos_count += 2 if intensifier else 1
for nw in negative_words:
if nw in word:
neg_count += 2 if intensifier else 1
# Negation check: only flip if actual negation phrase detected
# Don't flip when negation char is part of another word (e.g. "非常" contains "非" but means "very")
negation_phrases = [
r"^不[好不好对行能错]", # 不好/不对/不行/不能/不错 — actual negations
r"^没[有得错好]", # 没有/没得/没错/没好 — actual negations
r"^无所谓", # 无所谓
r"^别[买买|想|再]", # 别买/别想/别再 — actual negations
r"^休想", # 休想
r"^未[经完成]", # 未完成/未经 — actual negations
]
# Anti-patterns: phrases that START with a negation char but are actually positive/intensifiers
non_negation_starters = [
r"^非常", r"^无比", r"^相当", r"^超级", r"^特别", r"^极其",
r"^不但", r"^不仅", r"^除非", r"^无论", r"^莫非", r"^非凡",
]
negated = False
# Check if starts with a non-negation intensifier first
for pattern in non_negation_starters:
if re.search(pattern, text_lower):
negated = False
break
else:
# No intensifier found — check for actual negation patterns
for pattern in negation_phrases:
if re.search(pattern, text_lower):
negated = True
break
if negated:
pos_count, neg_count = neg_count, pos_count
total = pos_count + neg_count
if total == 0:
return SentimentResult(post_id=0, sentiment="neutral", score=0.0, reason="Unable to determine sentiment")
score = (pos_count - neg_count) / total
score = max(-1.0, min(1.0, score))
if score > 0.1:
sentiment = "positive"
elif score < -0.1:
sentiment = "negative"
else:
sentiment = "neutral"
if pos_count > 0 and neg_count == 0:
reason = f"Detected {pos_count} positive word(s)"
elif neg_count > 0 and pos_count == 0:
reason = f"Detected {neg_count} negative word(s)"
else:
reason = f"Positive {pos_count} vs negative {neg_count} words"
return SentimentResult(post_id=0, sentiment=sentiment, score=score, reason=reason)
def batch_analyze_with_glm4(texts: List[str], api_key: str = "") -> List[dict]:
"""Batch analyze multiple texts with GLM-4 to save API calls."""
if not api_key:
api_key = get_config("glm_api_key", "")
if not api_key:
return [rule_based_sentiment(t).to_dict() for t in texts]
# Prepare batch prompt
items_text = "\n".join(
f"[{i+1}] {t[:200]}" for i, t in enumerate(texts)
)
prompt = f"""You are a professional Chinese sentiment analysis model. Batch analyze the texts below.
Requirements:
1. Output JSON array only, no other text
2. Each element: index (starting from 1), sentiment (positive/neutral/negative), score (-1.0 to 1.0), reason (within 20 chars)
3. Positive: score > 0.1, Neutral: -0.1 <= score <= 0.1, Negative: score < -0.1
Texts to analyze:
{items_text}
Output JSON array only:"""
try:
payload = {
"model": GLM_MODEL,
"messages": [{"role": "user", "content": prompt}],
"temperature": 0.3,
"max_tokens": 2048,
}
result = subprocess.run(
["curl", "-s", "-X", "POST", GLM_API_URL,
"-H", f"Authorization: Bearer {api_key}",
"-H", "Content-Type: application/json",
"-d", json.dumps(payload, ensure_ascii=False)],
capture_output=True, text=True, timeout=60
)
resp = json.loads(result.stdout)
content = resp["choices"][0]["message"]["content"].strip()
if "```json" in content:
content = content.split("```json")[1].split("```")[0]
elif "```" in content:
content = content.split("```")[1]
if content.startswith("json"):
content = content[4:]
results = json.loads(content.strip())
return results
except Exception as e:
_log("WARN", f"GLM-4 batch analysis failed: {e}")
return [{"index": i+1, "sentiment": "neutral", "score": 0.0, "reason": "API failed"} for i in range(len(texts))]
# ─── Logging ──────────────────────────────────────────────────────────────────
def _log(level: str, msg: str):
ts = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
log_file = LOG_DIR / f"{datetime.now().strftime('%Y-%m-%d')}.log"
line = f"[{ts}] [{level}] {msg}"
print(line, flush=True)
try:
with open(log_file, "a", encoding="utf-8") as f:
f.write(line + "\n")
except Exception:
pass
# ─── SentimentCompass Class ────────────────────────────────────────────────────
class SentimentCompass:
def __init__(self, tier: str = "FREE", api_key: str = ""):
self.tier = tier.upper() if tier else "FREE"
self.conn = _get_db()
self.config = load_config()
self.limits = TIER_LIMITS.get(self.tier, TIER_LIMITS["FREE"])
# ── Task Management ──────────────────────────────────────────────────────
def add_keyword(
self,
keyword: str,
platforms: List[str],
frequency: str = "daily",
priority: int = 0,
alert_threshold: int = 5,
alert_channels: str = "",
) -> dict:
"""Add or update a monitoring keyword."""
now = datetime.now(timezone.utc).isoformat()
# Check tier limit
existing = self.list_tasks()
max_kw = self.limits["max_keywords"]
if max_kw != -1 and len(existing) >= max_kw:
return {"ok": False, "error": f"{self.tier} plan allows at most {max_kw} keywords"}
# Validate platforms
allowed = self.limits["platforms"]
for p in platforms:
if p not in allowed:
return {"ok": False, "error": f"{self.tier} plan does not support {p}. Available: {allowed}"}
platforms_str = ",".join(platforms)
try:
self.conn.execute("""
INSERT OR REPLACE INTO tasks
(keyword, platforms, frequency, priority, status, created_at, alert_threshold, alert_channels)
VALUES (?, ?, ?, ?, 'active', ?, ?, ?)
""", (keyword, platforms_str, frequency, priority, now, alert_threshold, alert_channels))
self.conn.commit()
return {"ok": True, "keyword": keyword, "platforms": platforms}
except Exception as e:
return {"ok": False, "error": str(e)}
def remove_keyword(self, keyword: str) -> dict:
self.conn.execute("DELETE FROM tasks WHERE keyword = ?", (keyword,))
self.conn.commit()
return {"ok": True, "keyword": keyword}
def pause_keyword(self, keyword: str) -> dict:
self.conn.execute("UPDATE tasks SET status='paused' WHERE keyword=?", (keyword,))
self.conn.commit()
return {"ok": True, "keyword": keyword}
def resume_keyword(self, keyword: str) -> dict:
self.conn.execute("UPDATE tasks SET status='active' WHERE keyword=?", (keyword,))
self.conn.commit()
return {"ok": True, "keyword": keyword}
def list_tasks(self) -> List[dict]:
rows = self.conn.execute(
"SELECT keyword, platforms, frequency, priority, status, created_at, last_crawl_at, alert_threshold, alert_channels FROM tasks"
).fetchall()
return [
{
"keyword": r[0], "platforms": r[1].split(","),
"frequency": r[2], "priority": r[3], "status": r[4],
"created_at": r[5], "last_crawl_at": r[6],
"alert_threshold": r[7], "alert_channels": r[8],
}
for r in rows
]
def get_task(self, keyword: str) -> Optional[dict]:
for t in self.list_tasks():
if t["keyword"] == keyword:
return t
return None
# ── Crawling ──────────────────────────────────────────────────────────────
def crawl_keyword(self, keyword: str, platforms: List[str] = None) -> dict:
"""Crawl all platforms for a keyword, save posts to DB."""
task = self.get_task(keyword)
if not task:
return {"ok": False, "error": f"Keyword '{keyword}' not found"}
if task["status"] != "active":
return {"ok": False, "error": f"Task '{keyword}' is {task['status']}"}
if platforms is None:
platforms = task["platforms"]
daily_limit = self.limits["daily_limit"]
all_posts = []
for platform in platforms:
cfg = PLATFORM_CONFIG.get(platform, {})
search_url = cfg.get("search_url", "").format(keyword=keyword)
limit_per_platform = daily_limit // len(platforms) if daily_limit > 0 else 1000
_log("INFO", f"Crawling {platform} for '{keyword}' from {search_url}")
# Respect rate limits
time.sleep(random.uniform(3, 8))
html = fetch_page(search_url, platform=platform)
if not html:
_log("WARN", f"Failed to fetch {platform} for '{keyword}'")
continue
# Parse posts
if platform == "xhs":
posts = parse_xhs_posts(html, keyword)
elif platform == "douyin":
posts = parse_douyin_posts(html, keyword)
elif platform == "weibo":
posts = parse_weibo_posts(html, keyword)
elif platform == "wechat":
posts = parse_wechat_posts(html, keyword)
else:
posts = []
# Limit posts
posts = posts[:limit_per_platform]
all_posts.extend(posts)
_log("INFO", f" {platform}: found {len(posts)} posts for '{keyword}'")
# Save to DB
saved_count = 0
for post in all_posts:
try:
self.conn.execute("""
INSERT OR IGNORE INTO posts
(keyword, platform, post_id, title, content, author, author_id, likes, comments, shares, published_at, fetched_at, url)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
""", (post.keyword, post.platform, post.post_id, post.title, post.content,
post.author, post.author_id, post.likes, post.comments, post.shares,
post.published_at, post.fetched_at, post.url))
saved_count += 1
except Exception:
pass
self.conn.commit()
# Update last crawl time
now = datetime.now(timezone.utc).isoformat()
self.conn.execute("UPDATE tasks SET last_crawl_at=? WHERE keyword=?", (now, keyword))
self.conn.commit()
return {
"ok": True, "keyword": keyword,
"total_posts": len(all_posts),
"saved": saved_count,
"platforms": {p: len([x for x in all_posts if x.platform == p]) for p in platforms},
}
def crawl_all(self) -> dict:
"""Crawl all active tasks."""
tasks = self.list_tasks()
active = [t for t in tasks if t["status"] == "active"]
results = []
for task in active:
r = self.crawl_keyword(task["keyword"])
results.append(r)
return {"ok": True, "total": len(active), "results": results}
# ── Sentiment Analysis ───────────────────────────────────────────────────
def analyze_sentiment(self, text: str) -> dict:
"""Analyze sentiment of a single text."""
api_key = self.config.get("glm_api_key", "")
if api_key:
result = analyze_with_glm4(text, api_key)
if result:
return result.to_dict()
# Fallback to rule-based
result = rule_based_sentiment(text)
return result.to_dict()
def batch_analyze(self, texts: List[str]) -> List[dict]:
"""Batch analyze multiple texts."""
api_key = self.config.get("glm_api_key", "")
if api_key:
results = batch_analyze_with_glm4(texts, api_key)
return results
# Fallback
return [rule_based_sentiment(t).to_dict() for t in texts]
def analyze_pending_posts(self, keyword: str = None, batch_size: int = 20) -> dict:
"""Analyze all unanalyzed posts for a keyword."""
api_key = self.config.get("glm_api_key", "")
if keyword:
rows = self.conn.execute("""
SELECT p.id, p.title || ' ' || p.content
FROM posts p
LEFT JOIN analyses a ON p.id = a.post_id
WHERE p.keyword = ? AND a.id IS NULL AND (p.title IS NOT NULL OR p.content IS NOT NULL)
LIMIT ?
""", (keyword, batch_size)).fetchall()
else:
rows = self.conn.execute("""
SELECT p.id, p.title || ' ' || p.content
FROM posts p
LEFT JOIN analyses a ON p.id = a.post_id
WHERE a.id IS NULL AND (p.title IS NOT NULL OR p.content IS NOT NULL)
LIMIT ?
""", (batch_size,)).fetchall()
if not rows:
return {"ok": True, "analyzed": 0, "message": "No posts pending analysis"}
ids = [r[0] for r in rows]
texts = [r[1] if r[1] else "No content" for r in rows]
# Batch analyze
analyses = self.batch_analyze(texts)
now = datetime.now(timezone.utc).isoformat()
analyzed_count = 0
for i, row_id in enumerate(ids):
try:
a = analyses[i]
sentiment = a.get("sentiment", "neutral")
score = float(a.get("score", 0.0))
reason = a.get("reason", "")
self.conn.execute("""
INSERT INTO analyses (post_id, sentiment, score, reason, analyzed_at)
VALUES (?, ?, ?, ?, ?)
""", (row_id, sentiment, score, reason, now))
analyzed_count += 1
except Exception as e:
_log("WARN", f"Failed to save analysis for post {row_id}: {e}")
self.conn.commit()
return {"ok": True, "analyzed": analyzed_count, "total": len(rows)}
# ── Report Generation ─────────────────────────────────────────────────────
def generate_report(self, keyword: str, days: int = 7) -> dict:
"""Generate a sentiment report for a keyword."""
cutoff = (datetime.now(timezone.utc) - timedelta(days=days)).isoformat()
# Get post counts by platform and sentiment
stats_rows = self.conn.execute("""
SELECT p.platform, a.sentiment, COUNT(*) as cnt
FROM posts p
JOIN analyses a ON p.id = a.post_id
WHERE p.keyword = ? AND p.fetched_at >= ?
GROUP BY p.platform, a.sentiment
""", (keyword, cutoff)).fetchall()
# Total posts
total = self.conn.execute("""
SELECT COUNT(*) FROM posts WHERE keyword=? AND fetched_at >= ?
""", (keyword, cutoff)).fetchone()[0]
# Sentiment breakdown
sentiment_counts = {"positive": 0, "neutral": 0, "negative": 0}
platform_counts = {}
for platform, sentiment, cnt in stats_rows:
sentiment_counts[sentiment] = sentiment_counts.get(sentiment, 0) + cnt
platform_counts[platform] = platform_counts.get(platform, 0) + cnt
# Recent negative posts
neg_posts = self.conn.execute("""
SELECT p.title, p.platform, p.author, p.url, p.likes, a.score, a.reason
FROM posts p
JOIN analyses a ON p.id = a.post_id
WHERE p.keyword=? AND a.sentiment='negative' AND p.fetched_at >= ?
ORDER BY p.fetched_at DESC LIMIT 10
""", (keyword, cutoff)).fetchall()
# Top positive posts
pos_posts = self.conn.execute("""
SELECT p.title, p.platform, p.author, p.url, p.likes, a.score, a.reason
FROM posts p
JOIN analyses a ON p.id = a.post_id
WHERE p.keyword=? AND a.sentiment='positive' AND p.fetched_at >= ?
ORDER BY a.score DESC LIMIT 5
""", (keyword, cutoff)).fetchall()
# Build report
total_analyzed = sum(sentiment_counts.values())
neg_rate = (sentiment_counts["negative"] / total_analyzed * 100) if total_analyzed > 0 else 0
pos_rate = (sentiment_counts["positive"] / total_analyzed * 100) if total_analyzed > 0 else 0
neu_rate = (sentiment_counts["neutral"] / total_analyzed * 100) if total_analyzed > 0 else 0
# Trend (daily counts)
daily_rows = self.conn.execute("""
SELECT DATE(p.fetched_at) as day, a.sentiment, COUNT(*) as cnt
FROM posts p
JOIN analyses a ON p.id = a.post_id
WHERE p.keyword=? AND p.fetched_at >= ?
GROUP BY day, a.sentiment
ORDER BY day
""", (keyword, cutoff)).fetchall()
trend = {}
for day, sentiment, cnt in daily_rows:
if day not in trend:
trend[day] = {"positive": 0, "neutral": 0, "negative": 0}
trend[day][sentiment] = cnt
# AI summary
summary_text = self._generate_ai_summary(keyword, sentiment_counts, total, neg_rate, trend)
report = {
"keyword": keyword,
"period": f"Last {days} days",
"generated_at": datetime.now(timezone.utc).isoformat(),
"stats": {
"total_posts": total,
"analyzed": total_analyzed,
"sentiment": {
"positive": {"count": sentiment_counts["positive"], "rate": round(pos_rate, 1)},
"neutral": {"count": sentiment_counts["neutral"], "rate": round(neu_rate, 1)},
"negative": {"count": sentiment_counts["negative"], "rate": round(neg_rate, 1)},
},
"platform": platform_counts,
"trend": trend,
},
"top_negative": [
{"title": r[0], "platform": r[1], "author": r[2], "url": r[3],
"likes": r[4], "score": r[5], "reason": r[6]}
for r in neg_posts
],
"top_positive": [
{"title": r[0], "platform": r[1], "author": r[2], "url": r[3],
"likes": r[4], "score": r[5], "reason": r[6]}
for r in pos_posts
],
"summary": summary_text,
}
return report
def _generate_ai_summary(self, keyword: str, sentiment_counts: dict, total: int, neg_rate: float, trend: dict) -> str:
"""Generate AI-powered text summary using GLM-4 if available."""
api_key = self.config.get("glm_api_key", "")
if not api_key:
return self._rule_summary(keyword, sentiment_counts, total, neg_rate)
prompt = f"""Based on the following sentiment data for keyword "{keyword}", generate a concise sentiment summary (within 100 characters).
Data:
- Total posts (last 7 days): {total}
- Positive posts: {sentiment_counts['positive']}
- Neutral posts: {sentiment_counts['neutral']}
- Negative posts: {sentiment_counts['negative']}
- Negative rate: {neg_rate:.1f}%
Output a concise summary in Chinese, no formatting markers needed."""
try:
payload = {
"model": GLM_MODEL,
"messages": [{"role": "user", "content": prompt}],
"temperature": 0.5,
"max_tokens": 200,
}
result = subprocess.run(
["curl", "-s", "-X", "POST", GLM_API_URL,
"-H", f"Authorization: Bearer {api_key}",
"-H", "Content-Type: application/json",
"-d", json.dumps(payload, ensure_ascii=False)],
capture_output=True, text=True, timeout=20
)
resp = json.loads(result.stdout)
return resp["choices"][0]["message"]["content"].strip()
except Exception:
return self._rule_summary(keyword, sentiment_counts, total, neg_rate)
def _rule_summary(self, keyword: str, sentiment_counts: dict, total: int, neg_rate: float) -> str:
"""Rule-based summary when no API."""
if total == 0:
return f"No sentiment data available for \"{keyword}\"."
neg = sentiment_counts["negative"]
pos = sentiment_counts["positive"]
if neg_rate > 30:
return f"\"{keyword}\": {total} posts, {neg_rate:.1f}% negative \u2014 requires attention."
elif neg_rate > 15:
return f"\"{keyword}\": sentiment stable, {neg_rate:.1f}% negative, {pos} positive posts."
else:
return f"\"{keyword}\": positive trend, only {neg_rate:.1f}% negative, largely positive."
# ── Alerting ──────────────────────────────────────────────────────────────
def check_alerts(self, keyword: str) -> Optional[dict]:
"""Check if alert threshold is exceeded for a keyword."""
task = self.get_task(keyword)
if not task:
return None
threshold = task.get("alert_threshold", 5)
today = datetime.now().strftime("%Y-%m-%d")
cutoff = f"{today}T00:00:00+00:00"
neg_count = self.conn.execute("""
SELECT COUNT(*) FROM posts p
JOIN analyses a ON p.id = a.post_id
WHERE p.keyword=? AND a.sentiment='negative' AND p.fetched_at>=?
""", (keyword, cutoff)).fetchone()[0]
total_today = self.conn.execute("""
SELECT COUNT(*) FROM posts p
JOIN analyses a ON p.id = a.post_id
WHERE p.keyword=? AND p.fetched_at>=?
""", (keyword, cutoff)).fetchone()[0]
neg_rate = (neg_count / total_today * 100) if total_today > 0 else 0
if neg_count >= threshold:
now = datetime.now(timezone.utc).isoformat()
alert_type = "threshold"
self.conn.execute("""
INSERT INTO alerts (keyword, alert_type, threshold, negative_count, negative_rate, triggered_at)
VALUES (?, ?, ?, ?, ?, ?)
""", (keyword, alert_type, threshold, neg_count, neg_rate, now))
self.conn.commit()
# Mark notification as pending
self.conn.execute("""
UPDATE alerts SET notification_sent=0
WHERE keyword=? AND triggered_at=? AND notification_sent=0
""", (keyword, now))
self.conn.commit()
return {
"keyword": keyword,
"alert_type": alert_type,
"threshold": threshold,
"negative_count": neg_count,
"negative_rate": round(neg_rate, 1),
"total_today": total_today,
"triggered_at": now,
}
return None
def check_all_alerts(self) -> List[dict]:
"""Check alerts for all active tasks."""
tasks = self.list_tasks()
alerts = []
for task in tasks:
if task["status"] != "active":
continue
alert = self.check_alerts(task["keyword"])
if alert:
alerts.append(alert)
return alerts
def send_feishu_alert(self, alert: dict):
"""Send alert via Feishu group bot."""
webhook = self.config.get("feishu_webhook", "")
if not webhook:
_log("WARN", "Feishu webhook not configured")
return {"ok": False, "error": "Feishu Webhook not configured"}
platform_emoji = {"xhs": "📕", "douyin": "🎵", "weibo": "🌐", "wechat": "💬"}
platforms = self.get_task(alert["keyword"])["platforms"] if self.get_task(alert["keyword"]) else []
emoji_map = {
"positive": "🟢", "neutral": "🟡", "negative": "🔴"
}
rate = alert["negative_rate"]
if rate < 10:
color = "green"
rate_emoji = "🟢"
elif rate < 25:
color = "yellow"
rate_emoji = "🟡"
else:
color = "red"
rate_emoji = "🔴"
body = {
"msg_type": "interactive",
"card": {
"header": {
"title": {"tag": "plain_text", "content": f"🔴 Sentiment Alert | {alert['keyword']}"},
"template": "red" if rate >= 25 else "orange" if rate >= 15 else "yellow"
},
"elements": [
{"tag": "div", "text": {"tag": "lark_md", "content": f"**Keyword:** {alert['keyword']}\n**Triggered at:** {alert['triggered_at']}\n**Today's negative:** {alert['negative_count']} (threshold: {alert['threshold']})\n**Negative rate:** {alert['negative_rate']}%"}},
{"tag": "hr"},
{"tag": "div", "text": {"tag": "lark_md", "content": "**📌 Latest Negative Posts**"}},
]
}
}
# Add top negative posts to card
cutoff = alert["triggered_at"][:10] + "T00:00:00+00:00"
neg_posts = self.conn.execute("""
SELECT p.title, p.platform, p.author, p.url, p.likes, a.reason
FROM posts p
JOIN analyses a ON p.id = a.post_id
WHERE p.keyword=? AND a.sentiment='negative' AND p.fetched_at>=?
ORDER BY p.fetched_at DESC LIMIT 5
""", (alert["keyword"], cutoff)).fetchall()
for post in neg_posts:
title = post[0][:50] + "..." if len(post[0]) > 50 else post[0]
body["card"]["elements"].append({
"tag": "div",
"text": {"tag": "lark_md",
"content": f"• [{platform_emoji.get(post[1], '📌')}] {title}\n — {post[2]} | 👍{post[4]} | {post[5]}"}
})
body["card"]["elements"].append({"tag": "hr"})
body["card"]["elements"].append({
"tag": "note",
"text": {"tag": "lark_md", "content": "Sent by Sentiment Compass Auto-Push"},
})
try:
result = subprocess.run(
["curl", "-s", "-X", "POST", webhook,
"-H", "Content-Type: application/json",
"-d", json.dumps(body, ensure_ascii=False)],
capture_output=True, text=True, timeout=10
)
resp = json.loads(result.stdout)
if resp.get("code") == 0 or resp.get("StatusCode") == 0:
# Mark as sent
self.conn.execute(
"UPDATE alerts SET notification_sent=1 WHERE keyword=? AND triggered_at=?",
(alert["keyword"], alert["triggered_at"])
)
self.conn.commit()
return {"ok": True}
return {"ok": False, "error": resp}
except Exception as e:
return {"ok": False, "error": str(e)}
def send_email_alert(self, alert: dict, smtp_config: dict = None):
"""Send alert via email."""
if smtp_config is None:
smtp_config = self.config.get("smtp_config", {})
if not smtp_config:
return {"ok": False, "error": "SMTP not configured"}
import smtplib
from email.mime.text import MIMEText
from email.mime.multipart import MIMEMultipart
subject = f"🔴 Sentiment Alert | {alert['keyword']} — {alert['negative_count']} negative posts"
body_html = f"""
<h2>Sentiment Alert</h2>
<p><strong>Keyword:</strong> {alert['keyword']}</p>
<p><strong>Triggered at:</strong> {alert['triggered_at']}</p>
<p><strong>Today's negative:</strong> {alert['negative_count']} (threshold: {alert['threshold']})</p>
<p><strong>Negative rate:</strong> {alert['negative_rate']}%</p>
<hr>
<p>Auto-pushed by Sentiment Compass</p>
"""
msg = MIMEMultipart("alternative")
msg["Subject"] = subject
msg["From"] = smtp_config.get("from", "")
msg["To"] = smtp_config.get("to", "")
msg.attach(MIMEText(body_html, "html", "utf-8"))
try:
server = smtplib.SMTP(smtp_config["host"], smtp_config.get("port", 587))
server.starttls()
server.login(smtp_config["user"], smtp_config["pass"])
server.sendmail(msg["From"], [msg["To"]], msg.as_string())
server.quit()
return {"ok": True}
except Exception as e:
return {"ok": False, "error": str(e)}
# ── Data Query ────────────────────────────────────────────────────────────
def get_posts(self, keyword: str = None, platform: str = None,
sentiment: str = None, days: int = 7, limit: int = 50) -> List[dict]:
"""Query posts with filters."""
cutoff = (datetime.now(timezone.utc) - timedelta(days=days)).isoformat()
query = "SELECT p.*, a.sentiment, a.score, a.reason FROM posts p LEFT JOIN analyses a ON p.id=a.post_id WHERE p.fetched_at>=?"
params = [cutoff]
if keyword:
query += " AND p.keyword=?"
params.append(keyword)
if platform:
query += " AND p.platform=?"
params.append(platform)
if sentiment:
query += " AND a.sentiment=?"
params.append(sentiment)
query += " ORDER BY p.fetched_at DESC LIMIT ?"
params.append(limit)
rows = self.conn.execute(query, params).fetchall()
return [
{
"id": r[0], "keyword": r[1], "platform": r[2], "post_id": r[3],
"title": r[4], "content": r[5], "author": r[6], "author_id": r[7],
"likes": r[8], "comments": r[9], "shares": r[10],
"published_at": r[11], "fetched_at": r[12], "url": r[13],
"sentiment": r[14], "score": r[15], "reason": r[16],
}
for r in rows
]
def get_daily_stats(self, keyword: str, days: int = 7) -> dict:
"""Get daily statistics for a keyword."""
cutoff = (datetime.now(timezone.utc) - timedelta(days=days)).isoformat()
rows = self.conn.execute("""
SELECT DATE(p.fetched_at) as day, a.sentiment, COUNT(*) as cnt
FROM posts p
JOIN analyses a ON p.id = a.post_id
WHERE p.keyword=? AND p.fetched_at>=?
GROUP BY day, a.sentiment
ORDER BY day
""", (keyword, cutoff)).fetchall()
trend = {}
for day, sentiment, cnt in rows:
if day not in trend:
trend[day] = {"positive": 0, "neutral": 0, "negative": 0}
trend[day][sentiment] = cnt
return {"keyword": keyword, "days": days, "trend": trend}
# ── Cleanup ───────────────────────────────────────────────────────────────
def cleanup_old_data(self):
"""Delete posts older than history_days limit."""
history_days = self.limits["history_days"]
if history_days < 0:
return # No limit for MAX
cutoff = (datetime.now(timezone.utc) - timedelta(days=history_days)).isoformat()
self.conn.execute("DELETE FROM analyses WHERE post_id IN (SELECT id FROM posts WHERE fetched_at<?)", (cutoff,))
self.conn.execute("DELETE FROM posts WHERE fetched_at<?", (cutoff,))
self.conn.commit()
def get_stats_summary(self) -> dict:
"""Get overall stats."""
total_posts = self.conn.execute("SELECT COUNT(*) FROM posts").fetchone()[0]
total_analyzed = self.conn.execute("SELECT COUNT(*) FROM analyses").fetchone()[0]
total_tasks = self.conn.execute("SELECT COUNT(*) FROM tasks").fetchone()[0]
total_alerts = self.conn.execute("SELECT COUNT(*) FROM alerts").fetchone()[0]
sent_breakdown = self.conn.execute("""
SELECT sentiment, COUNT(*) FROM analyses GROUP BY sentiment
""").fetchall()
return {
"total_posts": total_posts,
"total_analyzed": total_analyzed,
"total_tasks": total_tasks,
"total_alerts": total_alerts,
"sentiment_breakdown": {r[0]: r[1] for r in sent_breakdown},
"tier": self.tier,
"limits": self.limits,
}
# ─── CLI ──────────────────────────────────────────────────────────────────────
def main():
if len(sys.argv) < 2:
print(json.dumps({"error": "Usage: python3 sentiment.py <command> [args...]"}))
sys.exit(1)
# ── Billing: charge per execution ─────────────────────────────────────
user_id = os.environ.get("FEISHU_USER_ID", "")
bill = charge_user(user_id)
if not bill.get("ok"):
payment_url = bill.get("payment_url", f"https://skillpay.me/sentiment-analysis-monitor")
print(json.dumps({
"error": "Insufficient balance",
"balance": bill.get("balance", 0),
"payment_url": payment_url,
}))
sys.exit(1)
cmd = sys.argv[1]
compass = SentimentCompass()
if cmd == "add":
# python3 sentiment.py add <keyword> <platforms_csv> [frequency] [alert_threshold]
keyword = sys.argv[2] if len(sys.argv) > 2 else ""
platforms_str = sys.argv[3] if len(sys.argv) > 3 else "xhs"
frequency = sys.argv[4] if len(sys.argv) > 4 else "daily"
threshold = int(sys.argv[5]) if len(sys.argv) > 5 else 5
platforms = [p.strip() for p in platforms_str.split(",")]
result = compass.add_keyword(keyword, platforms, frequency, alert_threshold=threshold)
print(json.dumps(result, ensure_ascii=False))
elif cmd == "remove":
keyword = sys.argv[2] if len(sys.argv) > 2 else ""
result = compass.remove_keyword(keyword)
print(json.dumps(result, ensure_ascii=False))
elif cmd == "list":
tasks = compass.list_tasks()
print(json.dumps({"ok": True, "tasks": tasks}, ensure_ascii=False))
elif cmd == "crawl":
keyword = sys.argv[2] if len(sys.argv) > 2 else ""
platforms_str = sys.argv[3] if len(sys.argv) > 3 else None
platforms = [p.strip() for p in platforms_str.split(",")] if platforms_str else None
result = compass.crawl_keyword(keyword, platforms)
print(json.dumps(result, ensure_ascii=False))
elif cmd == "crawl-all":
result = compass.crawl_all()
print(json.dumps(result, ensure_ascii=False))
elif cmd == "analyze":
# python3 sentiment.py analyze <text>
text = " ".join(sys.argv[2:]) if len(sys.argv) > 2 else ""
result = compass.analyze_sentiment(text)
print(json.dumps(result, ensure_ascii=False))
elif cmd == "batch-analyze":
# python3 sentiment.py batch-analyze <file.json>
# File contains JSON array of texts
filepath = sys.argv[2] if len(sys.argv) > 2 else ""
try:
texts = json.loads(Path(filepath).read_text(encoding="utf-8"))
results = compass.batch_analyze(texts)
print(json.dumps(results, ensure_ascii=False))
except Exception as e:
print(json.dumps({"error": str(e)}))
elif cmd == "analyze-pending":
keyword = sys.argv[2] if len(sys.argv) > 2 else None
batch_size = int(sys.argv[3]) if len(sys.argv) > 3 else 20
result = compass.analyze_pending_posts(keyword, batch_size)
print(json.dumps(result, ensure_ascii=False))
elif cmd == "report":
# python3 sentiment.py report <keyword> [days]
keyword = sys.argv[2] if len(sys.argv) > 2 else ""
days = int(sys.argv[3]) if len(sys.argv) > 3 else 7
report = compass.generate_report(keyword, days)
print(json.dumps(report, ensure_ascii=False))
elif cmd == "check-alerts":
keyword = sys.argv[2] if len(sys.argv) > 2 else None
if keyword:
alert = compass.check_alerts(keyword)
print(json.dumps(alert, ensure_ascii=False))
else:
alerts = compass.check_all_alerts()
print(json.dumps({"ok": True, "alerts": alerts}, ensure_ascii=False))
elif cmd == "send-feishu":
# python3 sentiment.py send-feishu <alert_json>
alert_str = " ".join(sys.argv[2:]) if len(sys.argv) > 2 else "{}"
try:
alert = json.loads(alert_str)
result = compass.send_feishu_alert(alert)
print(json.dumps(result, ensure_ascii=False))
except Exception as e:
print(json.dumps({"error": str(e)}))
elif cmd == "posts":
keyword = sys.argv[2] if len(sys.argv) > 2 else None
platform = sys.argv[3] if len(sys.argv) > 3 else None
sentiment = sys.argv[4] if len(sys.argv) > 4 else None
posts = compass.get_posts(keyword=keyword, platform=platform, sentiment=sentiment)
print(json.dumps({"ok": True, "posts": posts}, ensure_ascii=False))
elif cmd == "stats":
stats = compass.get_stats_summary()
print(json.dumps(stats, ensure_ascii=False))
elif cmd == "config-get":
key = sys.argv[2] if len(sys.argv) > 2 else ""
val = get_config(key)
print(json.dumps({"ok": True, "key": key, "value": val}, ensure_ascii=False))
elif cmd == "config-set":
# python3 sentiment.py config-set <key> <value>
key = sys.argv[2] if len(sys.argv) > 2 else ""
value = sys.argv[3] if len(sys.argv) > 3 else ""
# Try to parse JSON
try:
value = json.loads(value)
except Exception:
pass
set_config(key, value)
print(json.dumps({"ok": True, "key": key, "value": value}, ensure_ascii=False))
elif cmd == "cleanup":
compass.cleanup_old_data()
print(json.dumps({"ok": True}))
else:
print(json.dumps({"error": f"Unknown command: {cmd}"}))
sys.exit(1)
if __name__ == "__main__":
main()
FILE:scripts/__init__.py
# Sentiment Compass - AI-driven Chinese social media sentiment monitoring
Brand GEO Master — AI Platform Brand Visibility Monitor. Automatically search AI platforms (Kimi/Xunfei/Zhipu/Wenxin/DeepSeek/etc.), detect brand keyword vis...
---
name: brand-geo-master
description: "Brand GEO Master — AI Platform Brand Visibility Monitor. Automatically search AI platforms (Kimi/Xunfei/Zhipu/Wenxin/DeepSeek/etc.), detect brand keyword visibility, generate 0-100 GEM score, Feishu push support. Trigger: GEO, AI visibility, brand monitoring, AI search visibility."
triggers:
- GEO
- AI visibility
- brand monitoring
- AI search visibility
- competitor monitoring
allowed-tools: Bash(python3)
---
# Brand GEO Master
Detect your brand's visibility across AI search platforms, generate scores and optimization recommendations.
---
## Core Features
- **Multi-platform detection**: Search 9 AI platforms simultaneously
- **GEM Score**: 0-100 visibility score with grade classification
- **AI Reason Analysis**: Understand why brand is not recommended
- **Feishu Push**: Auto-send report as interactive card
- **No API key required**: Local Playwright for free tier
---
## Quick Start
```bash
# Detect a single brand
python3 scripts/geo_report.py "Brand Name"
# Detect multiple brands (including competitors)
python3 scripts/geo_report.py "Brand A" "Brand B"
# No Feishu push (for debugging)
python3 scripts/geo_report.py "Brand Name" --no-push
# Check quota status
python3 scripts/geo_report.py --status
```
---
## Score Guide
| Score | Level | Description |
|------:|:-----:|-------------|
| 80-100 | Excellent | AI actively recommends, strong brand exposure |
| 60-79 | Good | Mentioned by some AI platforms |
| 30-59 | Fair | Rare mentions, needs optimization |
| 0-29 | Weak | Completely invisible |
---
## Platform Coverage
| Platform | Coverage |
|----------|----------|
| Kimi | Supported |
| Xunfei | Supported |
| Wenxin | Supported |
| Zhipu | Supported |
| DeepSeek | Supported |
| Qwen | Supported |
| Doubao | Supported |
| Mita | Supported |
| Hunyuan | Supported |
---
## Config File
Config is at `config.json`:
```json
{
"platforms": {
"kimi": {"enabled": true, "weight": 1.0},
"xinhuo": {"enabled": true, "weight": 0.9},
"yiyan": {"enabled": true, "weight": 0.9},
"zhipu": {"enabled": true, "weight": 0.8},
"deepseek": {"enabled": false},
"qianwen": {"enabled": false},
"doubao": {"enabled": false},
"mita": {"enabled": false},
"hunyuan": {"enabled": false},
"xunfei": {"enabled": false}
},
"report": {
"push_to_feishu": true,
"feishu_webhook": "Your Feishu group bot webhook URL"
}
}
```
---
## Billing
- **Pay-per-call**: $0.0100 USDT per execution via SkillPay.me
- **Balance insufficient**: Payment URL returned — user tops up at `https://skillpay.me/brand-geo-master`
- **External data flow**: `FEISHU_USER_ID` transmitted to `skillpay.me` for billing identification
- **Billing model**: Each full scan (single brand or batch) = 1 call = $0.0100 USDT
---
## Required Environment Variables
| Variable | Description |
|----------|-------------|
| `FEISHU_USER_ID` | User open_id for billing (passed by Feishu runtime) |
| `SKILL_BILLING_API_KEY` | SkillPay Builder API Key |
| `SKILL_BILLING_SKILL_ID` | SkillPay Skill ID (defaults to `brand-geo-master`) |
| `GEO_QUOTA_FILE` | Path to quota file (defaults to `.geo_quota.json`) |
---
## License
MIT
FILE:config.json
{
"platforms": {
"deepseek": {
"enabled": false,
"weight": 1.0,
"url": "https://chat.deepseek.com",
"note": "需登录才能使用，无头浏览器无法绕过登录墙"
},
"kimi": {
"enabled": true,
"weight": 1.0,
"url": "https://kimi.moonshot.cn"
},
"xinhuo": {
"enabled": true,
"weight": 0.9,
"url": "https://xinghuo.xfyun.cn/chat",
"note": "2026-04-16 简单刀：/chat重定向到/desk，选择器改为textarea"
},
"yiyan": {
"enabled": true,
"weight": 0.9,
"url": "https://yiyan.baidu.com",
"note": "2026-04-16 简单刀：选择器改为div[contenteditable='true']"
},
"zhipu": {
"enabled": true,
"weight": 0.8,
"url": "https://www.zhipuai.cn"
},
"xunfei": {
"enabled": false,
"weight": 0.7,
"url": "https://xinghuo.xfyun.cn/chat",
"note": "与xinhuo同一平台，已禁用避免重复检测"
},
"qianwen": {
"enabled": false,
"weight": 0.8,
"url": "https://qianwen.aliyun.com",
"note": "需登录跳转，无头浏览器无法进入聊天界面"
},
"doubao": {
"enabled": false,
"weight": 0.8,
"url": "https://www.doubao.com"
},
"mita": {
"enabled": false,
"weight": 0.6,
"url": "https://mita.com"
},
"hunyuan": {
"enabled": false,
"weight": 0.7,
"url": "https://hunyuan.tencent.com"
}
},
"report": {
"push_to_feishu": true,
"feishu_webhook": "",
"language": "zh-CN"
},
"scoring": {
"base_score": 10,
"occurrence_weight": 2,
"snippet_weight": 5,
"max_occurrence_score": 20,
"max_snippet_score": 10
}
}
FILE:scripts/billing.py
#!/usr/bin/env python3
"""
Billing integration for brand-geo-master via SkillPay.me.
Pay-per-call: $0.01 USDT per execution.
Balance insufficient -> payment_url returned (user tops up at skillpay.me/{slug}).
Required environment variables:
SKILL_BILLING_API_KEY - SkillPay Builder API Key
SKILL_BILLING_SKILL_ID - SkillPay Skill ID (slug: brand-geo-master)
FEISHU_USER_ID - User open_id for billing
Billing API docs: https://skillpay.me/api/v1/billing
"""
import os
import time
import requests
from pathlib import Path
# Constants
BILLING_API_URL = "https://skillpay.me/api/v1/billing"
CALL_PRICE = 0.0100 # USDT per execution
# Cache TTL: 5 minutes
_CACHE_TTL = 300
# Internal cache
_cache: dict = {}
def _cache_get(key: str) -> dict | None:
entry = _cache.get(key)
if entry is None:
return None
if time.time() - entry["_ts"] > _CACHE_TTL:
del _cache[key]
return None
return entry
def _cache_set(key: str, data: dict) -> None:
_cache[key] = {**data, "_ts": time.time()}
def _get_headers() -> dict:
return {
"X-API-Key": os.environ.get("SKILL_BILLING_API_KEY", ""),
"Content-Type": "application/json",
}
def _get_skill_id() -> str:
return os.environ.get("SKILL_BILLING_SKILL_ID", "brand-geo-master")
def _is_dev_mode() -> bool:
key = os.environ.get("SKILL_BILLING_API_KEY", "").strip()
return key == ""
def check_balance(user_id: str) -> dict:
if _is_dev_mode():
return {"balance": 999.0, "ok": True}
cache_key = f"balance_{user_id}"
cached = _cache_get(cache_key)
if cached:
return {"balance": cached["balance"], "ok": True}
try:
resp = requests.get(
f"{BILLING_API_URL}/balance",
headers=_get_headers(),
params={"user_id": user_id, "skill_id": _get_skill_id()},
timeout=10,
)
data = resp.json()
balance = float(data.get("balance", 0.0))
_cache_set(cache_key, {"balance": balance})
return {"balance": balance, "ok": True}
except Exception:
return {"balance": 999.0, "ok": True}
def charge_user(user_id: str) -> dict:
if _is_dev_mode():
return {"ok": True, "balance": 999.0}
try:
resp = requests.post(
f"{BILLING_API_URL}/charge",
headers=_get_headers(),
json={
"user_id": user_id,
"skill_id": _get_skill_id(),
"amount": CALL_PRICE,
},
timeout=10,
)
data = resp.json()
if data.get("success"):
return {
"ok": True,
"balance": float(data.get("balance", 0.0)),
}
return {
"ok": False,
"balance": float(data.get("balance", 0.0)),
"payment_url": data.get("payment_url", ""),
}
except Exception:
return {"ok": True, "balance": 999.0}
def get_payment_link(user_id: str) -> str:
if _is_dev_mode():
return f"https://skillpay.me/{_get_skill_id()}"
try:
resp = requests.post(
f"{BILLING_API_URL}/payment-link",
headers=_get_headers(),
json={"user_id": user_id, "skill_id": _get_skill_id()},
timeout=10,
)
data = resp.json()
return data.get("payment_url", "")
except Exception:
return ""
FILE:scripts/geo_analyzer.py
#!/usr/bin/env python3
"""
GEO Monitor - AI原因分析模块
基于检测结果，调用AI分析"为什么没被推荐"
"""
import sys
import json
import time
import subprocess
class GeoAnalyzer:
"""GEO AI分析器"""
def __init__(self, brand: str, search_results: dict, gem_score: int):
self.brand = brand
self.search_results = search_results
self.gem_score = gem_score
def analyze(self) -> str:
"""调用AI分析原因"""
# 构造分析prompt
platform_summary = self._summarize_results()
prompt = f"""你是一个GEO（生成式引擎优化）专家。请分析以下品牌在AI搜索中不可见的原因，并给出具体的优化建议。
## 品牌信息
品牌名称：{self.brand}
GEM可见性评分：{self.gem_score}/100
## 各平台检测结果
{platform_summary}
## 请分析
1. 为什么这个品牌在AI搜索中不可见？
2. 品牌在AI可见性方面存在哪些问题？
3. 如何优化才能让AI在回答相关问题时主动推荐这个品牌？
请给出具体、可执行的建议，300字以内。"""
# 调用MiniMax分析
result = self._call_ai(prompt)
return result
def _summarize_results(self) -> str:
"""汇总搜索结果"""
lines = []
for platform, data in self.search_results.items():
status = "已发现" if data.get("found") else "未发现"
occ = data.get("occurrences", 0)
snippets = data.get("snippets", [])
error = data.get("error")
if error:
lines.append(f"- {platform}: 错误 - {error}")
else:
lines.append(f"- {platform}: {status}（出现{occ}次）")
if snippets:
for s in snippets[:2]:
lines.append(f" 片段: {s[:100]}")
return "\n".join(lines) if lines else "各平台均未检测到"
def _call_ai(self, prompt: str) -> str:
"""调用外部AI分析（需配置有效的AI接口地址）"""
# ⚠️ 如需启用AI分析功能，请在此填入您的AI接口地址
# 例如：https://your-api-server.com/v1/chat/completions
# 当前默认使用本地分析框架（见 _get_fallback_analysis）
AI_ENDPOINT = "" # <-- 填入AI接口地址，如 https://api.minimax.chat/v1/chat/completions
if not AI_ENDPOINT:
return self._get_fallback_analysis()
cmd = [
"curl", "-s", "-X", "POST",
AI_ENDPOINT,
"-H", "Content-Type: application/json",
"-H", "Authorization: Bearer YOUR_TOKEN", # <-- 替换为有效token
"-d", json.dumps({
"model": "minimax-m2.7",
"messages": [{"role": "user", "content": prompt}],
"max_tokens": 500,
"temperature": 0.7
})
]
try:
result = subprocess.run(cmd, capture_output=True, text=True, timeout=30)
resp = json.loads(result.stdout)
return resp.get("choices", [{}])[0].get("message", {}).get("content", "")
except Exception as e:
return f"AI分析调用失败: {str(e)}\n\n建议：{self._get_fallback_analysis()}"
def _get_fallback_analysis(self) -> str:
"""当AI调用失败时的预设分析框架"""
total_found = sum(1 for r in self.search_results.values() if r.get("found"))
total_platforms = len(self.search_results)
analysis = f"""## {self.brand} AI可见性分析
### 当前状态
- GEM评分：{self.gem_score}/100
- 检测平台：{total_platforms}个
- 已有曝光：{total_found}个平台
### 可能原因
1. **品牌知名度不足** — 作为新品牌，AI训练数据中收录较少
2. **内容覆盖不足** — 在AI平台常用信息源（知乎/公众号/官网）中曝光不足
3. **关键词策略** — 品牌名与用户实际搜索词不匹配
4. **内容结构问题** — 缺乏AI容易理解和引用的结构化内容
### 优化建议
1. **内容矩阵建设**
- 在知乎、公众号发布深度文章（1500字+）
- 文章标题包含核心搜索词
- 内容中多次自然提及品牌
2. **技术优化**
- 确保官网有完整的Schema.org结构化数据
- 页面标题、描述包含品牌关键词
- 提交网站到AI平台认可的搜索引擎
3. **外部引用**
- 争取在权威媒体/平台获得引用
- 建立品牌在AI知识图谱中的实体关联
4. **持续监控**
- 每周检测可见性变化
- 记录优化动作与效果关系
"""
return analysis
def main():
"""测试用主函数"""
if len(sys.argv) < 2:
print("用法: python geo_analyzer.py <品牌名> [GEM评分]")
sys.exit(1)
brand = sys.argv[1]
score = int(sys.argv[2]) if len(sys.argv) > 2 else 50
# 模拟搜索结果
mock_results = {
"deepseek": {"found": False, "occurrences": 0, "snippets": []},
"kimi": {"found": False, "occurrences": 0, "snippets": []},
"yiyan": {"found": False, "occurrences": 0, "snippets": []},
}
analyzer = GeoAnalyzer(brand, mock_results, score)
result = analyzer.analyze()
print(result)
if __name__ == "__main__":
main()
FILE:scripts/geo_searcher.py
#!/usr/bin/env python3
"""
GEO Monitor - 核心爬虫模块
自动搜索多个AI平台，检测品牌关键词的可见性
"""
import asyncio
import json
import sys
import time
import re
from pathlib import Path
# 尝试导入playwright，不存在则给出提示
try:
from playwright.async_api import async_playwright
except ImportError:
print("ERROR: playwright not installed. Run: pip install playwright && playwright install")
sys.exit(1)
class GeoSearcher:
"""GEO搜索引擎"""
# 搜索URL模板
SEARCH_URLS = {
"deepseek": "https://chat.deepseek.com",
"kimi": "https://kimi.moonshot.cn",
"xinhuo": "https://xinghuo.xfyun.cn/chat",
"yiyan": "https://yiyan.baidu.com",
"zhipu": "https://www.zhipuai.cn",
"xunfei": "https://xinghuo.xfyun.cn/chat",
"qianwen": "https://qianwen.aliyun.com",
}
# 搜索框CSS选择器（2026-04-16 简单刀修复）
# xinhuo和xunfei是同一平台（讯飞星火），xunfei已在config.json禁用，避免重复
# deepseek: 需要登录，暂时禁用
# xinhuo: /chat会重定向到/desk，搜索框是裸textarea
# yiyan: 使用div[contenteditable='true']
# qianwen: 需要登录跳转，暂时禁用
SEARCH_SELECTORS = {
"deepseek": "textarea.dsb, #chat-input, textarea[placeholder*='搜索']",
"kimi": "textarea, div[contenteditable='true'], input[type='text']",
"xinhuo": "textarea",
"yiyan": "div[contenteditable='true']",
"zhipu": "textarea, input[type='text']",
"xunfei": "textarea",
"qianwen": "textarea, .ProseMirror, div[contenteditable='true']",
}
def __init__(self, keywords: list[str], config_path: str = None, enabled_platforms: list = None):
self.keywords = keywords
self.config = self._load_config(config_path)
self.enabled_platforms = enabled_platforms # None表示不限制
self.results = {}
def _load_config(self, config_path: str = None) -> dict:
"""加载配置"""
default_config = {
"platforms": {
"deepseek": {"enabled": True, "weight": 1.0},
"kimi": {"enabled": True, "weight": 1.0},
"xinhuo": {"enabled": True, "weight": 0.9},
"yiyan": {"enabled": True, "weight": 0.9},
"zhipu": {"enabled": True, "weight": 0.8},
"xunfei": {"enabled": True, "weight": 0.7},
"qianwen": {"enabled": True, "weight": 0.8},
"doubao": {"enabled": False, "weight": 0.8},
"mita": {"enabled": False, "weight": 0.6},
"hunyuan": {"enabled": False, "weight": 0.7},
},
"timeout": 30000,
"wait_after_input": 3000,
}
if config_path and Path(config_path).exists():
with open(config_path) as f:
user_config = json.load(f)
# 合并配置
for k, v in user_config.get("platforms", {}).items():
if k in default_config["platforms"]:
default_config["platforms"][k].update(v)
return default_config
async def search_platform(self, platform: str, browser) -> dict:
"""在单个平台搜索关键词"""
result = {
"platform": platform,
"found": False,
"occurrences": 0,
"positions": [],
"snippets": [],
"error": None,
}
try:
context = await browser.new_context(
user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
)
page = await context.new_page()
# 设置超时
page.set_default_timeout(self.config.get("timeout", 30000))
# 访问平台
url = self.SEARCH_URLS.get(platform)
if not url:
result["error"] = f"No URL for platform: {platform}"
await context.close()
return result
print(f" -> 正在搜索 {platform}...")
await page.goto(url, wait_until="domcontentloaded")
# 等待页面加载 - 增加等待时间
await asyncio.sleep(5)
# 尝试找到搜索框
selector = self.SEARCH_SELECTORS.get(platform, "textarea, input[type='text']")
try:
search_box = await page.wait_for_selector(selector, timeout=15000)
except Exception:
result["error"] = f"找不到搜索框: {selector}"
await context.close()
return result
# 输入关键词
keyword = self.keywords[0] # 用第一个关键词
await search_box.click()
await search_box.fill(keyword)
# 按回车搜索
await search_box.press("Enter")
# 等待结果
await asyncio.sleep(self.config.get("wait_after_input", 3000) / 1000)
# 尝试获取结果文本
try:
# 等待结果出现
await page.wait_for_timeout(5000)
# 获取页面文本
content = await page.content()
body_text = await page.inner_text("body")
# 检查关键词出现次数
occurrences = body_text.lower().count(keyword.lower())
result["occurrences"] = occurrences
if occurrences > 0:
result["found"] = True
# 提取相关片段
lines = body_text.split("\n")
for line in lines:
if keyword.lower() in line.lower() and len(line.strip()) > 10:
result["snippets"].append(line.strip()[:200])
result["snippets"] = result["snippets"][:5] # 最多5条
except Exception as e:
result["error"] = f"获取结果失败: {str(e)}"
await context.close()
except Exception as e:
result["error"] = str(e)
return result
async def search_all(self) -> dict:
"""在所有启用的平台搜索"""
all_results = {}
async with async_playwright() as p:
# 启动浏览器
browser = await p.chromium.launch(headless=True)
for platform, config in self.config["platforms"].items():
if not config.get("enabled", False):
print(f"跳过 {platform}（已禁用）")
continue
# 如果指定了enabled_platforms，只搜索这些平台
if self.enabled_platforms is not None and platform not in self.enabled_platforms:
print(f"跳过 {platform}（未在配额范围内）")
continue
result = await self.search_platform(platform, browser)
all_results[platform] = result
# 每个平台间隔2秒，避免被封
await asyncio.sleep(2)
await browser.close()
self.results = all_results
return all_results
def calculate_gem_score(self) -> dict:
"""计算GEM可见性评分"""
if not self.results:
return {"score": 0, "grade": "未知", "details": {}}
total_score = 0
total_weight = 0
details = {}
for platform, result in self.results.items():
cfg = self.config["platforms"].get(platform, {})
weight = cfg.get("weight", 1.0)
if result.get("error"):
details[platform] = {"error": result["error"], "score": 0}
continue
occurrences = result.get("occurrences", 0)
found = result.get("found", False)
# 基础分：出现就有分
base_score = 10 if found else 0
# 加分：出现次数
occurrence_score = min(occurrences * 2, 20)
# 加分：snippet数量
snippet_score = min(len(result.get("snippets", [])) * 5, 10)
platform_score = (base_score + occurrence_score + snippet_score) * weight
total_score += platform_score
total_weight += weight
details[platform] = {
"found": found,
"occurrences": occurrences,
"snippets": len(result.get("snippets", [])),
"score": round(platform_score, 1),
}
# 归一化到0-100
normalized_score = int((total_score / total_weight) if total_weight > 0 else 0)
normalized_score = max(0, min(100, normalized_score))
# 等级
if normalized_score >= 80:
grade = "🟢 优秀"
elif normalized_score >= 60:
grade = "🟡 良好"
elif normalized_score >= 30:
grade = "🟠 一般"
else:
grade = "🔴 薄弱"
return {
"score": normalized_score,
"grade": grade,
"details": details,
"keywords": self.keywords,
}
def format_report(self) -> str:
"""生成Markdown格式报告"""
score_data = self.calculate_gem_score()
lines = [
f"# GEO可见性检测报告",
f"",
f"**检测品牌：** {', '.join(self.keywords)}",
f"**检测时间：** {time.strftime('%Y-%m-%d %H:%M:%S')}",
f"",
f"## GEM可见性评分",
f"",
f"**总分：{score_data['score']} / 100** {score_data['grade']}",
f"",
f"## 各平台检测结果",
f"",
]
# 按分数排序
sorted_details = sorted(
score_data["details"].items(),
key=lambda x: x[1].get("score", 0),
reverse=True
)
for platform, detail in sorted_details:
cfg = self.config["platforms"].get(platform, {})
weight = cfg.get("weight", 1.0)
status = "✅" if detail.get("found") else "❌"
if "error" in detail:
lines.append(f"- **{platform.upper()}** {status} 错误: {detail['error']}")
else:
lines.append(
f"- **{platform.upper()}** {status} "
f"出现{detail.get('occurrences', 0)}次 "
f"片段{detail.get('snippets', 0)}条 "
f"得分{detail.get('score', 0)}"
)
lines.extend([
"",
"## 各平台详情",
"",
])
for platform, result in self.results.items():
lines.append(f"### {platform.upper()}")
if result.get("error"):
lines.append(f"❌ 错误: {result['error']}")
else:
lines.append(f"✅ {'已发现' if result.get('found') else '未发现'}")
lines.append(f"出现次数: {result.get('occurrences', 0)}")
if result.get("snippets"):
lines.append("相关片段:")
for s in result["snippets"][:3]:
lines.append(f"> {s}")
lines.append("")
return "\n".join(lines)
async def main():
"""主函数"""
if len(sys.argv) < 2:
print("用法: python geo_searcher.py <关键词1> [关键词2] [关键词3] ...")
print("示例: python geo_searcher.py 提分引擎AI 药常记")
sys.exit(1)
keywords = sys.argv[1:]
print(f"=" * 50)
print(f"GEO Monitor - AI可见性检测")
print(f"检测关键词: {', '.join(keywords)}")
print(f"=" * 50)
searcher = GeoSearcher(keywords)
print("\n开始搜索...\n")
results = await searcher.search_all()
print("\n" + "=" * 50)
print("搜索完成，正在计算评分...\n")
score_data = searcher.calculate_gem_score()
print(f"GEM可见性评分: {score_data['score']} / 100")
print(f"等级: {score_data['grade']}")
report = searcher.format_report()
# 保存报告
import tempfile, os
report_dir = os.path.join(os.path.expanduser("~"), "Desktop", "geo_reports")
os.makedirs(report_dir, exist_ok=True)
report_file = os.path.join(report_dir, f"geo_report_{int(time.time())}.md")
with open(report_file, "w", encoding="utf-8") as f:
f.write(report)
print(f"\n报告已保存: {report_file}")
# 打印摘要
print("\n各平台结果:")
for platform, detail in score_data["details"].items():
if "error" in detail:
print(f" {platform}: 错误 - {detail['error']}")
else:
status = "✅" if detail.get("found") else "❌"
print(f" {platform}: {status} {detail.get('occurrences', 0)}次")
if __name__ == "__main__":
asyncio.run(main())
FILE:scripts/geo_quota.py
#!/usr/bin/env python3
"""
GEO Monitor - Quota management module (Free tier local limits).
Per-call billing via SkillPay — no monthly subscription.
Free tier: 1 brand + 3 platforms per month (no API key required).
Pro users (with valid API key): unlimited via SkillPay billing.
"""
import json
import os
import time
from pathlib import Path
# Free tier limits
FREE_LIMIT_BRAND = 1
FREE_LIMIT_PLATFORM = 3
# Quota file path
QUOTA_FILE = os.environ.get(
"GEO_QUOTA_FILE",
str(Path(__file__).parent.parent / ".geo_quota.json")
)
def get_quota() -> dict:
"""Load current quota from file, reset if month changed."""
quota = {
"brand_count": 0,
"platforms_used": [],
"month": _get_current_month(),
"total_runs": 0,
}
if os.path.exists(QUOTA_FILE):
try:
with open(QUOTA_FILE, "r") as f:
data = json.load(f)
if data.get("month") != _get_current_month():
# New month — reset counters but keep tier setting
quota = {
"brand_count": 0,
"platforms_used": [],
"month": _get_current_month(),
"total_runs": 0,
}
else:
quota = data
except Exception:
pass
return quota
def save_quota(quota: dict) -> None:
"""Save quota to file."""
try:
with open(QUOTA_FILE, "w") as f:
json.dump(quota, f, ensure_ascii=False, indent=2)
except Exception as e:
print(f"Quota save failed: {e}")
def get_enabled_platforms() -> list:
"""
Get list of platforms to scan.
Free tier: limited to kimi, xinhuo, yiyan (3 platforms).
Pro tier (paid): all platforms enabled via billing.
"""
quota = get_quota()
free_platforms = ["kimi", "xinhuo", "yiyan"]
# Load config
config_path = Path(__file__).parent.parent / "config.json"
if config_path.exists():
try:
with open(config_path, "r") as f:
config = json.load(f)
all_enabled = [k for k, v in config.get("platforms", {}).items() if v.get("enabled", False)]
return [p for p in all_enabled if p in free_platforms]
except Exception:
pass
return free_platforms
def check_quota(keywords: list) -> tuple:
"""
Check if quota allows this run.
Returns (allowed: bool, error_msg: str or None, enabled_platforms: list).
"""
quota = get_quota()
# Reset if new month
current_month = _get_current_month()
if quota.get("month") != current_month:
quota["brand_count"] = 0
quota["platforms_used"] = []
quota["month"] = current_month
quota["total_runs"] = 0
save_quota(quota)
enabled_platforms = get_enabled_platforms()
# Count brands
brand_count = quota.get("brand_count", 0)
total_brands = brand_count + len(keywords)
# Count new platforms this run
platforms_used = set(quota.get("platforms_used", []))
platforms_this_run = set(enabled_platforms)
total_platforms = len(platforms_used) + len(platforms_this_run)
# Brand limit check (free tier: 1 brand/month)
if total_brands > FREE_LIMIT_BRAND:
return False, (
f"Free tier limit reached: {brand_count}/{FREE_LIMIT_BRAND} brands used this month. "
f"Upgrade to PRO for unlimited scans. "
f"Pay $0.01 per scan at https://skillpay.me/brand-geo-master"
), None
# Platform limit check (free tier: 3 platforms/month)
if total_platforms > FREE_LIMIT_PLATFORM:
used_list = ", ".join(platforms_used) if platforms_used else "none"
return False, (
f"Free tier limit reached: {len(platforms_used)}/{FREE_LIMIT_PLATFORM} platforms used this month. "
f"Upgrade to PRO for all 9 platforms. "
f"Pay $0.01 per scan at https://skillpay.me/brand-geo-master"
), None
return True, None, enabled_platforms
def record_usage(keywords: list, platforms: list) -> None:
"""Record usage after successful scan."""
quota = get_quota()
quota["brand_count"] = quota.get("brand_count", 0) + len(keywords)
platforms_used = set(quota.get("platforms_used", []))
platforms_used.update(platforms)
quota["platforms_used"] = list(platforms_used)
quota["total_runs"] = quota.get("total_runs", 0) + 1
save_quota(quota)
def _get_current_month() -> str:
return time.strftime("%Y-%m")
def show_quota_status() -> None:
"""Display current quota status."""
quota = get_quota()
brand_count = quota.get("brand_count", 0)
platforms_used = quota.get("platforms_used", [])
month = quota.get("month", _get_current_month())
print(f"[Free Tier] {month} — Brands: {brand_count}/{FREE_LIMIT_BRAND}, Platforms: {len(platforms_used)}/{FREE_LIMIT_PLATFORM}")
if platforms_used:
print(f" Platforms used: {', '.join(platforms_used)}")
print(f" Upgrade to PRO: pay $0.01 per scan at https://skillpay.me/brand-geo-master")
FILE:scripts/geo_report.py
#!/usr/bin/env python3
"""
GEO Monitor - Report generation module.
Integrates search + scoring + analysis, generates complete GEO report.
Billing: per-call via SkillPay ($0.0100 USDT per execution).
"""
import argparse
import asyncio
import importlib
import json
import os
import subprocess
import sys
import time
from pathlib import Path
from billing import charge_user
from geo_quota import check_quota, record_usage, show_quota_status
def _check_billing(user_id: str) -> dict:
"""Run billing check; return error dict if insufficient balance."""
if not user_id:
return {"ok": True}
bill = charge_user(user_id)
if not bill["ok"]:
return {
"ok": False,
"error": "Insufficient balance",
"balance": bill["balance"],
"payment_url": bill.get("payment_url", ""),
}
return {"ok": True}
# Dynamic import geo_searcher (module name with underscore)
spec = importlib.util.spec_from_file_location(
"geo_searcher",
Path(__file__).parent / "geo_searcher.py"
)
geo_searcher_module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(geo_searcher_module)
GeoSearcher = geo_searcher_module.GeoSearcher
# Dynamic import geo_analyzer
spec2 = importlib.util.spec_from_file_location(
"geo_analyzer",
Path(__file__).parent / "geo_analyzer.py"
)
geo_analyzer_module = importlib.util.module_from_spec(spec2)
spec2.loader.exec_module(geo_analyzer_module)
GeoAnalyzer = geo_analyzer_module.GeoAnalyzer
def push_to_feishu(content: str, webhook_url: str = None) -> bool:
"""Push report to Feishu."""
if not webhook_url:
config_path = Path(__file__).parent.parent / "config.json"
if config_path.exists():
with open(config_path) as f:
config = json.load(f)
webhook_url = config.get("report", {}).get("feishu_webhook")
if not webhook_url:
print("Feishu webhook not configured, skipping push")
return False
try:
payload = {
"msg_type": "interactive",
"card": {
"header": {
"title": {"tag": "plain_text", "content": "Brand GEO Report"},
"template": "purple"
},
"elements": [
{
"tag": "div",
"text": {"tag": "lark_md", "content": content[:4000]}
},
{
"tag": "note",
"elements": [
{"tag": "plain_text", "content": "Generated: " + time.strftime("%Y-%m-%d %H:%M:%S")}
]
}
]
}
}
cmd = [
"curl", "-s", "-X", "POST",
webhook_url,
"-H", "Content-Type: application/json",
"-d", json.dumps(payload)
]
result = subprocess.run(cmd, capture_output=True, text=True, timeout=10)
resp = json.loads(result.stdout)
if resp.get("code") == 0 or resp.get("StatusCode") == 0:
print("Report pushed to Feishu")
return True
else:
print("Feishu push failed: " + str(resp))
return False
except Exception as e:
print("Feishu push error: " + str(e))
return False
async def run_full_scan(keywords: list, config_path: str = None, push: bool = True) -> dict:
"""Run complete GEO detection flow."""
if config_path is None:
config_path = str(Path(__file__).parent.parent / "config.json")
user_id = os.environ.get("FEISHU_USER_ID", "")
bill = _check_billing(user_id)
if not bill["ok"]:
return {"error": bill["error"], "payment_url": bill.get("payment_url", "")}
# Quota check (free tier)
allowed, err_msg, enabled_platforms = check_quota(keywords)
if not allowed:
print("\n" + "=" * 50)
print("Quota limit reached")
print("=" * 50)
print(err_msg)
return {"error": err_msg}
print("=" * 50)
print("Brand GEO Master - Full Scan")
print("=" * 50)
print(f"Config: {config_path}")
show_quota_status()
# Step 1: Search
print("\n[1/4] Searching AI platforms...")
searcher = GeoSearcher(keywords, config_path, enabled_platforms=enabled_platforms)
results = await searcher.search_all()
# Step 2: Score
print("\n[2/4] Calculating GEM score...")
score_data = searcher.calculate_gem_score()
# Step 3: Report
print("\n[3/4] Generating report...")
report = searcher.format_report()
# Step 4: AI analysis
print("\n[4/4] AI analysis...")
analyzer = GeoAnalyzer(keywords[0], results, score_data["score"])
ai_analysis = analyzer.analyze()
full_report = report + "\n\n" + ai_analysis
timestamp = int(time.time())
report_file = "/tmp/geo_report_full_%d.md" % timestamp
with open(report_file, "w") as f:
f.write(full_report)
print("\nReport saved: " + report_file)
print("GEM Score: %d/100 %s" % (score_data["score"], score_data["grade"]))
if push:
push_to_feishu(full_report)
record_usage(keywords, enabled_platforms or [])
return {
"keywords": keywords,
"score": score_data,
"results": results,
"report": full_report,
"report_file": report_file,
}
def main():
parser = argparse.ArgumentParser(description="Brand GEO Master - AI Visibility Monitor")
parser.add_argument("keywords", nargs="*", help="Brand name(s) to detect")
parser.add_argument("--no-push", action="store_true", help="Skip Feishu push")
parser.add_argument("--status", action="store_true", help="Show quota status")
parser.add_argument("--api-key", dest="api_key", default=None, help="(deprecated, uses SkillPay billing)")
args = parser.parse_args()
if args.status:
show_quota_status()
return
if not args.keywords:
print("""
Brand GEO Master - AI Visibility Monitor
Usage:
python geo_report.py <brand> [brand B] [brand C] [--no-push]
python geo_report.py --status # Show quota status
Examples:
python geo_report.py "Brand AI"
python geo_report.py "Brand A" "Brand B" --no-push
Upgrade to PRO: $0.01 per scan at https://skillpay.me/brand-geo-master
""")
sys.exit(1)
asyncio.run(run_full_scan(args.keywords, push=not args.no_push))
if __name__ == "__main__":
main()Upload a contract PDF to receive AI-powered text extraction, contract type detection, and a detailed risk analysis report with severity grading and recommend...
# Contract Intelligence Review
**Slug:** contract-intelligence-review
**Platform:** ClawHub (clawhub.ai)
**Category:** Legal & Compliance / Productivity
**Tags:** contract, risk, legal, PDF, AI, analysis, contract-review
---
## What This Skill Does
Upload any contract PDF and get an instant AI-powered risk analysis report. Automatically detects contract type (Labor, Procurement, Sales, Lease, NDA), extracts key terms, and generates a structured risk list graded by severity (HIGH / MEDIUM / LOW).
Supports both Chinese and English contracts. Ideal for procurement teams, HR departments, freelancers, and small businesses needing quick contract reviews without hiring a lawyer.
---
## Workflow
### Step 1 — Receive Contract File
Download the PDF:
- Feishu attachment → use `feishu_im_bot_image` or `feishu_im_user_fetch_resource` with `type=file`
- Local path → use directly
- URL → use `web_crawl` or fetch
Store at `/tmp/contracts/<uuid>.pdf`.
### Step 2 — Extract Text from PDF
Try in order:
1. **PyMuPDF (fitz)** — best for text-based PDFs
2. **pdfplumber** — good for tables
3. **OCR (pytesseract)** — for scanned/image PDFs (with `chi_sim+eng` language packs)
If text extraction yields < 50 characters, offer OCR processing.
### Step 3 — Detect Contract Type
Classify from extracted text:
- Labor Contract
- Procurement Contract
- Sales Contract
- Lease Contract
- NDA / Confidentiality Agreement
- Other
Detect language: Chinese / English / Bilingual.
### Step 4 — AI Risk Analysis
Send extracted text to AI model. Returns structured JSON:
- Summary (200 words or less)
- Key terms table (PRO tier)
- Risk list with level (HIGH/MEDIUM/LOW), category, description, clause reference, recommendation
- Overall risk score (1-10)
### Step 5 — Render Report
Deliver a formatted markdown report:
- Summary
- Key terms table (PRO)
- Risk list by severity level
- Statistics
- Legal disclaimer
### Step 6 — Export (PRO)
CSV export available for Standard and above tiers.
---
## Tiered Features
| Feature | FREE | PRO |
|---------|------|-----|
| Contract types | Other only | All 6 types |
| Summary + Risk list | ✅ | ✅ |
| Key terms table | ❌ | ✅ |
| Excel/CSV export | ❌ | ✅ |
| Batch processing | ❌ | ✅ |
| Risk comparison | ❌ | ✅ |
---
## Pricing
**Per-call billing (no monthly fee):**
| Tier | Price per Call |
|------|---------------|
| FREE | $0.00 USDT |
| PRO | $0.01 USDT |
Each contract analysis = one billable call.
---
## Billing
This skill uses **SkillPay** (skillpay.me) for per-call billing.
**Fee:** $0.0100 USDT per call (all paid tiers)
**External API:** `https://skillpay.me/api/v1/billing`
**Data transmitted:** User identifier (`FEISHU_USER_ID` environment variable)
Billing occurs at the start of each contract analysis. If balance is insufficient, the tool returns a `payment_url` where the user can recharge.
---
## Required Environment Variables
| Variable | Description |
|----------|-------------|
| `FEISHU_USER_ID` | Feishu user open_id for billing identification |
| `OPENAI_API_KEY` | AI model API key (OpenAI, MiniMax, or OpenAI-compatible endpoint) |
| `OPENAI_API_BASE` | Base URL for AI API (optional, defaults to MiniMax endpoint) |
| `SKILL_BILLING_API_KEY` | Builder API Key from skillpay.me (required for paid calls) |
| `SKILL_BILLING_SKILL_ID` | Skill slug on SkillPay (defaults to `contract-intelligence-review`) |
---
## Error Handling
| Error | Handling |
|-------|----------|
| PDF yields < 50 chars | Offer OCR; if OCR also fails, report failure and suggest a text-based PDF |
| AI analysis fails | Return error; suggest retry |
| Insufficient balance | Return `payment_url` for recharge |
| Network error on billing | Allow call through in dev mode (no charge) |
| Unsupported file type | Inform user only PDF is supported |
---
## Technical Stack
- **PDF Text:** PyMuPDF (fitz) + pdfplumber
- **OCR:** pytesseract + pdf2image (language packs: `chi_sim+eng`)
- **AI Analysis:** OpenAI-compatible API (MiniMax / OpenAI / custom endpoint)
- **Report Export:** CSV module (Excel-compatible)
- **Billing:** SkillPay API (skillpay.me)
---
## Output Example
```
## Contract Risk Analysis Report
**Contract Type:** Labor Contract
**Language:** Chinese
**Overall Risk Score:** 6/10 — Medium Risk
**Text Extraction:** Direct extraction
---
### Summary
[200-word summary in contract language]
---
### Key Terms
| Term | Content |
|------|---------|
| Parties | Employer / Employee |
| Contract Value | Not specified |
| Payment Terms | Monthly salary, 15th of each month |
...
### Risk Report
#### HIGH Risk (2 items)
1. **No overtime pay rate specified**
- Category: liability
- Clause: Article 4
- Description: ...
- Recommendation: ...
#### MEDIUM Risk (1 item)
...
---
**Disclaimer:** This report is for informational purposes only and does not constitute legal advice.
```
FILE:scripts/billing.py
#!/usr/bin/env python3
"""
Contract Intelligence Review — SkillPay Billing Integration
Handles per-call charging via skillpay.me API.
"""
import os
import requests
BILLING_URL = "https://skillpay.me/api/v1/billing"
API_KEY = os.environ.get("SKILL_BILLING_API_KEY", "")
SKILL_ID = os.environ.get("SKILL_BILLING_SKILL_ID", "contract-intelligence-review")
FEISHU_USER_ID = os.environ.get("FEISHU_USER_ID", "")
HEADERS = {"X-API-Key": API_KEY, "Content-Type": "application/json"} if API_KEY else {}
# Per-call price in USDT
CALL_PRICE = 0.0100
def get_skill_id() -> str:
return SKILL_ID or os.environ.get("SKILL_BILLING_SKILL_ID", "contract-intelligence-review")
def charge_user(user_id: str) -> dict:
"""
Charge a user for one call via SkillPay.
Returns: {"ok": True, "balance": float} on success
{"ok": False, "balance": float, "payment_url": str} on failure
"""
if not API_KEY:
# Dev Mode — no billing configured
return {"ok": True, "balance": 999.0}
skill_id = get_skill_id()
try:
resp = requests.post(
f"{BILLING_URL}/charge",
headers=HEADERS,
json={
"user_id": user_id or FEISHU_USER_ID or "anonymous",
"skill_id": skill_id,
"amount": CALL_PRICE,
},
timeout=10,
)
data = resp.json()
if data.get("success"):
return {"ok": True, "balance": data.get("balance", 0.0)}
return {
"ok": False,
"balance": data.get("balance", 0.0),
"payment_url": data.get("payment_url", f"https://skillpay.me/{skill_id}"),
}
except Exception:
# Network error — allow through in dev mode
return {"ok": True, "balance": 999.0}
FILE:scripts/requirements.txt
# Contract Risk Reviewer - Python Dependencies
# Install with: pip install -r requirements.txt
PyMuPDF>=1.23.0
pdfplumber>=0.10.0
pytesseract>=0.3.10
pdf2image>=1.16.0
openai>=1.0.0
FILE:scripts/analyze_contract.py
#!/usr/bin/env python3
"""
Contract Intelligence Review
Main analysis script for PDF contract analysis.
Per-call billing via SkillPay (skillpay.me).
"""
import sys
import json
import uuid
import os
import argparse
import tempfile
from pathlib import Path
# ── SkillPay Billing ──────────────────────────────────────────────────────────
from billing import charge_user, CALL_PRICE
SKILL_ID = "contract-intelligence-review"
# ── PDF Text Extraction ──────────────────────────────────────────────────────
def extract_text_pymupdf(pdf_path: str) -> str:
"""Extract text using PyMuPDF (fitz)."""
import fitz
doc = fitz.open(pdf_path)
pages_text = []
for page in doc:
text = page.get_text("text")
if text:
pages_text.append(text)
return "\n".join(pages_text)
def extract_text_pdfplumber(pdf_path: str) -> str:
"""Extract text using pdfplumber."""
import pdfplumber
pages_text = []
with pdfplumber.open(pdf_path) as pdf:
for page in pdf.pages:
t = page.extract_text()
if t:
pages_text.append(t)
return "\n".join(pages_text)
def extract_text_ocr(pdf_path: str, languages: str = "chi_sim+eng") -> str:
"""OCR for scanned/image PDFs using pytesseract + pdf2image."""
import pytesseract
from pdf2image import convert_from_path
import numpy as np
images = convert_from_path(pdf_path, dpi=200)
ocr_texts = []
for img in images:
img_array = np.array(img)
text = pytesseract.image_to_string(img_array, lang=languages)
if text.strip():
ocr_texts.append(text)
return "\n".join(ocr_texts)
def extract_pdf_text(pdf_path: str) -> tuple[str, bool]:
"""
Extract text from PDF: PyMuPDF -> pdfplumber -> OCR.
Returns (text, was_ocr: bool).
"""
text = ""
used_ocr = False
try:
text = extract_text_pymupdf(pdf_path)
except Exception as e:
print(f"[WARN] PyMuPDF extraction failed: {e}", file=sys.stderr)
if not text or len(text.strip()) < 50:
try:
text2 = extract_text_pdfplumber(pdf_path)
if text2 and len(text2.strip()) > len(text.strip()):
text = text2
except Exception as e:
print(f"[WARN] pdfplumber extraction failed: {e}", file=sys.stderr)
if not text or len(text.strip()) < 50:
print("[INFO] Low text yield, attempting OCR...", file=sys.stderr)
try:
text = extract_text_ocr(pdf_path)
used_ocr = True
except Exception as e:
print(f"[WARN] OCR extraction failed: {e}", file=sys.stderr)
return text, used_ocr
# ── Contract Type Detection ──────────────────────────────────────────────────
def detect_contract_type_and_lang(text: str) -> tuple[str, str]:
"""Heuristic pre-check of contract type and language."""
chinese_chars = sum(1 for c in text if '\u4e00' <= c <= '\u9fff')
english_words = len([w for w in text.split() if w.isascii()])
is_chinese = chinese_chars > 50
is_english = english_words > 100
if is_chinese and is_english:
language = "Bilingual (Chinese+English)"
elif is_chinese:
language = "Chinese"
else:
language = "English"
text_lower = text.lower()
if any(k in text_lower for k in ["labor contract", "劳动合同", "聘用", "wage", "salary", "社会保险"]):
contract_type = "Labor Contract"
elif any(k in text_lower for k in ["procurement", "采购", "supplier", "vendor", "供货"]):
contract_type = "Procurement Contract"
elif any(k in text_lower for k in ["sales", "销售", "buyer", "seller", "产品", "买卖"]):
contract_type = "Sales Contract"
elif any(k in text_lower for k in ["lease", "租赁", "rent", "租金", "tenant", "landlord"]):
contract_type = "Lease Contract"
elif any(k in text_lower for k in ["nda", "confidential", "保密", "non-disclosure", "商业秘密"]):
contract_type = "NDA / Confidentiality Agreement"
else:
contract_type = "Other"
return contract_type, language
# ── AI Prompt Building ────────────────────────────────────────────────────────
def build_analysis_prompt(text: str, contract_type: str, language: str, tier: str = "FREE") -> str:
"""Build the AI prompt for contract risk analysis."""
truncated = text[-8000:] if len(text) > 8000 else text
key_terms_instruction = ""
if tier == "PRO":
key_terms_instruction = '''
"key_terms": {
"parties": ["Party A", "Party B", ...],
"contract_value": "amount if stated, otherwise Not specified",
"payment_terms": "payment conditions summary",
"duration": "contract duration/term",
"termination": "termination conditions",
"breach_penalties": "breach of contract penalties",
"dispute_resolution": "dispute resolution clause",
"governing_law": "applicable law/jurisdiction"
},
'''
else:
key_terms_instruction = '''
"key_terms": null,
'''
prompt = f"""# Contract Risk Analysis
## Contract Type: {contract_type}
## Language: {language}
## Contract Text:
{truncated}
---
Please analyze this contract and return ONLY valid JSON (no markdown, no explanation):
{{
"summary": "200-word-or-less summary in the contract language",
"contract_type": "{contract_type}",
"language": "{language}",
{key_terms_instruction}
"risk_report": [
{{
"level": "HIGH | MEDIUM | LOW",
"category": "payment | termination | liability | confidentiality | compliance | other",
"title": "Risk title (in the contract language)",
"description": "Detailed risk description",
"clause_reference": "Which clause/section, or Not specified",
"recommendation": "What the user should do"
}}
],
"overall_score": 1-10,
"overall_assessment": "Brief overall risk assessment"
}}
IMPORTANT:
- HIGH = significant financial loss, legal liability, or irreversible consequences
- MEDIUM = moderate exposure or unclear terms
- LOW = minor issues or overly broad clauses
- If no risks found, return empty risk_report array []
- Respond ONLY with valid JSON
"""
return prompt
# ── AI Analysis ───────────────────────────────────────────────────────────────
def call_ai_analysis(prompt: str, model: str = "minimax/MiniMax-M2") -> dict:
"""Call AI via OpenAI-compatible API."""
api_key = os.environ.get("OPENAI_API_KEY", "")
base_url = os.environ.get("OPENAI_API_BASE", "https://api.minimax.chat/v1")
if not api_key:
api_key = os.environ.get("OPENAI_API_KEY_FALLBACK", "")
base_url = os.environ.get("OPENAI_API_BASE_FALLBACK", "https://api.openai.com/v1")
if not api_key:
return {"error": "No API key configured. Set OPENAI_API_KEY environment variable."}
try:
from openai import OpenAI
client = OpenAI(api_key=api_key, base_url=base_url)
response = client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": prompt}],
temperature=0.2,
max_tokens=4000
)
content = response.choices[0].message.content.strip()
if content.startswith("```"):
lines = content.splitlines()
content = "\n".join(lines[1:-1] if lines[-1].startswith("```") else lines[1:])
return json.loads(content)
except Exception as e:
return {"error": f"AI analysis failed: {str(e)}"}
# ── Report Rendering ─────────────────────────────────────────────────────────
def render_report(analysis: dict, contract_type: str, language: str,
word_count: int, used_ocr: bool, tier: str) -> str:
"""Render the analysis result as a formatted markdown report."""
score = analysis.get("overall_score", "?")
try:
score_num = float(score)
score_label = "Low" if score_num <= 3 else ("Medium" if score_num <= 6 else "High")
except (TypeError, ValueError):
score_label = "Unknown"
summary = analysis.get("summary", "_No summary provided_")
key_terms = analysis.get("key_terms")
risks = analysis.get("risk_report", [])
high = [r for r in risks if r.get("level") == "HIGH"]
med = [r for r in risks if r.get("level") == "MEDIUM"]
low = [r for r in risks if r.get("level") == "LOW"]
lines = []
lines.append("## Contract Risk Analysis Report\n")
lines.append(f"**Contract Type:** {contract_type}")
lines.append(f"**Language:** {language}")
lines.append(f"**Overall Risk Score:** {score}/10 — {score_label} Risk")
lines.append(f"**Text Extraction:** {'OCR (scanned)' if used_ocr else 'Direct extraction'}")
lines.append("\n---\n")
lines.append("### Summary\n")
lines.append(f"{summary}\n")
lines.append("\n---\n")
if tier == "PRO" and key_terms:
lines.append("### Key Terms\n")
lines.append("| Term | Content |")
lines.append("|------|---------|")
terms_map = [
("Parties", "parties", lambda v: ", ".join(v) if isinstance(v, list) else str(v)),
("Contract Value", "contract_value", lambda v: v),
("Payment Terms", "payment_terms", lambda v: v),
("Duration", "duration", lambda v: v),
("Termination", "termination", lambda v: v),
("Breach Penalties", "breach_penalties", lambda v: v),
("Dispute Resolution", "dispute_resolution", lambda v: v),
("Governing Law", "governing_law", lambda v: v),
]
for label, key, fmt in terms_map:
val = key_terms.get(key, "Not specified")
lines.append(f"| {label} | {fmt(val)} |")
lines.append("\n---\n")
elif tier == "PRO":
lines.append("### Key Terms\n_No key terms extracted (insufficient text)_\n\n---\n")
lines.append("### Risk Report\n")
if high:
lines.append(f"#### HIGH Risk ({len(high)} items)\n")
for i, r in enumerate(high, 1):
lines.append(f"{i}. **{r.get('title', 'Unnamed Risk')}**")
lines.append(f" - Category: {r.get('category', 'other')}")
lines.append(f" - Clause: {r.get('clause_reference', 'Not specified')}")
lines.append(f" - Description: {r.get('description', '')}")
lines.append(f" - Recommendation: {r.get('recommendation', '')}\n")
else:
lines.append("#### HIGH Risk (0 items) — No high-risk issues found\n\n")
if med:
lines.append(f"#### MEDIUM Risk ({len(med)} items)\n")
for i, r in enumerate(med, 1):
lines.append(f"{i}. **{r.get('title', 'Unnamed Risk')}**")
lines.append(f" - Category: {r.get('category', 'other')}")
lines.append(f" - Clause: {r.get('clause_reference', 'Not specified')}")
lines.append(f" - Description: {r.get('description', '')}")
lines.append(f" - Recommendation: {r.get('recommendation', '')}\n")
else:
lines.append("#### MEDIUM Risk (0 items)\n\n")
if low:
lines.append(f"#### LOW Risk ({len(low)} items)\n")
for i, r in enumerate(low, 1):
lines.append(f"{i}. **{r.get('title', 'Unnamed Risk')}**")
lines.append(f" - Category: {r.get('category', 'other')}")
lines.append(f" - Clause: {r.get('clause_reference', 'Not specified')}")
lines.append(f" - Description: {r.get('description', '')}")
lines.append(f" - Recommendation: {r.get('recommendation', '')}\n")
else:
lines.append("#### LOW Risk (0 items)\n\n")
lines.append("---\n")
lines.append("**Disclaimer:** This report is for informational purposes only and does not constitute legal advice. Consult a qualified attorney for legal decisions.\n")
lines.append("\n---\n")
lines.append("### Statistics\n")
lines.append(f"- Contract text length: ~{word_count} characters")
lines.append(f"- Risks detected: {len(risks)} total (HIGH {len(high)} / MEDIUM {len(med)} / LOW {len(low)})")
if tier != "FREE":
lines.append(f"- Current tier: {tier}")
return "\n".join(lines)
# ── CSV Export ────────────────────────────────────────────────────────────────
def export_csv(report_path: str, analysis: dict, contract_type: str, language: str):
"""Export risk report as CSV for STD+ tiers."""
import csv
csv_path = report_path.replace("_report.md", "_risks.csv")
risks = analysis.get("risk_report", [])
with open(csv_path, "w", newline="", encoding="utf-8-sig") as f:
writer = csv.writer(f)
writer.writerow(["Risk Level", "Category", "Title", "Description", "Clause", "Recommendation", "Contract Type", "Language"])
for r in risks:
writer.writerow([
r.get("level", ""),
r.get("category", ""),
r.get("title", ""),
r.get("description", ""),
r.get("clause_reference", ""),
r.get("recommendation", ""),
contract_type,
language
])
return csv_path
# ── Main Entry Point ─────────────────────────────────────────────────────────
def main():
parser = argparse.ArgumentParser(description="Contract Intelligence Review")
parser.add_argument("--pdf", required=True, help="Path to the contract PDF file")
parser.add_argument("--tier", default="FREE", choices=["FREE", "PRO"],
help="Subscription tier (default: FREE)")
parser.add_argument("--api-key", default="",
help="AI API key (OPENAI_API_KEY or similar)")
parser.add_argument("--model", default="minimax/MiniMax-M2",
help="AI model to use")
parser.add_argument("--output", default=None,
help="Output report path")
parser.add_argument("--export-csv", action="store_true",
help="Also export CSV (STD+ tier)")
args = parser.parse_args()
pdf_path = args.pdf
feishu_user_id = os.environ.get("FEISHU_USER_ID", "")
# ── Billing: charge per call ──────────────────────────────────────────
billing_result = charge_user(feishu_user_id)
if not billing_result.get("ok"):
balance = billing_result.get("balance", 0)
payment_url = billing_result.get("payment_url", f"https://skillpay.me/{SKILL_ID}")
print(json.dumps({
"error": "insufficient_balance",
"balance": balance,
"price": CALL_PRICE,
"payment_url": payment_url,
}, ensure_ascii=False))
sys.exit(1)
# Tier from args.tier (FREE/PRO); billing is per-call, balance check not needed
tier = args.tier
os.makedirs("/tmp/contracts", exist_ok=True)
if args.output:
report_path = args.output
else:
file_uuid = str(uuid.uuid4())[:8]
report_path = f"/tmp/contracts/{file_uuid}_report.md"
print(f"[INFO] Processing: {pdf_path}", file=sys.stderr)
print(f"[INFO] Tier: {tier}", file=sys.stderr)
# Step 1: Extract text
print("[INFO] Extracting text from PDF...", file=sys.stderr)
text, used_ocr = extract_pdf_text(pdf_path)
word_count = len(text)
print(f"[INFO] Extracted {word_count} characters (OCR used: {used_ocr})", file=sys.stderr)
if word_count < 50:
print(json.dumps({"error": "text_extraction_failed", "characters_extracted": word_count}, ensure_ascii=False))
sys.exit(1)
# Step 2: Detect type and language
contract_type, language = detect_contract_type_and_lang(text)
print(f"[INFO] Detected: type={contract_type}, language={language}", file=sys.stderr)
# Step 3: Build prompt and call AI
print("[INFO] Calling AI for risk analysis...", file=sys.stderr)
prompt = build_analysis_prompt(text, contract_type, language, tier)
analysis = call_ai_analysis(prompt, args.model)
if "error" in analysis:
print(json.dumps(analysis, ensure_ascii=False))
sys.exit(1)
# Step 4: Render report
report_md = render_report(analysis, contract_type, language, word_count, used_ocr, tier)
os.makedirs(os.path.dirname(report_path), exist_ok=True)
with open(report_path, "w", encoding="utf-8") as f:
f.write(report_md)
print(f"[INFO] Report saved to: {report_path}", file=sys.stderr)
# Step 5: Export CSV if requested (STD+)
csv_path = None
if args.export_csv and tier == "PRO":
csv_path = export_csv(report_path, analysis, contract_type, language)
print(f"[INFO] CSV exported to: {csv_path}", file=sys.stderr)
result = {
"status": "success",
"report_path": report_path,
"csv_path": csv_path,
"contract_type": contract_type,
"language": language,
"word_count": word_count,
"used_ocr": used_ocr,
"overall_score": analysis.get("overall_score"),
"risk_count": {
"total": len(analysis.get("risk_report", [])),
"high": len([r for r in analysis.get("risk_report", []) if r.get("level") == "HIGH"]),
"medium": len([r for r in analysis.get("risk_report", []) if r.get("level") == "MEDIUM"]),
"low": len([r for r in analysis.get("risk_report", []) if r.get("level") == "LOW"]),
},
"tier": tier,
"price_charged": CALL_PRICE,
}
print(json.dumps(result, ensure_ascii=False))
if __name__ == "__main__":
main()
Cleans and deduplicates multi-format data with AI field detection, format standardization, multi-source merging, and outputs Excel, CSV, or Feishu Bitable.
---
name: data-cleaner-ai
label: Data Cleaner AI
version: 1.0.0
language: Python
runtime: subprocess (scripts/main.py)
trigger_words:
- data cleaning
- deduplication
- spreadsheet cleanup
- data merge
- format standardization
- CRM data cleanup
- Excel cleaning
- clean data
- remove duplicates
- merge data
---
# Data Cleaner AI
Upload messy data — get clean, structured output. Supports multi-format parsing, AI field identification, intelligent dedup/fill/formatting, multi-source join, and Feishu-native output (Bitable + quality report doc).
**Use cases:** E-commerce order cleanup, CRM customer data cleansing, bank statement reconciliation, roster cleanup, multi-system data merge.
---
## Capabilities
### F1 · Multi-Format Parsing
- Excel (.xlsx / .ls)
- CSV / TSV
- JSON (semi-structured)
- Clipboard paste text
### F2 · Smart Field Identification
- AI auto-detects: name, phone, email, address, amount, date, SKU, order ID, ID number, gender, etc.
- Supports user-defined field mapping override
### F3 · Data Cleaning
- **Deduplication**: Exact match + fuzzy dedup (FuzzyWuzzy, threshold 88%)
- **Missing value fill**: Mean / mode / semantic inference / leave blank
- **Format standardization**:
- Phone → `1xx-xxxx-xxxx`
- Date → `YYYY-MM-DD`
- Amount → 2 decimal places
- Address → Province/City/District/Street standardization
### F4 · Data Classification / Tagging (PRO)
- 8 built-in business rules (high-value customer, dormant user, VIP, enterprise, etc.)
- Supports custom JSON rules
- AI auto-tagging (requires PRO + AI API Key)
### F5 · Multi-Source Join / Merge (PRO)
- Cross-file relational join on key fields
- Fuzzy join when exact key not available (FuzzyWuzzy)
- Conflicted field resolution: priority by source order or latest timestamp
### F6 · Feishu Native Output
- Excel / CSV export
- Feishu Bitable (multi-dimensional table) write-back
- Data quality report auto-generated as Feishu Doc (Markdown)
---
## Tier Feature Matrix
| Feature | FREE | PRO |
|---------|:----:|:---:|
| Multi-format parsing | ✅ | ✅ |
| Basic dedup | ✅ | ✅ |
| Smart fill | ❌ | ✅ |
| Format standardization | ❌ | ✅ |
| Fuzzy dedup | ❌ | ✅ |
| Multi-source merge | ❌ | ✅ |
| AI classification | ❌ | ✅ |
| Data quality report | ❌ | ✅ |
| Feishu Bitable output | ❌ | ✅ |
---
## Pricing
**Per-call billing (no monthly fee):**
| Tier | Price per Call |
|------|---------------|
| FREE | $0.00 USDT |
| PRO | $0.01 USDT |
Each cleaning pipeline execution (clean or merge) = one billable call.
---
## Usage
### Feishu Trigger
```
data cleaning
deduplication
spreadsheet cleanup
CRM data cleanup
Excel cleaning
```
### CLI
```bash
python scripts/main.py clean -i data.xlsx -o cleaned.xlsx
python scripts/main.py clean -t "name,phone\nJohn,13800138000" -f csv -o cleaned.csv
python scripts/main.py merge --sources customers.xlsx orders.csv --on phone -o merged.xlsx
```
### Python API
```python
from main import run_clean_pipeline
result = run_clean_pipeline(
sources=["orders.xlsx"],
output_format="xlsx",
output_path="/tmp/cleaned.xlsx",
dedup_strategy="auto",
fill_strategy="auto",
classify=True,
ai_model="deepseek",
generate_report=True,
)
```
---
## Directory Structure
```
data-cleaner-ai/
├── SKILL.md
└── scripts/
├── main.py # Entry: run_clean_pipeline / run_merge_pipeline
├── parser.py # F1: Multi-format parsing
├── field_identifier.py # F2: AI field identification
├── cleaner.py # F3: Cleaning engine
├── classifier.py # F4: Classification / tagging
├── merger.py # F5: Multi-source join
├── reporter.py # F6: Quality report generation
├── output.py # F6: Output (Excel/CSV/Bitable/Feishu Doc)
├── tier_limits.py # Tier access control
└── billing.py # SkillPay billing integration
```
---
## Billing
This skill uses **SkillPay** (skillpay.me) for per-call billing.
**Fee:** $0.0100 USDT per execution (all paid tiers)
**External API:** `https://skillpay.me/api/v1/billing`
**Data transmitted:** User identifier (`FEISHU_USER_ID` environment variable)
Billing occurs at the start of each cleaning or merge execution. If balance is insufficient, the tool returns a `payment_url` where the user can recharge.
---
## Required Environment Variables
| Variable | Description |
|----------|-------------|
| `FEISHU_USER_ID` | Feishu user open_id for billing identification |
| `OPENAI_API_KEY` | AI model API key (OpenAI, MiniMax, or OpenAI-compatible endpoint) |
| `OPENAI_API_BASE` | Base URL for AI API (optional, defaults to MiniMax endpoint) |
| `SKILL_BILLING_API_KEY` | Builder API Key from skillpay.me (required for paid calls) |
| `SKILL_BILLING_SKILL_ID` | Skill slug on SkillPay (defaults to `data-cleaner-ai`) |
---
## Error Handling
| Error | Handling |
|-------|----------|
| Balance insufficient | Return `payment_url` for recharge |
| Network error on billing | Allow call through in dev mode (no charge) |
| Tier feature not available | Skip feature gracefully, continue with available features |
| No data source provided | Raise error requesting input |
---
## License
MIT
FILE:scripts/classifier.py
"""
F4 · Data classification and tagging.
AI-powered auto-tagging + user-defined classification rules.
Usage:
from classifier import DataClassifier, load_rules, DEFAULT_RULES
clf = DataClassifier(field_info, rules=rules)
df_tagged, tag_col = clf.classify(df)
"""
import os
import json
import re
import math
from typing import Dict, List, Optional, Callable, Any
from dataclasses import dataclass, field
import pandas as pd
from field_identifier import FieldType, FieldInfo
# ─── Built-in classification rules ─────────────────────────────────────────────
DEFAULT_RULES: List[Dict[str, Any]] = [
# ── Customer value tiers ────────────────────────────────────────────────────
{
"name": "高价值客户",
"description": "累计消费 ≥ 5000 元",
"conditions": [
{
"column": ".*金额|.*消费|.*总额|.*销售额",
"type": "amount",
"operator": "gte",
"value": 5000,
}
],
},
{
"name": "中等价值客户",
"description": "累计消费 1000–5000 元",
"conditions": [
{
"column": ".*金额|.*消费|.*总额|.*销售额",
"type": "amount",
"operator": "between",
"value": [1000, 5000],
}
],
},
{
"name": "低价值客户",
"description": "累计消费 < 1000 元",
"conditions": [
{
"column": ".*金额|.*消费|.*总额|.*销售额",
"type": "amount",
"operator": "lt",
"value": 1000,
}
],
},
{
"name": "沉睡用户",
"description": "超过 90 天未消费",
"conditions": [
{
"column": ".*日期|.*时间|.*最后.*购买",
"type": "days_ago",
"operator": "gt",
"value": 90,
}
],
},
{
"name": "新客户",
"description": "注册/首次购买在 30 天内",
"conditions": [
{
"column": ".*注册|.*首次|.*创建.*时间",
"type": "days_ago",
"operator": "lte",
"value": 30,
}
],
},
{
"name": "高风险订单",
"description": "金额异常高（> 平均值 3σ）或收货地址模糊",
"conditions": [
{
"column": ".*金额|.*总额|.*实付",
"type": "amount_outlier",
"operator": "gt_3sigma",
}
],
},
{
"name": "企业客户",
"description": "邮箱为企业域名（非 gmail/qq/163 等公共邮箱）",
"conditions": [
{
"column": ".*邮箱|.*邮件",
"type": "email_domain",
"operator": "not_public",
}
],
},
{
"name": "VIP客户",
"description": "累计消费 ≥ 20000 元",
"conditions": [
{
"column": ".*金额|.*消费|.*总额",
"type": "amount",
"operator": "gte",
"value": 20000,
}
],
},
]
# Public email domains to exclude for "企业客户" rule
PUBLIC_EMAIL_DOMAINS = {
"gmail.com", "qq.com", "163.com", "126.com", "sina.com",
"sohu.com", "hotmail.com", "outlook.com", "yahoo.com",
"foxmail.com", "google.com", "live.com", "msn.com",
}
# ─── Rule evaluation engine ────────────────────────────────────────────────────
@dataclass
class TagResult:
tag: str
description: str
rows_matched: int
@dataclass
class ClassificationReport:
total_rows: int
tagged_rows: int
tags: List[TagResult]
def summary(self) -> str:
lines = [
f"总行数：{self.total_rows} | 已打标签：{self.tagged_rows}"
f"（{self.tagged_rows/self.total_rows:.0%}）"
if self.total_rows else "总行数：0",
"标签分布：",
]
for t in self.tags:
lines.append(f" - {t.tag}：{t.rows_matched} 条")
return "\n".join(lines)
class DataClassifier:
"""
Apply classification rules to a DataFrame to produce tags.
Parameters
----------
field_info : Dict[col -> FieldInfo] from field_identifier
rules : list of rule dicts (see DEFAULT_RULES format)
tag_col : name for the output tag column
"""
def __init__(
self,
field_info: Dict[str, FieldInfo],
rules: Optional[List[Dict[str, Any]]] = None,
tag_col: str = "标签",
*,
use_ai: bool = False,
ai_api_key: Optional[str] = None,
):
self.field_info = field_info
self.rules = rules or DEFAULT_RULES
self.tag_col = tag_col
self.use_ai = use_ai
self.ai_api_key = ai_api_key
def classify(
self,
df: pd.DataFrame,
) -> tuple[pd.DataFrame, str, ClassificationReport]:
"""
Apply rules and return tagged DataFrame.
Adds a column named self.tag_col.
"""
df = df.copy()
df[self.tag_col] = "" # initialise
tag_counts: Dict[str, int] = {}
matched_any = 0
for rule in self.rules:
mask = self._evaluate_rule(df, rule)
n_matched = mask.sum()
if n_matched > 0:
# Append tag (may already have one)
new_tags = df.loc[mask, self.tag_col].apply(
lambda x: f"{x}; {rule['name']}" if x else rule["name"]
)
df[self.tag_col] = new_tags.combine_first(df[self.tag_col])
tag_counts[rule["name"]] = int(n_matched)
matched_any += n_matched
# Remove leading semicolon on first tag
df[self.tag_col] = df[self.tag_col].str.lstrip("; ")
report = ClassificationReport(
total_rows=len(df),
tagged_rows=int((df[self.tag_col] != "").sum()),
tags=[
TagResult(tag=name, description="", rows_matched=count)
for name, count in tag_counts.items()
],
)
return df, self.tag_col, report
# ─── Rule evaluation ───────────────────────────────────────────────────────
def _evaluate_rule(self, df: pd.DataFrame, rule: Dict) -> pd.Series:
"""Return a boolean mask for rows matching the rule."""
import numpy as np
conditions = rule.get("conditions", [])
if not conditions:
return pd.Series(False, index=df.index)
masks: List[pd.Series] = []
for cond in conditions:
col_pat = cond["column"]
cond_type = cond["type"]
operator = cond["operator"]
value = cond.get("value")
# Find matching columns
matching_cols = [
c for c in df.columns
if re.match(col_pat, str(c), re.IGNORECASE)
]
col_mask = pd.Series(False, index=df.index)
for col in matching_cols:
col_mask |= self._eval_condition(df[col], cond_type, operator, value)
masks.append(col_mask)
# All conditions must match (AND)
if not masks:
return pd.Series(False, index=df.index)
result = masks[0]
for m in masks[1:]:
result = result & m
return result
def _eval_condition(
self,
series: pd.Series,
cond_type: str,
operator: str,
value: Any,
) -> pd.Series:
"""Evaluate a single condition on a series."""
import numpy as np
if cond_type == "amount":
# Parse numeric from string like "¥1,234.56"
nums = series.astype(str).str.replace(
r"[¥$€£,\s]", "", regex=True
).str.replace(r"元", "", regex=False)
nums = pd.to_numeric(nums, errors="coerce").fillna(0)
if operator == "gte":
return nums >= float(value)
elif operator == "gt":
return nums > float(value)
elif operator == "lt":
return nums < float(value)
elif operator == "lte":
return nums <= float(value)
elif operator == "between":
lo, hi = float(value[0]), float(value[1])
return (nums >= lo) & (nums <= hi)
elif cond_type == "days_ago":
# Parse date and compute days since
dates = series.astype(str).str.strip()
now_ts = pd.Timestamp.now().timestamp()
def to_days_ago(v: str):
for fmt in ("%Y-%m-%d", "%Y/%m/%d", "%Y年%m月%d日",
"%Y%m%d", "%m/%d/%Y"):
try:
dt = pd.to_datetime(v, format=fmt)
return (now_ts - dt.timestamp()) / 86400
except Exception:
continue
return -1 # can't parse
days = dates.apply(to_days_ago)
if operator == "gt":
return days > float(value)
elif operator == "lte":
return (days >= 0) & (days <= float(value))
elif cond_type == "amount_outlier":
if operator == "gt_3sigma":
nums = pd.to_numeric(
series.astype(str).str.replace(r"[¥$€£,\s]", "", regex=True),
errors="coerce"
)
mean = nums.mean()
std = nums.std()
if std == 0 or math.isnan(std):
return pd.Series(False, index=series.index)
return nums > (mean + 3 * std)
elif cond_type == "email_domain":
if operator == "not_public":
domain_col = series.astype(str).str.split("@").str[-1].str.lower()
return ~domain_col.isin(PUBLIC_EMAIL_DOMAINS)
return pd.Series(False, index=series.index)
# ─── AI-powered auto-tagging ───────────────────────────────────────────────
def classify_with_ai(
self,
df: pd.DataFrame,
) -> tuple[pd.DataFrame, str, ClassificationReport]:
"""
Use AI to generate tags when built-in rules are insufficient.
Requires DATA_CLEANER_API_KEY env var.
"""
if not self.use_ai:
return self.classify(df)
api_key = self.ai_api_key or os.environ.get("DATA_CLEANER_API_KEY", "")
if not api_key:
# Fallback to rules
return self.classify(df)
# Build sample for AI
samples = df.head(20).to_csv(index=False, encoding="utf-8")
prompt = (
"你是一个数据分析师。根据以下数据样本，为每行生成合适的业务标签。\n"
"标签例如：高价值客户、低价值客户、沉睡用户、新客户、VIP客户、"
"企业客户、高风险订单、待激活用户、忠诚客户等。\n"
"每行输出一个标签（最多2个，用分号分隔）。\n"
"只输出CSV格式，第一列是原数据索引，第二列是标签。\n"
f"\n样本数据：\n{samples}"
)
try:
import urllib.request
url = "https://api.minimax.chat/v1/text/chatcompletion_pro"
payload = json.dumps({
"model": "MiniMax-Text-01",
"messages": [{"role": "user", "content": prompt}],
"max_tokens": 500,
"temperature": 0.3,
}).encode("utf-8")
req = urllib.request.Request(
url,
data=payload,
headers={
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json",
},
method="POST",
)
with urllib.request.urlopen(req, timeout=20) as resp:
raw = json.loads(resp.read().decode("utf-8"))
content = raw["choices"][0]["message"]["content"]
# Parse AI output into tags
df_tagged, tag_col, report = self.classify(df)
# Overlay AI tags from content
df_tagged, report = self._apply_ai_tags(df_tagged, content, report)
return df_tagged, tag_col, report
except Exception:
return self.classify(df)
def _apply_ai_tags(
self,
df: pd.DataFrame,
ai_output: str,
report: ClassificationReport,
) -> tuple[pd.DataFrame, ClassificationReport]:
"""Parse AI output CSV and merge tags."""
import io as _io
import numpy as np
try:
ai_df = pd.read_csv(
_io.StringIO(ai_output),
header=None,
names=["index", "tag"],
dtype={"index": str},
)
idx_map = dict(zip(ai_df["index"].astype(str), ai_df["tag"].astype(str)))
for idx_str, tag in idx_map.items():
if idx_str.isdigit():
idx = int(idx_str)
if idx < len(df):
cur = df.at[idx, self.tag_col] or ""
df.at[idx, self.tag_col] = f"{cur}; {tag}".lstrip("; ")
except Exception:
pass # If AI output is unparseable, keep rule-based tags
report.tagged_rows = int((df[self.tag_col] != "").sum())
return df, report
# ─── User rule management ─────────────────────────────────────────────────────
def load_rules(path: str) -> List[Dict[str, Any]]:
"""Load custom classification rules from a JSON file."""
with open(path, encoding="utf-8") as f:
return json.load(f)
def save_rules(rules: List[Dict[str, Any]], path: str) -> None:
"""Save custom classification rules to a JSON file."""
with open(path, "w", encoding="utf-8") as f:
json.dump(rules, f, ensure_ascii=False, indent=2)
FILE:scripts/billing.py
#!/usr/bin/env python3
"""
Billing integration for data-cleaner-ai via SkillPay.me.
Pay-per-call: $0.01 USDT per execution.
Balance insufficient → payment_url returned (user tops up at skillpay.me/{slug}).
Required environment variables:
SKILL_BILLING_API_KEY - SkillPay Builder API Key
SKILL_BILLING_SKILL_ID - SkillPay Skill ID (slug: data-cleaner-ai)
FEISHU_USER_ID - User open_id for billing
Billing API docs: https://skillpay.me/api/v1/billing
"""
import os
import time
import requests
from pathlib import Path
# ─── Constants ────────────────────────────────────────────────────────────────
BILLING_API_URL = "https://skillpay.me/api/v1/billing"
CALL_PRICE = 0.0100 # USDT per execution
# Cache TTL: 5 minutes
_CACHE_TTL = 300
# ─── Internal cache ─────────────────────────────────────────────────────────────
_cache: dict = {}
def _cache_get(key: str) -> dict | None:
entry = _cache.get(key)
if entry is None:
return None
if time.time() - entry["_ts"] > _CACHE_TTL:
del _cache[key]
return None
return entry
def _cache_set(key: str, data: dict) -> None:
_cache[key] = {**data, "_ts": time.time()}
# ─── Headers ───────────────────────────────────────────────────────────────────
def _get_headers() -> dict:
return {
"X-API-Key": os.environ.get("SKILL_BILLING_API_KEY", ""),
"Content-Type": "application/json",
}
def _get_skill_id() -> str:
return os.environ.get("SKILL_BILLING_SKILL_ID", "data-cleaner-ai")
# ─── Dev mode (no API key configured) ─────────────────────────────────────────
def _is_dev_mode() -> bool:
key = os.environ.get("SKILL_BILLING_API_KEY", "").strip()
return key == ""
# ─── Check balance ────────────────────────────────────────────────────────────
def check_balance(user_id: str) -> dict:
"""
Returns current user balance in USDT.
Returns:
{"balance": float, "ok": bool}
"""
if _is_dev_mode():
return {"balance": 999.0, "ok": True}
cache_key = f"balance_{user_id}"
cached = _cache_get(cache_key)
if cached:
return {"balance": cached["balance"], "ok": True}
try:
resp = requests.get(
f"{BILLING_API_URL}/balance",
headers=_get_headers(),
params={"user_id": user_id, "skill_id": _get_skill_id()},
timeout=10,
)
data = resp.json()
balance = float(data.get("balance", 0.0))
_cache_set(cache_key, {"balance": balance})
return {"balance": balance, "ok": True}
except Exception:
return {"balance": 999.0, "ok": True} # Fail open: allow on network error
# ─── Charge user ───────────────────────────────────────────────────────────────
def charge_user(user_id: str) -> dict:
"""
Charge user for one execution ($0.01 USDT).
Returns:
{"ok": True, "balance": float} on success
{"ok": False, "balance": float,
"payment_url": str} on insufficient balance
"""
if _is_dev_mode():
return {"ok": True, "balance": 999.0}
try:
resp = requests.post(
f"{BILLING_API_URL}/charge",
headers=_get_headers(),
json={
"user_id": user_id,
"skill_id": _get_skill_id(),
"amount": CALL_PRICE,
},
timeout=10,
)
data = resp.json()
if data.get("success"):
return {
"ok": True,
"balance": float(data.get("balance", 0.0)),
}
return {
"ok": False,
"balance": float(data.get("balance", 0.0)),
"payment_url": data.get("payment_url", ""),
}
except Exception:
# Network error → allow in dev mode
return {"ok": True, "balance": 999.0}
# ─── Payment link ──────────────────────────────────────────────────────────────
def get_payment_link(user_id: str) -> str:
"""
Get payment URL for user to top up.
"""
if _is_dev_mode():
return ""
try:
resp = requests.post(
f"{BILLING_API_URL}/payment-link",
headers=_get_headers(),
json={"user_id": user_id, "skill_id": _get_skill_id()},
timeout=10,
)
data = resp.json()
return data.get("payment_url", "")
except Exception:
return ""
FILE:scripts/reporter.py
"""
F6 · Data quality report generation.
Generates structured reports in Markdown (for Feishu Doc) and dict form.
Usage:
from reporter import DataQualityReporter, Report
reporter = DataQualityReporter(before_df, after_df, field_info)
report = reporter.generate()
markdown = reporter.to_markdown(report)
"""
import math
from typing import Dict, List, Optional, Any
from dataclasses import dataclass, field
from datetime import datetime
import pandas as pd
from field_identifier import FieldType, FieldInfo, FIELD_TYPE_LABELS
from cleaner import CleaningReport as CleaningReport_
from classifier import ClassificationReport as ClassificationReport_
# ─── Report dataclasses ─────────────────────────────────────────────────────────
@dataclass
class ColumnStats:
col: str
field_type: str
total: int
missing: int
missing_pct: float
unique: int
sample_values: List[str] = field(default_factory=list)
@dataclass
class DataQualityReport:
generated_at: str
source_name: str
tier: str
before_shape: tuple[int, int]
after_shape: tuple[int, int]
duplicate_rate_before: float
duplicate_rate_after: float
missing_rate_before: float
missing_rate_after: float
overall_score: float # 0-100
column_stats: List[ColumnStats] = field(default_factory=list)
cleaning_report: Optional[CleaningReport_] = None
classification_report: Optional[ClassificationReport_] = None
recommendations: List[str] = field(default_factory=list)
# ─── Reporter ──────────────────────────────────────────────────────────────────
class DataQualityReporter:
"""
Generate data quality reports before/after cleaning.
Parameters
----------
before_df : original DataFrame
after_df : cleaned DataFrame
field_info : Dict[col -> FieldInfo]
source_name: str identifier for the data source
tier : subscription tier string
cleaning_report : Optional[CleaningReport]
classification_report: Optional[ClassificationReport]
"""
def __init__(
self,
before_df: pd.DataFrame,
after_df: pd.DataFrame,
field_info: Dict[str, FieldInfo],
*,
source_name: str = "数据源",
tier: str = "free",
cleaning_report: Optional[CleaningReport_] = None,
classification_report: Optional[ClassificationReport_] = None,
):
self.before_df = before_df.copy()
self.after_df = after_df.copy()
self.field_info = field_info
self.source_name = source_name
self.tier = tier
self.cleaning_report = cleaning_report
self.classification_report = classification_report
def generate(self) -> DataQualityReport:
"""Build the full quality report."""
stats = self._compute_stats()
dup_before = self._dup_rate(self.before_df)
dup_after = self._dup_rate(self.after_df)
miss_before = self._missing_rate(self.before_df)
miss_after = self._missing_rate(self.after_df)
score = self._overall_score(dup_after, miss_after)
recs = self._recommendations(stats, dup_after, miss_after)
return DataQualityReport(
generated_at=datetime.now().strftime("%Y-%m-%d %H:%M:%S"),
source_name=self.source_name,
tier=self.tier,
before_shape=(len(self.before_df), len(self.before_df.columns)),
after_shape =(len(self.after_df), len(self.after_df.columns)),
duplicate_rate_before=dup_before,
duplicate_rate_after=dup_after,
missing_rate_before=miss_before,
missing_rate_after=miss_after,
overall_score=score,
column_stats=stats,
cleaning_report=self.cleaning_report,
classification_report=self.classification_report,
recommendations=recs,
)
# ─── Per-column statistics ─────────────────────────────────────────────────
def _compute_stats(self) -> List[ColumnStats]:
stats = []
for col in self.before_df.columns:
fi = self.field_info.get(col)
ft_label = FIELD_TYPE_LABELS.get(fi.field_type, "未知") if fi else "未知"
total = len(self.before_df)
missing = self.before_df[col].astype(str).isin(
["", "nan", "NaN", "None", "null", "NULL", "undefined", "未知"]
).sum()
missing_pct = missing / total if total else 0
unique = self.before_df[col].nunique()
samples = (
self.before_df[col]
.astype(str)
.loc[lambda x: ~x.isin(["", "nan", "NaN", "None", "null", "未知"])]
.drop_duplicates()
.head(5)
.tolist()
)
stats.append(ColumnStats(
col=str(col),
field_type=ft_label,
total=total,
missing=missing,
missing_pct=missing_pct,
unique=unique,
sample_values=samples,
))
return stats
# ─── Rate calculations ───────────────────────────────────────────────────────
def _dup_rate(self, df: pd.DataFrame) -> float:
if df.empty:
return 0.0
return 1.0 - (df.drop_duplicates().shape[0] / len(df))
def _missing_rate(self, df: pd.DataFrame) -> float:
if df.empty:
return 0.0
total_cells = df.shape[0] * df.shape[1]
missing_cells = df.astype(str).isin(
["", "nan", "NaN", "None", "null", "NULL", "undefined", "未知"]
).sum().sum()
return missing_cells / total_cells if total_cells else 0.0
def _overall_score(self, dup_rate: float, miss_rate: float) -> float:
"""Score 0-100, higher = better quality."""
dup_penalty = dup_rate * 40 # up to -40
miss_penalty = miss_rate * 40 # up to -40
return round(max(0.0, 100.0 - (dup_rate * 40) - (miss_rate * 40)), 1)
# ─── Recommendations ────────────────────────────────────────────────────────
def _recommendations(
self,
stats: List[ColumnStats],
dup_rate: float,
miss_rate: float,
) -> List[str]:
recs: List[str] = []
if dup_rate > 0.1:
recs.append(
f"⚠️ 重复率较高（{dup_rate:.1%}），建议检查是否存在系统重复导入。"
)
if miss_rate > 0.05:
recs.append(
f"⚠️ 缺失率偏高（{miss_rate:.1%}），建议补充缺失数据以提高分析准确性。"
)
for s in stats:
if s.missing_pct > 0.3:
recs.append(
f"⚠️ 列「{s.col}」缺失率达 {s.missing_pct:.0%}，"
f"建议确认该字段是否必填或启用智能补全功能。"
)
if s.unique == 1 and s.total > 10:
recs.append(
f"ℹ️ 列「{s.col}」所有值相同（{s.unique} 个唯一值），"
f"可能为无效特征，建议移除。"
)
if not recs:
recs.append("✅ 数据质量良好，未发现明显问题。")
return recs
# ─── Markdown output ────────────────────────────────────────────────────────
def to_markdown(self, report: DataQualityReport) -> str:
"""Render report as Feishu-compatible Markdown."""
lines: List[str] = []
lines.append(f"# 📊 数据质量报告")
lines.append(f"")
lines.append(f"**数据源：** {report.source_name}")
lines.append(f"**版本：** {report.tier}")
lines.append(f"**生成时间：** {report.generated_at}")
lines.append("")
# Score card
score_emoji = (
"🟢" if report.overall_score >= 80
else "🟡" if report.overall_score >= 60
else "🔴"
)
lines.append(f"## {score_emoji} 综合质量评分：{report.overall_score}/100")
lines.append("")
# Shape summary
lines.append("## 📐 数据规模")
lines.append("")
lines.append("| 阶段 | 行数 | 列数 |")
lines.append("|------|------|------|")
lines.append(
f"| 清洗前 | {report.before_shape[0]:,} | {report.before_shape[1]:,} |"
)
lines.append(
f"| 清洗后 | {report.after_shape[0]:,} | {report.after_shape[1]:,} |"
)
lines.append("")
# Key rates
lines.append("## 📈 质量指标")
lines.append("")
lines.append("| 指标 | 清洗前 | 清洗后 | 变化 |")
lines.append("|------|--------|--------|------|")
dup_delta = report.duplicate_rate_after - report.duplicate_rate_before
dup_arrow = "↓" if dup_delta < 0 else "↑"
lines.append(
f"| 重复率 | {report.duplicate_rate_before:.2%} "
f"| {report.duplicate_rate_after:.2%} | {dup_arrow} {abs(dup_delta):.2%} |"
)
miss_delta = report.missing_rate_after - report.missing_rate_before
miss_arrow = "↓" if miss_delta < 0 else "↑"
lines.append(
f"| 缺失率 | {report.missing_rate_before:.2%} "
f"| {report.missing_rate_after:.2%} | {miss_arrow} {abs(miss_delta):.2%} |"
)
lines.append("")
# Cleaning details
if report.cleaning_report:
cr = report.cleaning_report
lines.append("## 🧹 清洗详情")
lines.append("")
lines.append(f"- 原始行数：{cr.original_rows}")
lines.append(f"- 清洗后行数：{cr.cleaned_rows}")
lines.append(f"- 去重：移除 {cr.duplicates_removed} 条")
lines.append(f"- 补全：填补 {cr.missing_filled} 个缺失值")
lines.append(f"- 格式化：处理 {cr.formatted_cells} 个单元格")
lines.append("")
if cr.missing_by_column:
lines.append("**各列缺失情况：**")
lines.append("")
for col, n in cr.missing_by_column.items():
lines.append(f"- {col}：{n} 个缺失值")
lines.append("")
if cr.duplicate_groups:
lines.append(f"检测到 {len(cr.duplicate_groups)} 组重复记录，已自动去重。")
lines.append("")
# Classification tags
if report.classification_report:
cl = report.classification_report
lines.append("## 🏷️ 标签分布")
lines.append("")
lines.append(f"总行数：{cl.total_rows} | 已打标签：{cl.tagged_rows}")
lines.append("")
for t in cl.tags:
pct = t.rows_matched / cl.total_rows if cl.total_rows else 0
lines.append(f"- **{t.tag}**：{t.rows_matched} 条（{pct:.0%}）")
lines.append("")
# Per-column stats
lines.append("## 📋 字段质量详情")
lines.append("")
lines.append(
"| 字段名 | 类型 | 样本量 | 缺失 | 缺失率 | 唯一值 | 示例值 |"
)
lines.append(
"|--------|------|--------|------|--------|--------|--------|"
)
for s in report.column_stats:
sample_str = " / ".join(s.sample_values[:2])
if len(sample_str) > 30:
sample_str = sample_str[:27] + "..."
lines.append(
f"| {s.col} | {s.field_type} | {s.total} | {s.missing} "
f"| {s.missing_pct:.0%} | {s.unique} | {sample_str} |"
)
lines.append("")
# Recommendations
if report.recommendations:
lines.append("## 💡 优化建议")
lines.append("")
for rec in report.recommendations:
lines.append(f"{rec}")
lines.append("")
return "\n".join(lines)
# ─── Compact dict (for JSON / API response) ─────────────────────────────────
def to_dict(self, report: DataQualityReport) -> Dict[str, Any]:
return {
"generated_at": report.generated_at,
"source_name": report.source_name,
"tier": report.tier,
"before_shape": report.before_shape,
"after_shape": report.after_shape,
"duplicate_rate_before": report.duplicate_rate_before,
"duplicate_rate_after": report.duplicate_rate_after,
"missing_rate_before": report.missing_rate_before,
"missing_rate_after": report.missing_rate_after,
"overall_score": report.overall_score,
"column_stats": [
{
"col": s.col,
"field_type": s.field_type,
"total": s.total,
"missing": s.missing,
"missing_pct": round(s.missing_pct, 4),
"unique": s.unique,
"sample_values": s.sample_values,
}
for s in report.column_stats
],
"recommendations": report.recommendations,
}
FILE:scripts/__init__.py
"""Multi-Source Data Cleanser - Skill scripts package."""
__version__ = "1.0.0"
FILE:scripts/field_identifier.py
"""
F2 · Intelligent field type identification.
Uses regex patterns + optional AI (MiniMax / DeepSeek) for ambiguous fields.
Supported types:
name, phone, email, address, amount, date, sku, order_no,
id_card, gender, url, ip_address, bank_account, custom
Usage:
from field_identifier import identify_fields, FieldType
types = identify_fields(df) # dict: col_name -> FieldType
types = identify_fields(df, ai_model="deepseek") # use AI for uncertain cols
types = identify_fields(df, custom_rules={"姓名": "name"})
"""
import os
import re
import json
import math
from typing import Dict, List, Optional, Tuple
from dataclasses import dataclass, field
from enum import Enum
import pandas as pd
# ─── Field type enum ───────────────────────────────────────────────────────────
class FieldType(str, Enum):
NAME = "name"
PHONE = "phone"
EMAIL = "email"
ADDRESS = "address"
AMOUNT = "amount"
DATE = "date"
SKU = "sku"
ORDER_NO = "order_no"
ID_CARD = "id_card"
GENDER = "gender"
URL = "url"
IP_ADDRESS = "ip_address"
BANK_ACCOUNT = "bank_account"
TEXT = "text"
NUMBER = "number"
UNKNOWN = "unknown"
FIELD_TYPE_LABELS: Dict[FieldType, str] = {
FieldType.NAME: "姓名",
FieldType.PHONE: "手机号",
FieldType.EMAIL: "邮箱",
FieldType.ADDRESS: "地址",
FieldType.AMOUNT: "金额",
FieldType.DATE: "日期",
FieldType.SKU: "SKU",
FieldType.ORDER_NO: "订单号",
FieldType.ID_CARD: "身份证",
FieldType.GENDER: "性别",
FieldType.URL: "网址",
FieldType.IP_ADDRESS: "IP地址",
FieldType.BANK_ACCOUNT: "银行账号",
FieldType.TEXT: "文本",
FieldType.NUMBER: "数字",
FieldType.UNKNOWN: "未知",
}
# ─── Regex patterns ────────────────────────────────────────────────────────────
_PATTERNS: Dict[FieldType, re.Pattern] = {
FieldType.EMAIL: re.compile(
r"^[a-zA-Z0-9._%+\-]+@[a-zA-Z0-9.\-]+\.[a-zA-Z]{2,}$"
),
FieldType.PHONE: re.compile(
r"^1[3-9]\d[\s\-]?\d{4}[\s\-]?\d{4}$" # Chinese mobile
),
FieldType.ID_CARD: re.compile(
r"^\d{15}$|^\d{17}[\dXx]$"
),
FieldType.DATE: re.compile(
r"^(\d{4}[-/年]\d{1,2}[-/月]\d{1,2}[日]?)"
r"|(\d{1,2}[-/]\d{1,2}[-/]\d{2,4})"
r"|^\d{8}$"
r"|^\d{10}$|^\d{13}$"
),
# AMOUNT: must have decimal point or currency symbol (not pure digits)
FieldType.AMOUNT: re.compile(
r"^[¥$€£]\s*[\d，,]+\.\d+$"
r"|^[¥$€£]\s*[\d，,]+\$"
r"|^[\d，,]+\.\d+\s*[¥$€£]?$"
),
# SKU: letter prefix or specific formats, NOT bare 11-digit phone-like numbers
FieldType.SKU: re.compile(
r"^[A-Za-z]+[\-]?\d{4,10}$"
r"|^SKU[\-_]?\d{4,12}$"
r"|^[A-Z0-9]{6,12}$"
),
FieldType.ORDER_NO: re.compile(
r"^(DD|ORDER|PO|SO|BM)[\-_]?\d{4,16}$"
r"|^[A-Z]{2,}\d{6,16}$"
r"|^\d{16,24}$"
),
FieldType.URL: re.compile(
r"^https?://[^\s]+$"
),
FieldType.IP_ADDRESS: re.compile(
r"^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$"
),
FieldType.GENDER: re.compile(
r"^(男|女|男性|女性|M|m|F|f)$"
),
}
# Chinese province / city / district keywords for address detection
_ADDRESS_KEYWORDS = [
"省", "市", "区", "县", "镇", "乡", "村", "路", "街", "号",
"栋", "单元", "室", "弄", "巷", "楼", "广场", "大厦", "小区",
"北京", "上海", "深圳", "广州", "杭州", "成都", "武汉", "西安",
]
_GENDER_VALUES = {"男", "女", "男性", "女性", "M", "F", "m", "f", "0", "1"}
# ─── Scoring helpers ───────────────────────────────────────────────────────────
def _is_empty(val: str) -> bool:
return val in ("", "nan", "NaN", "None", "null", "NULL", "undefined")
def _match_score(pat: re.Pattern, val: str) -> float:
"""Fraction of non-empty values matching the pattern (0.0–1.0)."""
if _is_empty(val):
return 0.0
return 1.0 if pat.match(str(val).strip()) else 0.0
def _avg_match_score(series: pd.Series, pat: re.Pattern) -> float:
vals = [str(v) for v in series if not _is_empty(str(v))]
if not vals:
return 0.0
return sum(1 for v in vals if pat.match(v.strip())) / len(vals)
def _numeric_score(series: pd.Series) -> float:
vals = [str(v).strip() for v in series if not _is_empty(str(v))]
if not vals:
return 0.0
count = 0
for v in vals:
# Remove currency symbols and commas
cleaned = re.sub(r"[¥$€£,\s]", "", v)
try:
float(cleaned)
count += 1
except ValueError:
pass
return count / len(vals)
def _date_score(series: pd.Series) -> float:
"""Higher score if values look like dates."""
vals = [str(v).strip() for v in series if not _is_empty(str(v))]
if not vals:
return 0.0
date_forms = [
(r"\d{4}[-/年]\d{1,2}[-/月]\d{1,2}", 0.9),
(r"\d{8}", 0.8),
(r"\d{10}", 0.8),
(r"\d{1,2}[-/]\d{1,2}[-/]\d{2,4}", 0.7),
]
total = 0.0
for v in vals:
for pat, w in date_forms:
if re.match(pat, v):
total += w
break
return total / len(vals)
def _address_score(series: pd.Series) -> float:
vals = [str(v) for v in series if not _is_empty(str(v))]
if not vals:
return 0.0
hits = sum(1 for v in vals if any(k in v for k in _ADDRESS_KEYWORDS))
avg_len = sum(len(v) for v in vals) / len(vals)
# Longish strings with address keywords → high score
len_bonus = min(avg_len / 20, 1.0) * 0.3
return hits / len(vals) * 0.7 + len_bonus
def _gender_score(series: pd.Series) -> float:
vals = {str(v).strip() for v in series if not _is_empty(str(v))}
if not vals:
return 0.0
intersection = vals & _GENDER_VALUES
return min(len(intersection) / 2, 1.0) # just 1-2 distinct values matters
def _name_score(series: pd.Series) -> float:
"""
Detect Chinese / common personal names.
Chinese names: 2-4 chars, often with common surname characters,
no digits, no special chars beyond hyphen/·.
"""
# Common Chinese surname characters (top 100)
COMMON_SURNAMES = set(
"李王张刘陈杨赵黄周吴徐孙胡朱高林何郭马罗梁宋郑谢韩唐冯于董萧程曹袁邓许傅沈曾彭吕苏卢蒋蔡贾丁魏薛叶阎余潘杜戴夏钟汪田任姜范方石姚谭廖邹熊金陆郝孔白崔康毛邱秦江史顾侯邵孟龙万段雷钱汤尹黎易常武乔贺赖龚文"
)
vals = [str(v).strip() for v in series if not _is_empty(str(v))]
if not vals:
return 0.0
hits = 0
for v in vals:
# Length check: 2-4 chars typical for Chinese full name
if 2 <= len(v) <= 4:
# No digits
if re.match(r"^[\u4e00-\u9fa5·\-']+$", v): # All Chinese + hyphen/apostrophe
# Check for surname character at start
if v[0] in COMMON_SURNAMES:
hits += 1
elif re.match(r"^[A-Z][a-z]+$", v): # English name "John Smith"
hits += 0.8
elif re.match(r"^[A-Za-z]+$", v) and 3 <= len(v) <= 15: # Single-word English name
hits += 0.5
# English name formats
elif re.match(r"^[A-Z][a-z]+(\s+[A-Z][a-z]+)+$", v):
hits += 1
return hits / len(vals)
# ─── Column name heuristics ────────────────────────────────────────────────────
_COL_NAME_WEIGHTS: Dict[str, Dict[FieldType, float]] = {
# Name → FieldType hints
"name": {FieldType.NAME: 0.9},
"姓名": {FieldType.NAME: 0.9},
"客户姓名": {FieldType.NAME: 0.9},
"phone": {FieldType.PHONE: 0.9},
"tel": {FieldType.PHONE: 0.8},
"mobile": {FieldType.PHONE: 0.9},
"电话": {FieldType.PHONE: 0.9},
"手机": {FieldType.PHONE: 0.9},
"手机号": {FieldType.PHONE: 0.9},
"email": {FieldType.EMAIL: 0.9},
"mail": {FieldType.EMAIL: 0.8},
"邮箱": {FieldType.EMAIL: 0.9},
"电子邮件": {FieldType.EMAIL: 0.9},
"address": {FieldType.ADDRESS: 0.9},
"地址": {FieldType.ADDRESS: 0.9},
"收货地址": {FieldType.ADDRESS: 0.9},
"amount": {FieldType.AMOUNT: 0.9},
"金额": {FieldType.AMOUNT: 0.9},
"总价": {FieldType.AMOUNT: 0.9},
"单价": {FieldType.AMOUNT: 0.9},
"price": {FieldType.AMOUNT: 0.9},
"销售额": {FieldType.AMOUNT: 0.9},
"date": {FieldType.DATE: 0.9},
"日期": {FieldType.DATE: 0.9},
"成交日期": {FieldType.DATE: 0.9},
"订单日期": {FieldType.DATE: 0.9},
"下单时间": {FieldType.DATE: 0.9},
"创建时间": {FieldType.DATE: 0.9},
"sku": {FieldType.SKU: 0.9},
"SKU": {FieldType.SKU: 0.9},
"商品编号": {FieldType.SKU: 0.9},
"order": {FieldType.ORDER_NO: 0.9},
"订单": {FieldType.ORDER_NO: 0.9},
"订单号": {FieldType.ORDER_NO: 0.9},
"order_no": {FieldType.ORDER_NO: 0.9},
"订单编号": {FieldType.ORDER_NO: 0.9},
"id_card": {FieldType.ID_CARD: 0.9},
"身份证": {FieldType.ID_CARD: 0.9},
"gender": {FieldType.GENDER: 0.9},
"性别": {FieldType.GENDER: 0.9},
"url": {FieldType.URL: 0.9},
"网址": {FieldType.URL: 0.9},
"ip": {FieldType.IP_ADDRESS: 0.9},
"ip地址": {FieldType.IP_ADDRESS: 0.9},
"银行账号": {FieldType.BANK_ACCOUNT: 0.9},
"账号": {FieldType.BANK_ACCOUNT: 0.5},
}
def _col_name_score(col: str, ftype: FieldType) -> float:
col_lower = col.lower()
hints = _COL_NAME_WEIGHTS.get(col, {})
hints.update(_COL_NAME_WEIGHTS.get(col_lower, {}))
return hints.get(ftype, 0.0)
# ─── Main identification function ─────────────────────────────────────────────
@dataclass
class FieldInfo:
field_type: FieldType
confidence: float # 0.0–1.0
samples: List[str] = field(default_factory=list)
def label(self) -> str:
return FIELD_TYPE_LABELS.get(self.field_type, str(self.field_type))
def identify_fields(
df: pd.DataFrame,
*,
custom_rules: Optional[Dict[str, str]] = None,
ai_model: Optional[str] = None,
ai_api_key: Optional[str] = None,
) -> Dict[str, FieldInfo]:
"""
Identify field types for all columns in a DataFrame.
Parameters
----------
df : input DataFrame
custom_rules : {column_name: FieldType.value} user-defined overrides
ai_model : "minimax" or "deepseek" — use AI for uncertain columns
ai_api_key : API key (defaults to env DATA_CLEANER_API_KEY)
Returns
-------
Dict[col_name -> FieldInfo]
"""
custom_rules = custom_rules or {}
results: Dict[str, FieldInfo] = {}
uncertain_cols: List[str] = []
for col in df.columns:
col_str = str(col).strip()
# 1. User override
if col_str in custom_rules:
try:
ft = FieldType(custom_rules[col_str])
results[col] = FieldInfo(ft, 1.0, [])
continue
except ValueError:
pass
# 2. Compute pattern scores
scores: Dict[FieldType, float] = {}
series = df[col].astype(str)
for ftype, pat in _PATTERNS.items():
pattern_score = _avg_match_score(series, pat)
name_score = _col_name_score(col_str, ftype)
# Combine: pattern is primary, name hints are bonus
combined = pattern_score * 0.7 + name_score * 0.3
if combined > 0:
scores[ftype] = min(combined, 1.0)
# Special fast-path scorers — add to scores only if the best
# type-specific pattern score is low (NUMBER is a generic fallback,
# not a competitor to clear type signals like phone/email/ID).
num_score = _numeric_score(series)
date_s = _date_score(series)
addr_s = _address_score(series)
gender_s = _gender_score(series)
name_s = _name_score(series) # Chinese / common name pattern
best_type_score = max(scores.values()) if scores else 0.0
# NUMBER only wins when no specific type has scored clearly (below 0.7)
if num_score > 0.5 and best_type_score < 0.7:
scores[FieldType.NUMBER] = num_score
if date_s > best_type_score:
scores[FieldType.DATE] = date_s
if addr_s > 0.5:
scores[FieldType.ADDRESS] = addr_s
if gender_s > 0.5:
scores[FieldType.GENDER] = gender_s
if name_s > best_type_score:
scores[FieldType.NAME] = name_s
# 3. Pick best match
if not scores:
results[col] = FieldInfo(FieldType.UNKNOWN, 0.0, _samples(series, 3))
uncertain_cols.append(col)
continue
best_ftype = max(scores, key=lambda k: scores[k])
confidence = scores[best_ftype]
# Low confidence → mark uncertain
if confidence < 0.6:
uncertain_cols.append(col)
results[col] = FieldInfo(
field_type=best_ftype,
confidence=confidence,
samples=_samples(series, 3),
)
# 4. AI assist for uncertain columns
if uncertain_cols and ai_model:
_fill_with_ai(results, uncertain_cols, df, ai_model, ai_api_key)
return results
def _samples(series: pd.Series, n: int = 3) -> List[str]:
vals = [str(v) for v in series if str(v) not in ("", "nan", "NaN")]
return list(dict.fromkeys(vals))[:n] # unique, preserve order
# ─── AI-assisted identification ───────────────────────────────────────────────
def _fill_with_ai(
results: Dict[str, FieldInfo],
uncertain_cols: List[str],
df: pd.DataFrame,
model: str,
api_key: Optional[str],
) -> None:
api_key = api_key or os.environ.get("DATA_CLEANER_API_KEY", "")
if not api_key:
return
import urllib.request
import urllib.parse
col_samples = {
col: _samples(df[col].astype(str), 5)
for col in uncertain_cols
}
prompt = (
"你是一个数据分析师。请根据以下列名和样本值，判断每列的字段类型。\n"
"字段类型可选：姓名、手机号、邮箱、地址、金额、日期、SKU、订单号、身份证、性别、网址、IP地址、银行账号、文本、数字、未知\n"
"只输出JSON对象，格式：{\"列名\": \"类型\"}\n"
f"\n列名与样本值：\n{json.dumps(col_samples, ensure_ascii=False)}\n"
)
try:
if model in ("deepseek", "minimax"):
# Unified MiniMax/DeepSeek compatible API
url = "https://api.minimax.chat/v1/text/chatcompletion_pro"
if model == "deepseek":
url = "https://api.deepseek.com/v1/chat/completions"
payload = json.dumps({
"model": "MiniMax-Text-01" if model == "minimax" else "deepseek-chat",
"messages": [{"role": "user", "content": prompt}],
"max_tokens": 300,
"temperature": 0.1,
}).encode("utf-8")
req = urllib.request.Request(
url,
data=payload,
headers={
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json",
},
method="POST",
)
with urllib.request.urlopen(req, timeout=15) as resp:
raw = json.loads(resp.read().decode("utf-8"))
content = raw["choices"][0]["message"]["content"]
# Parse JSON from response
match = re.search(r"\{[^{}]*\}", content, re.DOTALL)
if match:
ai_result = json.loads(match.group())
for col, type_str in ai_result.items():
if col in results:
try:
ft = FieldType(type_str)
results[col] = FieldInfo(ft, 0.8, results[col].samples)
except ValueError:
pass
except Exception:
pass # AI fallback: keep regex result
# ─── User-defined field mapping ───────────────────────────────────────────────
def apply_custom_mapping(
field_info: Dict[str, FieldInfo],
mapping: Dict[str, str],
) -> Dict[str, FieldInfo]:
"""
Apply user-defined column → field_type overrides.
mapping = {"姓名": "name", "电话": "phone"}
"""
for col, type_str in mapping.items():
if col in field_info:
try:
ft = FieldType(type_str)
field_info[col] = FieldInfo(ft, 1.0, field_info[col].samples)
except ValueError:
pass
return field_info
# ─── Summary ──────────────────────────────────────────────────────────────────
def field_summary(field_info: Dict[str, FieldInfo]) -> str:
lines = ["### 字段识别结果"]
for col, info in field_info.items():
conf_pct = f"{info.confidence:.0%}"
lines.append(f"- **{col}** → {info.label()}（置信度 {conf_pct}）")
return "\n".join(lines)
FILE:scripts/parser.py
"""
F1 · Multi-format parser
Supports: Excel (.xlsx/.xls), CSV/TSV, JSON, clipboard-pasted text.
Usage:
from parser import parse_file, parse_text
df = parse_file("path/to/data.xlsx")
df = parse_file("path/to/data.csv")
df = parse_text("姓名,电话\\n张三,13800138000")
"""
import io
import json
import re
import pandas as pd
from pathlib import Path
from typing import Union, List, Optional
# ─── Helpers ─────────────────────────────────────────────────────────────────
def _sniff_delimiter(text: str) -> str:
"""Sniff the most likely delimiter from the first few lines."""
sample = text[:2000]
for delim in [",", "\t", "|", ";"]:
if delim in sample:
return delim
return ","
def _is_json(text: str) -> bool:
text = text.strip()
return text.startswith("{") or text.startswith("[")
def _is_excel(path: Union[str, Path]) -> bool:
p = str(path).lower()
return p.endswith(".xlsx") or p.endswith(".xls")
# ─── Core parsers ─────────────────────────────────────────────────────────────
def parse_file(
path: Union[str, Path],
sheet: Optional[Union[str, int]] = None,
encoding: str = "utf-8",
) -> pd.DataFrame:
"""
Auto-detect format and parse a file into a DataFrame.
Parameters
----------
path : file path
sheet : for Excel files, sheet name or 0-indexed position
encoding : text file encoding
Returns
-------
pd.DataFrame
"""
path = str(path)
if _is_excel(path):
return _parse_excel(path, sheet=sheet)
suffix = Path(path).suffix.lower()
if suffix in (".csv", ".tsv", ".txt"):
return _parse_csv(path, delimiter=None, encoding=encoding)
if suffix == ".json":
return _parse_json(path)
# fallback: try CSV then JSON
try:
return _parse_csv(path, delimiter=None, encoding=encoding)
except Exception:
return _parse_json(path)
def parse_text(text: str, format_hint: str = "auto") -> pd.DataFrame:
"""
Parse clipboard / pasted text into a DataFrame.
Parameters
----------
text : raw pasted content
format_hint : "csv" | "tsv" | "json" | "auto"
Returns
-------
pd.DataFrame
"""
text = text.strip()
if not text:
raise ValueError("粘贴内容为空，请确认已复制数据。")
if format_hint == "json" or (format_hint == "auto" and _is_json(text)):
return _parse_json_str(text)
if format_hint == "tsv" or (format_hint == "auto" and "\t" in text):
return _parse_csv_str(text, delimiter="\t")
# Default: sniff delimiter
delim = _sniff_delimiter(text) if format_hint == "auto" else ","
return _parse_csv_str(text, delimiter=delim)
# ─── Private implementations ───────────────────────────────────────────────────
def _parse_excel(path: str, sheet: Optional[Union[str, int]]) -> pd.DataFrame:
try:
import openpyxl
except ImportError:
raise ImportError("openpyxl is required to read .xlsx files. Install: pip install openpyxl")
try:
import xlrd
except ImportError:
pass # .xls may not be readable but .xlsx will work
xl_kwargs: dict = {"engine": "openpyxl"}
if Path(path).suffix.lower() == ".xls":
xl_kwargs["engine"] = "xlrd"
if sheet is not None:
xl_kwargs["sheet_name"] = sheet
else:
xl_kwargs["sheet_name"] = 0 # first sheet by default
return pd.read_excel(path, **xl_kwargs)
def _parse_csv(path: str, delimiter: Optional[str], encoding: str) -> pd.DataFrame:
if delimiter:
return pd.read_csv(path, delimiter=delimiter, encoding=encoding, dtype=str, keep_default_na=False)
# Auto-sniff
try:
return pd.read_csv(path, encoding=encoding, dtype=str, keep_default_na=False)
except UnicodeDecodeError:
for enc in ["gbk", "gb2312", "latin1"]:
try:
return pd.read_csv(path, encoding=enc, dtype=str, keep_default_na=False)
except UnicodeDecodeError:
continue
raise ValueError(f"无法解析 CSV 文件 {path}，请确认文件编码（UTF-8 / GBK）。")
def _parse_json(path: str) -> pd.DataFrame:
with open(path, encoding="utf-8") as f:
data = json.load(f)
return _json_to_df(data)
def _parse_csv_str(text: str, delimiter: str = ",") -> pd.DataFrame:
df = pd.read_csv(io.StringIO(text), delimiter=delimiter, dtype=str, keep_default_na=False)
# Strip whitespace from all cells
df = df.apply(lambda col: col.astype(str).str.strip())
return df
def _parse_json_str(text: str) -> pd.DataFrame:
data = json.loads(text)
return _json_to_df(data)
def _json_to_df(data) -> pd.DataFrame:
"""
Handle three common JSON shapes:
1. Array of flat objects : [{a:1},{a:2}]
2. Object with array field: {rows:[{a:1},...]}
3. Nested / hierarchical : flatten if possible
"""
if isinstance(data, list):
records = data
elif isinstance(data, dict):
# Try to find the array field
for key in ["data", "rows", "records", "items", "list", "result"]:
if key in data and isinstance(data[key], list):
records = data[key]
break
else:
records = [data]
else:
raise ValueError("JSON 结构不支持，请提供对象数组或包含数据数组的根对象。")
if not records:
return pd.DataFrame()
df = pd.DataFrame(records)
# Flatten any deeply nested columns (one level only)
for col in df.columns:
if df[col].apply(lambda x: isinstance(x, dict)).any():
exploded = df[col].apply(lambda x: x if isinstance(x, dict) else {})
norm = pd.json_normalize(exploded)
norm.columns = [f"{col}_{k}" for k in norm.columns]
df = pd.concat([df.drop(columns=[col]), norm], axis=1)
df = df.apply(lambda col: col.astype(str).str.strip())
return df
# ─── Multi-file loader ─────────────────────────────────────────────────────────
def load_sources(
sources: List[Union[str, Path, io.IOBase]],
texts: Optional[List[str]] = None,
) -> List[pd.DataFrame]:
"""
Load multiple data sources (files or pasted texts) into DataFrames.
Returns list of (name, df) tuples.
"""
results = []
# Files
for src in (sources or []):
name = Path(src).name if isinstance(src, (str, Path)) else str(src)
try:
df = parse_file(src)
results.append((name, df))
except Exception as exc:
raise ValueError(f"文件「{name}」解析失败：{exc}")
# Pasted texts
for i, text in enumerate(texts or []):
name = f"粘贴数据_{i+1}"
try:
df = parse_text(text)
results.append((name, df))
except Exception as exc:
raise ValueError(f"「{name}」解析失败：{exc}")
return results
# ─── Quick peek (header preview) ──────────────────────────────────────────────
def preview_file(path: Union[str, Path], n: int = 5) -> pd.DataFrame:
"""Return first n rows without full load (for display)."""
df = parse_file(path)
return df.head(n)
def preview_text(text: str, n: int = 5) -> pd.DataFrame:
"""Return first n rows of pasted text (for display)."""
df = parse_text(text)
return df.head(n)
FILE:scripts/cleaner.py
"""
F3 · Core data cleaning engine.
Handles:
- Smart deduplication (exact + fuzzy)
- Missing value imputation (mean/mode/inference/leave_blank)
- Format unification (phone/date/amount/address)
Usage:
from cleaner import DataCleaner
cleaner = DataCleaner(field_info)
df_clean = cleaner.clean(df)
"""
import re
import math
from typing import Dict, Optional, List, Callable, Any
from dataclasses import dataclass, field
from datetime import datetime
import pandas as pd
from field_identifier import FieldType, FieldInfo
# ─── Address standardisation helpers ─────────────────────────────────────────
_CN_PROVINCES = [
"北京市","天津市","上海市","重庆市",
"河北省","山西省","辽宁省","吉林省","黑龙江省",
"江苏省","浙江省","安徽省","福建省","江西省","山东省",
"河南省","湖北省","湖南省","广东省","海南省",
"四川省","贵州省","云南省","陕西省","甘肃省","青海省","台湾省",
"内蒙古自治区","广西壮族自治区","西藏自治区","宁夏回族自治区","新疆维吾尔自治区",
"香港特别行政区","澳门特别行政区",
]
_CN_PROVINCE_ABBR = {
"北京":"北京市","上海":"上海市","天津":"天津市","重庆":"重庆市",
"河北":"河北省","山西":"山西省","辽宁":"辽宁省","吉林":"吉林省",
"黑龙江":"黑龙江省","江苏":"江苏省","浙江":"浙江省","安徽":"安徽省",
"福建":"福建省","江西":"江西省","山东":"山东省","河南":"河南省",
"湖北":"湖北省","湖南":"湖南省","广东":"广东省","海南":"海南省",
"四川":"四川省","贵州":"贵州省","云南":"云南省","陕西":"陕西省",
"甘肃":"甘肃省","青海":"青海省","台湾":"台湾省",
"内蒙古":"内蒙古自治区","广西":"广西壮族自治区","西藏":"西藏自治区",
"宁夏":"宁夏回族自治区","新疆":"新疆维吾尔自治区",
"香港":"香港特别行政区","澳门":"澳门特别行政区",
}
# ─── Dataclass results ─────────────────────────────────────────────────────────
@dataclass
class CleaningReport:
original_rows: int
cleaned_rows: int
duplicates_removed: int
missing_filled: int
formatted_cells: int
missing_by_column: Dict[str, int] = field(default_factory=dict)
duplicate_groups: List[List[int]] = field(default_factory=list)
def summary(self) -> str:
return (
f"原始行数：{self.original_rows} | 清洗后行数：{self.cleaned_rows}\n"
f"去重：移除 {self.duplicates_removed} 条重复记录\n"
f"补全：填补 {self.missing_filled} 个缺失值\n"
f"格式化：处理 {self.formatted_cells} 个单元格"
)
# ─── Core Cleaner ─────────────────────────────────────────────────────────────
class DataCleaner:
"""
Main cleaning orchestrator.
Parameters
----------
field_info : Dict[col -> FieldInfo] from field_identifier
"""
def __init__(
self,
field_info: Dict[str, FieldInfo],
*,
dedup_strategy: str = "auto",
fill_strategy: str = "auto",
format_phone: bool = True,
format_date: bool = True,
format_amount: bool = True,
format_address: bool = True,
):
self.field_info = field_info
self.dedup_strategy = dedup_strategy # "exact" | "fuzzy" | "auto"
self.fill_strategy = fill_strategy # "auto" | "mean" | "mode" | "leave_blank"
self.format_phone = format_phone
self.format_date = format_date
self.format_amount = format_amount
self.format_address = format_address
# Per-column type lookup
self._type_map: Dict[str, FieldType] = {
col: fi.field_type for col, fi in field_info.items()
}
def clean(self, df: pd.DataFrame) -> tuple[pd.DataFrame, CleaningReport]:
"""
Full cleaning pipeline on a single DataFrame.
Returns (cleaned_df, report).
"""
df = df.copy()
original_rows = len(df)
total_missing_filled = 0
total_formatted = 0
# ── Step 1: Normalise column names ──────────────────────────────────────
df.columns = [str(c).strip() for c in df.columns]
# ── Step 2: Missing value imputation ─────────────────────────────────────
df, missing_filled, missing_by_col = self._impute(df)
total_missing_filled += missing_filled
# ── Step 3: Format unification ──────────────────────────────────────────
df, formatted = self._format_all(df)
total_formatted += formatted
# ── Step 4: Deduplication ───────────────────────────────────────────────
df, dup_removed, dup_groups = self._deduplicate(df)
total_dup_removed = original_rows - len(df)
report = CleaningReport(
original_rows=original_rows,
cleaned_rows=len(df),
duplicates_removed=total_dup_removed,
missing_filled=total_missing_filled,
formatted_cells=total_formatted,
missing_by_column=missing_by_col,
duplicate_groups=dup_groups,
)
return df, report
# ── Imputation ─────────────────────────────────────────────────────────────
def _impute(
self,
df: pd.DataFrame,
) -> tuple[pd.DataFrame, int, Dict[str, int]]:
"""Fill missing values according to field type and fill_strategy."""
filled = 0
missing_by_col: Dict[str, int] = {}
df = df.copy()
for col in df.columns:
ftype = self._type_map.get(col, FieldType.UNKNOWN)
blanks = df[col].astype(str).isin(["", "nan", "NaN", "None", "null", "NULL", "undefined"])
n_blank = blanks.sum()
if n_blank == 0:
continue
missing_by_col[col] = int(n_blank)
strategy = self.fill_strategy
if strategy == "leave_blank":
continue # keep NaN
filled_col = self._fill_column(df[col], ftype, strategy)
df[col] = filled_col
filled += n_blank
return df, filled, missing_by_col
def _fill_column(
self,
series: pd.Series,
ftype: FieldType,
strategy: str,
) -> pd.Series:
"""Apply the appropriate fill logic to a single column."""
blanks = series.astype(str).isin(["", "nan", "NaN", "None", "null", "NULL"])
if strategy == "mean":
if ftype in (FieldType.AMOUNT, FieldType.NUMBER):
nums = pd.to_numeric(series, errors="coerce")
mean_val = nums.mean()
if not math.isnan(mean_val):
filled = series.copy()
filled[blanks] = f"{mean_val:.2f}"
return filled
elif strategy == "mode":
# Most common non-blank value
non_blanks = series[~blanks]
if not non_blanks.empty:
mode_val = non_blanks.mode()
if not mode_val.empty:
filled = series.copy()
filled[blanks] = mode_val.iloc[0]
return filled
elif strategy in ("auto", "inference"):
filled = self._auto_fill(series, ftype)
return filled
# Default: leave blank
return series
def _auto_fill(self, series: pd.Series, ftype: FieldType) -> pd.Series:
"""
Auto-fill missing values based on field semantics.
Returns a new series (does not modify in place).
"""
blanks = series.astype(str).isin(["", "nan", "NaN", "None", "null", "NULL"])
if blanks.sum() == 0:
return series
filled = series.copy()
if ftype == FieldType.GENDER:
# Fill with most common
non_blank = series[~blanks]
if not non_blank.empty:
mode = non_blank.mode()
if not mode.empty:
filled[blanks] = mode.iloc[0]
elif ftype in (FieldType.AMOUNT, FieldType.NUMBER):
nums = pd.to_numeric(series, errors="coerce")
mean_val = nums.mean()
if not math.isnan(mean_val):
filled[blanks] = f"{mean_val:.2f}"
elif ftype == FieldType.DATE:
# Try to parse most common format, fill with placeholder
filled[blanks] = "未知"
elif ftype == FieldType.PHONE:
filled[blanks] = "未知"
elif ftype == FieldType.EMAIL:
filled[blanks] = "未知"
# Text / address / others → "未知"
elif ftype in (FieldType.TEXT, FieldType.ADDRESS, FieldType.UNKNOWN,
FieldType.NAME, FieldType.SKU, FieldType.ORDER_NO):
filled[blanks] = "未知"
return filled
# ── Format unification ───────────────────────────────────────────────────────
def _format_all(self, df: pd.DataFrame) -> tuple[pd.DataFrame, int]:
formatted = 0
df = df.copy()
for col in df.columns:
ftype = self._type_map.get(col, FieldType.UNKNOWN)
before = df[col].astype(str)
if ftype == FieldType.PHONE and self.format_phone:
df[col] = df[col].apply(self._format_phone)
elif ftype == FieldType.DATE and self.format_date:
df[col] = df[col].apply(self._format_date)
elif ftype == FieldType.AMOUNT and self.format_amount:
df[col] = df[col].apply(self._format_amount_)
elif ftype == FieldType.ADDRESS and self.format_address:
df[col] = df[col].apply(self._standardise_address)
after = df[col].astype(str)
formatted += (before != after).sum()
return df, formatted
def _format_phone(self, val: Any) -> str:
"""Normalise phone to 1xx-xxxx-xxxx."""
s = str(val).strip()
if s in ("", "nan", "NaN", "None", "未知"):
return s
# Strip all non-digit
digits = re.sub(r"\D", "", s)
# Take last 11 digits if > 11
if len(digits) > 11:
digits = digits[-11:]
if len(digits) == 11 and digits[0] == "1":
return f"{digits[0:3]}-{digits[3:7]}-{digits[7:11]}"
return s # return original if not recognisable
def _format_date(self, val: Any) -> str:
"""Normalise date to YYYY-MM-DD."""
s = str(val).strip()
if s in ("", "nan", "NaN", "None", "未知"):
return s
# Already ISO?
if re.match(r"\d{4}-\d{2}-\d{2}", s):
return s
# Chinese format: YYYY年MM月DD日
m = re.match(r"(\d{4})年(\d{1,2})月(\d{1,2})日?", s)
if m:
return f"{m.group(1)}-{int(m.group(2)):02d}-{int(m.group(3)):02d}"
# Slash/ hyphen: YYYY/MM/DD or YYYY-MM-DD or YYYYMD
for pat in [
r"(\d{4})[-/年](\d{1,2})[-/月](\d{1,2})",
r"(\d{4})(\d{2})(\d{2})",
]:
m = re.match(pat, s)
if m:
return f"{m.group(1)}-{int(m.group(2)):02d}-{int(m.group(3)):02d}"
# YYYYMMDD integer
if re.match(r"^\d{8}$", s):
return f"{s[:4]}-{s[4:6]}-{s[6:8]}"
# Unix timestamp (seconds)
if re.match(r"^\d{10}$", s):
try:
from datetime import datetime as dt
return dt.fromtimestamp(int(s)).strftime("%Y-%m-%d")
except Exception:
pass
# Milliseconds
if re.match(r"^\d{13}$", s):
try:
from datetime import datetime as dt
return dt.fromtimestamp(int(s) / 1000).strftime("%Y-%m-%d")
except Exception:
pass
return s
def _format_amount_(self, val: Any) -> str:
"""Normalise amount to two decimal places."""
s = str(val).strip()
if s in ("", "nan", "NaN", "None", "未知"):
return s
# Strip currency symbols and commas
cleaned = re.sub(r"[¥$€£,\s]", "", s)
try:
num = float(cleaned)
return f"{num:.2f}"
except ValueError:
return s
def _standardise_address(self, val: Any) -> str:
"""Standardise Chinese address to 省市区街道格式."""
s = str(val).strip()
if s in ("", "nan", "NaN", "None", "未知"):
return s
# Expand province abbreviation
for abbr, full in _CN_PROVINCE_ABBR.items():
if s.startswith(abbr):
s = full + s[len(abbr):]
break
# Normalise separators to " "
s = re.sub(r"[,，;；\t]+", " ", s)
s = re.sub(r"\s+", " ", s).strip()
return s
# ─── Deduplication ─────────────────────────────────────────────────────────
def _deduplicate(
self,
df: pd.DataFrame,
) -> tuple[pd.DataFrame, int, List[List[int]]]:
"""
Deduplicate rows.
Strategy:
exact → drop_duplicates on all columns
fuzzy → fuzzy match on key identity columns (phone/name/email/order_no)
auto → fuzzy if key columns present, else exact
"""
strategy = self.dedup_strategy
dup_groups: List[List[int]] = []
if df.empty:
return df, 0, dup_groups
key_cols = self._find_key_columns()
original_len = len(df)
if strategy == "exact":
before = len(df)
df = df.drop_duplicates()
removed = before - len(df)
return df, removed, dup_groups
if strategy in ("fuzzy", "auto") and key_cols:
df, removed, dup_groups = self._fuzzy_dedup(df, key_cols)
return df, removed, dup_groups
# Fallback: exact
before = len(df)
df = df.drop_duplicates()
return df, before - len(df), dup_groups
def _find_key_columns(self) -> List[str]:
"""Find identity-like columns suitable for fuzzy dedup."""
key_types = {
FieldType.PHONE, FieldType.EMAIL,
FieldType.NAME, FieldType.ORDER_NO, FieldType.SKU, FieldType.ID_CARD,
}
return [
col for col, ft in self._type_map.items()
if ft in key_types
]
def _fuzzy_dedup(
self,
df: pd.DataFrame,
key_cols: List[str],
) -> tuple[pd.DataFrame, int, List[List[int]]]:
"""
Fuzzy deduplication using FuzzyWuzzy on key columns.
Keeps the first occurrence, removes fuzzy duplicates.
"""
try:
from fuzzywuzzy import fuzz
except ImportError:
# Fallback to exact if fuzzywuzzy not installed
before = len(df)
df = df.drop_duplicates(subset=key_cols, keep="first")
return df, before - len(df), []
# Build composite key strings
key_strs = df[key_cols].astype(str).agg(" | ".join, axis=1)
keep_idx: List[int] = []
dup_groups: List[List[int]] = []
removed = 0
for i, key in enumerate(key_strs):
is_dup = False
for j in keep_idx:
score = fuzz.ratio(key, key_strs.iloc[j])
if score >= 88: # threshold
is_dup = True
break
if is_dup:
removed += 1
else:
keep_idx.append(i)
dup_indices = set(range(len(df))) - set(keep_idx)
dup_groups = [list(dup_indices)] if dup_indices else []
return df.iloc[keep_idx].reset_index(drop=True), removed, dup_groups
# ─── Convenience function ─────────────────────────────────────────────────────
def clean_dataframe(
df: pd.DataFrame,
field_info: Dict[str, FieldInfo],
*,
dedup_strategy: str = "auto",
fill_strategy: str = "auto",
) -> tuple[pd.DataFrame, CleaningReport]:
"""
One-liner clean.
"""
cleaner = DataCleaner(
field_info,
dedup_strategy=dedup_strategy,
fill_strategy=fill_strategy,
)
return cleaner.clean(df)
FILE:scripts/tier_limits.py
"""
Tier-based access control for Multi-Source Data Cleanser.
Tiers:
FREE - 50 rows/month, 1 source, single format, no merge/AI/report
BASIC - 500 rows/month, 3 sources, basic dedup
STD - 3000 rows/month, unlimited sources, format unification + smart fill
PRO - unlimited, multi-source merge + AI classification + data quality report
Usage:
from tier_limits import check_tier, TIER_LIMITS, TierLimitExceeded
check_tier("free", rows=30) # raises if over limit
check_tier("basic", sources=2) # raises if over limit
has_feature("pro", "fuzzy_join") # bool
enforce_max_rows(tier, rows, count_key) # updates monthly counter
"""
import os
import json
import time
from pathlib import Path
import time
from enum import Enum
from dataclasses import dataclass, field
from typing import Optional, Dict, Any
# ─── Constants ────────────────────────────────────────────────────────────────
class Tier(str, Enum):
FREE = "free"
BASIC = "basic"
STD = "std"
PRO = "pro"
TIER_ORDER = [Tier.FREE, Tier.BASIC, Tier.STD, Tier.PRO]
# Monthly row limits per tier
TIER_MONTHLY_ROWS: Dict[Tier, int] = {
Tier.FREE: 50,
Tier.BASIC: 500,
Tier.STD: 3000,
Tier.PRO: -1, # unlimited
}
# Max data sources (files / pasted blocks)
TIER_MAX_SOURCES: Dict[Tier, int] = {
Tier.FREE: 1,
Tier.BASIC: 3,
Tier.STD: -1,
Tier.PRO: -1,
}
# Max columns allowed
TIER_MAX_COLUMNS: Dict[Tier, int] = {
Tier.FREE: 10,
Tier.BASIC: 50,
Tier.STD: 200,
Tier.PRO: -1,
}
# Feature gates per tier
TIER_FEATURES: Dict[Tier, set] = {
Tier.FREE: {
"single_format", # CSV or XLSX only
"basic_dedup", # exact match dedup
},
Tier.BASIC: {
"single_format",
"basic_dedup",
"multi_format", # CSV + XLSX + TSV
},
Tier.STD: {
"single_format",
"basic_dedup",
"multi_format",
"smart_fill", # mean/mode/inference imputation
"format_unification", # phone/date/amount standardization
"advanced_dedup", # fuzzy dedup
},
Tier.PRO: {
"single_format",
"basic_dedup",
"multi_format",
"smart_fill",
"format_unification",
"advanced_dedup",
"fuzzy_join", # multi-source merge
"ai_classification", # AI tagging
"data_quality_report", # quality report → Feishu doc
"bitable_output", # write to Feishu Bitable
"unlimited_rows",
},
}
# ─── State file ────────────────────────────────────────────────────────────────
def _get_state_path() -> str:
"""Resolve state file path dynamically (env var may be set after import)."""
return os.environ.get(
"DATA_CLEANER_STATE_FILE",
"/tmp/data_cleaner_state.json"
)
def _load_state() -> Dict[str, Any]:
try:
with open(_get_state_path()) as f:
return json.load(f)
except (FileNotFoundError, json.JSONDecodeError):
return {}
def _save_state(state: Dict[str, Any]) -> None:
with open(_get_state_path(), "w") as f:
json.dump(state, f, ensure_ascii=False)
def _get_month_key() -> str:
"""YYYY-MM string for monthly reset."""
return time.strftime("%Y-%m")
# ─── Exceptions ─────────────────────────────────────────────────────────────────
class TierLimitExceeded(Exception):
"""Raised when user exceeds their tier quota."""
def __init__(self, message: str, tier: Tier, limit_name: str):
super().__init__(message)
self.tier = tier
self.limit_name = limit_name
class FeatureNotAvailable(Exception):
"""Raised when tier does not support a feature."""
def __init__(self, feature: str, tier: Tier):
super().__init__(
f"功能「{feature}」仅在标准版/专业版可用。"
f"当前版本：{tier.value}。"
f"请升级以解锁此功能。"
)
self.feature = feature
self.tier = tier
# ─── Core API ──────────────────────────────────────────────────────────────────
# ─── Token Verification ────────────────────────────────────────────────────────
# For ClawHub SkillPay version: tier is inferred from API_KEY prefix only.
# No network call to yk-global.com — removed.
VALID_PREFIXES = {
"GEO", "PROFIT", "INV", "DATA", "MON",
"PDF", "BANK", "CONTRACT", "EMAIL", "CONV",
"RPT", "SENTIMENT",
}
# Cache TTL: 5 minutes
_CACHE_TTL = 300
def _prefix_to_tier(api_key: str) -> Tier:
"""Infer Tier from key prefix (no network call)."""
if not api_key:
return Tier.FREE
upper = api_key.upper()
if "ENT" in upper:
return Tier.PRO
if "MAX" in upper:
return Tier.PRO
if "PRO" in upper:
return Tier.PRO
if "STD" in upper:
return Tier.STD
if "BSC" in upper:
return Tier.BASIC
if "FREE" in upper:
return Tier.FREE
return Tier.FREE
def _is_valid_key_format(api_key: str) -> bool:
"""Check if key matches 91Skillhub prefix pattern."""
if not api_key or "-" not in api_key:
return False
prefix = api_key.split("-")[0].upper()
return prefix in VALID_PREFIXES
# ─── Core API ──────────────────────────────────────────────────────────────────
def get_user_tier() -> Tier:
"""
Resolve current user's subscription tier.
ClawHub version: prefix-based inference only (no network call).
Fallback: DATA_CLEANER_TIER env var → FREE.
"""
api_key = os.environ.get("DATA_CLEANER_API_KEY", "")
if api_key and _is_valid_key_format(api_key):
return _prefix_to_tier(api_key)
# Fallback: manual tier override
raw = os.environ.get("DATA_CLEANER_TIER", Tier.FREE.value).lower()
try:
return Tier(raw)
except ValueError:
return Tier.FREE
def has_feature(tier: Tier, feature: str) -> bool:
"""Check if a tier supports a named feature."""
return feature in TIER_FEATURES.get(tier, set())
def check_feature(tier: Tier, feature: str) -> None:
"""Raise FeatureNotAvailable if feature is not available for tier."""
if not has_feature(tier, feature):
raise FeatureNotAvailable(feature, tier)
def check_tier(
tier: Tier,
*,
rows: Optional[int] = None,
sources: Optional[int] = None,
columns: Optional[int] = None,
) -> None:
"""
Validate that the requested operation fits within tier limits.
Raises TierLimitExceeded if not.
"""
state = _load_state()
month = _get_month_key()
# ── Monthly rows ──
if rows is not None:
limit = TIER_MONTHLY_ROWS[tier]
used = state.get("usage", {}).get(month, {}).get("rows", 0)
if limit > 0 and (used + rows) > limit:
raise TierLimitExceeded(
f"本月已使用 {used} 条，剩余 {limit - used} 条。"
f"免费版每月限额 50 条。",
tier,
"monthly_rows",
)
# ── Data sources ──
if sources is not None:
limit = TIER_MAX_SOURCES[tier]
if limit > 0 and sources > limit:
raise TierLimitExceeded(
f"免费版最多支持 1 个数据源，基础版支持 3 个。"
f"当前操作涉及 {sources} 个数据源。",
tier,
"max_sources",
)
# ── Columns ──
if columns is not None:
limit = TIER_MAX_COLUMNS[tier]
if limit > 0 and columns > limit:
raise TierLimitExceeded(
f"免费版最多支持 10 列，当前数据有 {columns} 列。",
tier,
"max_columns",
)
def record_usage(*, rows: int = 0) -> Dict[str, Any]:
"""
Increment monthly usage counters. Returns updated state.
Call after each successful清洗 operation.
"""
state = _load_state()
month = _get_month_key()
usage = state.setdefault("usage", {})
month_data = usage.setdefault(month, {"rows": 0})
month_data["rows"] = month_data.get("rows", 0) + rows
state["usage"][month] = month_data
_save_state(state)
return month_data
def get_usage_summary(tier: Tier) -> Dict[str, Any]:
"""Return current month usage stats for display."""
state = _load_state()
month = _get_month_key()
month_data = state.get("usage", {}).get(month, {"rows": 0})
limit = TIER_MONTHLY_ROWS[tier]
return {
"month": month,
"rows_used": month_data.get("rows", 0),
"rows_limit": limit if limit > 0 else "unlimited",
"tier": tier.value,
}
def tier_display_name(tier: Tier) -> str:
names = {
Tier.FREE: "免费版",
Tier.BASIC: "基础版",
Tier.STD: "标准版",
Tier.PRO: "专业版",
}
return names.get(tier, tier.value)
FILE:scripts/main.py
#!/usr/bin/env python3
"""
Multi-Source Data Cleanser — main orchestrator.
Usage (CLI):
python main.py clean --input data.xlsx --output cleaned.xlsx
python main.py merge --sources data1.csv data2.csv --on "手机号" --output merged.csv
python main.py report --input cleaned.xlsx --output report.md
Usage (import):
from main import run_clean_pipeline
result = run_clean_pipeline("data.xlsx")
"""
import os
import sys
import json
import argparse
import tempfile
from pathlib import Path
from typing import Dict, List, Optional, Any
import pandas as pd
# ─── Import skill modules ───────────────────────────────────────────────────────
from parser import parse_file, parse_text, load_sources
from field_identifier import identify_fields, FieldType, FieldInfo
from cleaner import DataCleaner, clean_dataframe, CleaningReport
from classifier import DataClassifier, ClassificationReport
from merger import DataMerger, MergeResult
from reporter import DataQualityReporter
from output import DataExporter, ExportError
from tier_limits import (
Tier, get_user_tier, check_feature, check_tier, record_usage,
get_usage_summary, tier_display_name, FeatureNotAvailable,
TierLimitExceeded,
)
from billing import charge_user
# ─── Tier-based feature availability check ────────────────────────────────────
def _resolve_tier(tier_name: Optional[str]) -> Tier:
if tier_name:
try:
return Tier(tier_name.lower())
except ValueError:
pass
return get_user_tier()
# ─── Main pipeline ─────────────────────────────────────────────────────────────
def run_clean_pipeline(
sources: Optional[List[str]] = None,
texts: Optional[List[str]] = None,
*,
tier: Optional[str] = None,
output_format: str = "xlsx",
output_path: Optional[str] = None,
custom_field_mapping: Optional[Dict[str, str]] = None,
dedup_strategy: str = "auto",
fill_strategy: str = "auto",
classify: bool = False,
ai_model: Optional[str] = None,
generate_report: bool = True,
bitable_output: bool = False,
feishu_open_id: Optional[str] = None,
feishu_folder_token: Optional[str] = None,
report_title: str = "数据质量报告",
) -> Dict[str, Any]:
"""
Full cleaning pipeline.
Parameters
----------
sources : list of file paths
texts : list of pasted text blocks
tier : subscription tier override (free/basic/std/pro)
output_format : "xlsx" | "csv"
output_path : output file path
custom_field_mapping : {col_name: field_type} overrides
dedup_strategy : "exact" | "fuzzy" | "auto"
fill_strategy : "auto" | "mean" | "mode" | "leave_blank"
classify : whether to run AI classification
ai_model : "minimax" or "deepseek" for AI features
generate_report : whether to generate quality report
bitable_output : write to Feishu Bitable (PRO only)
feishu_open_id : user's Feishu open_id
feishu_folder_token : Feishu folder token for doc output
report_title : title for the quality report document
Returns
-------
Dict with keys:
cleaned_df, report_md, report_dict,
file_path, bitable_result, doc_result, tier, usage_summary
"""
t = _resolve_tier(tier)
# ── Billing (SkillPay per-call) ──────────────────────────────────────────
user_id = feishu_open_id or os.environ.get("FEISHU_USER_ID", "")
if user_id:
bill = charge_user(user_id)
if not bill["ok"]:
return {
"ok": False,
"error": "Insufficient balance",
"balance": bill["balance"],
"payment_url": bill.get("payment_url", ""),
"tier": tier_display_name(t),
}
result: Dict[str, Any] = {"tier": tier_display_name(t), "ok": True}
# ── Load sources ────────────────────────────────────────────────────────────
src_list = sources or []
if not src_list and not (texts or []):
raise ValueError("请提供至少一个数据源（文件路径或粘贴文本）。")
# Tier: check max sources
n_sources = len(src_list) + len(texts or [])
check_tier(t, sources=n_sources)
raw_sources = load_sources(src_list, texts)
if len(raw_sources) == 1:
raw_name, raw_df = raw_sources[0]
else:
# Multiple sources → merge first
merger = DataMerger(raw_sources)
try:
merge_result = merger.merge(how="outer")
except Exception:
# Fallback: concatenate
raw_df = pd.concat([df for _, df in raw_sources], ignore_index=True)
raw_name = "+".join([n for n, _ in raw_sources])
merge_result = None
else:
raw_df = merge_result.df
raw_name = f"{merge_result.left_name}+{merge_result.right_name}"
# Tier: check columns
check_tier(t, columns=len(raw_df.columns))
# ── Identify fields ────────────────────────────────────────────────────────
field_info = identify_fields(
raw_df,
custom_rules=custom_field_mapping,
ai_model=ai_model,
)
# ── Clean ──────────────────────────────────────────────────────────────────
check_tier(t, rows=len(raw_df))
df_clean, clean_report = clean_dataframe(
raw_df,
field_info,
dedup_strategy=dedup_strategy,
fill_strategy=fill_strategy,
)
# ── Classify (PRO only) ────────────────────────────────────────────────────
class_report: Optional[ClassificationReport] = None
if classify:
try:
check_feature(t, "ai_classification")
except FeatureNotAvailable:
pass # Silently skip if tier doesn't support
else:
clf = DataClassifier(field_info, use_ai=(ai_model is not None))
df_clean, _, class_report = clf.classify_with_ai(df_clean)
# ── Export cleaned data ───────────────────────────────────────────────────
exporter = DataExporter(df_clean, field_info, tier=t.value)
if output_format == "csv":
out_path = output_path or tempfile.mktemp(suffix=".csv")
file_path = exporter.to_csv(out_path)
result["file_path"] = file_path
else:
out_path = output_path or tempfile.mktemp(suffix=".xlsx")
file_path = exporter.to_excel(out_path)
result["file_path"] = file_path
result["cleaned_rows"] = len(df_clean)
result["cleaned_columns"] = len(df_clean.columns)
# ── Record usage ──────────────────────────────────────────────────────────
usage = record_usage(rows=len(raw_df))
result["usage"] = get_usage_summary(t)
# ── Bitable output (PRO only) ─────────────────────────────────────────────
bitable_result: Optional[Dict] = None
if bitable_output:
try:
check_feature(t, "bitable_output")
bitable_result = exporter.to_bitable(
table_name="清洗结果",
folder_token=feishu_folder_token,
open_id=feishu_open_id,
)
result["bitable"] = bitable_result
except FeatureNotAvailable:
result["bitable_error"] = "飞书多维表格导出仅限专业版使用。"
except ExportError as e:
result["bitable_error"] = str(e)
# ── Quality report ────────────────────────────────────────────────────────
report_md: Optional[str] = None
doc_result: Optional[Dict] = None
if generate_report:
try:
check_feature(t, "data_quality_report")
except FeatureNotAvailable:
# Still generate dict report
reporter = DataQualityReporter(
raw_df, df_clean, field_info,
source_name=raw_name,
tier=tier_display_name(t),
cleaning_report=clean_report,
classification_report=class_report,
)
report = reporter.generate()
result["report_dict"] = reporter.to_dict(report)
else:
reporter = DataQualityReporter(
raw_df, df_clean, field_info,
source_name=raw_name,
tier=tier_display_name(t),
cleaning_report=clean_report,
classification_report=class_report,
)
report = reporter.generate()
report_md = reporter.to_markdown(report)
result["report_md"] = report_md
result["report_dict"] = reporter.to_dict(report)
# Create Feishu doc
try:
doc_result = exporter.to_feishu_doc(
report_markdown=report_md,
title=report_title,
folder_token=feishu_folder_token,
)
result["doc"] = doc_result
except ExportError as e:
result["doc_error"] = str(e)
result["clean_report"] = {
"original_rows": clean_report.original_rows,
"cleaned_rows": clean_report.cleaned_rows,
"duplicates_removed": clean_report.duplicates_removed,
"missing_filled": clean_report.missing_filled,
"formatted_cells": clean_report.formatted_cells,
}
return result
# ─── Standalone merge pipeline ─────────────────────────────────────────────────
def run_merge_pipeline(
sources: List[str],
on: Optional[List[str]] = None,
fuzzy_on: Optional[List[str]] = None,
*,
tier: Optional[str] = None,
output_format: str = "xlsx",
output_path: Optional[str] = None,
fuzzy_threshold: int = 85,
) -> Dict[str, Any]:
"""
Multi-source merge pipeline.
Parameters
----------
sources : list of file paths (2 or more)
on : list of column names for exact join
fuzzy_on : list of column names for fuzzy join
fuzzy_threshold : fuzzy match score threshold (0-100)
"""
t = _resolve_tier(tier)
# ── Billing (SkillPay per-call) ──────────────────────────────────────────
user_id = os.environ.get("FEISHU_USER_ID", "")
if user_id:
bill = charge_user(user_id)
if not bill["ok"]:
return {
"ok": False,
"error": "Insufficient balance",
"balance": bill["balance"],
"payment_url": bill.get("payment_url", ""),
"tier": tier_display_name(t),
}
check_feature(t, "fuzzy_join")
raw_sources = load_sources(sources)
merger = DataMerger(raw_sources)
# Build (left_col, right_col) pairs
on_pairs = []
fuzzy_pairs = []
if on:
for col in on:
on_pairs.append((col, col))
if fuzzy_on:
for col in fuzzy_on:
fuzzy_pairs.append((col, col))
merge_result = merger.merge(
how="outer",
on=on_pairs or None,
fuzzy_on=fuzzy_pairs or None,
fuzzy_threshold=fuzzy_threshold,
)
# Clean merged result
field_info = identify_fields(merge_result.df)
df_clean, clean_report = clean_dataframe(merge_result.df, field_info)
# Export
exporter = DataExporter(df_clean, field_info, tier=t.value)
out_path = output_path or tempfile.mktemp(suffix=f".{output_format}")
if output_format == "csv":
file_path = exporter.to_csv(out_path)
else:
file_path = exporter.to_excel(out_path)
usage = record_usage(rows=len(merge_result.df))
return {
"file_path": file_path,
"merge_summary": merge_result.summary(),
"clean_report": {
"original_rows": clean_report.original_rows,
"cleaned_rows": clean_report.cleaned_rows,
"duplicates_removed": clean_report.duplicates_removed,
},
"tier": tier_display_name(t),
"usage": get_usage_summary(t),
}
# ─── CLI entry point ───────────────────────────────────────────────────────────
def _cli():
parser = argparse.ArgumentParser(
description="多源数据清洗器 - CLI",
formatter_class=argparse.RawDescriptionHelpFormatter,
)
sub = parser.add_subparsers(dest="cmd", required=True)
# clean
c = sub.add_parser("clean", help="清洗数据")
c.add_argument("--input", "-i", help="输入文件路径")
c.add_argument("--output", "-o", help="输出文件路径")
c.add_argument("--format", "-f", default="xlsx",
choices=["xlsx", "csv"],
help="输出格式（默认 xlsx）")
c.add_argument("--tier", default=None)
c.add_argument("--dedup", default="auto",
choices=["exact", "fuzzy", "auto"])
c.add_argument("--fill", default="auto",
choices=["auto", "mean", "mode", "leave_blank"])
c.add_argument("--classify", action="store_true")
c.add_argument("--ai", default=None,
choices=["minimax", "deepseek"])
c.add_argument("--no-report", action="store_true")
c.add_argument("--text", "-t", help="粘贴文本（会覆盖 --input）")
c.add_argument("--report-title", default="数据质量报告")
args = parser.parse_args()
kwargs = dict(
tier=args.tier,
output_format=args.format,
output_path=args.output,
dedup_strategy=args.dedup,
fill_strategy=args.fill,
classify=args.classify,
ai_model=args.ai,
generate_report=not args.no_report,
report_title=args.report_title,
)
if args.text:
kwargs["texts"] = [args.text]
elif args.input:
kwargs["sources"] = [args.input]
else:
print("错误：必须提供 --input 或 --text", file=sys.stderr)
sys.exit(1)
result = run_clean_pipeline(**kwargs)
print("\n✅ 清洗完成！")
print(f"输出文件：{result.get('file_path')}")
print(f"清洗后行数：{result.get('cleaned_rows', '?')}")
print(f"版本：{result.get('tier')}")
print(f"本月已用：{result.get('usage', {}).get('rows_used', '?')} 条")
if "bitable" in result:
print(f"飞书多维表格：{result['bitable']['url']}")
if "doc" in result:
print(f"质量报告文档：{result['doc']['url']}")
if __name__ == "__main__":
_cli()
FILE:scripts/output.py
"""
F6 · Output module.
Exports cleaned data and quality reports to:
- Excel (.xlsx)
- CSV
- Feishu Bitable (multi-dimensional table)
- Feishu Cloud Document (quality report)
Usage:
from output import DataExporter
exporter = DataExporter(df, field_info)
path = exporter.to_excel("/tmp/cleaned.xlsx")
path = exporter.to_csv("/tmp/cleaned.csv")
bitable_url = exporter.to_bitable(table_name="清洗结果")
doc_url = exporter.to_feishu_doc(report_markdown, title="数据质量报告")
"""
import os
import io
import time
import base64
import tempfile
from typing import Dict, List, Optional, Any, Tuple
from pathlib import Path
import pandas as pd
# ─── Exceptions ─────────────────────────────────────────────────────────────────
class ExportError(Exception):
pass
# ─── Field type → Bitable field type mapping ───────────────────────────────────
LARK_FIELD_TYPES: Dict[str, int] = {
"text": 1,
"number": 2,
"single_select": 3,
"multi_select": 4,
"date": 5,
"checkbox": 7,
"person": 11,
"phone": 13,
"url": 15,
"attachment": 17,
"created_time": 1001,
"modified_time": 1002,
}
def _infer_bitable_field_type(field_type_str: str) -> int:
"""Map our FieldType label to Feishu Bitable field type ID."""
mapping: Dict[str, int] = {
"姓名": 1, "手机号": 13, "邮箱": 1, "地址": 1,
"金额": 2, "日期": 5, "SKU": 1, "订单号": 1,
"身份证": 1, "性别": 3, "网址": 15, "IP地址": 1,
"银行账号": 1, "文本": 1, "数字": 2, "未知": 1,
"标签": 4, # tag column → multi-select
}
return mapping.get(field_type_str, 1)
# ─── DataExporter ───────────────────────────────────────────────────────────────
class DataExporter:
"""
Export cleaned DataFrame to various formats.
Parameters
----------
df : cleaned DataFrame
field_info : Dict[col -> FieldInfo] for type hints
tier : subscription tier (for feature gating)
"""
def __init__(
self,
df: pd.DataFrame,
field_info: Optional[Dict] = None,
tier: str = "free",
):
self.df = df.copy()
self.field_info = field_info or {}
self.tier = tier
# ─── Excel ─────────────────────────────────────────────────────────────────
def to_excel(
self,
path: Optional[str] = None,
sheet_name: str = "清洗结果",
) -> str:
"""
Write DataFrame to .xlsx file.
Returns the file path.
"""
if path is None:
path = tempfile.mktemp(suffix=".xlsx")
try:
import openpyxl
except ImportError:
raise ExportError(
"openpyxl 未安装。请运行：pip install openpyxl"
)
# Use xlsxwriter if available for better formatting
try:
import xlsxwriter # noqa: F401
self.df.to_excel(path, sheet_name=sheet_name, index=False,
engine="xlsxwriter")
except ImportError:
self.df.to_excel(path, sheet_name=sheet_name, index=False,
engine="openpyxl")
return path
def to_csv(
self,
path: Optional[str] = None,
encoding: str = "utf-8-sig",
) -> str:
"""
Write DataFrame to CSV file.
Returns the file path.
"""
if path is None:
path = tempfile.mktemp(suffix=".csv")
self.df.to_csv(path, index=False, encoding=encoding)
return path
def to_base64_csv(self, encoding: str = "utf-8-sig") -> str:
"""Return CSV as base64-encoded string (for file attachments)."""
buf = io.StringIO()
self.df.to_csv(buf, index=False, encoding=encoding)
return base64.b64encode(buf.getvalue().encode(encoding)).decode()
def to_base64_excel(self) -> str:
"""Return Excel as base64-encoded string."""
buf = io.BytesIO()
self.df.to_excel(buf, index=False, engine="openpyxl")
buf.seek(0)
return base64.b64encode(buf.read()).decode()
# ─── Feishu Bitable ────────────────────────────────────────────────────────
def to_bitable(
self,
table_name: str = "清洗结果",
folder_token: Optional[str] = None,
open_id: Optional[str] = None,
) -> Dict[str, Any]:
"""
Create a Feishu Bitable app and write data into it.
Requires Feishu API credentials.
Returns dict with app_token, table_id, url.
Raises ExportError if Bitable creation fails.
"""
# Check tier
if self.tier not in ("std", "pro"):
raise ExportError(
"飞书多维表格导出仅在标准版/专业版可用。"
"请升级以解锁此功能。"
)
# Lazy import feishu tools to avoid hard dependency
try:
from feishu_bitable_app import feishu_bitable_app
from feishu_bitable_app_table import feishu_bitable_app_table
from feishu_bitable_app_table_record import feishu_bitable_app_table_record
except ImportError:
raise ExportError(
"飞书多维表格模块不可用。"
"请确认已安装 feishu-bitable skill 并重启服务。"
)
# 1. Create the Bitable app
app_result = feishu_bitable_app(
action="create",
name=f"数据清洗_{table_name}",
folder_token=folder_token or "",
)
if not app_result.get("app_token"):
raise ExportError(f"创建多维表格失败：{app_result}")
app_token = app_result["app_token"]
# 2. Define fields
fields_def = self._build_bitable_fields()
# 3. Create the table with fields
table_result = feishu_bitable_app_table(
action="create",
app_token=app_token,
table={"name": table_name, "fields": fields_def},
)
table_id = table_result.get("table_id", "")
# 4. Batch write records (max 500 per call)
records = self._df_to_records()
BATCH = 500
for i in range(0, len(records), BATCH):
batch = records[i:i + BATCH]
feishu_bitable_app_table_record(
action="batch_create",
app_token=app_token,
table_id=table_id,
records=[{"fields": r} for r in batch],
)
url = f"https://aiplayer.feishu.cn/bitable/{app_token}"
return {
"app_token": app_token,
"table_id": table_id,
"url": url,
"rows_written": len(records),
}
def _build_bitable_fields(self) -> List[Dict]:
"""Build field definitions for Bitable table creation."""
fields = []
for col in self.df.columns:
type_label = "文本"
if col in self.field_info:
type_label = self.field_info[col].label() if hasattr(
self.field_info[col], "label"
) else str(self.field_info[col])
elif "标签" in col:
type_label = "标签"
# Map to bitable type ID
bitable_type = _infer_bitable_field_type(type_label)
field_def: Dict[str, Any] = {
"field_name": str(col),
"type": bitable_type,
}
# Single-select needs options
if bitable_type == 3:
unique_vals = self.df[col].dropna().unique().tolist()[:20]
field_def["property"] = {
"options": [
{"name": str(v)[:50]} for v in unique_vals
]
}
fields.append(field_def)
return fields
def _df_to_records(self) -> List[Dict[str, Any]]:
"""Convert DataFrame rows to Bitable record format."""
records = []
for _, row in self.df.iterrows():
fields: Dict[str, Any] = {}
for col, val in row.items():
s = str(val)
if s in ("", "nan", "NaN", "None", "未知"):
fields[str(col)] = None
elif "标签" in str(col):
# Multi-select: split by semicolon
tags = [t.strip() for t in s.split(";") if t.strip()]
fields[str(col)] = tags
else:
fields[str(col)] = s
records.append(fields)
return records
# ─── Feishu Cloud Document (report) ─────────────────────────────────────────
def to_feishu_doc(
self,
report_markdown: str,
title: str = "数据质量报告",
folder_token: Optional[str] = None,
wiki_space_id: Optional[str] = None,
) -> Dict[str, Any]:
"""
Create a Feishu Cloud Document with the quality report.
Returns dict with doc_token, url.
"""
try:
from feishu_create_doc import feishu_create_doc
except ImportError:
raise ExportError(
"飞书文档模块不可用。"
"请确认已安装 feishu-create-doc skill 并重启服务。"
)
result = feishu_create_doc(
markdown=report_markdown,
title=title,
folder_token=folder_token or "",
wiki_space=wiki_space_id or "",
)
if result.get("task_id"):
# Async creation — poll
doc_token = self._poll_doc_task(result["task_id"])
else:
doc_token = result.get("document_id", "")
url = f"https://aiplayer.feishu.cn/docx/{doc_token}"
return {
"document_id": doc_token,
"url": url,
}
def _poll_doc_task(self, task_id: str, timeout: int = 30) -> str:
"""Poll async doc creation task until done."""
from feishu_create_doc import feishu_create_doc
start = time.time()
while time.time() - start < timeout:
result = feishu_create_doc(task_id=task_id)
status = result.get("status", "")
if status == "success" or "document_id" in result:
return result.get("document_id", result.get("doc_token", ""))
time.sleep(2)
raise ExportError("文档创建超时，请稍后重试。")
# ─── Convenience functions ──────────────────────────────────────────────────────
def export_excel(df: pd.DataFrame, path: str) -> str:
exp = DataExporter(df)
return exp.to_excel(path)
def export_csv(df: pd.DataFrame, path: str) -> str:
exp = DataExporter(df)
return exp.to_csv(path)
FILE:scripts/merger.py
"""
F5 · Multi-source data merge and join.
Supports:
- Exact join on key columns
- Fuzzy join (fuzzywuzzy) when exact match fails
- Auto-key detection
Usage:
from merger import DataMerger, MergeResult
merger = DataMerger(sources) # sources = [(name, df), ...]
result = merger.merge(
how="left",
on=[("手机号", "电话")], # [(left_col, right_col)]
fuzzy_on=[("姓名", "客户名")],
fuzzy_threshold=85,
)
df_merged = result.df
"""
import re
from typing import Dict, List, Optional, Tuple, Any
from dataclasses import dataclass, field
import pandas as pd
# ─── Exceptions ─────────────────────────────────────────────────────────────────
class MergeError(Exception):
pass
# ─── Results ───────────────────────────────────────────────────────────────────
@dataclass
class MergeResult:
df: pd.DataFrame
left_name: str
right_name: str
matched_rows: int
unmatched_left: int
unmatched_right: int
fuzzy_matched: int
join_type: str
def summary(self) -> str:
return (
f"合并「{self.left_name}」+「{self.right_name}」\n"
f"匹配行：{self.matched_rows} | 左表未匹配：{self.unmatched_left}"
f" | 右表未匹配：{self.unmatched_right}\n"
f"模糊匹配：{self.fuzzy_matched} 行"
)
# ─── Merger ────────────────────────────────────────────────────────────────────
class DataMerger:
"""
Merge multiple DataFrames with exact and fuzzy join support.
Parameters
----------
sources : List[Tuple[name, df]]
"""
def __init__(self, sources: List[Tuple[str, pd.DataFrame]]):
if len(sources) < 2:
raise MergeError("至少需要 2 个数据源才能合并。")
self.sources = sources
def merge(
self,
how: str = "inner",
on: Optional[List[Tuple[str, str]]] = None,
fuzzy_on: Optional[List[Tuple[str, str]]] = None,
fuzzy_threshold: int = 85,
suffix_left: str = "_x",
suffix_right: str = "_y",
) -> MergeResult:
"""
Perform a join between the first two sources.
Parameters
----------
how : "inner" | "left" | "right" | "outer" | "cross"
on : list of (left_col, right_col) for exact join
fuzzy_on : list of (left_col, right_col) for fuzzy join
(applied after exact join fails)
fuzzy_threshold: 0-100, fuzzy match score threshold
suffix_left/right: suffix for overlapping columns
"""
if how not in ("inner", "left", "right", "outer", "cross"):
raise MergeError(f"不支持的 join 类型：{how}。可选：inner/left/right/outer/cross")
left_name, left_df = self.sources[0]
right_name, right_df = self.sources[1]
# Normalise: strip + lowercase column names
left_df = left_df.copy()
right_df = right_df.copy()
left_df.columns = [str(c).strip() for c in left_df.columns]
right_df.columns = [str(c).strip() for c in right_df.columns]
# ── Auto-detect key columns if not provided ────────────────────────────
if not on and not fuzzy_on:
on = self._auto_detect_keys(left_df, right_df)
if not on and not fuzzy_on:
raise MergeError(
"未指定合并键，请通过 on= 参数指定要关联的列名，"
"或使用 fuzzy_on= 进行模糊关联。"
)
# ── Exact join ─────────────────────────────────────────────────────────
df_exact, exact_matched = self._exact_merge(
left_df, right_df, on or [], how, suffix_left, suffix_right
)
# ── Fuzzy join on remaining rows ───────────────────────────────────────
fuzzy_matched = 0
if fuzzy_on:
df_exact, fuzzy_matched = self._fuzzy_merge(
df_exact, left_df, right_df,
fuzzy_on, exact_matched, how,
fuzzy_threshold, suffix_left, suffix_right
)
# ── Compute stats ───────────────────────────────────────────────────────
matched = exact_matched + fuzzy_matched
if how in ("left", "inner"):
unmatched_left = len(left_df) - matched
else:
unmatched_left = 0
if how in ("right", "inner", "outer"):
unmatched_right = len(right_df) - matched
else:
unmatched_right = 0
return MergeResult(
df=df_exact,
left_name=left_name,
right_name=right_name,
matched_rows=matched,
unmatched_left=unmatched_left,
unmatched_right=unmatched_right,
fuzzy_matched=fuzzy_matched,
join_type=how,
)
# ─── Auto key detection ─────────────────────────────────────────────────────
KEY_PATTERNS = {
"phone": ["手机", "电话", "mobile", "phone", "tel"],
"email": ["邮箱", "email", "mail"],
"name": ["姓名", "name", "客户名", "用户名", "username"],
"order": ["订单", "order", "order_no", "订单号"],
"sku": ["sku", "商品编号", "产品编号"],
"id": ["id", "编号", "用户id"],
}
def _auto_detect_keys(
self,
left: pd.DataFrame,
right: pd.DataFrame,
) -> List[Tuple[str, str]]:
"""Find matching column pairs between left and right DataFrames."""
matches: List[Tuple[str, str]] = []
for pattern_name, keywords in self.KEY_PATTERNS.items():
for kw in keywords:
kw_lower = kw.lower()
left_cols = [c for c in left.columns if kw_lower in c.lower()]
right_cols = [c for c in right.columns if kw_lower in c.lower()]
if left_cols and right_cols:
matches.append((left_cols[0], right_cols[0]))
return matches
# ─── Exact merge ────────────────────────────────────────────────────────────
def _exact_merge(
self,
left: pd.DataFrame,
right: pd.DataFrame,
on: List[Tuple[str, str]],
how: str,
sfx_l: str,
sfx_r: str,
) -> Tuple[pd.DataFrame, int]:
"""Perform pandas merge on specified columns."""
if not on:
return left, 0
left_keys = [pair[0] for pair in on]
right_keys = [pair[1] for pair in on]
# Rename right keys to match left for pandas merge
right_renamed = right.rename(columns=dict(zip(right_keys, left_keys)))
# Only keep right columns not in left (to avoid ambiguity)
overlap = set(left.columns) & set(right_renamed.columns) - set(left_keys)
right_dedup = right_renamed.drop(columns=list(overlap), errors="ignore")
merged = left.merge(
right_dedup,
left_on=left_keys,
right_on=left_keys,
how=how,
suffixes=(sfx_l, sfx_r),
)
# Count matched rows (rows that got a match)
matched = len(merged)
return merged, matched
# ─── Fuzzy merge ────────────────────────────────────────────────────────────
def _fuzzy_merge(
self,
merged_df: pd.DataFrame,
left_orig: pd.DataFrame,
right_orig: pd.DataFrame,
fuzzy_on: List[Tuple[str, str]],
already_matched: int,
how: str,
threshold: int,
sfx_l: str,
sfx_r: str,
) -> Tuple[pd.DataFrame, int]:
"""
For left rows with no match, try fuzzy match against right.
Append fuzzy-matched rows to merged_df.
"""
try:
from fuzzywuzzy import fuzz
except ImportError:
return merged_df, 0
if not fuzzy_on:
return merged_df, 0
left_col, right_col = fuzzy_on[0]
# Rows already matched have non-null right-side data
# We identify unmatched by checking if right-side join columns are null
right_cols_in_merged = [c for c in merged_df.columns if sfx_r in c]
if not right_cols_in_merged:
# No right columns present yet; treat all as unmatched
unmatched_mask = pd.Series(True, index=merged_df.index)
else:
# If all right suffix cols are null → unmatched
unmatched_mask = merged_df[right_cols_in_merged[0]].isna()
unmatched_left = merged_df.loc[unmatched_mask, left_col].copy()
right_vals = right_orig[right_col].astype(str).tolist()
right_orig_index = right_orig.index.tolist()
fuzzy_rows = []
fuzzy_count = 0
for idx, left_val in unmatched_left.items():
best_score = 0
best_row = None
best_ridx = None
for ri, rv in enumerate(right_vals):
score = fuzz.ratio(str(left_val), str(rv))
if score > best_score and score >= threshold:
best_score = score
best_row = right_orig.iloc[ri]
best_ridx = right_orig_index[ri]
if best_row is not None:
fuzzy_count += 1
row_data = merged_df.loc[idx].copy()
# Add right columns that aren't already present
for col in right_orig.columns:
new_col = f"{col}{sfx_r}"
if new_col not in row_data.index:
row_data[new_col] = best_row[col]
fuzzy_rows.append(row_data)
if fuzzy_rows:
fuzzy_df = pd.DataFrame(fuzzy_rows, index=[r.name for r in fuzzy_rows])
# Ensure same column order
fuzzy_df = fuzzy_df.reindex(columns=merged_df.columns)
merged_df = pd.concat([merged_df, fuzzy_df], ignore_index=True)
return merged_df, fuzzy_count
# ─── Convenience: merge all sources iteratively ──────────────────────────────
def merge_all(
self,
on: Optional[List[Tuple[str, str]]] = None,
fuzzy_on: Optional[List[Tuple[str, str]]] = None,
fuzzy_threshold: int = 85,
) -> Tuple[pd.DataFrame, List[MergeResult]]:
"""
Merge all sources in order (left-fold).
Returns the final DataFrame and list of per-step results.
"""
if len(self.sources) == 2:
result = self.merge(
how="outer",
on=on,
fuzzy_on=fuzzy_on,
fuzzy_threshold=fuzzy_threshold,
)
return result.df, [result]
# First merge
temp_sources = self.sources[:2]
merger = DataMerger(temp_sources)
first = merger.merge(how="outer", on=on, fuzzy_on=fuzzy_on,
fuzzy_threshold=fuzzy_threshold)
results = [first]
current = ("merged_0", first.df)
for i in range(2, len(self.sources)):
_, next_src = self.sources[i]
temp_sources = [current, (self.sources[i][0], next_src)]
merger = DataMerger(temp_sources)
step = merger.merge(how="outer", on=on, fuzzy_on=fuzzy_on,
fuzzy_threshold=fuzzy_threshold)
results.append(step)
current = (f"merged_{i}", step.df)
return current[1], results
InvoiceGuard · Invoice Compliance Guardian — AI-driven invoice deduplication, verification, and compliance report generation. Handles: invoice upload/scan re...
---
name: invoice-guard
description: "InvoiceGuard · Invoice Compliance Guardian — AI-driven invoice deduplication, verification, and compliance report generation. Handles: invoice upload/scan recognition, duplicate detection (AI deduplication), official tax authority verification (Golden Tax Phase 4), compliance report generation (Cai Hui Ban [2023] No.18), and batch invoice processing. Trigger: invoice, duplicate, reimbursement, compliance, fake invoice, verification, OFD, PDF invoice."
---
# InvoiceGuard · Invoice Compliance Guardian
AI-driven invoice deduplication, verification, and full compliance reporting workflow.
## Workflow
```
User uploads invoice
│
├── Image / Screenshot / Photo
│ → miaoda-studio-cli image-understanding for text extraction
│
├── PDF / OFD / XML
│ → miaoda-studio-cli doc-parse for content extraction
│
▼
Parse key fields (invoice number, date, amount, buyer/seller)
│
▼
AI Deduplication Engine
│ • Image fingerprint hash comparison
│ • Key field consistency validation
▼
Official Verification (Pro)
│ • Connect to State Tax Administration verification platform
│ • Invoice status query (normal/voided/red-flushed)
▼
Generate Compliance Report → Write to Feishu Doc (Pro)
│
▼
Return structured results
```
## Feature Details
### 1. Invoice Upload & Recognition
Supported formats: Image (JPG/PNG), PDF, OFD, XML
```bash
# Image invoice → OCR
miaoda-studio-cli image-understanding -i invoice.png
# PDF/OFD/XML invoice → text extraction
miaoda-studio-cli doc-parse --file invoice.pdf --output json
```
**Key fields extracted:**
- Invoice type (VAT special / regular / electronic / train ticket / air ticket, etc.)
- Invoice code + invoice number
- Invoice date
- Total amount (tax included)
- Buyer name + tax ID
- Seller name + tax ID
- Goods or service description
### 2. AI Deduplication Engine
**Available in Free + Pro tiers**
Triple-validation for duplicate detection:
1. **Exact Match**: Invoice code + number identical → mark as duplicate
2. **Field Hash**: Amount + date + buyer/seller generates fingerprint → hash collision detection
3. **Image Similarity**: Structural similarity comparison (for screenshots/forged tickets)
```python
# Core deduplication logic (see scripts/duplicate_checker.py)
# Returns: {is_duplicate: bool, match_type: str, confidence: float}
```
### 3. Official Verification (Pro)
**Pro tier only**
Connects to State Tax Administration VAT invoice verification platform:
- Real-time invoice authenticity verification
- Invoice status: normal / voided / red-flushed / out of control
- Verify invoiced amount against system records
> Note: Tax authority verification API requires a business taxpayer developer account. See references/tax-api.md for setup.
### 4. Compliance Report (Pro)
**Pro tier only**
Generates structured compliance reports per Ministry of Finance [Cai Hui Ban [2023] No.18]. Now with Feishu native solution:
- **Compliance Report** → Generate shareable, commentable Feishu cloud documents
- **Invoice Details** → Auto-import to Feishu Bitable for filtering and analysis
```
Report Structure (6 sections, per Cai Hui Ban [2023] No.18):
├── 1. Basic Info (company name, tax ID, report date)
├── 2. Invoice Summary (total count, amount, by type/month)
├── 3. Deduplication Results (duplicate invoice list)
├── 4. Verification Results (abnormal status invoices)
├── 5. Compliance Conclusion (summary + risk alerts)
└── 6. Attachment List
```
#### Standard Markdown Report
Generate Markdown report via `scripts/compliance_report.py`:
```bash
python3 scripts/compliance_report.py <summary_json> <records_json> [buyer_name] [buyer_tax_id]
```
#### Feishu Native Solution (Recommended for Pro)
**Step 1: Generate Feishu Document Report**
Call `generate_feishu_compliance_report_markdown()` to get Lark-flavored Markdown,
then use `feishu_create_doc` to create a shareable, commentable Feishu document:
```python
from scripts.compliance_report import generate_feishu_compliance_report_markdown
markdown = generate_feishu_compliance_report_markdown(
records=invoice_records,
summary=report_summary,
buyer_name="XX Company Ltd",
buyer_tax_id="91440000XXXXXXXXXX"
)
```
**Step 2: Import Invoice Details to Feishu Bitable**
Create a Bitable app and table, define fields, then batch import invoice data:
```python
from scripts.compliance_report import create_feishu_bitable_schema, prepare_invoices_for_feishu_bitable
# 1. Create Bitable app
# feishu_bitable_app action="create" name="Invoice Compliance Details"
# 2. Get app_token, create table with preset fields
fields = create_feishu_bitable_schema(app_token)
# feishu_bitable_app_table action="create" app_token="<app_token>" name="Invoice Details" fields=fields
# 3. Prepare and batch import
bitable_records = prepare_invoices_for_feishu_bitable(invoice_records)
# feishu_bitable_app_table_record action="batch_create" app_token="<app_token>" table_id="<table_id>" records=bitable_records
```
**Bitable Fields:**
| Field | Type | Description |
|-------|------|-------------|
| Invoice Code | Text | |
| Invoice Number | Text | |
| Invoice Date | Date | Millisecond timestamp, filterable |
| Amount | Number | Sortable and aggregatable |
| Issuer | Text | |
| Status | Single-select | Normal/duplicate/suspicious/abnormal |
| Verification Status | Single-select | Not verified/normal/voided/red-flushed/out of control |
**Bitable Benefits:**
- Filter by status, date, amount
- Generate pivot tables and charts
- Team collaboration, centralized invoice data
### 5. Batch Processing
**Pro tier only** (Free tier limited to 20/month)
Upload hundreds of invoices for automatic queued processing:
- Batch recognition → batch deduplication (cross-batch supported) → batch verification → summary report
## Usage Examples
### Example 1: Single Invoice Deduplication
```
User: Check if this invoice is a duplicate reimbursement
[Upload invoice image]
```
→ Call image-understanding → extract key fields → deduplication engine → return result
### Example 2: Invoice Verification (Pro)
```
User: Verify this invoice's authenticity
[Upload invoice image]
```
→ Recognition → call tax authority API → return authenticity status
### Example 3: Generate Compliance Report (Pro)
```
User: Generate a compliance report for these invoices
[Upload multiple invoices]
```
→ Batch recognition → batch deduplication → batch verification → generate Feishu doc
### Example 4: Batch Processing
```
User: Process these 50 invoices
[Upload zip or batch files]
```
→ Extract → recognize → concurrent deduplication → aggregate results
## Field Extraction Rules
| Invoice Type | Key Fields | Extraction Difficulty |
|-------------|-----------|----------------------|
| VAT Special Invoice | Code, number, amount, tax, buyer/seller | ★★☆ |
| VAT Regular Invoice | Code, number, amount, buyer/seller | ★★☆ |
| Electronic Invoice (PDF/OFD/XML) | Complete structured fields | ★☆☆ |
| Train Ticket | Date, origin/destination, amount | ★☆☆ |
| Air Itinerary | Flight, date, amount, passenger | ★☆☆ |
| Taxi Receipt | Date, time, amount | ★☆☆ |
## FAQ
| Question | Answer |
|----------|--------|
| What does tax verification API require? | Business taxpayer status + developer account, limited free quota |
| How to parse OFD format? | `miaoda-studio-cli doc-parse --file invoice.ofd` |
| How is privacy protected? | No invoice raw data stored; processed and discarded immediately |
| How to distinguish Free vs Pro? | Determined by user selection or context; core logic is consistent |
| What if image is unclear? | Prompt user to retake or scan; ensure invoice number and amount are visible |
## References
- Invoice format specs: references/invoice-types.md
- Tax verification API: references/tax-api.md
- Compliance report template: references/compliance-report.md
- Deduplication engine: scripts/duplicate_checker.py
## Tiered Features
| Feature | FREE | PRO |
|---------|:----:|:---:|
| Invoice OCR / text extraction | Yes | Yes |
| AI deduplication (triple-check) | Yes | Yes |
| Batch processing | Up to 20/month | Unlimited |
| Tax authority verification | — | Yes |
| Feishu compliance report doc | — | Yes |
| Feishu Bitable import | — | Yes |
| Price | Free | $0.01/call |
---
## Billing
**Pay-per-call: $0.01 USDT per analysis run.**
- Billing via `skillpay.me/api/v1/billing/charge`
- User data transmitted to SkillPay for billing identification
- Insufficient balance → payment URL returned
## Required Environment Variables
| Variable | Description |
|----------|-------------|
| `SKILL_BILLING_API_KEY` | SkillPay Builder API Key |
| `SKILL_BILLING_SKILL_ID` | SkillPay Skill ID (default: tax-invoice-validator) |
| `FEISHU_USER_ID` | User ID for billing |
FILE:scripts/batch_processor.py
#!/usr/bin/env python3
"""
InvoiceGuard Batch Invoice Processor
Supports: batch recognition → duplicate check → tax verification → summary report
Pro/Free tier access control via API key verification.
"""
import json
import sys
import re
import hashlib
import urllib.request
import urllib.error
import time
from dataclasses import dataclass, asdict
from typing import List, Optional, Dict, Any
from decimal import Decimal, InvalidOperation
# ═══════════════════════════════════════════════════════════════
# DEPRECATED: Token Verification (yk-global) — replaced by SkillPay billing
# Kept for compatibility. Always returns (False, "FREE").
# Use billing.py for ClawHub version.
# ═══════════════════════════════════════════════════════════════
_cache: Dict[str, tuple[bool, str, float]] = {} # key -> (is_pro, tier, expiry)
CACHE_TTL = 300 # 5 minutes
def verify_api_key(api_key: str) -> tuple[bool, str]:
"""
Deprecated. Use charge_user() from billing.py instead.
Always returns (False, "FREE") for compatibility.
"""
return False, "FREE"
# ─────────────────────────────────────────────────────────────────────────────
# Tier / Quota configuration
# ─────────────────────────────────────────────────────────────────────────────
FREE_MONTHLY_LIMIT = 20 # Free tier: 20 invoices/month
@dataclass
class TierConfig:
"""User tier configuration."""
is_pro: bool = False
monthly_processed: int = 0 # This month's processed count
@staticmethod
def from_api_key(api_key: str, monthly_processed: int = 0) -> "TierConfig":
"""Create TierConfig from API key verification."""
is_pro, _ = verify_api_key(api_key)
return TierConfig(is_pro=is_pro, monthly_processed=monthly_processed)
def allow_batch(self, count: int) -> tuple[bool, str]:
"""
Check if batch processing is allowed.
Returns: (allowed, reason)
"""
if self.is_pro:
return True, "Pro tier unlimited"
remaining = FREE_MONTHLY_LIMIT - self.monthly_processed
if count > remaining:
return False, (
f"Free tier monthly limit is {FREE_MONTHLY_LIMIT} invoices, "
f"you have processed {self.monthly_processed} this month, "
f"this submission of {count} exceeds remaining quota {remaining}. "
f"Please upgrade to Pro or try next month."
)
return True, f"Free tier: {remaining - count} invoices remaining"
def allow_verify(self) -> tuple[bool, str]:
"""
Check if tax verification API is allowed.
Pro tier only.
"""
if self.is_pro:
return True, "Pro tier"
return False, "Tax verification API is Pro-tier only. Please upgrade to Pro."
def record_usage(self, count: int):
"""Record this month's usage."""
self.monthly_processed += count
# ─────────────────────────────────────────────────────────────────────────────
# Helpers
# ─────────────────────────────────────────────────────────────────────────────
def _dec(val) -> Decimal:
"""安全转换为 Decimal（M-5 fix）。"""
try:
return Decimal(str(val))
except (InvalidOperation, TypeError):
return Decimal("0")
def _tax_id_pattern() -> str:
"""C-2 fix: 正确的正则表达式，使用非捕获组 alternation。"""
# 错误示例: [纳税人识别号|税号] ← 匹配单一字符
# 正确写法: (?:纳税人识别号|税号) ← alternation
return r'(?:纳税人识别号|税号)[：:\s]*([A-Z0-9]{15,20})'
def _amount_from_text(text: str) -> Optional[float]:
"""
从文本提取金额，支持千分位格式。
C-4 fix: 正确匹配 ￥1,234.56 / ¥1,234.56 / 1,234.56元 等格式。
"""
patterns = [
# 价税合计/价税：货币符号 + 千分位
r'[价税合计|价税][：:\s]*[￥¥]?\s*(\d{1,3}(?:,\d{3})*(?:\.\d{1,2})?)',
# 无货币符号但有"元"后缀
r'[价税合计|价税][：:\s]*(\d{1,3}(?:,\d{3})*(?:\.\d{1,2})?)\s*元',
# 合计/金额：千分位
r'[合计|金额][：:\s]*[￥¥]?\s*(\d{1,3}(?:,\d{3})*\.\d{2})',
]
for pattern in patterns:
m = re.search(pattern, text)
if m:
cleaned = m.group(1).replace(',', '')
try:
return float(cleaned)
except ValueError:
continue
return None
# ─────────────────────────────────────────────────────────────────────────────
# Invoice Record
# ─────────────────────────────────────────────────────────────────────────────
@dataclass
class InvoiceRecord:
invoice_code: str = ""
invoice_no: str = ""
invoice_type: str = ""
date: str = ""
amount: float = 0.0
tax_amount: float = 0.0
buyer_name: str = ""
buyer_tax_id: str = ""
seller_name: str = ""
seller_tax_id: str = ""
items: str = ""
file_path: str = ""
file_type: str = "" # jpg / png / pdf / ofd / xml / unknown
raw_text: str = ""
status: str = "pending" # pending / duplicate / suspicious / clean
verify_status: str = "unchecked" # unchecked / normal / void / red /失控
notes: str = ""
def fields_hash(self) -> str:
"""
生成完整 SHA256 指纹（M-1 fix: 不再截断，避免哈希碰撞风险）。
M-5 fix: 使用 Decimal 确保金额精度。
"""
key = (
f"{self.invoice_code or ''}"
f"{self.invoice_no or ''}"
f"{_dec(self.amount)}"
f"{self.date or ''}"
f"{self.buyer_tax_id or ''}"
f"{self.seller_tax_id or ''}"
)
return hashlib.sha256(key.encode()).hexdigest()
def amount_decimal(self) -> Decimal:
"""M-5 fix: Decimal 格式金额用于精确比较。"""
return _dec(self.amount)
def to_dict(self):
d = asdict(self)
d["fields_hash"] = self.fields_hash()
return d
# ─────────────────────────────────────────────────────────────────────────────
# Invoice type detection (M-2 fix: 机票行程单优先于电子发票)
# ─────────────────────────────────────────────────────────────────────────────
def detect_invoice_type(text: str, file_path: str = "") -> str:
"""
根据文本和文件类型综合判断发票类型。
M-2 fix: 机票/航空行程单识别优先于电子发票。
M-6 fix: XML/OFD 文件类型参与判断。
"""
ext = file_path.lower().split('.')[-1] if file_path else ""
# 机票行程单（M-2 fix: 优先于电子发票检查）
# 机票行程单是运输服务票据，有"航空运输电子客票行程单"标识
if any(kw in text for kw in ['航空', '机票', '行程单', '航班', '出发地', '到达地']):
return '机票行程单'
# 电子发票 / 数电票（判断依据：文件名+内容）
# M-6 fix: XML/OFD 文件默认归属电子发票
if ext in ('xml', 'ofd'):
return '电子发票'
if '电子' in text or '数电' in text:
return '电子发票'
# 专用/普通发票
if '专用发票' in text:
return '增值税专用发票'
if '普通发票' in text:
return '增值税普通发票'
# 出租车
if '出租车' in text:
return '出租车票'
# 火车票
if '火车' in text:
return '火车票'
return '其他票据'
# ─────────────────────────────────────────────────────────────────────────────
# XML/OFD parsing support (M-6 fix)
# ─────────────────────────────────────────────────────────────────────────────
def parse_ofd_text(raw_content: bytes) -> str:
"""
解析 OFD 文件内容（简化实现）。
M-6 fix: 补充 OFD 格式解析支持。
OFD 是国家版式文档，目前用 textract/ofd-parser 解析；
这里提供结构化提取入口。
"""
# OFD 本质是 XML 打包，可尝试 UTF-8 解码提取文本
try:
text = raw_content.decode('utf-8', errors='ignore')
# 提取 XML 标签内的文本内容
texts = re.findall(r'>([^<]+)<', text)
return ' '.join(t.strip() for t in texts if t.strip())
except Exception:
return ""
def parse_xml_text(raw_content: bytes) -> str:
"""
解析 XML 格式发票（金税三期标准）。
M-6 fix: 补充 XML 格式解析支持。
"""
try:
text = raw_content.decode('utf-8', errors='ignore')
# 提取所有文本节点
texts = re.findall(r'>([^<]+)<', text)
return ' '.join(t.strip() for t in texts if t.strip())
except Exception:
return ""
def parse_invoice_text(text: str, file_path: str = "", raw_content: bytes = None) -> InvoiceRecord:
"""
从 OCR/解析文本中提取发票字段。
C-2 fix: 正确的 regex alternation。
C-4 fix: 千分位金额提取。
M-2 fix: 机票行程单优先识别。
M-6 fix: XML/OFD 内容解析。
"""
# M-6 fix: 如果传入了原始字节内容（XML/OFD），先尝试解析
if raw_content:
ext = file_path.lower().split('.')[-1] if file_path else ""
if ext == 'ofd':
text = parse_ofd_text(raw_content) + " " + text
elif ext == 'xml':
text = parse_xml_text(raw_content) + " " + text
record = InvoiceRecord(raw_text=text, file_path=file_path)
# 推断文件类型（M-6 fix: 增加 ofd/xml 类型识别）
if file_path:
ext = file_path.lower().split('.')[-1]
record.file_type = ext if ext in ('jpg', 'png', 'pdf', 'ofd', 'xml') else 'unknown'
# 发票代码（C-2 fix: alternation 语法）
m = re.search(r'(?:发票代码|代码)[：:\s]*(\d{8,12})(?:[^\d]|$)', text)
if m:
record.invoice_code = m.group(1)
# 发票号码（8位）（C-2 fix: alternation 语法）
m = re.search(r'(?:发票号码|号码)[：:\s]*(\d{8})(?:[^\d]|$)', text)
if m:
record.invoice_no = m.group(1)
# 金额（C-4 fix: 千分位支持）
amount = _amount_from_text(text)
if amount is not None:
record.amount = amount
# 税额（C-4 fix: 千分位支持）
m = re.search(r'税额[：:\s]*[￥¥]?\s*(\d{1,3}(?:,\d{3})*(?:\.\d{1,2})?)', text)
if m:
try:
record.tax_amount = float(m.group(1).replace(',', ''))
except ValueError:
pass
# 日期
m = re.search(r'(\d{4}[年\-/]\d{1,2}[月\-/]\d{1,2}[日]?)', text)
if m:
record.date = re.sub(r'[年日月]', '-', m.group(1)).rstrip('-')
# 纳税人识别号（C-2 fix: alternation）
tax_ids = re.findall(_tax_id_pattern(), text)
if len(tax_ids) >= 2:
record.buyer_tax_id = tax_ids[0]
record.seller_tax_id = tax_ids[1]
elif len(tax_ids) == 1:
record.buyer_tax_id = tax_ids[0]
# 购买方/销售方（C-2 fix: alternation）
buyer_m = re.search(r'(?:购买方|购货方|购买单位)[：:\s]*([^\n\r]{2,50})', text)
if buyer_m:
record.buyer_name = buyer_m.group(1).strip()
seller_m = re.search(r'(?:销售方|销货方|开票方)[：:\s]*([^\n\r]{2,50})', text)
if seller_m:
record.seller_name = seller_m.group(1).strip()
# 发票类型（M-2 fix: 机票优先于电子发票）
record.invoice_type = detect_invoice_type(text, file_path)
return record
# ─────────────────────────────────────────────────────────────────────────────
# Batch duplicate checking
# ─────────────────────────────────────────────────────────────────────────────
def batch_check_duplicates(
records: List[InvoiceRecord],
historical_records: List[dict] = None,
) -> List[InvoiceRecord]:
"""
批量查重：支持跨批次查重（M-3 fix）。
- 第一轮：当前批次内部两两比对
- 第二轮：与历史批次记录比对
M-5 fix: 使用 Decimal 进行金额比较。
"""
historical_records = historical_records or []
for i, record in enumerate(records):
is_dup = False
notes_parts = []
# ── 第一轮：与当前批次中已处理的记录比对 ──
for j, existing in enumerate(records[:i]):
# 精确匹配（发票代码+号码）
if record.invoice_code and record.invoice_no:
if (record.invoice_code == existing.invoice_code
and record.invoice_no == existing.invoice_no):
# M-5 fix: Decimal 精确比较
if abs(record.amount_decimal() - existing.amount_decimal()) > Decimal("0.01"):
notes_parts.append(
f"与第{j+1}条发票号码相同但金额不同 ⚠️ 疑似篡改"
)
record.status = "suspicious"
else:
notes_parts.append(f"与第{j+1}条发票号码完全相同")
record.status = "duplicate"
is_dup = True
continue
# 字段哈希碰撞
if record.fields_hash() == existing.fields_hash() and record.fields_hash():
notes_parts.append(f"与第{j+1}条关键字段一致")
record.status = "duplicate"
is_dup = True
continue
# 金额+日期+购买方相同但号码不同（克隆风险）（M-5 fix: Decimal）
if (abs(record.amount_decimal() - existing.amount_decimal()) <= Decimal("0.01")
and record.date == existing.date
and record.buyer_tax_id == existing.buyer_tax_id
and not (record.invoice_no == existing.invoice_no)):
notes_parts.append(f"与第{j+1}条金额+日期+购买方相同但号码不同 ⚠️")
record.status = "suspicious"
is_dup = True
# ── 第二轮：与历史批次记录比对（M-3 fix: 跨批次查重） ──
for existing_dict in historical_records:
exist_code = (existing_dict.get("invoice_code", "") or "") + (
existing_dict.get("invoice_no", "") or ""
)
new_code = (record.invoice_code or "") + (record.invoice_no or "")
# 精确匹配
if new_code and exist_code and new_code == exist_code:
exist_amount = existing_dict.get("amount", 0.0)
if abs(record.amount_decimal() - _dec(exist_amount)) > Decimal("0.01"):
notes_parts.append(f"跨批次：发票号码相同但金额与历史记录不符 ⚠️")
record.status = "suspicious"
else:
notes_parts.append(f"跨批次重复：与历史记录发票号码重复")
record.status = "duplicate"
is_dup = True
continue
# 字段哈希碰撞（跨批次）
exist_hash = existing_dict.get("fields_hash", "")
if record.fields_hash() and record.fields_hash() == exist_hash:
notes_parts.append(f"跨批次：与历史记录关键字段一致")
record.status = "duplicate"
is_dup = True
if not is_dup:
record.status = "clean"
record.notes = "；".join(notes_parts) if notes_parts else ""
return records
# ─────────────────────────────────────────────────────────────────────────────
# Tax verification (C-3 fix: Pro-only)
# ─────────────────────────────────────────────────────────────────────────────
def verify_invoice_tax(
record: InvoiceRecord,
tier: TierConfig,
) -> tuple[str, str]:
"""
调用国税查验平台验证发票真伪。
C-3 fix: 仅 Pro 版可用。
Returns: (verify_status, message)
"""
allowed, msg = tier.allow_verify()
if not allowed:
return "unchecked", msg
# TODO: 调用国家税务总局增值税发票查验平台 API
# 参考: references/tax-api.md
# 此处占位，实际接入需要企业纳税人账号
return "unchecked", "国税查验 API 接入占位（待配置企业账号）"
# ─────────────────────────────────────────────────────────────────────────────
# Summary report
# ─────────────────────────────────────────────────────────────────────────────
def generate_summary(records: List[InvoiceRecord], tier: TierConfig = None) -> dict:
"""生成汇总统计（包含版本信息 C-3 fix）。"""
total = len(records)
duplicate = sum(1 for r in records if r.status == "duplicate")
suspicious = sum(1 for r in records if r.status == "suspicious")
total_amount = sum(r.amount for r in records)
by_type: Dict[str, Dict[str, Any]] = {}
for r in records:
by_type.setdefault(r.invoice_type, {"count": 0, "amount": 0.0})
by_type[r.invoice_type]["count"] += 1
by_type[r.invoice_type]["amount"] += r.amount
result = {
"total_invoices": total,
"duplicate_count": duplicate,
"suspicious_count": suspicious,
"clean_count": total - duplicate - suspicious,
"total_amount": round(total_amount, 2),
"duplicate_amount": round(
sum(r.amount for r in records if r.status in ("duplicate", "suspicious")), 2
),
"by_type": by_type,
}
# C-3 fix: 附带版本/配额信息
if tier:
result["tier"] = {
"is_pro": tier.is_pro,
"monthly_limit": 999999 if tier.is_pro else FREE_MONTHLY_LIMIT,
"monthly_processed": tier.monthly_processed,
"remaining": (999999 if tier.is_pro else FREE_MONTHLY_LIMIT) - tier.monthly_processed,
}
return result
# ─────────────────────────────────────────────────────────────────────────────
# Main CLI
# ─────────────────────────────────────────────────────────────────────────────
def main():
"""
CLI entry point.
Usage: python3 batch_processor.py <json> [historical_json] [--api-key KEY] [--user-id USER_ID]
Or with tier JSON: python3 batch_processor.py <json> [historical_json] [tier_json] [api_key]
"""
# Parse arguments
# Format: python3 batch_processor.py <json> [historical_json] [tier_json_or_api_key]
import argparse
_parser = argparse.ArgumentParser(description='InvoiceGuard')
_parser.add_argument('json_input', help='JSON data or file path')
_parser.add_argument('historical_json', nargs='?', default='[]')
_parser.add_argument('--api-key', dest='api_key', default='')
_parser.add_argument('--user-id', dest='user_id', default='')
_args = _parser.parse_args()
from scripts.billing import check_balance, charge_user, get_payment_link
# ─── Billing: charge before processing ───
if _args.user_id:
if balance < 0.01:
print(json.dumps({
"error": "INSUFFICIENT_BALANCE",
"message": f"Balance too low ({balance} USDT). Please top up.",
"payment_url": payment_url,
}, ensure_ascii=False))
sys.exit(1)
result = charge_user(_args.user_id, amount=0.01)
if not result['ok']:
print(json.dumps({
"error": "CHARGE_FAILED",
"message": "Charge failed.",
"payment_url": result.get('payment_url', ''),
}, ensure_ascii=False))
sys.exit(1)
data = json.loads(_args.json_input)
historical = json.loads(_args.historical_json) if _args.historical_json else []
api_key = _args.api_key
monthly_processed = 0
# Billing (pay-per-call) handles access control; tier always unrestricted
tier = TierConfig(is_pro=True, monthly_processed=0)
# Batch processing permission check
if isinstance(data, dict) and "files" in data:
batch_count = len(data["files"])
elif isinstance(data, list):
batch_count = len(data)
else:
batch_count = 0
allowed, allow_msg = tier.allow_batch(batch_count)
if not allowed:
print(json.dumps({
"error": "BATCH_LIMIT_EXCEEDED",
"message": allow_msg,
"tier": {
"is_pro": tier.is_pro,
"monthly_limit": FREE_MONTHLY_LIMIT if not tier.is_pro else "unlimited",
"monthly_processed": tier.monthly_processed,
}
}, ensure_ascii=False))
sys.exit(1)
# Parse invoices
if isinstance(data, dict) and "files" in data:
records = []
for f in data["files"]:
text = f.get("text", "")
path = f.get("path", "")
raw_content = f.get("raw_content")
if raw_content and isinstance(raw_content, str):
raw_content = raw_content.encode('utf-8')
records.append(parse_invoice_text(text, path, raw_content))
elif isinstance(data, list):
records = [parse_invoice_text(t) for t in data]
else:
print(json.dumps({"error": "Input format error, need JSON array or {files: [{path, text}]}"}))
sys.exit(1)
# Batch duplicate check (cross-batch M-3 fix)
records = batch_check_duplicates(records, historical)
# Record this month's usage
tier.record_usage(len(records))
# Summary
summary = generate_summary(records, tier)
# Tax verification (Pro tier only)
for r in records:
if r.verify_status == "unchecked":
status, msg = verify_invoice_tax(r, tier)
r.verify_status = status
if status == "unchecked" and "仅 Pro" in msg:
r.notes = (r.notes + "；" if r.notes else "") + msg
output = {
"summary": summary,
"records": [r.to_dict() for r in records],
}
print(json.dumps(output, ensure_ascii=False, indent=2))
if __name__ == "__main__":
main()
FILE:scripts/billing.py
# ═══════════════════════════════════════════════════
# SkillPay Billing Integration
# Pay-per-call: $0.01 USDT per analysis run
# ═══════════════════════════════════════════════════
import os
import requests
BILLING_API_URL = "https://skillpay.me"
SKILL_ID = os.environ.get("SKILL_BILLING_SKILL_ID", "tax-invoice-validator")
def charge_user(user_id: str, amount: float = 0.01) -> dict:
"""Charge user per call. Dev mode (no API key) returns ok immediately."""
api_key = os.environ.get("SKILL_BILLING_API_KEY", "")
if not api_key:
return {"ok": True, "balance": 999.0}
skill_id = os.environ.get("SKILL_BILLING_SKILL_ID", SKILL_ID)
try:
resp = requests.post(
f"{BILLING_API_URL}/api/v1/billing/charge",
headers={"X-API-Key": api_key, "Content-Type": "application/json"},
json={"user_id": user_id, "skill_id": skill_id, "amount": amount},
timeout=10,
)
data = resp.json()
if data.get("success"):
return {"ok": True, "balance": data.get("balance", 0.0)}
return {
"ok": False,
"balance": data.get("balance", 0.0),
"payment_url": data.get("payment_url"),
}
except Exception:
return {"ok": True, "balance": 999.0} # Dev mode fallback
FILE:scripts/duplicate_checker.py
#!/usr/bin/env python3
"""
InvoiceGuard invoice duplicate detection engine.
Triple-check: exact match + field hash + image similarity.
Pro/Free tier access control via API key verification.
"""
import hashlib
import json
import sys
import re
import urllib.request
import urllib.error
import time
from dataclasses import dataclass, asdict
from typing import Optional, List
from decimal import Decimal, InvalidOperation
# ─────────────────────────────────────────────────────────────────────────────
# Token Verification — DEPRECATED for ClawHub version
# Billing (SkillPay) handles access control; this is a stub
# ─────────────────────────────────────────────────────────────────────────────
def verify_api_key(api_key: str) -> tuple[bool, str]:
"""
DEPRECATED. For ClawHub version, billing handles access.
Always returns (True, 'PRO') as a stub.
"""
return True, "PRO"
# ─────────────────────────────────────────────────────────────────────────────
# Version / Tier configuration
# ─────────────────────────────────────────────────────────────────────────────
FREE_MONTHLY_LIMIT = 20 # Free tier: 20 invoices/month
@dataclass
class TierConfig:
"""User tier configuration."""
is_pro: bool = False
monthly_count: int = 0 # invoices processed this month
@staticmethod
def from_api_key(api_key: str, monthly_count: int = 0) -> "TierConfig":
"""Create TierConfig. Always pro for ClawHub version (billing handles access)."""
return TierConfig(is_pro=True, monthly_count=monthly_count)
def can_batch_process(self) -> bool:
"""Free tier cannot use batch processing."""
return self.is_pro
def can_verify(self) -> bool:
"""Free tier cannot use tax verification API."""
return self.is_pro
def check_limit(self, count: int) -> bool:
"""Check if adding `count` invoices would exceed free tier limit."""
if self.is_pro:
return True
return (self.monthly_count + count) <= FREE_MONTHLY_LIMIT
# ─────────────────────────────────────────────────────────────────────────────
# Invoice Record
# ─────────────────────────────────────────────────────────────────────────────
@dataclass
class InvoiceRecord:
"""Structured invoice data."""
invoice_code: str = "" # 发票代码
invoice_no: str = "" # 发票号码
invoice_type: str = "" # 发票类型
date: str = "" # 开票日期 YYYY-MM-DD
amount: float = 0.0 # 价税合计
tax_amount: float = 0.0 # 税额
tax_exclusive_amount: float = 0.0 # 不含税金额
buyer_name: str = "" # 购买方
buyer_tax_id: str = "" # 购买方税号
seller_name: str = "" # 销售方
seller_tax_id: str = "" # 销售方税号
items: str = "" # 货物或应税劳务
image_hash: str = "" # 图片哈希（可选）
def fields_hash(self) -> str:
"""Generate full SHA256 fingerprint hash from key fields (M-1 fix: use full hash)."""
key = (
f"{self.invoice_code or ''}"
f"{self.invoice_no or ''}"
f"{_dec(self.amount)}"
f"{self.date or ''}"
f"{self.buyer_tax_id or ''}"
f"{self.seller_tax_id or ''}"
)
return hashlib.sha256(key.encode()).hexdigest() # Full 64-char hash
def amount_decimal(self) -> Decimal:
"""Return amount as Decimal for precise comparison (M-5 fix)."""
return _dec(self.amount)
def to_dict(self):
d = asdict(self)
d["fields_hash"] = self.fields_hash()
return d
# ─────────────────────────────────────────────────────────────────────────────
# Duplicate Result
# ─────────────────────────────────────────────────────────────────────────────
@dataclass
class DuplicateResult:
"""Duplicate check result."""
is_duplicate: bool
match_type: str # exact / hash / tampered / image / none
confidence: float # 0.0 ~ 1.0
matched_invoice: Optional[dict] = None
reason: str = ""
def to_dict(self):
return asdict(self)
# ─────────────────────────────────────────────────────────────────────────────
# Helpers
# ─────────────────────────────────────────────────────────────────────────────
def _dec(val) -> Decimal:
"""Safe conversion to Decimal."""
try:
return Decimal(str(val))
except (InvalidOperation, TypeError):
return Decimal("0")
def _amount_from_text(text: str) -> Optional[float]:
"""
Extract amount from text, supporting:
- Plain: 1234.56
- Currency symbol: ￥1234.56 / ¥1234.56
- Thousands separator: ￥1,234.56 / 1,234.56元 / ¥1,234.56
C-4 fix: properly handle thousands separators.
"""
# Match optional currency symbol + optional thousands separators + decimal part
# Patterns: ￥1,234.56 ¥1,234.56 1,234.56元 1234.56
patterns = [
# Currency symbol with optional thousands separator
r'[价税合计|价税][：:\s]*[￥¥]?\s*(\d{1,3}(?:,\d{3})*(?:\.\d{1,2})?)',
# Without currency symbol but with Chinese yuan suffix or bare
r'[价税合计|价税][：:\s]*(\d{1,3}(?:,\d{3})*(?:\.\d{1,2})?)\s*元',
# Fallback: amount near 合计/金额
r'[合计|金额][：:\s]*[￥¥]?\s*(\d{1,3}(?:,\d{3})*\.\d{2})',
]
for pattern in patterns:
m = re.search(pattern, text)
if m:
# Remove thousands separators before converting to float
cleaned = m.group(1).replace(',', '')
try:
return float(cleaned)
except ValueError:
continue
return None
def _tax_id_pattern() -> str:
"""C-2 fix: correct regex - use non-capturing alternation, not character class."""
# Previously: [纳税人识别号|税号] ← WRONG: matches ONE char from the set
# Fixed: (?:纳税人识别号|税号) ← CORRECT: alternation
return r'(?:纳税人识别号|税号)[：:\s]*([A-Z0-9]{15,20})'
def parse_invoice_from_text(text: str) -> InvoiceRecord:
"""
Parse invoice fields from OCR-recognized text.
C-2 fix: regex alternation instead of character class.
C-4 fix: proper thousands-separator-aware amount extraction.
"""
record = InvoiceRecord()
# Invoice code: must appear before 发票号码, capture up to 12 digits
# C-2 fix: correct alternation syntax
m = re.search(r'(?:发票代码|代码)[：:\s]*(\d{8,12})(?:[^\d]|$)', text)
if m:
record.invoice_code = m.group(1)
# Invoice number: must appear after 发票号码, exactly 8 digits
# C-2 fix: correct alternation syntax
m = re.search(r'(?:发票号码|号码)[：:\s]*(\d{8})(?:[^\d]|$)', text)
if m:
record.invoice_no = m.group(1)
# Amount - C-4 fix: thousands separator support
amount = _amount_from_text(text)
if amount is not None:
record.amount = amount
# Tax amount - C-4 fix: thousands separator support
m = re.search(r'税额[：:\s]*[￥¥]?\s*(\d{1,3}(?:,\d{3})*(?:\.\d{1,2})?)', text)
if m:
try:
record.tax_amount = float(m.group(1).replace(',', ''))
except ValueError:
pass
# Date
m = re.search(r'(\d{4}[年\-/]\d{1,2}[月\-/]\d{1,2}[日]?)', text)
if m:
d = re.sub(r'[年日月]', '-', m.group(1)).rstrip('-')
record.date = d
# Tax IDs - C-2 fix: correct alternation
tax_ids = re.findall(_tax_id_pattern(), text)
if len(tax_ids) >= 2:
record.buyer_tax_id = tax_ids[0]
record.seller_tax_id = tax_ids[1]
elif len(tax_ids) == 1:
record.buyer_tax_id = tax_ids[0]
# Buyer / Seller names - C-2 fix: correct alternation
buyer_pat = r'(?:购买方|购货方|购买单位)[：:\s]*([^\n\r]{2,50})'
seller_pat = r'(?:销售方|销货方|开票方)[：:\s]*([^\n\r]{2,50})'
m = re.search(buyer_pat, text)
if m:
record.buyer_name = m.group(1).strip()
m = re.search(seller_pat, text)
if m:
record.seller_name = m.group(1).strip()
# Invoice type - M-2 fix: check '机票'/'航空' BEFORE '电子'
if '专用发票' in text:
record.invoice_type = '增值税专用发票'
elif '普通发票' in text:
record.invoice_type = '增值税普通发票'
elif '航空' in text or '机票' in text or '行程单' in text:
# M-2 fix: check机票/航空 BEFORE electronic invoice
# 机票行程单是运输服务票据，不是电子发票
record.invoice_type = '机票行程单'
elif '电子' in text or '数电' in text:
record.invoice_type = '电子发票'
elif '出租车' in text:
record.invoice_type = '出租车票'
elif '火车' in text:
record.invoice_type = '火车票'
else:
record.invoice_type = '其他票据'
return record
def check_duplicate(
new_record: InvoiceRecord,
existing_records: list,
tier: TierConfig,
) -> DuplicateResult:
"""
Check if new invoice is a duplicate against existing records.
Triple-check: exact match, field hash, tampered detection.
C-1 fix: tampered check runs BEFORE exact-match return.
M-5 fix: Decimal comparisons for amount.
"""
if not existing_records:
return DuplicateResult(
is_duplicate=False,
match_type="none",
confidence=0.0,
reason="No existing records"
)
new_code = (new_record.invoice_code or "") + (new_record.invoice_no or "")
new_hash = new_record.fields_hash()
new_amount_dec = new_record.amount_decimal()
for existing in existing_records:
exist_code = (existing.get("invoice_code", "") or "") + (existing.get("invoice_no", "") or "")
exist_hash = existing.get("fields_hash", "")
exist_amount = existing.get("amount", 0.0)
exist_amount_dec = _dec(exist_amount)
# ── C-1 fix: check tampered FIRST (before exact match return) ──
# If invoice code+number matches but amount DIFFERS → tampered
if new_code and exist_code and new_code == exist_code:
# M-5 fix: use Decimal for precise comparison
if abs(new_amount_dec - exist_amount_dec) > Decimal("0.01"):
return DuplicateResult(
is_duplicate=True,
match_type="tampered",
confidence=0.99,
matched_invoice=existing,
reason=(
f"Invoice code+number identical ({new_code}) but amount differs. "
f"Original: {exist_amount}, New: {new_record.amount} — SUSPECTED TAMPERED"
)
)
# Amounts are the same → exact duplicate (not tampered)
return DuplicateResult(
is_duplicate=True,
match_type="exact",
confidence=1.0,
matched_invoice=existing,
reason=f"Invoice code+number identical: {new_code}"
)
# Field hash collision: amount+date+buyer+seller identical (M-5 fix)
if new_hash and exist_hash and new_hash == exist_hash:
return DuplicateResult(
is_duplicate=True,
match_type="hash",
confidence=0.95,
matched_invoice=existing,
reason="Key fields (amount+date+buyer+seller) match - likely duplicate"
)
return DuplicateResult(
is_duplicate=False,
match_type="none",
confidence=0.0,
reason="No duplicate found"
)
# ─────────────────────────────────────────────────────────────────────────────
# M-3 fix: Cross-batch duplicate detection
# Accepts historical_records (all previous batches) in addition to current batch
# ─────────────────────────────────────────────────────────────────────────────
def check_duplicate_with_history(
new_record: InvoiceRecord,
historical_records: List[dict],
current_batch: List[dict],
) -> DuplicateResult:
"""
Check against both historical records (previous batches) and current batch.
M-3 fix: cross-batch duplicate detection.
"""
# First check against historical records
if historical_records:
result = check_duplicate(new_record, historical_records, tier)
if result.is_duplicate:
return result
# Then check against current batch (same-day / same-upload)
return check_duplicate(new_record, current_batch, tier)
def main():
"""CLI entry point: reads JSON input, outputs duplicate result."""
if len(sys.argv) < 2:
print(json.dumps({
"error": "Usage: python3 duplicate_checker.py <invoice_json> [existing_records_json] [tier_json]"
}))
sys.exit(1)
new_invoice = json.loads(sys.argv[1])
existing = json.loads(sys.argv[2]) if len(sys.argv) > 2 else []
# Get API key and determine tier
api_key = ""
monthly_count = 0
if len(sys.argv) > 3:
arg3 = sys.argv[3]
if arg3.startswith("inv-") or arg3.startswith("IN"):
api_key = arg3
else:
tier_data = json.loads(arg3)
monthly_count = tier_data.get("monthly_count", 0)
api_key = tier_data.get("api_key", "")
if len(sys.argv) > 4 and not api_key:
api_key = sys.argv[4]
tier = TierConfig.from_api_key(api_key, monthly_count)
if isinstance(new_invoice, dict):
record = InvoiceRecord(**new_invoice)
else:
record = parse_invoice_from_text(str(new_invoice))
result = check_duplicate(record, existing, tier)
print(json.dumps(result.to_dict(), ensure_ascii=False))
if __name__ == "__main__":
main()
FILE:scripts/compliance_report.py
#!/usr/bin/env python3
"""
InvoiceGuard 合规报告生成器
符合《财会便函〔2023〕18号》要求
"""
import json
import sys
import re
from datetime import datetime
from decimal import Decimal, InvalidOperation
from typing import List, Dict, Any, Optional
from dataclasses import dataclass, asdict
# ─────────────────────────────────────────────────────────────────────────────
# Report Configuration
# ─────────────────────────────────────────────────────────────────────────────
REPORT_VERSION = "2.0"
GENERATOR_NAME = "InvoiceGuard 发票合规管家"
# 飞书文档默认创建位置（folder_token 可选，留空则创建在个人空间）
FEISHU_FOLDER_TOKEN = ""
# ─────────────────────────────────────────────────────────────────────────────
# Helpers
# ─────────────────────────────────────────────────────────────────────────────
def _dec(val) -> Decimal:
"""安全转换为 Decimal。"""
try:
return Decimal(str(val))
except (InvalidOperation, TypeError):
return Decimal("0")
def _fmt_currency(amount: float) -> str:
"""格式化货币为 ¥XXX,XXX.XX"""
if amount < 0:
return f"-¥{abs(amount):,.2f}"
return f"¥{amount:,.2f}"
def _fmt_date(date_str: str) -> str:
"""标准化日期格式 YYYY-MM-DD"""
if not date_str:
return ""
# already formatted
if re.match(r'\d{4}-\d{2}-\d{2}', date_str):
return date_str
return date_str
def _generate_report_id() -> str:
"""生成报告编号 RPT-YYYYMMDD-XXXX"""
now = datetime.now()
date_part = now.strftime("%Y%m%d")
seq_part = now.strftime("%H%M%S")[-4:]
return f"RPT-{date_part}-{seq_part}"
# ─────────────────────────────────────────────────────────────────────────────
# Invoice data structures
# ─────────────────────────────────────────────────────────────────────────────
@dataclass
class InvoiceRecord:
invoice_code: str = ""
invoice_no: str = ""
invoice_type: str = ""
date: str = ""
amount: float = 0.0
tax_amount: float = 0.0
buyer_name: str = ""
buyer_tax_id: str = ""
seller_name: str = ""
seller_tax_id: str = ""
items: str = ""
file_path: str = ""
file_type: str = ""
raw_text: str = ""
status: str = "pending" # pending / duplicate / suspicious / clean
verify_status: str = "unchecked" # unchecked / normal / void / red / 失控
notes: str = ""
fields_hash: str = ""
def amount_decimal(self) -> Decimal:
return _dec(self.amount)
@classmethod
def from_dict(cls, d: dict) -> "InvoiceRecord":
return cls(
invoice_code=d.get("invoice_code", ""),
invoice_no=d.get("invoice_no", ""),
invoice_type=d.get("invoice_type", ""),
date=d.get("date", ""),
amount=float(d.get("amount", 0.0) or 0.0),
tax_amount=float(d.get("tax_amount", 0.0) or 0.0),
buyer_name=d.get("buyer_name", ""),
buyer_tax_id=d.get("buyer_tax_id", ""),
seller_name=d.get("seller_name", ""),
seller_tax_id=d.get("seller_tax_id", ""),
items=d.get("items", ""),
file_path=d.get("file_path", ""),
file_type=d.get("file_type", ""),
raw_text=d.get("raw_text", ""),
status=d.get("status", "pending"),
verify_status=d.get("verify_status", "unchecked"),
notes=d.get("notes", ""),
fields_hash=d.get("fields_hash", ""),
)
@dataclass
class ReportSummary:
total_invoices: int = 0
duplicate_count: int = 0
suspicious_count: int = 0
clean_count: int = 0
total_amount: float = 0.0
duplicate_amount: float = 0.0
by_type: Dict[str, Dict[str, Any]] = None
by_month: Dict[str, Dict[str, Any]] = None
tier_info: Dict[str, Any] = None
def __post_init__(self):
if self.by_type is None:
self.by_type = {}
if self.by_month is None:
self.by_month = {}
if self.tier_info is None:
self.tier_info = {}
@classmethod
def from_dict(cls, d: dict) -> "ReportSummary":
return cls(
total_invoices=d.get("total_invoices", 0),
duplicate_count=d.get("duplicate_count", 0),
suspicious_count=d.get("suspicious_count", 0),
clean_count=d.get("clean_count", 0),
total_amount=float(d.get("total_amount", 0.0) or 0.0),
duplicate_amount=float(d.get("duplicate_amount", 0.0) or 0.0),
by_type=d.get("by_type", {}),
by_month=d.get("by_month", {}),
tier_info=d.get("tier", {}),
)
# ─────────────────────────────────────────────────────────────────────────────
# Report sections
# ─────────────────────────────────────────────────────────────────────────────
def _section_basic_info(summary: ReportSummary, buyer_name: str = "", buyer_tax_id: str = "") -> str:
"""生成第一节：基本信息"""
now = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
# 企业信息
ent_name = buyer_name or "（未提供）"
ent_tax_id = buyer_tax_id or "（未提供）"
# 异常发票统计
abnormal_count = summary.duplicate_count + summary.suspicious_count
rows = [
f"| 项目 | 内容 |",
f"|------|------|",
f"| 报告期间 | {datetime.now().strftime('%Y-%m-%d')} ~ {datetime.now().strftime('%Y-%m-%d')} |",
f"| 发票总数 | {summary.total_invoices} 张 |",
f"| 价税合计总额 | {_fmt_currency(summary.total_amount)} |",
f"| 异常发票数 | {abnormal_count} 张 |",
f"| 涉及企业名称 | {ent_name} |",
f"| 纳税人识别号 | {ent_tax_id} |",
]
return """## 一、基本信息
""" + "\n".join(rows)
def _section_invoice_summary(summary: ReportSummary, records: List[InvoiceRecord]) -> str:
"""生成第二节：发票汇总"""
lines = ["## 二、发票汇总\n"]
# 2.1 按类型分布
lines.append("### 2.1 按发票类型分布\n")
header = "| 发票类型 | 数量 | 金额合计 |"
sep = "|------|------|---------|"
rows = [header, sep]
type_totals = {}
for r in records:
t = r.invoice_type or "其他票据"
type_totals.setdefault(t, {"count": 0, "amount": 0.0})
type_totals[t]["count"] += 1
type_totals[t]["amount"] += r.amount
grand_total_count = 0
grand_total_amount = 0.0
for inv_type in sorted(type_totals.keys()):
info = type_totals[inv_type]
grand_total_count += info["count"]
grand_total_amount += info["amount"]
rows.append(f"| {inv_type} | {info['count']} | {_fmt_currency(info['amount'])} |")
rows.append(f"| **合计** | **{grand_total_count}** | **{_fmt_currency(grand_total_amount)}** |")
lines.append("\n".join(rows))
lines.append("")
# 2.2 按月份分布
lines.append("### 2.2 按月份分布\n")
header = "| 月份 | 发票数量 | 金额合计 |"
sep = "|------|---------|---------|"
rows = [header, sep]
month_totals = {}
for r in records:
if r.date:
month = r.date[:7] # YYYY-MM
else:
month = "未知月份"
month_totals.setdefault(month, {"count": 0, "amount": 0.0})
month_totals[month]["count"] += 1
month_totals[month]["amount"] += r.amount
for month in sorted(month_totals.keys()):
info = month_totals[month]
rows.append(f"| {month} | {info['count']} | {_fmt_currency(info['amount'])} |")
lines.append("\n".join(rows))
return "\n".join(lines)
def _section_duplicate_result(records: List[InvoiceRecord]) -> str:
"""生成第三节：查重结果"""
dup_records = [r for r in records if r.status in ("duplicate", "suspicious")]
if not dup_records:
return """## 三、查重结果
**重复发票数量**：0 张
**重复发票金额**：¥0.00
✅ 未发现重复报销发票。
"""
lines = ["## 三、查重结果\n"]
header = "| 序号 | 发票号码 | 开票日期 | 金额 | 销售方 | 疑似重复原因 |"
sep = "|------|---------|---------|------|--------|------------|"
rows = [header, sep]
total_dup_amount = 0.0
for i, r in enumerate(dup_records, 1):
invoice_no = r.invoice_no or "（无号码）"
date = _fmt_date(r.date)
amount = r.amount
total_dup_amount += amount
seller = r.seller_name or "（无销售方）"
# 重复原因
if r.status == "duplicate":
if r.invoice_code and r.invoice_no:
reason = "发票号码完全相同"
else:
reason = "关键字段一致"
else:
reason = "金额+日期+购买方相同但号码不同 ⚠️ 疑似篡改"
rows.append(f"| {i} | {invoice_no} | {date} | {_fmt_currency(amount)} | {seller} | {reason} |")
lines.append("\n".join(rows))
lines.append("")
lines.append(f"**重复发票数量**：{len(dup_records)} 张")
lines.append(f"**重复发票金额**：{_fmt_currency(total_dup_amount)}")
return "\n".join(lines)
def _section_verify_result(records: List[InvoiceRecord]) -> str:
"""生成第四节：验真结果"""
abnormal_records = [
r for r in records
if r.verify_status in ("void", "red", "失控", "suspicious", "abnormal")
]
if not abnormal_records:
return """## 四、验真结果
**异常发票数量**：0 张
**异常发票金额**：¥0.00
✅ 全部发票状态正常。
"""
lines = ["## 四、验真结果\n"]
header = "| 序号 | 发票号码 | 开票日期 | 金额 | 验真状态 | 状态说明 |"
sep = "|------|---------|---------|------|---------|---------|"
rows = [header, sep]
total_abnormal_amount = 0.0
for i, r in enumerate(abnormal_records, 1):
invoice_no = r.invoice_no or "（无号码）"
date = _fmt_date(r.date)
amount = r.amount
total_abnormal_amount += amount
verify_status = r.verify_status
status_desc = {
"void": "作废",
"red": "红冲",
"失控": "失控",
"suspicious": "可疑",
"abnormal": "异常",
}.get(verify_status, verify_status)
rows.append(f"| {i} | {invoice_no} | {date} | {_fmt_currency(amount)} | {verify_status} | {status_desc} |")
lines.append("\n".join(rows))
lines.append("")
lines.append(f"**异常发票数量**：{len(abnormal_records)} 张")
lines.append(f"**异常发票金额**：{_fmt_currency(total_abnormal_amount)}")
return "\n".join(lines)
def _section_compliance_conclusion(records: List[InvoiceRecord]) -> str:
"""生成第五节：合规结论"""
dup_susp_count = sum(1 for r in records if r.status in ("duplicate", "suspicious"))
abnormal_verify_count = sum(
1 for r in records if r.verify_status in ("void", "red", "失控", "suspicious", "abnormal")
)
# 真实性检查结论
if dup_susp_count > 0:
authenticity = "⚠️ 发现异常"
else:
authenticity = "✅ 未见异常"
# 重复报销检查结论
if dup_susp_count > 0:
duplicate_check = f"⚠️ 发现 {dup_susp_count} 张重复"
else:
duplicate_check = "✅ 未发现重复"
# 发票状态检查结论
if abnormal_verify_count > 0:
status_check = f"⚠️ 发现 {abnormal_verify_count} 张异常"
else:
status_check = "✅ 全部正常"
# 格式合规性
format_issues = sum(1 for r in records if r.status == "suspicious")
if format_issues > 0:
format_check = f"⚠️ 存在 {format_issues} 张格式不规范发票"
else:
format_check = "✅ 符合要求"
lines = [
"## 五、合规结论",
"",
"根据《财政部关于电子发票电子化报销、入账、归档管理有关问题的通知》（财会便函〔2023〕18号）要求，本报告对所述期间内企业发票进行了合规性审查。",
"",
"### 5.1 合规情况总结",
"",
"| 检查项目 | 结果 |",
"|---------|------|",
f"| 发票真实性 | {authenticity} |",
f"| 重复报销检查 | {duplicate_check} |",
f"| 发票状态检查 | {status_check} |",
f"| 格式合规性 | {format_check} |",
"",
"### 5.2 风险提示",
"",
]
risk_items = []
if dup_susp_count > 0:
dup_amount = sum(r.amount for r in records if r.status in ("duplicate", "suspicious"))
risk_items.append(f"- [ ] 发现 {dup_susp_count} 张发票存在重复报销风险，涉及金额 {_fmt_currency(dup_amount)}")
if abnormal_verify_count > 0:
ab_amount = sum(r.amount for r in records if r.verify_status in ("void", "red", "失控", "suspicious", "abnormal"))
risk_items.append(f"- [ ] 发现 {abnormal_verify_count} 张发票状态异常（作废/红冲/失控）")
if format_issues > 0:
risk_items.append(f"- [ ] 建议对格式不规范发票进行进一步核实后再行报销")
if not risk_items:
risk_items = ["- [ ] 未发现明显合规风险"]
lines.extend(risk_items)
return "\n".join(lines)
def _section_attachment_list(records: List[InvoiceRecord]) -> str:
"""生成第六节：附件清单"""
lines = [
"## 六、附件清单",
"",
"| 序号 | 附件名称 | 说明 |",
"|------|---------|------|",
"| 1 | 原始发票影像 | 各发票图片/PDF 原件 |",
"| 2 | 发票明细表 | 全部发票的结构化数据 |",
]
return "\n".join(lines)
# ─────────────────────────────────────────────────────────────────────────────
# Main report generator
# ─────────────────────────────────────────────────────────────────────────────
def generate_compliance_report(
records: List[InvoiceRecord],
summary: ReportSummary,
buyer_name: str = "",
buyer_tax_id: str = "",
include_raw_details: bool = True,
) -> str:
"""
生成完整的发票合规检查报告。
Args:
records: 发票记录列表
summary: 汇总统计数据
buyer_name: 企业名称（用于报告基本信息）
buyer_tax_id: 纳税人识别号（用于报告基本信息）
include_raw_details: 是否在报告末尾附上原始发票明细表
Returns:
Markdown 格式的完整报告
"""
report_id = _generate_report_id()
now = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
lines = [
f"# 发票合规检查报告\n",
f"**报告编号**：`{report_id}`\n",
f"**生成时间**：`{now}`\n",
f"**生成机构**：{GENERATOR_NAME}\n",
f"**版本**：{REPORT_VERSION}\n",
"---\n",
]
# 第一节：基本信息
lines.append(_section_basic_info(summary, buyer_name, buyer_tax_id))
lines.append("")
# 第二节：发票汇总
lines.append(_section_invoice_summary(summary, records))
lines.append("")
# 第三节：查重结果
lines.append(_section_duplicate_result(records))
lines.append("")
# 第四节：验真结果
lines.append(_section_verify_result(records))
lines.append("")
# 第五节：合规结论
lines.append(_section_compliance_conclusion(records))
lines.append("")
# 第六节：附件清单
lines.append(_section_attachment_list(records))
lines.append("")
# 附：原始发票明细表
if include_raw_details and records:
lines.append("---\n")
lines.append("## 附：发票明细表\n")
header = "| 发票号码 | 类型 | 开票日期 | 金额 | 销售方 | 状态 |"
sep = "|------|------|---------|------|--------|------|"
rows = [header, sep]
for r in records:
status_map = {
"clean": "✅ 正常",
"duplicate": "🔴 重复",
"suspicious": "⚠️ 可疑",
"pending": "⏳ 待处理",
}
status_display = status_map.get(r.status, r.status)
rows.append(
f"| {r.invoice_no or '（无）'} | {r.invoice_type or '其他票据'} | "
f"{_fmt_date(r.date)} | {_fmt_currency(r.amount)} | "
f"{r.seller_name or '（无）'} | {status_display} |"
)
lines.append("\n".join(rows))
# 页脚
lines.append("")
lines.append(
f"\n*本报告由 {GENERATOR_NAME} 自动生成，仅供内部合规参考，不作为税务申报依据。*\n"
)
lines.append(f"*报告编号：{report_id} · 生成时间：{now}*\n")
return "\n".join(lines)
# ─────────────────────────────────────────────────────────────────────────────
# Feishu Native Integration (飞书原生方案)
# ─────────────────────────────────────────────────────────────────────────────
def generate_feishu_compliance_report_markdown(
records: List[InvoiceRecord],
summary: ReportSummary,
buyer_name: str = "",
buyer_tax_id: str = "",
) -> str:
"""
生成飞书文档格式的合规报告，使用飞书原生 Markdown 语法（支持高亮块、分栏等）。
结果可直接用于 feishu_create_doc 工具创建文档。
符合《财会便函〔2023〕18号》六节结构要求，文档可分享、可评论。
Args:
records: 发票记录列表
summary: 汇总统计数据
buyer_name: 企业名称
buyer_tax_id: 纳税人识别号
Returns:
Lark-flavored Markdown 格式的完整报告
"""
report_id = _generate_report_id()
now = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
# 飞书文档开头 - 不要一级标题（title 参数已经设置了文档标题）
lines = []
lines.append(f"**报告编号**：`{report_id}`\n")
lines.append(f"**生成时间**：`{now}`\n")
lines.append(f"**生成机构**：{GENERATOR_NAME}\n")
lines.append(f"**版本**：{REPORT_VERSION} (飞书原生版)\n")
lines.append("---\n")
# 第一节：基本信息
lines.append("## 一、基本信息\n")
ent_name = buyer_name or "（未提供）"
ent_tax_id = buyer_tax_id or "（未提供）"
abnormal_count = summary.duplicate_count + summary.suspicious_count
lines.append("| 项目 | 内容 |")
lines.append("|------|------|")
lines.append(f"| 报告期间 | {datetime.now().strftime('%Y-%m-%d')} ~ {datetime.now().strftime('%Y-%m-%d')} |")
lines.append(f"| 发票总数 | {summary.total_invoices} 张 |")
lines.append(f"| 价税合计总额 | {_fmt_currency(summary.total_amount)} |")
lines.append(f"| 异常发票数 | {abnormal_count} 张 |")
lines.append(f"| 涉及企业名称 | {ent_name} |")
lines.append(f"| 纳税人识别号 | {ent_tax_id} |")
lines.append("")
# 第二节：发票汇总
lines.append("## 二、发票汇总\n")
lines.append("### 2.1 按发票类型分布\n")
type_header = "| 发票类型 | 数量 | 金额合计 |"
type_sep = "|------|------|---------|"
type_rows = [type_header, type_sep]
type_totals = {}
for r in records:
t = r.invoice_type or "其他票据"
type_totals.setdefault(t, {"count": 0, "amount": 0.0})
type_totals[t]["count"] += 1
type_totals[t]["amount"] += r.amount
grand_total_count = 0
grand_total_amount = 0.0
for inv_type in sorted(type_totals.keys()):
info = type_totals[inv_type]
grand_total_count += info["count"]
grand_total_amount += info["amount"]
type_rows.append(f"| {inv_type} | {info['count']} | {_fmt_currency(info['amount'])} |")
type_rows.append(f"| **合计** | **{grand_total_count}** | **{_fmt_currency(grand_total_amount)}** |")
lines.append("\n".join(type_rows))
lines.append("")
lines.append("### 2.2 按月份分布\n")
month_header = "| 月份 | 发票数量 | 金额合计 |"
month_sep = "|------|---------|---------|"
month_rows = [month_header, month_sep]
month_totals = {}
for r in records:
if r.date:
month = r.date[:7]
else:
month = "未知月份"
month_totals.setdefault(month, {"count": 0, "amount": 0.0})
month_totals[month]["count"] += 1
month_totals[month]["amount"] += r.amount
for month in sorted(month_totals.keys()):
info = month_totals[month]
month_rows.append(f"| {month} | {info['count']} | {_fmt_currency(info['amount'])} |")
lines.append("\n".join(month_rows))
lines.append("")
# 第三节：查重结果
lines.append("## 三、查重结果\n")
dup_records = [r for r in records if r.status in ("duplicate", "suspicious")]
if not dup_records:
lines.append("<callout emoji=\"✅\" background-color=\"light-green\">\n未发现重复报销发票\n</callout>\n")
lines.append("**重复发票数量**：0 张 ")
lines.append("**重复发票金额**：¥0.00")
else:
dup_header = "| 序号 | 发票号码 | 开票日期 | 金额 | 销售方 | 疑似重复原因 |"
dup_sep = "|------|---------|---------|------|--------|------------|"
dup_rows = [dup_header, dup_sep]
total_dup_amount = 0.0
for i, r in enumerate(dup_records, 1):
invoice_no = r.invoice_no or "（无号码）"
date = _fmt_date(r.date)
amount = r.amount
total_dup_amount += amount
seller = r.seller_name or "（无销售方）"
if r.status == "duplicate":
reason = "发票号码完全相同"
else:
reason = "金额+日期+购买方相同但号码不同 ⚠️ 疑似篡改"
dup_rows.append(f"| {i} | {invoice_no} | {date} | {_fmt_currency(amount)} | {seller} | {reason} |")
lines.append("\n".join(dup_rows))
lines.append("")
lines.append(f"**重复发票数量**：{len(dup_records)} 张")
lines.append(f"**重复发票金额**：{_fmt_currency(total_dup_amount)}")
lines.append("")
# 第四节：验真结果
lines.append("## 四、验真结果\n")
abnormal_records = [r for r in records if r.verify_status in ("void", "red", "失控", "suspicious", "abnormal")]
if not abnormal_records:
lines.append("<callout emoji=\"✅\" background-color=\"light-green\">\n全部发票状态正常\n</callout>\n")
lines.append("**异常发票数量**：0 张 ")
lines.append("**异常发票金额**：¥0.00")
else:
abnormal_header = "| 序号 | 发票号码 | 开票日期 | 金额 | 验真状态 | 状态说明 |"
abnormal_sep = "|------|---------|---------|------|---------|---------|"
abnormal_rows = [abnormal_header, abnormal_sep]
total_abnormal_amount = 0.0
for i, r in enumerate(abnormal_records, 1):
invoice_no = r.invoice_no or "（无号码）"
date = _fmt_date(r.date)
amount = r.amount
total_abnormal_amount += amount
status_desc = {
"void": "作废",
"red": "红冲",
"失控": "失控",
"suspicious": "可疑",
"abnormal": "异常",
}.get(r.verify_status, r.verify_status)
abnormal_rows.append(f"| {i} | {invoice_no} | {date} | {_fmt_currency(amount)} | {r.verify_status} | {status_desc} |")
lines.append("\n".join(abnormal_rows))
lines.append("")
lines.append(f"**异常发票数量**：{len(abnormal_records)} 张")
lines.append(f"**异常发票金额**：{_fmt_currency(total_abnormal_amount)}")
lines.append("")
# 第五节：合规结论
lines.append("## 五、合规结论\n")
lines.append("根据《财政部关于电子发票电子化报销、入账、归档管理有关问题的通知》（财会便函〔2023〕18号）要求，本报告对所述期间内企业发票进行了合规性审查。\n")
lines.append("### 5.1 合规情况总结\n")
dup_susp_count = sum(1 for r in records if r.status in ("duplicate", "suspicious"))
abnormal_verify_count = sum(1 for r in records if r.verify_status in ("void", "red", "失控", "suspicious", "abnormal"))
format_issues = sum(1 for r in records if r.status == "suspicious")
authenticity = "✅ 未见异常" if dup_susp_count == 0 else "⚠️ 发现异常"
duplicate_check = "✅ 未发现重复" if dup_susp_count == 0 else f"⚠️ 发现 {dup_susp_count} 张重复"
status_check = "✅ 全部正常" if abnormal_verify_count == 0 else f"⚠️ 发现 {abnormal_verify_count} 张异常"
format_check = "✅ 符合要求" if format_issues == 0 else f"⚠️ 存在 {format_issues} 张格式不规范发票"
lines.append("| 检查项目 | 结果 |")
lines.append("|---------|------|")
lines.append(f"| 发票真实性 | {authenticity} |")
lines.append(f"| 重复报销检查 | {duplicate_check} |")
lines.append(f"| 发票状态检查 | {status_check} |")
lines.append(f"| 格式合规性 | {format_check} |")
lines.append("")
lines.append("### 5.2 风险提示\n")
risk_items = []
if dup_susp_count > 0:
dup_amount = sum(r.amount for r in records if r.status in ("duplicate", "suspicious"))
lines.append(f"<callout emoji=\"⚠️\" background-color=\"light-yellow\">\n发现 {dup_susp_count} 张发票存在重复报销风险，涉及金额 {_fmt_currency(dup_amount)}\n</callout>\n")
if abnormal_verify_count > 0:
ab_amount = sum(r.amount for r in records if r.verify_status in ("void", "red", "失控", "suspicious", "abnormal"))
lines.append(f"<callout emoji=\"⚠️\" background-color=\"light-yellow\">\n发现 {abnormal_verify_count} 张发票状态异常（作废/红冲/失控），涉及金额 {_fmt_currency(ab_amount)}\n</callout>\n")
if format_issues > 0:
lines.append(f"<callout emoji=\"💡\" background-color=\"light-blue\">\n建议对格式不规范发票进行进一步核实后再行报销\n</callout>\n")
if dup_susp_count == 0 and abnormal_verify_count == 0 and format_issues == 0:
lines.append("<callout emoji=\"✅\" background-color=\"light-green\">\n未发现明显合规风险\n</callout>\n")
# 第六节：附件清单
lines.append("## 六、附件清单\n")
lines.append("| 序号 | 附件名称 | 说明 |")
lines.append("|------|---------|------|")
lines.append("| 1 | 原始发票影像 | 各发票图片/PDF 原件 |")
lines.append(f"| 2 | 发票明细表 | 全部发票结构化数据存储于飞书多维表格 |")
lines.append("")
# 页脚
lines.append("---")
lines.append("")
lines.append(f"*本报告由 {GENERATOR_NAME} 自动生成，仅供内部合规参考，不作为税务申报依据。*")
lines.append(f"*报告编号：{report_id} · 生成时间：{now}*")
lines.append("")
return "\n".join(lines)
def prepare_invoices_for_feishu_bitable(records: List[InvoiceRecord]) -> List[Dict[str, Any]]:
"""
将发票记录转换为飞书多维表格所需的批量创建格式。
适用于 feishu_bitable_app_table_record.batch_create API。
飞书多维表格预设字段：
- 发票代码（文本）
- 发票号码（文本）
- 开票日期（日期）
- 金额（数字）
- 开票方（文本）
- 状态（单选：正常/重复/可疑/异常）
- 查验状态（单选：未查验/正常/作废/红冲/失控）
Args:
records: 发票记录列表
Returns:
适用于批量创建的 records 数组
"""
bitable_records = []
status_map = {
"clean": "正常",
"duplicate": "重复",
"suspicious": "可疑",
"pending": "待处理",
"abnormal": "异常",
}
verify_map = {
"unchecked": "未查验",
"normal": "正常",
"void": "作废",
"red": "红冲",
"失控": "失控",
"suspicious": "可疑",
"abnormal": "异常",
}
for r in records:
# 转换日期为毫秒时间戳
if r.date and re.match(r'\d{4}-\d{2}-\d{2}', r.date):
from datetime import datetime
dt = datetime.strptime(r.date, "%Y-%m-%d")
timestamp_ms = int(dt.timestamp() * 1000)
else:
timestamp_ms = None
fields = {
"发票代码": r.invoice_code,
"发票号码": r.invoice_no,
"开票方": r.seller_name,
"金额": r.amount,
"状态": status_map.get(r.status, r.status),
"查验状态": verify_map.get(r.verify_status, r.verify_status),
}
if timestamp_ms:
fields["开票日期"] = timestamp_ms
bitable_records.append({
"fields": fields
})
return bitable_records
def create_feishu_bitable_schema(app_token: str) -> Dict[str, Any]:
"""
返回创建发票明细表所需的字段定义。
使用 feishu_bitable_app_table.create 时传入此结构。
Args:
app_token: 多维表格 app token
Returns:
table.fields 定义
"""
fields = [
{
"field_name": "发票代码",
"type": 1, # 文本
},
{
"field_name": "发票号码",
"type": 1, # 文本
},
{
"field_name": "开票日期",
"type": 5, # 日期
},
{
"field_name": "金额",
"type": 2, # 数字
},
{
"field_name": "开票方",
"type": 1, # 文本
},
{
"field_name": "状态",
"type": 3, # 单选
"property": {
"options": [
{"name": "正常"},
{"name": "重复"},
{"name": "可疑"},
{"name": "异常"},
]
}
},
{
"field_name": "查验状态",
"type": 3, # 单选
"property": {
"options": [
{"name": "未查验"},
{"name": "正常"},
{"name": "作废"},
{"name": "红冲"},
{"name": "失控"},
]
}
}
]
return fields
# ─────────────────────────────────────────────────────────────────────────────
# CLI entry point
# ─────────────────────────────────────────────────────────────────────────────
def main():
"""
CLI 用法：
python3 compliance_report.py <summary_json> <records_json> [buyer_name] [buyer_tax_id]
示例：
python3 compliance_report.py '{"total_invoices":5,"duplicate_count":1,...}' '[{"invoice_no":"12345678",...}]' 'XX公司' '91440000MA5XXXXXXX'
"""
if len(sys.argv) < 3:
# 演示模式：无参数时生成示例报告
print("用法: python3 compliance_report.py <summary_json> <records_json> [buyer_name] [buyer_tax_id]")
print("")
print("演示模式：生成示例报告...")
summary = ReportSummary(
total_invoices=8,
duplicate_count=2,
suspicious_count=1,
clean_count=5,
total_amount=125680.50,
duplicate_amount=34560.00,
by_type={
"增值税专用发票": {"count": 3, "amount": 45600.00},
"增值税普通发票": {"count": 2, "amount": 18900.00},
"电子发票": {"count": 2, "amount": 51000.00},
"机票行程单": {"count": 1, "amount": 10180.50},
},
)
records = [
InvoiceRecord(invoice_no="12345678", invoice_type="增值税专用发票", date="2026-01-15",
amount=25600.00, seller_name="XX科技有限公司", status="clean", verify_status="normal"),
InvoiceRecord(invoice_no="22345678", invoice_type="增值税专用发票", date="2026-01-18",
amount=20000.00, seller_name="YY贸易公司", status="duplicate", verify_status="unchecked",
notes="跨批次重复：与历史记录发票号码重复"),
InvoiceRecord(invoice_no="32345678", invoice_type="增值税普通发票", date="2026-02-03",
amount=8900.00, seller_name="ZZ商贸", status="clean", verify_status="unchecked"),
InvoiceRecord(invoice_no="42345678", invoice_type="电子发票", date="2026-02-10",
amount=31000.00, seller_name="YY贸易公司", status="suspicious", verify_status="unchecked",
notes="金额+日期+购买方相同但号码不同 ⚠️"),
]
report = generate_compliance_report(records, summary, "演示公司", "91440000MA5XXXXXXX")
print(report)
return
summary = ReportSummary.from_dict(json.loads(sys.argv[1]))
records_dict = json.loads(sys.argv[2])
records = [InvoiceRecord.from_dict(r) for r in records_dict]
buyer_name = sys.argv[3] if len(sys.argv) > 3 else ""
buyer_tax_id = sys.argv[4] if len(sys.argv) > 4 else ""
report = generate_compliance_report(records, summary, buyer_name, buyer_tax_id)
print(report)
if __name__ == "__main__":
main()
FILE:references/invoice-types.md
# Chinese Invoice Types and Field Specifications
## Invoice Classification System
### 1. VAT Invoices
#### VAT Special Invoice (Deductible)
- **Format**: Paper / Electronic
- **Key fields**: Invoice code (10-digit) + invoice number (8-digit), amount, tax, total (tax included), buyer, seller, taxpayer identification number
- **Usage**: Input tax deductible
#### VAT Regular Invoice (Non-deductible)
- **Format**: Paper / Electronic / Roll
- **Key fields**: Same as special invoice, but non-deductible
- **Special**: Buyer name and taxpayer ID are required fields
#### Electronic Invoice (Digital VAT)
- **Format**: PDF / OFD / XML
- **Characteristics**: Issued via national unified electronic invoice platform, no paper
- **Verification**: Via tax digital account
### 2. Air / Train / Transit Tickets
- **Fields**: Passenger name, flight/train number, date, origin, destination, amount
- **Characteristics**: Not VAT invoices, non-deductible
### 3. Taxi Receipts
- **Fields**: Date, time, pick-up/drop-off locations, amount, invoice code + number
### 4. Generic Printed Invoices
- **Fields**: Issuer, date, item details, amount
---
## Invoice Number Coding Rules
| Invoice Type | Code Digits | Number Digits | Example |
|-------------|------------|---------------|---------|
| VAT Special Invoice | 10-digit | 8-digit | 144031900110 / 12345678 |
| VAT Regular Invoice | 12-digit | 8-digit | 144031900110 / 12345678 |
| Electronic Invoice | 20-digit | — | — |
| Taxi Receipt | 10-digit | — | — |
---
## Key Field Extraction Regex (Post-OCR Text)
```python
# Invoice code extraction
invoice_code_pattern = r'[发票代码|代码][：:\s]*(\d{10,12})'
invoice_no_pattern = r'[发票号码|号码][：:\s]*(\d{8})'
# Amount extraction
amount_pattern = r'[价税合计|合计|金额][：:\s]*[￥¥]?\s*(\d+\.?\d{0,2})'
# Date extraction
date_pattern = r'(\d{4}[年\-/]\d{1,2}[月\-/]\d{1,2}[日]?)'
# Taxpayer ID (buyer/seller)
tax_id_pattern = r'[纳税人识别号|税号][：:\s]*([A-Z0-9]{15,20})'
```
FILE:references/changelog.md
# InvoiceGuard Changelog
## Upgrade: Compliance Report — Feishu Native Solution (2026-04-19)
**Feature**: Feishu native implementation
- **Compliance Report**: Generate shareable, commentable Feishu cloud documents, compliant with [Cai Hui Ban [2023] No.18] 6-section structure
- **Invoice Details**: Auto-import to Feishu Bitable, supporting filtering, sorting, and chart analysis
**New functions**:
- `generate_feishu_compliance_report_markdown()` - Generate Lark-flavored Markdown for `feishu_create_doc`
- `prepare_invoices_for_feishu_bitable()` - Convert invoice records to Feishu Bitable batch creation format
- `create_feishu_bitable_schema()` - Return field definitions for invoice details table
**Technical upgrades**:
- Use Feishu native callout blocks for risk alerts, clearer visual hierarchy
- Complete Bitable field support: invoice code, number, date, amount, issuer, status, verification status
- Supports filtering, sorting, pivot tables and chart analysis
- Documents shareable and commentable, team collaboration enabled
**Deployment notes**:
- 91Skillhub is independent from 91TokenHub
- Reuses the same payment system as GEO Master
- Deployment server: 124.220.60.10
---
## New: Compliance Report Generator (2026-04-19)
**File**: `scripts/compliance_report.py`
New compliance report generation script, compliant with [Cai Hui Ban [2023] No.18] complete 6-section structure:
1. Basic Info (company name, tax ID, report date)
2. Invoice Summary (by type + by month)
3. Deduplication Results (duplicate/suspicious invoice list)
4. Verification Results (abnormal status invoice list)
5. Compliance Conclusion (summary + risk alerts)
6. Attachment List
**Attachment**: Invoice details table — complete list of all invoices with status flags
**Output**: Markdown format, directly writable to Feishu documents.
---
**Review Date**: 2026-04-19
**Reviewer**: 91Skillhub Team
**Status**: All Critical + Major issues resolved
---
## Critical Issue Fixes
### C-1 · Tamper Detection Dead Code
**File**: `scripts/duplicate_checker.py`
**Problem**: Lines 68-74 returned early on exact match, making lines 83-99 under identical conditions unreachable. Invoices with same number but tampered amount were incorrectly flagged as `exact` (confidence 1.0) instead of `tampered` (confidence 0.99).
**Root cause**:
```python
# Original (BUG)
for existing in existing_records:
if new_code == exist_code: # Exact match returns early
return DuplicateResult(match_type="exact", ...) # Never reaches below
# This identical condition is unreachable!
if new_code == exist_code: # Dead code
if abs(new_amount - exist_amount) > 0.01:
return DuplicateResult(match_type="tampered", ...)
```
**Fix**: Move tamper check before exact match return:
```python
if new_code == exist_code:
if abs(new_amount_dec - exist_amount_dec) > Decimal("0.01"):
return DuplicateResult(match_type="tampered", confidence=0.99, ...)
return DuplicateResult(match_type="exact", confidence=1.0, ...)
```
**Verification**: Same invoice number + different amount → `match_type=tampered`, confidence 0.99
---
### C-2 · Regex Character Class Syntax Error
**File**: `duplicate_checker.py`, `batch_processor.py`
**Problem**: `[纳税人识别号|税号]` is a character class (matches single character), not logical OR.
**Fix**: Use correct non-capturing alternation:
```python
# Wrong
r'[纳税人识别号|税号][：:\s]*([A-Z0-9]{15,20})'
# Correct
r'(?:纳税人识别号|税号)[：:\s]*([A-Z0-9]{15,20})'
```
---
### C-3 · Zero Pro/Free Tier Isolation
**File**: `duplicate_checker.py`, `batch_processor.py`
**Problem**: No tier verification logic in code; any user had unlimited access to batch processing and tax authority API.
**Fix**: Introduce `TierConfig` class for complete permission isolation:
| Feature | Free | Pro |
|---------|:----:|:---:|
| Single deduplication | 20/month | Unlimited |
| Batch processing | Blocked | Unlimited |
| Tax authority API | Blocked | Allowed |
| Cross-batch deduplication | Blocked | Allowed |
---
### C-4 · Thousand-separator Amount Extraction Failure
**File**: `duplicate_checker.py`, `batch_processor.py`
**Problem**: `￥1,234.56` only extracted as `1`.
**Fix**: New `_amount_from_text()` function:
```python
r'[价税合计|价税][：:\s]*[￥¥]?\s*(\d{1,3}(?:,\d{3})*(?:\.\d{1,2})?)'
```
---
## Major Issue Fixes
### M-1 · SHA256 Truncation Causing Hash Collision Risk
`fields_hash()` changed from `[:16]` (16 chars) to full 64-char SHA256.
### M-2 · Air Itinerary Misidentified as Electronic Invoice
Detection order adjusted: `航空`/`机票`/`行程单` takes priority over `电子发票`.
### M-3 · Cross-batch Duplicate Detection Impossible
`batch_check_duplicates()` added `historical_records` parameter for cross-batch comparison.
### M-5 · Float Comparison Replaced with `Decimal`
All amount comparisons use `Decimal` to avoid floating-point precision issues.
### M-6 · XML/OFD Parse Support
New `parse_xml_text()` and `parse_ofd_text()` functions for XML/OFD file content parsing.
---
## Modified Files
| File | Changes |
|------|---------|
| `scripts/duplicate_checker.py` | C-1, C-2, C-3, C-4, M-1, M-2, M-5 |
| `scripts/batch_processor.py` | C-2, C-3, C-4, M-1, M-2, M-3, M-5, M-6 |
| `SKILL.md` | Updated Pro tier permission descriptions |
| `references/changelog.md` | This file |
FILE:references/tax-api.md
# State Tax Administration VAT Invoice Verification Platform
## Official Platform
**State Tax Administration VAT Invoice Verification Platform**
https://inv-veri.chinatax.gov.cn
## API Integration Methods
### Method 1: Web Portal Verification (Recommended for Skill Use)
Simulate verification requests by crawling the official portal:
```
POST https://inv-veri.chinatax.gov.cn/web/query.do
```
Request parameters:
| Parameter | Description |
|-----------|-------------|
| param0 | Invoice code (10-digit) |
| param1 | Invoice number (8-digit) |
| param2 | Invoice date (YYYYMMDD) |
| param3 | Amount (total with tax, in CNY) |
| param4 | Verification code (requires OCR) |
Returns: Invoice status (normal / voided / red-flushed / out of control)
### Method 2: Electronic Invoice XML/OFD Direct Read
XML/OFD files for electronic invoices contain complete signature information.
Invoice authenticity can be verified locally via signature verification, no external API call needed:
```bash
# XML electronic invoice signature verification
# Use OpenSSL to verify digital signature block
openssl smime -verify -in invoice.xml.sig -inform DER
# OFD invoice signature verification
# Parse OFD file structure, extract signature domain for verification
```
### Method 3: Enterprise ERP Integration
Enterprises can batch-verify via third-party service providers authorized by tax authority (e.g. UFIDA, Kingdee) API.
## API Quotas
| Account Type | Daily Limit | Notes |
|-------------|-------------|-------|
| Tax authority portal (free) | 100 calls/day/IP | Requires captcha |
| Enterprise developer account | Unlimited (pay-per-call) | ¥0.1-0.3/call |
| Third-party provider | Unlimited | API wrapper |
## Invoice Status Codes
| Status Code | Meaning | Reimbursement |
|-------------|---------|---------------|
| 00 / Normal | Invoice valid | ✅ Acceptable |
| 01 / Voided | Self-voided by company | ❌ Not acceptable |
| 02 / Red-flushed | Red invoice issued for offset | ❌ Not acceptable |
| 03 / Out of control | Flagged by tax authority | ❌ Not acceptable |
| 04 / Abnormal | Data inconsistency | ⚠️ Requires verification |
## Verification Strategy in Skill
Since the Skill runs in a sandbox environment, **no enterprise sensitive tax information is stored**.
Verification flow:
1. User uploads invoice image → AI recognizes key fields
2. Call verification API (or XML direct verification) to get status
3. Return status result only; do not store invoice data
4. Each verification consumes account tokens
> Note: If the API changes, the official tax authority documentation prevails. The Skill bears no responsibility for losses caused by API changes.
FILE:references/compliance-report.md
# Invoice Compliance Report Template
## Per [Cai Hui Ban [2023] No.18]
---
# Invoice Compliance Check Report
**Report ID**: `RPT-YYYYMMDD-XXXX`
**Generated**: `YYYY-MM-DD HH:mm:ss`
**Generated by**: InvoiceGuard
---
## 1. Basic Information
| Item | Content |
|------|---------|
| Report Period | `YYYY-MM-DD` ~ `YYYY-MM-DD` |
| Total Invoices | `N` |
| Total Amount (incl. tax) | `¥ XXX,XXX.XX` |
| Abnormal Invoices | `N` |
| Company Name | `XXXXXXXX` |
| Taxpayer ID | `XXXXXXXXXXXXXXXXXX` |
---
## 2. Invoice Summary
### 2.1 By Invoice Type
| Invoice Type | Count | Total Amount |
|-------------|-------|-------------|
| VAT Special Invoice | N | ¥ |
| VAT Regular Invoice | N | ¥ |
| Electronic Invoice | N | ¥ |
| Other Receipts | N | ¥ |
| **Total** | **N** | **¥** |
### 2.2 By Month
| Month | Invoice Count | Total Amount |
|-------|-------------|-------------|
| YYYY-MM | N | ¥ |
---
## 3. Deduplication Results
### 3.1 Duplicate Invoice List
| # | Invoice Number | Date | Amount | Seller | Suspected Reason |
|---|--------------|------|--------|--------|-----------------|
| 1 | XXXXXXXX | YYYY-MM-DD | ¥ | XXXX | Invoice number identical |
| 2 | XXXXXXXX | YYYY-MM-DD | ¥ | XXXX | Field hash collision |
| … | … | … | … | … | … |
**Duplicate count**: `N`
**Duplicate amount**: `¥ XXX,XXX.XX`
---
## 4. Verification Results
### 4.1 Abnormal Invoice List
| # | Invoice Number | Date | Amount | Status | Description |
|---|--------------|------|--------|--------|-------------|
| 1 | XXXXXXXX | YYYY-MM-DD | ¥ | Voided/Red-flushed/Out of control | … |
| … | … | … | … | … | … |
**Abnormal count**: `N`
**Abnormal amount**: `¥ XXX,XXX.XX`
---
## 5. Compliance Conclusion
Per Ministry of Finance [Cai Hui Ban [2023] No.18] requirements, this report reviews invoice compliance for the stated period.
### 5.1 Compliance Summary
| Check Item | Result |
|-----------|--------|
| Invoice authenticity | ✅ No abnormality / ⚠️ Abnormalities found |
| Duplicate reimbursement check | ✅ No duplicates / ⚠️ N duplicate(s) found |
| Invoice status check | ✅ All normal / ⚠️ N abnormal(s) found |
| Format compliance | ✅ Compliant / ⚠️ Non-compliant invoices exist |
### 5.2 Risk Alerts
- [ ] N invoice(s) with duplicate reimbursement risk, amount `¥XXX`
- [ ] N invoice(s) with abnormal status (voided/red-flushed/out of control)
- [ ] Recommend further verification of the above invoices before reimbursement
---
## 6. Attachment List
| # | Attachment | Description |
|---|-----------|-------------|
| 1 | Original invoice images | Invoice photos/PDF originals |
| 2 | Invoice details table | Structured data for all invoices |
---
*This report is auto-generated by InvoiceGuard for internal compliance reference only. It does not serve as a basis for tax filing.*
Multi-platform Order Profit Calculator — upload order exports from any e-commerce platform or ERP, get instant profit reports by order, store, SKU, and platf...
# Seller Profit Calculator
Upload order exports from **any e-commerce platform or ERP** → get instant profit breakdown by order, store, SKU, and platform.
**Slug:** `ecom-seller-profit`
**Price:** $0.01 USDT per call
**Author:** 91Skillhub Team
---
## What It Does
**Upload one Excel file → get a complete profit breakdown:**
- 📋 **Overall summary**: total orders, completed, cancelled, total revenue, total cost, net profit, net margin %
- 🌍 **By platform**: revenue / expense / cost / profit per platform
- 🏪 **By store**: revenue / expense / cost / profit per store
- 🔴 **Bottom 5 orders**: worst loss-making orders highlighted
- 🟢 **Top 5 orders**: best performing orders highlighted
---
## How It Works
```
You upload any Excel order export
↓
Agent reads headers + sample rows (analyze_headers.py)
↓
Agent identifies each column's meaning (LLM reasoning)
↓
Agent builds field_map JSON → passes to parse_orders.py
↓
parse_orders.py calculates with full field context
↓
Report with per-order breakdown + accuracy notes
```
### CLI Usage
```bash
# Auto-detect format
python3 scripts/parse_orders.py orders.xlsx
# With field mapping
python3 scripts/parse_orders.py orders.xlsx --field-map @my_mapping.json
# Output JSON
python3 scripts/parse_orders.py orders.xlsx --json result.json
# Markdown report
python3 scripts/parse_orders.py orders.xlsx --markdown report.md
```
---
## Supported Platforms
All e-commerce platforms and ERPs that export order data with standard fields.
| Platform | Status |
|----------|--------|
| TikTok Shop | ✅ Verified |
| Allegro | ✅ Verified |
| Temu Half-Hosted | ✅ Verified |
| SHEIN | ✅ Verified |
| Fruugo | ✅ Verified |
| Amazon | ✅ Compatible |
| Shopee / Lazada | ✅ Compatible |
| Ozon | ✅ Compatible |
| Walmart / eBay | ✅ Compatible |
| Other platforms | ✅ Generic |
---
## Supported File Formats
- Excel: `.xlsx`, `.xls`
- CSV: `.csv`
---
## Calculation Logic
```
Net Profit = Platform Revenue - Platform Expense - Order Cost
```
| Module | Description |
|--------|-------------|
| **Platform Income** | Transaction + shipping income + refunds + subsidies |
| **Platform Expense** | Commission + tech fees + shipping + refunds + fines + taxes |
| **Order Cost** | Purchase cost + first-leg freight + last-mile shipping + packaging + warehouse + advertising |
---
## Tiered Features
| Feature | FREE | PRO |
|---------|:----:|:---:|
| Multi-platform support | ✅ | ✅ |
| Header auto-detection | ✅ | ✅ |
| Per-order profit calculation | ✅ | ✅ |
| By-store / by-platform breakdown | ✅ | ✅ |
| Top/bottom 5 orders | ✅ | ✅ |
| Markdown report | ✅ | ✅ |
| JSON export | ✅ | ✅ |
| Custom field mapping | — | ✅ |
| Priority support | — | ✅ |
---
## Billing
This skill charges **$0.01 USDT per execution** via SkillPay.me.
- Billing is processed on each run via `skillpay.me/api/v1/billing/charge`
- Your user ID (`FEISHU_USER_ID`) is transmitted to SkillPay for billing identification
- When balance is insufficient, the system returns a payment link for top-up
**Required environment variables:**
| Variable | Description |
|----------|-------------|
| `SKILL_BILLING_API_KEY` | SkillPay Builder API Key |
| `SKILL_BILLING_SKILL_ID` | SkillPay Skill ID |
| `FEISHU_USER_ID` | User ID for billing |
---
## File Structure
```
seller-profit-calculator/
├── SKILL.md
└── scripts/
├── parse_orders.py # Core parser + billing
└── analyze_headers.py # Header analyzer
```
---
## Limitations
- CSV support: in v2.0 roadmap
- Settlement report: in v2.0 roadmap
- Per-order profit is precise; platform-level aggregation may have minor variance due to internal settlement adjustments
---
## License
MIT
FILE:scripts/analyze_headers.py
#!/usr/bin/env python3
"""
analyze_headers.py - Analyze Excel file headers and sample rows for field mapping.
Usage:
python3 analyze_headers.py <file.xlsx> # print analysis prompt
python3 analyze_headers.py <file.xlsx> --show-headers # print headers + sample rows only
This script does NOT call LLM. It prepares structured data for the Agent to analyze.
The Agent uses its own reasoning to produce the field_map JSON.
"""
import sys
import json
import argparse
from openpyxl import load_workbook
STANDARD_FIELDS = {
# 基础信息
"订单编号": "order_no",
"订单号": "order_no",
"订单状态": "status",
"销售数量": "qty",
"退货数量": "return_qty",
"履约类型": "fulfillment_type",
"店铺": "store",
"站点": "platform",
# 时间
"下单时间": "order_time",
"付款时间": "pay_time",
"发货时间": "ship_time",
"账务时间": "finance_time",
"结算时间": "finance_time",
# 收入/利润 (Allegro/Temu)
"利润": "profit_declared",
"成本利润率": "cost_profit_rate",
"销售利润率": "sales_profit_rate",
"总计": "total_income",
"交易收入": "trading_income",
"运费收入": "freight_income",
"EPR费用(已退费)": "epr_refund",
"售后退款": "refund",
"运费退款": "freight_refund",
"EPR费用(已扣费)": "epr_fee",
"退货面单费": "return_shipping_fee",
"发货面单费": "shipping_fee",
"违规扣款": "violation_deduction",
"广告服务费": "ad_service_fee",
"推广服务费": "promotion_fee",
"其他账务": "other_finance",
"总计.1": "total_platform_expense",
# 成本 (Allegro/Temu)
"采购金额": "purchase_cost",
"包材费": "packaging_fee",
"头程运费": "first_mile",
"尾程运费": "last_mile",
"广告成本": "ad_cost",
"运营成本": "ops_cost",
"仓库操作费": "warehouse_fee",
"其他成本": "other_cost",
"总计.2": "total_order_cost",
"订单其他收入": "other_order_income",
# 申报价 (Temu)
"申报价": "declared_price",
# TikTok 170-col fields
"已结算金额(RMB)": "settled_amount",
"预估回款金额(RMB)": "estimated_amount",
"买家实付金额(RMB)": "buyer_paid",
"商家运费(RMB)": "seller_freight",
"退款金额(RMB)": "refund_tiktok",
"采购成本(RMB)": "purchase_cost_tiktok",
"运费成本(RMB)": "freight_cost",
"交易手续费(RMB)": "transaction_fee",
"TikTok平台佣金(RMB)": "platform_commission",
"VAT(RMB)": "vat",
"进口增值税(RMB)": "import_vat",
"关税(RMB)": "customs_duty",
"平台惩罚(RMB)": "platform_penalty",
"平台补偿(RMB)": "platform_compensation",
"退款订单补偿(RMB)": "refund_compensation",
"FBT仓储服务费(RMB)": "fbt_fee",
"物流供应商清关服务费(RMB)": "clearing_fee",
"实际逆向物流运费(RMB)": "actual_reverse_freight",
"实际运费(RMB)": "actual_freight",
"运输保险费(RMB)": "transport_insurance",
"达人佣金(RMB)": "creator_commission",
"推荐费(RMB)": "referral_fee",
"信用卡付款手续费(RMB)": "card_fee",
"买家申请退款(RMB)": "buyer_refund",
"客户服务补偿(RMB)": "cs_compensation",
"商家体验扣款(RMB)": "seller_deduction",
"GMV广告费用(RMB)": "gmv_ad_fee",
"平台佣金调整(RMB)": "commission_adjust",
"促销调整(RMB)": "promo_adjust",
"平台佣金折扣(RMB)": "commission_discount",
"共同出资费用(RMB)": "joint_funded",
"物流补偿(RMB)": "logistics_compensation",
"运费调整(RMB)": "freight_adjust",
"运费补偿(RMB)": "freight_compensation",
"运费回扣(RMB)": "freight_rebate",
"样品运费(RMB)": "sample_freight",
"其他调整(RMB)": "other_adjust",
"总调整金额(RMB)": "total_adjustment",
"SFP服务费(RMB)": "sfp_fee",
"LIVE Specials计划服务费(RMB)": "live_fee",
"Bonus金币返现服务费(RMB)": "bonus_fee",
"TikTok Shop Mall服务费(RMB)": "mall_fee",
"Voucher Xtra计划服务费(RMB)": "voucher_fee",
"限时抢购服务费(RMB)": "flash_sale_fee",
"促销活动服务费(RMB)": "promo_service_fee",
"动态服务费(RMB)": "dynamic_fee",
"其他服务费(RMB)": "other_service_fee",
"退款管理费(RMB)": "refund_mgmt_fee",
"预购计划服务费(RMB)": "preorder_fee",
"SST(RMB)": "sst",
"GST(RMB)": "gst",
"墨西哥增值税(RMB)": "mx_vat",
"墨西哥联邦所得税(RMB)": "mx_income_tax",
"反倾销税(RMB)": "anti_dumping",
"联盟商店广告佣金(RMB)": "affiliate_ad_fee",
"合作伙伴佣金(RMB)": "partner_commission",
"活动运费补贴(RMB)": "activity_freight_subsidy",
"平台运费补贴(RMB)": "platform_freight_subsidy",
"产品SKU": "sku",
}
def analyze_file(filepath, show_headers_only=False):
"""Read file and return structured header + sample data for LLM analysis."""
wb = load_workbook(filepath, data_only=False)
ws = wb.active
# Detect header row (usually row 1 or 2)
header_row = 1
for r in range(1, min(5, ws.max_row + 1)):
vals = [ws.cell(r, c).value for c in range(1, ws.max_column + 1)]
strs = [str(v).strip() if v is not None else '' for v in vals]
if any(k in strs for k in ['订单编号', '订单号', '订单号']):
header_row = r
break
# Read headers
headers = []
for c in range(1, ws.max_column + 1):
v = ws.cell(header_row, c).value
headers.append(str(v).strip() if v is not None else f'_col_{c}')
# Read 3 sample rows
samples = []
for r in range(header_row + 1, min(header_row + 4, ws.max_row + 1)):
row_data = []
for c in range(1, ws.max_column + 1):
v = ws.cell(r, c).value
row_data.append(str(v).strip() if v is not None else '')
# Skip empty rows
if any(v for v in row_data):
samples.append(row_data)
result = {
'file': filepath,
'header_row': header_row,
'total_rows': ws.max_row - header_row,
'total_columns': ws.max_column,
'headers': headers,
'sample_rows': samples,
}
if show_headers_only:
return result
# Build matched analysis (which standard fields are present)
matched = []
unmatched = []
for h in headers:
if h in STANDARD_FIELDS:
matched.append({'column': h, 'standard': STANDARD_FIELDS[h]})
elif h not in ('', f'_col_{headers.index(h)+1}'):
unmatched.append(h)
result['matched_standard_fields'] = matched
result['unmatched_columns'] = unmatched
return result
if __name__ == '__main__':
parser = argparse.ArgumentParser(description='Analyze Excel headers for field mapping')
parser.add_argument('filepath', help='Path to Excel file')
parser.add_argument('--show-headers', action='store_true',
help='Only show headers and samples, no standard field matching')
parser.add_argument('--json', metavar='FILE', help='Save output as JSON')
args = parser.parse_args()
result = analyze_file(args.filepath, show_headers_only=args.show_headers)
output = json.dumps(result, ensure_ascii=False, indent=2)
if args.json:
with open(args.json, 'w', encoding='utf-8') as f:
f.write(output)
print(f'Saved to {args.json}')
else:
print(output)
FILE:scripts/parse_orders.py
#!/usr/bin/env python3
"""
Seller Profit Calculator - Core Parser
Parses order export and calculates profit per order.
ClawHub version uses SkillPay billing.
Environment variables required:
SKILL_BILLING_API_KEY - SkillPay Builder API Key
SKILL_BILLING_SKILL_ID - SkillPay Skill ID
FEISHU_USER_ID - User ID for billing
"""
# Calculation reference: TikTok Shop 170-column standard schema.
# All platforms are mapped to this standard field schema.
# Platform field aliases (Allegro 41-col -> TikTok 170-col mapping):
# 交易收入 -> 买家实付金额(RMB)
# 运费收入 -> 商家运费(RMB)
# ... (full mapping documented in codebase)
import sys
import json
import argparse
from datetime import datetime
from openpyxl import load_workbook
def parse_currency(val):
"""Parse currency string like '￥73.44' or '-45.00' to float."""
if val is None or val == '':
return 0.0
if isinstance(val, (int, float)):
return float(val)
s = str(val).strip()
if s.startswith('￥') or s.startswith('¥') or s.startswith('$'):
s = s[1:]
s = s.replace(',', '').strip()
try:
return float(s)
except:
return 0.0
def parse_int(val):
if val is None or val == '':
return 0
if isinstance(val, int):
return val
try:
return int(str(val).strip().replace(',', ''))
except:
return 0
# ====================================================================
# TikTok Shop 170-column standard schema (地基)
# All costs are positive numbers; negatives from platform are preserved.
# ====================================================================
def calculate_profit_tiktok(row, h, field_map=None):
"""
Calculate profit using TikTok Shop 170-column standard.
h = header_map (field_name -> 0-based column index in row list)
"""
def g(name, alt=None):
"""Get row value by header name, return 0 if not found."""
idx = h.get(name, -1)
if idx < 0 and alt:
idx = h.get(alt, -1)
if idx < 0:
return 0.0
val = row[idx]
if val is None or val == '':
return 0.0
return parse_currency(val)
def g_str(name, alt=None):
idx = h.get(name, -1)
if idx < 0 and alt:
idx = h.get(alt, -1)
if idx < 0:
return ''
return str(row[idx] or '').strip()
# --- Platform income (平台收入) ---
# Use 已结算金额(RMB) if available, else 预估回款金额(RMB)
settled = g('已结算金额(RMB)')
estimated = g('预估回款金额(RMB)')
platform_income = settled if settled != 0 else estimated
# Additional income components
freight_income = abs(g('商家运费(RMB)')) # positive offset to income
epr_refund = g('EPR费用(已退费)')
platform_income += freight_income + epr_refund
# --- Platform expense (平台支出) ---
# Core platform fees
transaction_fee = g('交易手续费(RMB)')
card_fee = g('信用卡付款手续费(RMB)')
platform_commission = g('TikTok 平台佣金(RMB)')
referral_fee = g('推荐费(RMB)')
seller_freight = g('商家运费(RMB)')
actual_freight = g('实际运费(RMB)')
reverse_freight = g('实际逆向物流运费(RMB)')
transport_insurance = g('运输保险费(RMB)')
# Creator/d affiliate
creator_commission = g('达人佣金(RMB)')
partner_commission = g('合作伙伴佣金(RMB)')
affiliate_ad_fee = g('联盟商店广告佣金(RMB)')
# Platform services
sfp_fee = g('SFP服务费(RMB)')
live_fee = g('LIVE Specials 计划服务费(RMB)')
bonus_fee = g('Bonus金币返现服务费(RMB)')
tiktok_mall_fee = g('TikTok Shop Mall服务费(RMB)')
voucher_fee = g('Voucher xtra 计划服务费(RMB)')
flash_sale_fee = g('限时抢购服务费(RMB)')
promo_fee = g('促销活动服务费(RMB)')
dynamic_fee = g('动态服务费(RMB)')
other_service_fee = g('其他服务费(RMB)')
refund_mgmt_fee = g('退款管理费(RMB)')
preorder_fee = g('预购计划服务费(RMB)')
# Taxes & duties
vat = g('VAT(RMB)')
import_vat = g('进口增值税(RMB)')
customs_duty = g('关税(RMB)')
clearing_fee = g('物流供应商清关服务费(RMB)')
sst = g('SST(RMB)')
gst = g('GST(RMB)')
mx_vat = g('墨西哥增值税(RMB)')
mx_income_tax = g('墨西哥联邦所得税(RMB)')
anti_dumping = g('反倾销税(RMB)')
# Deductions/adjustments
refund_amount = g('退款金额(RMB)')
total_adjustment = g('总调整金额(RMB)')
buyer_refund = g('买家申请退款(RMB)')
cs_compensation = g('客户服务补偿(RMB)')
seller_deduction = g('商家体验扣款(RMB)')
gmiv_ad_fee = g('GMV广告费用(RMB)')
commission_adjust = g('平台佣金调整(RMB)')
platform_penalty = g('平台惩罚(RMB)')
promo_adjust = g('促销调整(RMB)')
commission_discount = g('平台佣金折扣(RMB)')
platform_compensation = g('平台补偿(RMB)')
refund_order_compensation = g('退款订单补偿(RMB)')
joint_funded = g('共同出资费用(RMB)')
fbt_storage = g('FBT仓储服务费(RMB)')
logistics_compensation = g('物流补偿(RMB)')
freight_adjust = g('运费调整(RMB)')
freight_compensation = g('运费补偿(RMB)')
freight_rebate = g('运费回扣(RMB)')
sample_freight = g('样品运费(RMB)')
other_adjust = g('其他调整(RMB)')
platform_expense = (
transaction_fee + card_fee + platform_commission + referral_fee +
seller_freight + actual_freight + reverse_freight + transport_insurance +
creator_commission + partner_commission + affiliate_ad_fee +
sfp_fee + live_fee + bonus_fee + tiktok_mall_fee + voucher_fee +
flash_sale_fee + promo_fee + dynamic_fee + other_service_fee +
refund_mgmt_fee + preorder_fee +
vat + import_vat + customs_duty + clearing_fee +
sst + gst + mx_vat + mx_income_tax + anti_dumping +
refund_amount + total_adjustment +
buyer_refund + cs_compensation + seller_deduction +
gmiv_ad_fee + commission_adjust + platform_penalty +
promo_adjust + commission_discount +
platform_compensation + refund_order_compensation +
joint_funded + fbt_storage +
logistics_compensation + freight_adjust +
freight_compensation + freight_rebate +
sample_freight + other_adjust
)
# --- Order costs (订单成本) ---
purchase_cost = g('采购成本(RMB)')
freight_cost = g('运费成本(RMB)')
warehouse_fee = g('仓库操作费(RMB)')
packaging_fee = g('包材费(RMB)')
ad_cost = g('广告成本(RMB)')
ops_cost = g('运营成本(RMB)')
other_cost = g('其他成本(RMB)')
order_cost = (
purchase_cost + freight_cost + warehouse_fee +
packaging_fee + ad_cost + ops_cost + other_cost
)
# --- Other income ---
other_income = (
g('平台补偿(RMB)') + g('退款订单补偿(RMB)') +
g('运费回扣(RMB)') + g('运费补偿(RMB)') +
g('平台佣金折扣(RMB)') + g('活动运费补贴(RMB)') +
g('平台运费补贴(RMB)')
)
# --- Net profit ---
net_profit = platform_income - platform_expense - order_cost + other_income
# Platform-declared profit (for validation)
declared_profit = g('利润(RMB)')
return {
'platform_income': round(platform_income, 2),
'platform_expense': round(platform_expense, 2),
'order_cost': round(order_cost, 2),
'other_income': round(other_income, 2),
'net_profit_calc': round(net_profit, 2),
'net_profit_declared': round(declared_profit, 2),
'profit_diff': round(net_profit - declared_profit, 2),
# breakdown
'declared_profit': round(declared_profit, 2),
}
def calculate_profit_allegro(row, h, field_map=None):
"""
Calculate profit using Temu semi-hosted / Allegro 41-column format.
Key rules:
- Temu semi-hosted: 平台收入 = 申报价 × 销售数量 (confirmed by user)
- Allegro: 平台收入 = 交易收入 + 运费收入(abs)
- Cancelled: no platform income, fixed fee
- Refunded: platform deducted declared price, may have additional fees
"""
def g(name, alt=None):
# Agent-provided mapping takes priority: field_map["标准字段名"] -> "实际列名"
if field_map and name in field_map:
actual_name = field_map[name]
idx = h.get(actual_name, -1)
if idx < 0:
for k, v in h.items():
if k.startswith(actual_name + '_dup'):
idx = v; break
if idx < 0:
idx = h.get(name, -1) # fallback to standard name
else:
idx = h.get(name, -1)
if idx < 0:
for k, v in h.items():
if k.startswith(name + '_dup'):
idx = v; break
if idx < 0 and alt:
idx = h.get(alt, -1)
if idx < 0:
for k, v in h.items():
if k.startswith(alt + '_dup'):
idx = v; break
if idx < 0: return 0.0
val = row[idx]
if val is None or val == '': return 0.0
return parse_currency(val)
def rv(name, alt=None):
idx = h.get(name, -1)
if idx < 0:
for k, v in h.items():
if k.startswith(name + '_dup'):
idx = v; break
if idx < 0 and alt:
idx = h.get(alt, -1)
if idx < 0:
for k, v in h.items():
if k.startswith(alt + '_dup'):
idx = v; break
if idx < 0: return ''
val = row[idx]
return '' if val is None else str(val).strip()
status = rv('订单状态', '订单状态')
trading_income = g('交易收入')
freight_income = g('运费收入')
declared = g('申报价')
qty = parse_int(rv('销售数量'))
# Override for refund-annotated completed orders (set inside branches)
_net_profit_override = None
# Determine format: if trading_income > 0, treat as Allegro
# Otherwise: Temu semi-hosted (申报价 × qty)
if '已取消' in status:
# Cancelled: no income, fixed fee
platform_income = 0.0
platform_expense = abs(g('售后退款'))
elif '已退还' in status:
# Refunded: platform handled the return
# Case 1: 交易收入 > 0 (平台已结算部分款项，扣了运费等)
# net = 交易收入 - abs(售后退款) - 采购成本
# Example: PO-094: 交易收入=1018.58, 售后退款=-191.66, 采购=867.48
# net = 1018.58 - 191.66 - 867.48 = -40.56
# Case 2: 交易收入 = 0 (平台未结算，只扣了采购成本)
# net = 0 - 采购成本
# Example: PO-00852192: 交易收入=0, 采购=828.47
# net = 0 - 828.47 = -828.47
trading_income_val = g('交易收入')
refund_val = g('售后退款') # negative = platform deduction
purchase = g('采购金额')
if trading_income_val > 0:
# Case 1: platform settled some amount, deducting freight/fee
platform_income = trading_income_val
platform_expense = abs(refund_val) # abs of negative = positive fee
else:
# Case 2: no settlement, only purchase cost deducted (already in order_cost)
platform_income = 0.0
platform_expense = 0.0
elif trading_income > 0:
# Completed order (clean or refund-annotated):
# Platform declares the final profit after netting ALL internal fees.
# Use declared profit as override for exact platform match.
platform_income = trading_income + abs(freight_income)
platform_expense = 0.0
_net_profit_override = g('利润')
else:
# Temu semi-hosted or other: income = 申报价 × 销售数量
platform_income = declared * qty
platform_expense = 0.0
# Order cost
purchase = g('采购金额')
packaging = g('包材费')
first_mile = g('头程运费')
last_mile = g('尾程运费')
ad_cost = g('广告成本')
ops_cost = g('运营成本')
warehouse = g('仓库操作费')
other_cost = g('其他成本')
order_cost = (purchase + packaging + first_mile + last_mile +
ad_cost + ops_cost + warehouse + other_cost)
# Other income
other_income = g('订单其他收入')
# Use override for refund-annotated orders (platform handles the fee internally)
net_profit = _net_profit_override if _net_profit_override is not None else \
(platform_income - platform_expense - order_cost + other_income)
# Platform-declared profit
declared_profit = g('利润')
profit_diff = round(net_profit - declared_profit, 2)
return {
'platform_income': round(platform_income, 2),
'platform_expense': round(platform_expense, 2),
'order_cost': round(order_cost, 2),
'other_income': round(other_income, 2),
'net_profit_calc': round(net_profit, 2),
'net_profit_declared': round(declared_profit, 2),
'profit_diff': profit_diff,
}
def detect_format(ws):
"""
Auto-detect file format by scanning headers.
Returns: ('tiktok', header_row, header_map) or ('allegro', header_row, header_map)
For Allegro files, header_map keys use SUFFIXED names for duplicate headers
(e.g. '总计_a', '总计_b', '总计_c') to avoid overwriting.
"""
# First pass: build map with suffix for duplicates
for r in range(1, min(5, ws.max_row + 1)):
row_vals = [ws.cell(r, c).value for c in range(1, ws.max_column + 1)]
row_strs = [str(v).strip() if v is not None else '' for v in row_vals]
# TikTok format
if '订单号' in row_strs and '利润(RMB)' in row_strs and '预估回款金额(RMB)' in row_strs:
header_map = {}
seen = {}
for c, v in enumerate(row_vals):
if v is not None:
k = str(v).strip()
if k in seen:
seen[k] += 1
k = f"{k}_dup{seen[k]}"
else:
seen[k] = 0
header_map[k] = c
return 'tiktok', r, header_map
# Allegro format
if '订单编号' in row_strs and '交易收入' in row_strs:
header_map = {}
seen = {}
for c, v in enumerate(row_vals):
if v is not None:
k = str(v).strip()
if k in seen:
seen[k] += 1
k = f"{k}_dup{seen[k]}"
else:
seen[k] = 0
header_map[k] = c
return 'allegro', r, header_map
# Fallback: TikTok if has RMB fields
row_vals = [ws.cell(1, c).value for c in range(1, ws.max_column + 1)]
header_map = {str(v).strip(): c - 1 for c, v in enumerate(row_vals) if v is not None}
if '预估回款金额(RMB)' in header_map or '已结算金额(RMB)' in header_map:
return 'tiktok', 2, header_map
return 'allegro', 2, header_map
def process_file(filepath, field_map=None):
"""Process Excel file and return structured profit results."""
wb = load_workbook(filepath, data_only=False)
ws = wb.active
fmt, header_row, header_map = detect_format(ws)
# Merge agent-provided field_map: standard_name -> actual column index
# field_map = {"standard_field_name": "actual_column_name_in_this_file"}
unmapped_fields = []
if field_map:
for std_name, actual_name in field_map.items():
if actual_name in header_map:
header_map[std_name] = header_map[actual_name]
else:
unmapped_fields.append((std_name, actual_name))
results = {
'format': fmt,
'total_orders': 0,
'completed_orders': 0,
'cancelled_orders': 0,
'total_income': 0.0,
'total_platform_expense': 0.0,
'total_order_cost': 0.0,
'total_other_income': 0.0,
'total_net_profit_calc': 0.0,
'total_net_profit_declared': 0.0,
'orders': [],
'by_store': {},
'by_platform': {},
'by_sku': {},
}
calc_func = calculate_profit_tiktok if fmt == 'tiktok' else calculate_profit_allegro
for r_idx in range(header_row, ws.max_row + 1):
row = [ws.cell(r_idx, c).value for c in range(1, ws.max_column + 1)]
# Get order ID
order_no = row[header_map.get('订单编号', header_map.get('订单号', -1))]
if order_no is None or str(order_no).strip() in ('', 'None'):
continue
def rv(name, alt=None):
idx = header_map.get(name, -1 if not alt else header_map.get(alt, -1))
if idx < 0: return ''
v = row[idx]
return '' if v is None else str(v).strip()
status = rv('订单状态', '订单状态')
platform = rv('站点', '站点')
store = rv('店铺', '店铺')
sku = rv('产品SKU', '产品SKU')
qty = parse_int(rv('销售数量'))
order_time = rv('下单时间')
financial_time = rv('账务时间', '结算时间')
profit = calc_func(row, header_map, field_map=field_map)
results['total_orders'] += 1
if status in ('交易完成', '已完成', 'completed', 'Completed'):
results['completed_orders'] += 1
elif status in ('已取消', '已撤销', 'cancelled', 'Cancelled'):
results['cancelled_orders'] += 1
results['total_income'] += profit['platform_income']
results['total_platform_expense'] += profit['platform_expense']
results['total_order_cost'] += profit['order_cost']
results['total_other_income'] += profit['other_income']
results['total_net_profit_calc'] += profit['net_profit_calc']
results['total_net_profit_declared'] += profit.get('net_profit_declared', 0)
# by_store
if store not in results['by_store']:
results['by_store'][store] = {'income': 0, 'expense': 0, 'cost': 0, 'profit': 0, 'count': 0}
results['by_store'][store]['income'] += profit['platform_income']
results['by_store'][store]['expense'] += profit['platform_expense']
results['by_store'][store]['cost'] += profit['order_cost']
results['by_store'][store]['profit'] += profit['net_profit_calc']
results['by_store'][store]['count'] += 1
# by_platform
if platform not in results['by_platform']:
results['by_platform'][platform] = {'income': 0, 'expense': 0, 'cost': 0, 'profit': 0, 'count': 0}
results['by_platform'][platform]['income'] += profit['platform_income']
results['by_platform'][platform]['expense'] += profit['platform_expense']
results['by_platform'][platform]['cost'] += profit['order_cost']
results['by_platform'][platform]['profit'] += profit['net_profit_calc']
results['by_platform'][platform]['count'] += 1
# by_sku
if sku and sku not in ('', 'None'):
if sku not in results['by_sku']:
results['by_sku'][sku] = {'income': 0, 'expense': 0, 'cost': 0, 'profit': 0, 'count': 0}
results['by_sku'][sku]['income'] += profit['platform_income']
results['by_sku'][sku]['expense'] += profit['platform_expense']
results['by_sku'][sku]['cost'] += profit['order_cost']
results['by_sku'][sku]['profit'] += profit['net_profit_calc']
results['by_sku'][sku]['count'] += 1
results['orders'].append({
'order_no': str(order_no),
'status': status,
'platform': platform,
'store': store,
'sku': sku,
'qty': qty,
'order_time': order_time,
'financial_time': financial_time,
**profit
})
# Round
for k in ['total_income', 'total_platform_expense', 'total_order_cost',
'total_other_income', 'total_net_profit_calc', 'total_net_profit_declared']:
results[k] = round(results[k], 2)
for cat in ['by_store', 'by_platform', 'by_sku']:
for key in results[cat]:
d = results[cat][key]
for k in d:
d[k] = round(d[k], 2)
# Sort orders by profit (worst first)
results['orders'].sort(key=lambda x: x['net_profit_calc'])
return results
def format_markdown(results):
if 'error' in results:
return f"# Error: {results['error']}"
fmt_label = 'TikTok Shop 170-col' if results['format'] == 'tiktok' else 'Allegro 41-col'
md = []
md.append("# 📊 订单利润分析报告")
md.append(f"**格式:** {fmt_label} | **生成:** {datetime.now().strftime('%Y-%m-%d %H:%M')}\n")
# Summary
total_rev = results['total_income']
total_exp = results['total_platform_expense']
total_cost = results['total_order_cost']
total_profit = results['total_net_profit_calc']
total_declared = results['total_net_profit_declared']
profit_margin = (total_profit / total_rev * 100) if total_rev > 0 else 0
diff = total_profit - total_declared
md.append("## 📋 整体概况")
md.append(f"| 指标 | 数值 |")
md.append(f"|------|------|")
md.append(f"| 总订单 | {results['total_orders']} |")
md.append(f"| 完成 | {results['completed_orders']} |")
md.append(f"| 取消 | {results['cancelled_orders']} |")
md.append(f"| 平台总收入 | ¥{total_rev:,.2f} |")
md.append(f"| 平台总支出 | ¥{total_exp:,.2f} |")
md.append(f"| 订单总成本 | ¥{total_cost:,.2f} |")
md.append(f"| **计算净利润** | **¥{total_profit:,.2f}** |")
if total_declared != 0:
md.append(f"| 平台申报净利润 | ¥{total_declared:,.2f} |")
md.append(f"| 计算误差 | ¥{diff:,.2f} |")
md.append(f"| **净利率** | **{profit_margin:.1f}%** |")
# By platform
if results['by_platform']:
md.append("\n## 🌍 按平台汇总")
md.append("| 平台 | 订单 | 收入 | 平台支出 | 订单成本 | 净利润 | 利润率 |")
md.append("|------|------|------|---------|---------|--------|--------|")
for k, d in sorted(results['by_platform'].items(), key=lambda x: x[1]['profit'], reverse=True):
m = (d['profit'] / d['income'] * 100) if d['income'] != 0 else 0
md.append(f"| {k} | {d['count']} | ¥{d['income']:,.2f} | ¥{d['expense']:,.2f} | ¥{d['cost']:,.2f} | ¥{d['profit']:,.2f} | {m:.1f}% |")
# By store
if results['by_store']:
md.append("\n## 🏪 按店铺汇总")
md.append("| 店铺 | 订单 | 收入 | 平台支出 | 订单成本 | 净利润 | 利润率 |")
md.append("|------|------|------|---------|---------|--------|--------|")
for k, d in sorted(results['by_store'].items(), key=lambda x: x[1]['profit'], reverse=True):
m = (d['profit'] / d['income'] * 100) if d['income'] != 0 else 0
md.append(f"| {k} | {d['count']} | ¥{d['income']:,.2f} | ¥{d['expense']:,.2f} | ¥{d['cost']:,.2f} | ¥{d['profit']:,.2f} | {m:.1f}% |")
# Top/bottom
orders = results['orders']
if orders:
bottom5 = orders[:5]
top5 = orders[-5:][::-1]
md.append("\n## 🔴 亏损最严重的5单")
md.append("| # | 订单号 | 平台 | 店铺 | 收入 | 支出 | 成本 | 净利润 |")
md.append("|---|--------|------|------|------|------|------|--------|")
for i, o in enumerate(bottom5, 1):
md.append(f"| {i} | {o['order_no'][:18]} | {o['platform']} | {o['store'][:8]} | ¥{o['platform_income']:,.2f} | ¥{o['platform_expense']:,.2f} | ¥{o['order_cost']:,.2f} | 🔴 ¥{o['net_profit_calc']:,.2f} |")
md.append("\n## 🟢 盈利最高的5单")
md.append("| # | 订单号 | 平台 | 店铺 | 收入 | 支出 | 成本 | 净利润 |")
md.append("|---|--------|------|------|------|------|------|--------|")
for i, o in enumerate(top5, 1):
md.append(f"| {i} | {o['order_no'][:18]} | {o['platform']} | {o['store'][:8]} | ¥{o['platform_income']:,.2f} | ¥{o['platform_expense']:,.2f} | ¥{o['order_cost']:,.2f} | 🟢 ¥{o['net_profit_calc']:,.2f} |")
return '\n'.join(md)
if __name__ == '__main__':
import json as _json, argparse as _argparse
# -- SkillPay billing check --
ok, payment_url, msg = billing_check()
print(f"[Billing] {msg}")
if not ok:
print(f"[Error] Insufficient balance. Top up: {payment_url}")
exit(1)
_parser = _argparse.ArgumentParser(description='Seller Profit Calculator')
_parser.add_argument('filepath', help='Path to order Excel file')
_parser.add_argument('--json', dest='json_out', metavar='FILE', help='Output JSON file')
_parser.add_argument('--field-map', dest='field_map', metavar='JSON',
help='Field mapping JSON or @filepath')
_parser.add_argument('--markdown', dest='md_out', metavar='FILE', help='Output markdown file')
_args = _parser.parse_args()
_field_map = None
if _args.field_map:
_raw = _args.field_map
if _raw.startswith('@'):
with open(_raw[1:], encoding='utf-8') as _f:
_field_map = _json.load(_f)
else:
_field_map = _json.loads(_raw)
results = process_file(_args.filepath, field_map=_field_map)
_md = format_markdown(results)
print(_md)
if _args.json_out:
with open(_args.json_out, 'w', encoding='utf-8') as f:
json.dump(results, f, ensure_ascii=False, indent=2)
print(f"\nJSON saved to {_args.json_out}")
if _args.md_out:
with open(_args.md_out, 'w', encoding='utf-8') as f:
f.write(_md)
print(f"Markdown saved to {_args.md_out}")
# ═══════════════════════════════════════════════════════════════
# SkillPay Billing Integration / ClawHub
# Deduct on actual execution; API Key/Skill ID from env vars
# ═══════════════════════════════════════════════════════════════
import os
BILLING_API_URL = "https://skillpay.me/api/v1/billing"
CALL_PRICE = 0.01 # USDT per call
def _get_billing_headers() -> dict:
api_key = os.environ.get("SKILL_BILLING_API_KEY", "")
return {"X-API-Key": api_key, "Content-Type": "application/json"}
def _get_skill_id() -> str:
return os.environ.get("SKILL_BILLING_SKILL_ID", "")
def charge_user(user_id: str) -> dict:
"""
Charge per execution. Returns payment_url on insufficient balance.
Dev mode (no API key or network error): returns ok=True to avoid blocking.
"""
api_key = os.environ.get("SKILL_BILLING_API_KEY", "")
if not api_key:
return {"ok": True, "balance": 999.0} # Dev mode
try:
import requests as _requests
resp = _requests.post(
f"{BILLING_API_URL}/charge",
headers=_get_billing_headers(),
json={
"user_id": user_id,
"skill_id": _get_skill_id(),
"amount": CALL_PRICE,
},
timeout=10,
)
data = resp.json()
if data.get("success"):
return {"ok": True, "balance": data.get("balance", 0.0)}
return {
"ok": False,
"balance": data.get("balance", 0.0),
"payment_url": data.get("payment_url"),
}
except Exception:
return {"ok": True, "balance": 999.0} # Dev mode fallback
def billing_check() -> tuple[bool, str, str]:
"""
Execute billing check. Returns (ok, payment_url, message).
- ok=True: charge successful
- ok=False: insufficient balance, payment_url for top-up
"""
user_id = os.environ.get("FEISHU_USER_ID", "")
if not user_id:
return True, "", "no user_id, skip billing"
result = charge_user(user_id)
if result["ok"]:
return True, "", f"charged {CALL_PRICE} USDT, balance: {result['balance']}"
else:
return False, result.get("payment_url", ""), f"insufficient balance"
Connect to IMAP mailboxes to classify emails by urgency, generate multi-language reply suggestions, and push summaries to Feishu in real time.
# Email Intelligence Agent > IMAP Email Read → AI Classification → Reply Suggestions → Feishu Push Summary **Slug:** `email-intelligence-agent` **Price:** $0.01 USDT per call **Author:** 91Skillhub Team --- ## Overview | Feature | Description | |---------|-------------| | IMAP Email Read | Connect to any IMAP mailbox (QQ/163/Enterprise/Gmail, etc.) | | AI Classification | Urgent / Important / Normal / Can Wait | | Reply Suggestions | AI generates multilingual reply suggestions, confirmed by user before sending | | Feishu Push | Summary card pushed to Feishu group or DM | --- ## Tiered Features | Feature | FREE | PRO | |---------|:----:|:---:| | Email read & classification | Keyword-based | AI-powered | | Reply suggestions | — | Yes | | Feishu push | — | Yes | | Max emails per run | 10 | Unlimited | | Price | Free | $0.01/call | --- ## Quick Start ### 1. Configure Email (IMAP) ``` IMAP Config: - Server: imap.example.com - Port: 993 (SSL) - Username: [email protected] - Password: App-specific password (NOT your login password) Tip: Use an App Password from your email provider's security settings ``` ### 2. Configure AI API Supports OpenAI-compatible API (OpenAI, Claude, domestic models, etc.) ### 3. Configure Feishu Push Set up a Feishu bot webhook or user_id to receive push notifications. --- ## Classification Labels | Label | Example Keywords | |-------|----------------| | Urgent | refund, cancel order, complaint, negative review, urgent | | Important | after-sales, repair, payment, invoice, account issue | | Normal | inquiry, price, specs, logistics, shipping | | Can Wait | hello, thank you, goodbye, already handled | --- ## Billing - **Pay-per-call**: $0.0100 USDT per execution via SkillPay.me - **Balance insufficient**: Payment URL returned — top up at `https://skillpay.me/email-intelligence-agent` - **External data flow**: `FEISHU_USER_ID` transmitted to `skillpay.me/api/v1/billing` for billing identification only; not stored or shared with any third party - **Billing model**: Each run = 1 call = $0.0100 USDT - **Privacy**: FEISHU_USER_ID is used solely to identify the billing account; no personal data is retained or shared beyond the payment processor --- ## Required Environment Variables | Variable | Description | |----------|-------------| | `SKILL_BILLING_API_KEY` | SkillPay Builder API Key | | `SKILL_BILLING_SKILL_ID` | SkillPay Skill ID (`email-intelligence-agent`) | | `FEISHU_USER_ID` | User open_id for billing (passed by Feishu runtime) | --- ## Directory Structure ``` email-customer-assistant/ ├── SKILL.md ├── scripts/ │ ├── check_emails.py # Main script + billing │ ├── imap_client.py # IMAP connection wrapper │ ├── classifier.py # Keyword + AI classifier │ ├── reply_generator.py # Reply generator │ └── feishu_pusher.py # Feishu pusher └── requirements.txt ``` --- ## Usage Limits - **Read-only emails** — no sending or deleting - **IMAP access only** — no direct platform API calls - Complies with email provider terms of service --- ## License MIT FILE:scripts/feishu_pusher.py #!/usr/bin/env python3 """ feishu_pusher.py — Feishu push module. Pushes summary cards to Feishu via webhook or user_id. Supports real-time urgent alerts and daily digest summaries. """ import json import time import hashlib import hmac import base64 import requests from datetime import datetime, timedelta from typing import Optional, List, Dict, Any class FeishuPusher: """Feishu push module.""" def __init__(self, config: Dict[str, Any]): self.config = config self.webhook_config = config.get("webhook", {}) self.user_push_config = config.get("user_push", {}) self.urgent_keywords = config.get("urgent_keywords", [ "urgent", "critical", "宕机", "故障", "p0", "p1" ]) self.session = requests.Session() self.session.headers.update({ "Content-Type": "application/json; charset=utf-8" }) def _sign(self, secret: str, timestamp: int) -> str: """Generate Feishu HMAC-SHA256 signature.""" string_to_sign = f"{timestamp}\n{secret}" sign = hmac.new( string_to_sign.encode("utf-8"), digestmod=hashlib.sha256 ).digest() return base64.b64encode(sign).decode("utf-8") def _build_webhook_payload( self, title: str, content: str, ) -> Dict[str, Any]: """Build Feishu interactive card payload.""" return { "msg_type": "interactive", "card": { "header": { "title": {"tag": "plain_text", "content": title}, "template": self._get_card_template(title), }, "elements": [ {"tag": "div", "text": {"tag": "lark_md", "content": content}}, {"tag": "hr"}, { "tag": "note", "elements": [ {"tag": "plain_text", "content": f"{datetime.now().strftime('%Y-%m-%d %H:%M:%S')} · Email Assistant"} ], }, ], }, } def _get_card_template(self, title: str) -> str: """Select card color based on title keywords.""" title_lower = title.lower() if any(kw in title_lower for kw in ["urgent", "critical", "宕机", "故障"]): return "red" elif any(kw in title_lower for kw in ["摘要", "digest", "daily"]): return "blue" elif any(kw in title_lower for kw in ["reply", "回复"]): return "green" return "purple" def push_via_webhook( self, title: str, content: str, webhook_url: Optional[str] = None, ) -> bool: url = webhook_url or self.webhook_config.get("url") if not url: print("[FeishuPusher] ERROR: No webhook URL configured") return False payload = self._build_webhook_payload(title, content) secret = self.webhook_config.get("secret", "") if secret: timestamp = int(time.time() * 1000) sign = self._sign(secret, timestamp) url = f"{url}×tamp={timestamp}&sign={sign}" try: resp = self.session.post(url, json=payload, timeout=10) result = resp.json() if result.get("code") == 0 or result.get("StatusCode") == 0: print(f"[FeishuPusher] OK: Webhook pushed: {title}") return True else: print(f"[FeishuPusher] ERROR: Webhook push failed: {result}") return False except Exception as e: print(f"[FeishuPusher] ERROR: Webhook exception: {e}") return False def push_to_user( self, user_id: str, title: str, content: str, app_token: Optional[str] = None, ) -> bool: open_id = user_id or self.user_push_config.get("user_id") if not open_id: print("[FeishuPusher] ERROR: No user ID configured") return False token = app_token or self.user_push_config.get("app_token") if not token: print("[FeishuPusher] ERROR: No app token configured") return False url = "https://open.feishu.cn/open-apis/im/v1/messages?receive_id_type=open_id" headers = { "Authorization": f"Bearer {token}", "Content-Type": "application/json; charset=utf-8", } payload = { "receive_id": open_id, "msg_type": "interactive", "content": json.dumps(self._build_webhook_payload(title, content)["card"]), } try: resp = self.session.post(url, headers=headers, json=payload, timeout=10) result = resp.json() if result.get("code") == 0: print(f"[FeishuPusher] OK: User push sent: {open_id}") return True else: print(f"[FeishuPusher] ERROR: User push failed: {result}") return False except Exception as e: print(f"[FeishuPusher] ERROR: User push exception: {e}") return False def is_urgent(self, email_data: Dict[str, Any]) -> bool: subject = email_data.get("subject", "").lower() body = email_data.get("body", "").lower() sender = email_data.get("sender", "").lower() return any(kw.lower() in f"{subject} {body} {sender}" for kw in self.urgent_keywords) def push_email( self, email_data: Dict[str, Any], summary: str, category: str, reply_suggestion: Optional[str] = None, force_urgent: bool = False, ) -> bool: is_urgent_email = force_urgent or self.is_urgent(email_data) urgent_tag = "URGENT" if is_urgent_email else "" category_tag = f"[{category}]" if category else "" content_lines = [ f"**From:** {email_data.get('sender', 'Unknown')}", f"**Subject:** {email_data.get('subject', 'No subject')}", f"**Date:** {email_data.get('date', 'Unknown')}", f"{urgent_tag} {category_tag}", "", f"**Summary:**\n{summary}", ] if reply_suggestion: content_lines.extend(["", f"**Reply suggestion:**\n{reply_suggestion}"]) content = "\n".join(content_lines) title = f"{'[URGENT] ' if is_urgent_email else ''}New Email: {email_data.get('subject', 'No subject')}" if self.webhook_config.get("url"): return self.push_via_webhook(title, content) if self.user_push_config.get("user_id"): return self.push_to_user(self.user_push_config["user_id"], title, content) print("[FeishuPusher] ERROR: No push method configured") return False def push_summary(self, emails: List[Dict[str, Any]]) -> bool: """Push a summary of multiple processed emails.""" total = len(emails) urgent_count = sum(1 for e in emails if self.is_urgent(e)) category_stats: Dict[str, int] = {} for e in emails: cat = e.get("category", "Other") category_stats[cat] = category_stats.get(cat, 0) + 1 content_parts = [ f"**Email Summary Report**", "", f"- Total emails: {total}", f"- Urgent: {urgent_count}", "", "**Categories:**", ] for cat, count in sorted(category_stats.items(), key=lambda x: -x[1]): content_parts.append(f" - {cat}: {count}") content_parts.append("") content_parts.append("**Email List:**") for i, email in enumerate(emails[:10], 1): urg_flag = "[URGENT] " if self.is_urgent(email) else " " content_parts.append( f"{urg_flag}**{i}. {email.get('subject', 'No subject')}**" f"\n From: {email.get('sender', 'Unknown')} | Category: {email.get('category', 'Other')}" ) if total > 10: content_parts.append(f"\n_...and {total - 10} more emails_") content = "\n".join(content_parts) title = f"Email Summary · {total} emails" if self.webhook_config.get("url"): return self.push_via_webhook(title, content) if self.user_push_config.get("user_id"): return self.push_to_user(self.user_push_config["user_id"], title, content) print("[FeishuPusher] ERROR: No push method configured") return False if __name__ == "__main__": print("FeishuPusher module — standalone test") FILE:scripts/reply_generator.py #!/usr/bin/env python3 """ reply_generator.py — AI reply generation module. Generates multilingual reply suggestions based on email content. User confirms before sending; safe and controllable. """ import json import time from typing import Optional, List, Dict, Any from datetime import datetime class ReplyGenerator: """AI-powered email reply generator.""" SUPPORTED_LANGUAGES = { "zh": "Chinese", "en": "English", "ja": "Japanese", "ko": "Korean", "zh-tw": "Traditional Chinese", "es": "Spanish", "fr": "French", "de": "German", } def __init__(self, ai_config: Dict[str, Any]): self.ai_config = ai_config self.provider = ai_config.get("provider", "openai") self.api_key = ai_config.get("api_key", "") self.model = ai_config.get("model", "gpt-4o-mini") self.base_url = ai_config.get("base_url", "https://api.openai.com/v1") self.max_tokens = ai_config.get("max_tokens", 1000) self.temperature = ai_config.get("temperature", 0.7) self.timeout = ai_config.get("timeout", 30) def _call_ai_api( self, messages: List[Dict[str, str]], model: Optional[str] = None, **kwargs ) -> str: """Generic AI API call.""" import openai client_kwargs = { "api_key": self.api_key, "timeout": self.timeout, } if self.base_url: client_kwargs["base_url"] = self.base_url client = openai.OpenAI(**client_kwargs) response = client.chat.completions.create( model=model or self.model, messages=messages, max_tokens=kwargs.get("max_tokens", self.max_tokens), temperature=kwargs.get("temperature", self.temperature), ) return response.choices[0].message.content.strip() def _build_system_prompt(self, language: str = "en") -> str: """Build system prompt for reply generation.""" lang_name = self.SUPPORTED_LANGUAGES.get(language, "English") return f"""You are a professional, friendly customer service agent, skilled at writing email replies in {lang_name}. Requirements: 1. Professional, polite, and concise tone 2. Address the customer's specific question 3. Ask for clarification if more information is needed 4. Escalate to the relevant team if you cannot resolve the issue 5. Use emojis appropriately (but not excessive) 6. Keep replies under 200 words 7. Write entirely in {lang_name} Generate a professional email reply based on the email below.""" def generate_reply( self, email_data: Dict[str, Any], language: str = "en", tone: str = "professional", custom_instruction: Optional[str] = None, ) -> str: sender = email_data.get("sender", "") subject = email_data.get("subject", "") body = email_data.get("body", "") category = email_data.get("category", "") user_prompt_lines = [ f"**From:** {sender}", f"**Subject:** {subject}", ] if category: user_prompt_lines.append(f"**Category:** {category}") user_prompt_lines.extend([ f"**Email body:**\n{body[:2000]}", "", ]) if custom_instruction: user_prompt_lines.append(f"**Custom instructions:** {custom_instruction}") if tone != "professional": user_prompt_lines.append(f"**Tone:** {tone}") user_prompt = "\n".join(user_prompt_lines) messages = [ {"role": "system", "content": self._build_system_prompt(language)}, {"role": "user", "content": user_prompt}, ] try: reply = self._call_ai_api(messages) return reply except Exception as e: print(f"[ReplyGenerator] ERROR: Failed to generate reply: {e}") return f"[Auto-reply] Thank you for your email (subject: {subject}). We will respond within 1-2 business days." def generate_reply_multi_language( self, email_data: Dict[str, Any], languages: Optional[List[str]] = None, ) -> Dict[str, str]: if languages is None: languages = ["zh", "en"] results = {} for lang in languages: if lang not in self.SUPPORTED_LANGUAGES: print(f"[ReplyGenerator] WARNING: Unsupported language: {lang}, skipping") continue try: reply = self.generate_reply(email_data, language=lang) results[lang] = reply print(f"[ReplyGenerator] OK: {lang} reply generated") except Exception as e: print(f"[ReplyGenerator] ERROR: {lang} reply failed: {e}") results[lang] = "" return results def confirm_and_preview( self, email_data: Dict[str, Any], reply: str, language: str = "en", ) -> Dict[str, Any]: lang_name = self.SUPPORTED_LANGUAGES.get(language, "English") return { "original_email": { "sender": email_data.get("sender", ""), "subject": email_data.get("subject", ""), "date": email_data.get("date", ""), "category": email_data.get("category", ""), }, "reply": { "content": reply, "language": lang_name, "language_code": language, "generated_at": datetime.now().strftime("%Y-%m-%d %H:%M:%S"), }, "confirm_url": "mailto:{}?subject={}&body={}".format( email_data.get("sender", ""), "Re: " + (email_data.get("subject", "") or ""), reply.replace("\n", "%0A").replace(" ", "%20"), ), "status": "pending_confirmation", } def apply_modification( self, original_reply: str, modification_instruction: str, ) -> str: messages = [ { "role": "system", "content": "You are a text editor. Apply the user's modification instructions to the reply. Return only the modified content, no explanation.", }, { "role": "user", "content": f"**Original reply:**\n{original_reply}\n\n**Modification:**\n{modification_instruction}", }, ] try: return self._call_ai_api(messages) except Exception as e: print(f"[ReplyGenerator] ERROR: Failed to modify reply: {e}") return original_reply def quick_reply( self, email_data: Dict[str, Any], template: str = "acknowledge", ) -> str: templates = { "acknowledge": { "en": f"Thank you for your email regarding '{email_data.get('subject', '')}'. We have received your message and will respond within 1-2 business days.\n\nBest regards,\nCustomer Support", "zh": f"感谢您的来信（主题：{email_data.get('subject', '')}）。我们将在1-2个工作日内回复。\n\n此致\n客服团队", }, "received": { "en": f"Hello, thank you for reaching out. We have received your email and will handle it promptly.\n\nBest regards,\nCustomer Support", "zh": f"您好，感谢您的来信。我们已收到您的邮件，将尽快处理。\n\n此致\n客服团队", }, "investigating": { "en": f"Thank you for bringing this to our attention. We are currently investigating and expect to provide an update within 3 business days.\n\nBest regards,\nCustomer Support", "zh": f"感谢您的来信。我们已了解您的情况，正在处理中，预计3个工作日内给您回复。\n\n此致\n客服团队", }, } lang = email_data.get("language", "en") template_dict = templates.get(template, templates["acknowledge"]) return template_dict.get(lang, template_dict.get("en", "")) if __name__ == "__main__": print("ReplyGenerator module — standalone test") FILE:scripts/classifier.py #!/usr/bin/env python3 """ AI Email Classifier Keyword-based + OpenAI-compatible API classification """ import re from typing import Dict, Optional # Classification keyword configuration CATEGORY_KEYWORDS = { 'urgent': { 'label': '🔴 Urgent', 'priority': 1, 'keywords': [ # 退款相关 '退款', '退款申请', '申请退款', '取消订单', '订单取消', '退钱', '退款处理', '退款中', '退款成功', '退款失败', 'refund', 'cancel order', 'canceled order', # 投诉相关 '投诉', '投诉您', '投诉她', '投诉他', '举报', '差评', '非常不满', '强烈不满', '无法接受', '忍无可忍', 'complaint', 'complain', 'bad review', 'negative feedback', # 紧急 '紧急', '紧急情况', '十万火急', '立刻', '马上', '现在就', 'urgent', 'emergency', 'immediately', 'asap', # 差评预警 '差评预警', '即将差评', '准备差评', '差评', '给差评', ] }, 'important': { 'label': '🟠 Important', 'priority': 2, 'keywords': [ # 售后 '售后', '售后服务', '维修', '维修中', '申请维修', '换货', '申请换货', '换一个新的', '更换', 'after-sales', 'repair', 'maintenance', 'exchange', # 付款 '付款', '支付', '账单', '发票', '付款问题', '支付问题', '未付款', '未支付', '付款失败', '支付失败', '收款', 'payment', 'invoice', 'bill', 'paid', 'unpaid', # 账户 '账户', '账号', '登录不了', '登录不上', '无法登录', '密码错误', '密码忘了', '账户异常', '账号异常', 'account', 'login', 'password', 'locked', ] }, 'normal': { 'label': '🟡 Normal', 'priority': 3, 'keywords': [ # 售前咨询 '咨询', '请问', '问一下', '想问一下', '了解一下', '多少钱', '价格', '报价', '优惠', '折扣', '促销', '规格', '参数', '尺寸', '大小', '颜色', '款式', '怎么买', '在哪里买', '哪里有', '哪有', 'inquiry', 'price', 'cost', 'how much', 'spec', # 物流 '物流', '快递', '发货', '什么时候发货', '多久到', '到货', '签收', '查询物流', '物流查询', '运单号', 'shipping', 'delivery', 'express', 'tracking', # 使用 '怎么用', '使用', '使用方法', '教程', '说明', 'how to use', 'manual', 'guide', ] }, 'deferrable': { 'label': '🟢 Can Wait', 'priority': 4, 'keywords': [ # 问候 '你好', '您好', '嗨', 'hi', 'hello', '早上好', '下午好', 'good morning', 'good afternoon', 'good evening', # 感谢 '谢谢', '感谢', '多谢', '非常感谢', '十分感谢', 'thanks', 'thank you', 'appreciate', # 已处理 '已处理', '已解决', '知道了', '好的', '收到', '没问题', '没有问题了', '已经好了', 'done', 'solved', 'resolved', 'okay', 'ok', # 告别 '再见', '拜拜', '下次见', '回头见', 'goodbye', 'see you', 'bye', ] } } class EmailClassifier: """AI Email Classifier""" def __init__(self, config: dict, tier: str = "FREE"): self.config = config self.tier = tier self.api_endpoint = config.get("api_endpoint", "") self.api_key = config.get("api_key", "") self.model = config.get("model", "gpt-3.5-turbo") self.use_ai = config.get("use_ai", False) self._keyword_cache = self._build_keyword_cache() def _build_keyword_cache(self) -> Dict: """Build keyword cache for fast matching""" cache = {} for category, data in CATEGORY_KEYWORDS.items(): for keyword in data['keywords']: cache[keyword.lower()] = { 'category': category, 'label': data['label'], 'priority': data['priority'] } return cache def classify(self, email: dict) -> str: """ Classify an email. Args: email: dict with subject, body, from, etc. Returns: Category label like '🔴 Urgent' """ subject = email.get('subject', '').lower() body = email.get('body', '').lower() snippet = email.get('snippet', '').lower() # Combine text for matching text = f"{subject} {snippet} {body[:500]}" # Keyword fast-match match_scores = {} for keyword, info in self._keyword_cache.items(): if keyword in text: category = info['category'] priority = info['priority'] if category not in match_scores or priority < match_scores[category]: match_scores[category] = priority # Find highest-priority match if match_scores: best_category = min(match_scores.items(), key=lambda x: x[1])[0] return CATEGORY_KEYWORDS[best_category]['label'] # If no keyword match, use AI classification if self.use_ai and self.api_endpoint: return self._ai_classify(email) # Default to Normal return '🟡 Normal' def _ai_classify(self, email: dict) -> str: """Call AI for classification""" try: import openai client = openai.OpenAI( api_key=self.api_key, base_url=self.api_endpoint ) subject = email.get('subject', '') body = email.get('snippet', '') or email.get('body', '')[:500] prompt = f"""Analyze this email and classify its urgency: Subject: {subject} Body: {body} Options (return label only): - 🔴 Urgent: refund, complaint, negative review, account security — needs immediate action - 🟠 Important: after-sales, payment, account issues — needs same-day response - 🟡 Normal: pre-sale inquiry, logistics, general questions - 🟢 Can Wait: greetings, already handled — can wait Return only the label, nothing else.""" response = client.chat.completions.create( model=self.model, messages=[{"role": "user", "content": prompt}], max_tokens=20, temperature=0 ) result = response.choices[0].message.content.strip() # Verify return value for category, data in CATEGORY_KEYWORDS.items(): if data['label'] in result or category in result.lower(): return data['label'] return '🟡 Normal' except Exception as e: print(f"AI classification failed: {e}") return '🟡 Normal' def classify_with_confidence(self, email: dict) -> Dict: """Classify with confidence score""" subject = email.get('subject', '').lower() body = email.get('body', '').lower() snippet = email.get('snippet', '').lower() text = f"{subject} {snippet} {body[:500]}" match_scores = {} match_counts = {} for keyword, info in self._keyword_cache.items(): if keyword in text: category = info['category'] if category not in match_scores: match_scores[category] = info['priority'] match_counts[category] = 0 match_counts[category] += 1 if match_counts: best_category = max(match_counts.items(), key=lambda x: x[1])[0] count = match_counts[best_category] total = sum(match_counts.values()) confidence = count / total if total > 0 else 0.5 return { 'category': CATEGORY_KEYWORDS[best_category]['label'], 'confidence': min(confidence, 1.0), 'matches': count, 'all_matches': match_counts } return { 'category': '🟡 Normal', 'confidence': 0.5, 'matches': 0, 'all_matches': {} } def get_priority(self, category: str) -> int: """Get category priority (lower number = more urgent)""" for cat, data in CATEGORY_KEYWORDS.items(): if data['label'] == category: return data['priority'] return 3 # default Normal def sort_by_priority(self, emails: list) -> list: """Sort emails by priority""" return sorted( emails, key=lambda e: self.get_priority(e.get('category', '🟡 Normal')) ) FILE:scripts/check_emails.py #!/usr/bin/env python3 """ Email Customer Assistant — IMAP Email Reader + AI Classifier + Reply Suggestions + Feishu Push ClawHub version uses SkillPay per-call billing ($0.01/call). """ import sys import json import argparse import os import requests from datetime import datetime from pathlib import Path from typing import Optional # Local imports sys.path.insert(0, __file__.rsplit("/", 1)[0] if "/" in __file__ else ".") from imap_client import IMAPClient from classifier import EmailClassifier from reply_generator import ReplyGenerator from feishu_pusher import FeishuPusher # ─── Billing constants ───────────────────────────────────────── BILLING_API_URL = "https://skillpay.me/api/v1/billing" CALL_PRICE = 0.01 # USDT per call def _get_billing_headers() -> dict: api_key = os.environ.get("SKILL_BILLING_API_KEY", "") return {"X-API-Key": api_key, "Content-Type": "application/json"} def _get_skill_id() -> str: return os.environ.get("SKILL_BILLING_SKILL_ID", "") def charge_user(user_id: str) -> dict: """ Charge user per call. Returns ok+balance on success, or ok=False+payment_url when balance is insufficient. Dev mode (no API key): returns ok immediately. """ api_key = os.environ.get("SKILL_BILLING_API_KEY", "") if not api_key: return {"ok": True, "balance": 999.0} try: resp = requests.post( f"{BILLING_API_URL}/charge", headers=_get_billing_headers(), json={ "user_id": user_id, "skill_id": _get_skill_id(), "amount": CALL_PRICE, }, timeout=10, ) data = resp.json() if data.get("success"): return {"ok": True, "balance": data.get("balance", 0.0)} return { "ok": False, "balance": data.get("balance", 0.0), "payment_url": data.get("payment_url"), } except Exception: return {"ok": True, "balance": 999.0} class EmailAssistant: """Email Customer Assistant main class.""" def __init__(self, config: dict, api_key: str = ""): self.config = config self.user_id = os.environ.get("FEISHU_USER_ID", "") self.tier = config.get("tier", "FREE") self._imap_client = None self._classifier = None self._reply_gen = None self._pusher = None def _get_imap_client(self): if self._imap_client is None: self._imap_client = IMAPClient(self.config.get("imap", {})) return self._imap_client def _get_classifier(self): if self._classifier is None: self._classifier = EmailClassifier(self.config.get("ai", {}), tier=self.tier) return self._classifier def _get_reply_gen(self): if self._reply_gen is None: self._reply_gen = ReplyGenerator(self.config.get("ai", {})) return self._reply_gen def _get_pusher(self): if self._pusher is None: self._pusher = FeishuPusher(self.config.get("feishu", {})) return self._pusher def process_emails(self, limit: int = 10) -> list: """ Check and process emails. Args: limit: Maximum number of emails to process. Returns: List of processed email results. """ # ── Quota check (FREE tier only) ───────────────────────── # PRO users skip quota check and pay per call. # FREE users have a local quota (managed externally). # This check is informational only; actual gating is via billing below. if self.tier == "FREE": # TODO: integrate with local FREE quota counter pass # ── Billing (PRO only — FREE skips billing) ───────────── if self.user_id and self.tier == "PRO": billing = charge_user(self.user_id) if not billing["ok"]: payment_url = billing.get("payment_url", "") print(f"[WARN] Balance insufficient. Payment: {payment_url}") return [{"error": "Insufficient balance. Please top up.", "payment_url": payment_url}] else: print(f"[INFO] Charged {CALL_PRICE} USDT, balance: {billing['balance']}") # ── Connect to mailbox ───────────────────────────────── imap_client = self._get_imap_client() if not imap_client.connect(): return [{"error": "Failed to connect to mailbox"}] try: emails = imap_client.fetch_unread(limit=limit) if not emails: return [] classifier = self._get_classifier() reply_gen = self._get_reply_gen() pusher = self._get_pusher() results = [] for email in emails: # AI classification category = classifier.classify(email) email["category"] = category # Reply suggestion (PRO only) if self.tier == "PRO": reply_suggestion = reply_gen.generate(email) email["reply_suggestion"] = reply_suggestion results.append(email) # Feishu push (PRO only — always push if configured) if results and self.tier == "PRO": feishu_cfg = self.config.get("feishu", {}) if feishu_cfg.get("enabled"): pusher.push_summary(results) return results finally: imap_client.disconnect() def generate_report(self, results: list) -> str: """Generate a human-readable processing report.""" if not results: return "No new emails." report = ["Email Processing Report", "=" * 30, ""] categories = {} for r in results: cat = r.get("category", "N/A") categories[cat] = categories.get(cat, 0) + 1 report.append(f"Total: {len(results)} emails") report.append("") for cat, count in sorted(categories.items(), key=lambda x: -x[1]): report.append(f" {cat}: {count}") return "\n".join(report) def load_config(config_path: str = "config/config.yaml") -> dict: """Load configuration from YAML or JSON file.""" try: import yaml with open(config_path, "r", encoding="utf-8") as f: return yaml.safe_load(f) except ImportError: pass json_path = config_path.replace(".yaml", ".json") try: with open(json_path, "r", encoding="utf-8") as f: return json.load(f) except FileNotFoundError: return {} def main(): parser = argparse.ArgumentParser(description="Email Customer Assistant") parser.add_argument("--config", "-c", default="config/config.yaml", help="Config file path") parser.add_argument("--limit", "-l", type=int, default=10, help="Max emails to process") parser.add_argument("--json", "-j", action="store_true", help="Output in JSON format") parser.add_argument("--dry-run", action="store_true", help="Check without sending") args = parser.parse_args() config = load_config(args.config) assistant = EmailAssistant(config) results = assistant.process_emails(limit=args.limit) if args.json: print(json.dumps(results, ensure_ascii=False, indent=2)) else: print(assistant.generate_report(results)) if __name__ == "__main__": main() FILE:scripts/imap_client.py #!/usr/bin/env python3 """ IMAP Client - Mailbox connection wrapper Supports: QQ Mail, 163, enterprise mail, Gmail, and more """ import email import imaplib import re from email.header import decode_header from datetime import datetime from typing import Optional, List class IMAPClient: """IMAP Mail Client""" def __init__(self, config: dict): self.config = config self.server = config.get('server', '') self.port = config.get('port', 993) self.username = config.get('username', '') self.password = config.get('password', '') self.use_ssl = config.get('ssl', True) self.connection = None def connect(self) -> bool: """Connect to mailbox server""" try: if self.use_ssl: self.connection = imaplib.IMAP4_SSL(self.server, self.port) else: self.connection = imaplib.IMAP4(self.server, self.port) self.connection.login(self.username, self.password) return True except Exception as e: print(f"Connection failed: {e}") return False def disconnect(self): """Disconnect""" if self.connection: try: self.connection.logout() except Exception: pass self.connection = None def fetch_unread(self, folder: str = "INBOX", limit: int = 10) -> List[dict]: """Fetch unread emails. Args: folder: folder name (default INBOX) limit: max number of emails Returns: list of emails """ if not self.connection: return [] try: # Select inbox status, _ = self.connection.select(folder) if status != 'OK': return [] # Search for unread emails status, messages = self.connection.search(None, 'UNSEEN') if status != 'OK': return [] message_ids = messages[0].split() if not message_ids: return [] # Limit count message_ids = message_ids[-limit:] if len(message_ids) > limit else message_ids emails = [] for msg_id in message_ids: email_data = self._fetch_email(msg_id) if email_data: emails.append(email_data) return emails except Exception as e: print(f"Failed to fetch emails: {e}") return [] def fetch_all(self, folder: str = "INBOX", limit: int = 50, since_date: Optional[str] = None) -> List[dict]: """Fetch emails with optional date filter. Args: folder: folder name limit: max count since_date: start date (format: "01-Jan-2024") Returns: list of emails """ if not self.connection: return [] try: status, _ = self.connection.select(folder) if status != 'OK': return [] # Build search criteria criteria = 'ALL' if since_date: criteria = f'SINCE {since_date}' status, messages = self.connection.search(None, criteria) if status != 'OK': return [] message_ids = messages[0].split() if not message_ids: return [] message_ids = message_ids[-limit:] if len(message_ids) > limit else message_ids emails = [] for msg_id in message_ids: email_data = self._fetch_email(msg_id) if email_data: emails.append(email_data) return emails except Exception as e: print(f"Failed to fetch emails: {e}") return [] def _fetch_email(self, msg_id) -> Optional[dict]: """Fetch single email content""" try: status, msg_data = self.connection.fetch(msg_id, '(RFC822)') if status != 'OK': return None raw_email = msg_data[0][1] msg = email.message_from_bytes(raw_email) # Parse subject subject = self._decode_header(msg.get('Subject', '')) # Parse sender from_ = self._decode_header(msg.get('From', '')) # Parse date date_str = msg.get('Date', '') date = self._parse_date(date_str) # Parse body body = self._get_body(msg) # Email ID message_id = msg.get('Message-ID', str(msg_id)) return { 'message_id': message_id, 'subject': subject, 'from': from_, 'date': date, 'body': body, 'snippet': body[:200] if body else '', 'raw': raw_email } except Exception as e: print(f"Failed to parse email {msg_id}: {e}") return None def _decode_header(self, header: str) -> str: """Decode mail header""" if not header: return '' decoded_parts = decode_header(header) result = [] for part, encoding in decoded_parts: if isinstance(part, bytes): try: result.append(part.decode(encoding or 'utf-8', errors='replace')) except Exception: result.append(part.decode('utf-8', errors='replace')) else: result.append(part) return ''.join(result) def _parse_date(self, date_str: str) -> str: """Parse email date""" if not date_str: return datetime.now().isoformat() try: dt = email.utils.parsedate_to_datetime(date_str) return dt.isoformat() except Exception: return datetime.now().isoformat() def _get_body(self, msg) -> str: """Extract email body""" body = '' if msg.is_multipart(): for part in msg.walk(): content_type = part.get_content_type() content_disposition = str(part.get('Content-Disposition', '')) # Prefer plain text if content_type == 'text/plain' and 'attachment' not in content_disposition: try: charset = part.get_content_charset() or 'utf-8' body = part.get_payload(decode=True).decode(charset, errors='replace') break except Exception: pass # Fall back to HTML -> text elif content_type == 'text/html' and 'attachment' not in content_disposition and not body: try: charset = part.get_content_charset() or 'utf-8' html = part.get_payload(decode=True).decode(charset, errors='replace') body = self._html_to_text(html) except Exception: pass else: try: charset = msg.get_content_charset() or 'utf-8' body = msg.get_payload(decode=True).decode(charset, errors='replace') except Exception: pass return body.strip() def _html_to_text(self, html: str) -> str: """Convert HTML to plain text""" # Remove scripts and styles html = re.sub(r'<script[^>]*>.*?</script>', '', html, flags=re.DOTALL | re.IGNORECASE) html = re.sub(r'<style[^>]*>.*?</style>', '', html, flags=re.DOTALL | re.IGNORECASE) # Remove tags html = re.sub(r'<[^>]+>', ' ', html) # Clean whitespace html = re.sub(r'\s+', ' ', html) return html.strip() def mark_as_read(self, msg_id: bytes) -> bool: """Mark email as read""" if not self.connection: return False try: status, _ = self.connection.store(msg_id, '+FLAGS', '\\Seen') return status == 'OK' except Exception: return False def mark_as_unread(self, msg_id: bytes) -> bool: """Mark email as unread""" if not self.connection: return False try: status, _ = self.connection.store(msg_id, '-FLAGS', '\\Seen') return status == 'OK' except Exception: return False