@clawhub-zero2ai-hub-7e93ac7eec
Daily fact extraction from AI agent session history into a persistent learned.md memory file
---
name: skill-dreaming-extractor
version: 1.0.0
description: "Daily fact extraction from AI agent session history into a persistent learned.md memory file"
metadata:
openclaw:
requires: { bins: ["python3"] }
---
# Skill: Dreaming Extractor
Reads yesterday's real agent conversations, extracts structured facts using an LLM, and appends them to `memory/learned.md`. Designed to run daily via cron — builds a persistent, searchable knowledge base from your agent's actual interactions.
## What it does
- Scans yesterday's session JSONL files from `~/.openclaw/agents/`
- Filters out system noise (startup sequences, dreaming sessions, tool output)
- Extracts structured facts: decisions made, problems solved, config changes, corrections
- Appends extracted facts to `memory/learned.md` with confidence scores and source citations
- Enforces a daily token budget to cap cost
## Execution Steps
When triggered (by cron or manually), follow these steps:
### Step 1: Collect sessions
```bash
python3 skills/skill-dreaming-extractor/scripts/collect.py
```
Outputs a clean text file at `memory/.dreams/extraction-input/YYYY-MM-DD.txt` containing yesterday's real conversations. If output is "No real sessions found", stop here.
### Step 2: Check budget
```bash
python3 skills/skill-dreaming-extractor/scripts/budget.py --check
```
If output is "BUDGET EXCEEDED", stop. Do not proceed.
### Step 3: Extract facts
Read the collected input. For each meaningful exchange, extract structured facts:
```json
{
"facts": [
{
"subject": "specific entity or concept (e.g., 'API endpoint', 'deployment target')",
"predicate": "what happened or was decided (e.g., 'was deployed to', 'was updated to')",
"object": "the outcome or target (e.g., 'production on 2026-04-14', 'v2 with retry logic')",
"date": "YYYY-MM-DD",
"confidence": 0.0-1.0,
"source": "session-id:line-range"
}
]
}
```
#### Extraction Rules
**INCLUDE:**
- Decisions made (architectural, business, operational)
- Problems solved and how
- Configuration changes with rationale
- New capabilities deployed or verified
- Corrections or feedback from the user
- External facts learned (API changes, service incidents, pricing)
**REJECT (scaffolding — never extract):**
- Session startup greetings / "back online" messages
- Tool call results (file contents, grep output, git status)
- Progress updates ("working on X", "let me check")
- Repetitions of the same fact across sessions
- Vague themes with no concrete data
- Meta-conversation about the conversation itself
- Token counts, cost reports, cache stats
**Quality gates:**
- Each fact MUST have at least 3 concrete tokens (subject + predicate + date minimum)
- Confidence 0.9+ = directly stated by user or confirmed by system output
- Confidence 0.7-0.89 = strongly implied, single source
- Confidence 0.5-0.69 = inferred, use sparingly
- Below 0.5 = do not extract
### Step 4: Write to learned.md
Append extracted facts to `memory/learned.md`:
```markdown
## Learned — YYYY-MM-DD
- **[subject]** [predicate] [object] | confidence: X.XX | source: [session-id:lines]
- ...
```
If the file doesn't exist, create it with a header first.
### Step 5: Log budget
```bash
python3 skills/skill-dreaming-extractor/scripts/budget.py --log --facts-count N
```
Where N = number of facts extracted.
### Step 6: Report
Output a summary:
```
Dreaming extraction complete for YYYY-MM-DD:
- Sessions processed: X
- Facts extracted: Y
- Confidence range: X.XX - X.XX
- Budget remaining: $X.XX / $2.00
```
## Manual Run
```bash
python3 skills/skill-dreaming-extractor/scripts/collect.py --date 2026-04-14
# Then follow steps 2-6
```
## Cost
- Estimated: ~$0.50–1.50/day (Sonnet, ~30–200K chars of session history)
- Hard cap: $2.00/day (configurable in budget.py)
- Monthly projection: ~$15–45
## Cron Setup
```bash
openclaw cron add "Dreaming Extractor" "3 3 * * *" "Run task spec: skills/skill-dreaming-extractor/SKILL.md"
```
Fires daily at 03:03 UTC (after sessions have closed for the day).
## Success Metrics
| Metric | Target |
|---|---|
| Facts per cycle with 3+ concrete tokens | ≥ 3 |
| Confidence variance (std dev) | ≥ 0.15 |
| Scaffolding ratio | < 10% |
| Citations per week (facts actually recalled) | ≥ 1 |
If citation count stays at 0 for 30 days, the system isn't adding value — retire it.
FILE:scripts/budget.py
#!/usr/bin/env python3
"""
Dreaming v2 — Budget Meter
Tracks daily extraction costs against a $2/day hard cap.
Estimates cost from input size (conservative: $0.003 per 1K input tokens + $0.015 per 1K output tokens).
Usage:
python3 budget.py --check # Check if today's budget allows extraction
python3 budget.py --log --facts-count 5 # Log a completed extraction
python3 budget.py --status # Show budget status
python3 budget.py --reset # Reset today's budget (emergency only)
"""
import argparse
import json
import os
import sys
from datetime import datetime, timezone
from pathlib import Path
WORKSPACE = Path(os.environ.get("OPENCLAW_WORKSPACE", os.path.expanduser("~/.openclaw/workspace")))
BUDGET_FILE = WORKSPACE / "memory" / ".dreams" / "extraction-budget.json"
DAILY_CAP = 2.00 # USD
# Sonnet 4.6 pricing (conservative estimates via OpenRouter)
COST_PER_1K_INPUT = 0.003
COST_PER_1K_OUTPUT = 0.015
# Rough estimate: extraction produces ~100 output tokens per fact, plus ~500 overhead
EST_OUTPUT_TOKENS_PER_FACT = 100
EST_OUTPUT_OVERHEAD = 500
def load_budget() -> dict:
if BUDGET_FILE.exists():
try:
return json.loads(BUDGET_FILE.read_text())
except (json.JSONDecodeError, OSError):
pass
return {"version": 1, "days": {}}
def save_budget(data: dict):
BUDGET_FILE.parent.mkdir(parents=True, exist_ok=True)
BUDGET_FILE.write_text(json.dumps(data, indent=2))
def today_key() -> str:
return datetime.now(timezone.utc).strftime("%Y-%m-%d")
def get_today_spend(data: dict) -> float:
day = data.get("days", {}).get(today_key(), {})
return day.get("total_cost_usd", 0.0)
def estimate_cost(input_chars: int, facts_count: int) -> float:
input_tokens = input_chars / 4 # rough char-to-token ratio
output_tokens = facts_count * EST_OUTPUT_TOKENS_PER_FACT + EST_OUTPUT_OVERHEAD
cost = (input_tokens / 1000 * COST_PER_1K_INPUT) + (output_tokens / 1000 * COST_PER_1K_OUTPUT)
return round(cost, 4)
def cmd_check(data: dict):
spent = get_today_spend(data)
remaining = DAILY_CAP - spent
if remaining <= 0:
print(f"BUDGET EXCEEDED — spent .2f / .2f today")
sys.exit(1)
print(f"OK — .2f remaining (spent .2f / .2f)")
sys.exit(0)
def cmd_log(data: dict, facts_count: int):
key = today_key()
# Estimate cost from the extraction input file
input_dir = WORKSPACE / "memory" / ".dreams" / "extraction-input"
input_file = input_dir / f"{key}.txt"
input_chars = 0
if input_file.exists():
input_chars = len(input_file.read_text())
# If no file for today, check yesterday (extraction runs at 03:00 for previous day)
else:
from datetime import timedelta
yesterday = (datetime.now(timezone.utc) - timedelta(days=1)).strftime("%Y-%m-%d")
yesterday_file = input_dir / f"{yesterday}.txt"
if yesterday_file.exists():
input_chars = len(yesterday_file.read_text())
cost = estimate_cost(input_chars, facts_count)
if "days" not in data:
data["days"] = {}
if key not in data["days"]:
data["days"][key] = {"runs": [], "total_cost_usd": 0.0}
data["days"][key]["runs"].append({
"timestamp": datetime.now(timezone.utc).isoformat()[:19] + "Z",
"facts_extracted": facts_count,
"input_chars": input_chars,
"estimated_cost_usd": cost,
})
data["days"][key]["total_cost_usd"] = round(
sum(r["estimated_cost_usd"] for r in data["days"][key]["runs"]), 4
)
save_budget(data)
spent = data["days"][key]["total_cost_usd"]
print(f"Logged: {facts_count} facts, ~.4f estimated cost")
print(f"Today total: .4f / .2f")
# Prune entries older than 60 days
cutoff = (datetime.now(timezone.utc) - __import__("datetime").timedelta(days=60)).strftime("%Y-%m-%d")
data["days"] = {k: v for k, v in data["days"].items() if k >= cutoff}
save_budget(data)
def cmd_status(data: dict):
key = today_key()
spent = get_today_spend(data)
remaining = DAILY_CAP - spent
# Last 7 days
days = sorted(data.get("days", {}).items(), reverse=True)[:7]
print(f"Budget status — {key}")
print(f" Today: .4f / .2f (.4f remaining)")
print(f" Last 7 days:")
total_week = 0
total_facts = 0
for d, info in days:
day_cost = info.get("total_cost_usd", 0)
day_facts = sum(r.get("facts_extracted", 0) for r in info.get("runs", []))
total_week += day_cost
total_facts += day_facts
print(f" {d}: .4f ({day_facts} facts)")
print(f" Week total: .4f ({total_facts} facts)")
def cmd_reset(data: dict):
key = today_key()
if key in data.get("days", {}):
del data["days"][key]
save_budget(data)
print(f"Reset budget for {key}")
def main():
parser = argparse.ArgumentParser(description="Dreaming v2 budget meter")
parser.add_argument("--check", action="store_true", help="Check if budget allows extraction")
parser.add_argument("--log", action="store_true", help="Log a completed extraction")
parser.add_argument("--status", action="store_true", help="Show budget status")
parser.add_argument("--reset", action="store_true", help="Reset today's budget")
parser.add_argument("--facts-count", type=int, default=0, help="Number of facts extracted (for --log)")
args = parser.parse_args()
data = load_budget()
if args.check:
cmd_check(data)
elif args.log:
cmd_log(data, args.facts_count)
elif args.status:
cmd_status(data)
elif args.reset:
cmd_reset(data)
else:
parser.print_help()
if __name__ == "__main__":
main()
FILE:scripts/collect.py
#!/usr/bin/env python3
"""
Dreaming v2 — Session Collector
Reads yesterday's raw session JSONL files, filters out dreaming noise
and system sessions, outputs a clean text file for extraction.
Usage: python3 collect.py [--date YYYY-MM-DD] [--agents-dir DIR] [--output FILE]
"""
import argparse
import json
import os
import sys
from datetime import datetime, timedelta, timezone
from pathlib import Path
AGENTS_DIR = Path(os.environ.get("OPENCLAW_AGENTS_DIR", os.path.expanduser("~/.openclaw/agents")))
OUTPUT_DIR = Path(os.environ.get("OPENCLAW_WORKSPACE", os.path.expanduser("~/.openclaw/workspace"))) / "memory" / ".dreams" / "extraction-input"
# Patterns in first user message that indicate dreaming/system noise
NOISE_PATTERNS = [
"write a dream diary entry",
"dream diary entry from these memory fragments",
"continue where you left off. the previous model attempt failed",
"run your session startup sequence",
]
# Max characters per session to include (prevent one massive session from dominating)
MAX_CHARS_PER_SESSION = 15000
# Max total output chars (~50k tokens worth at ~4 chars/token)
MAX_TOTAL_CHARS = 200000
def is_noise_session(first_user_text: str) -> bool:
lower = first_user_text.lower()
return any(p in lower for p in NOISE_PATTERNS)
def extract_messages(jsonl_path: Path) -> list[dict]:
"""Extract user/assistant text messages from a session JSONL."""
messages = []
session_date = None
try:
with open(jsonl_path) as f:
for line in f:
if not line.strip():
continue
entry = json.loads(line)
if entry.get("type") == "session":
session_date = entry.get("timestamp", "")[:10]
if entry.get("type") != "message":
continue
msg = entry.get("message", {})
role = msg.get("role")
if role not in ("user", "assistant"):
continue
content = msg.get("content", [])
if isinstance(content, str):
texts = [content]
elif isinstance(content, list):
texts = [c.get("text", "") for c in content if c.get("type") == "text" and c.get("text")]
else:
continue
for text in texts:
if text.strip():
messages.append({
"role": role,
"text": text.strip(),
"timestamp": entry.get("timestamp", ""),
})
except (json.JSONDecodeError, OSError) as e:
print(f" WARN: failed to read {jsonl_path.name}: {e}", file=sys.stderr)
return []
return messages
def collect_sessions(target_date: str, agents_dir: Path) -> list[dict]:
"""Collect all real (non-noise) sessions for a given date."""
sessions = []
for agent_dir in sorted(agents_dir.iterdir()):
sessions_dir = agent_dir / "sessions"
if not sessions_dir.is_dir():
continue
for jsonl_file in sorted(sessions_dir.glob("*.jsonl")):
# Skip checkpoint files
if ".checkpoint." in jsonl_file.name:
continue
# Check file modification date matches target
mtime = datetime.fromtimestamp(jsonl_file.stat().st_mtime, tz=timezone.utc)
if mtime.strftime("%Y-%m-%d") != target_date:
# Also check session start date from first line
try:
with open(jsonl_file) as f:
first = json.loads(f.readline())
sess_date = first.get("timestamp", "")[:10]
if sess_date != target_date:
continue
except (json.JSONDecodeError, OSError):
continue
messages = extract_messages(jsonl_file)
if not messages:
continue
# Check if first user message is noise
first_user = next((m for m in messages if m["role"] == "user"), None)
if first_user and is_noise_session(first_user["text"]):
continue
# Filter to text messages only, truncate
text_parts = []
char_count = 0
for m in messages:
chunk = f"[{m['role']}] {m['text']}"
if char_count + len(chunk) > MAX_CHARS_PER_SESSION:
text_parts.append(f"[... truncated at {MAX_CHARS_PER_SESSION} chars ...]")
break
text_parts.append(chunk)
char_count += len(chunk)
sessions.append({
"session_id": jsonl_file.stem,
"agent": agent_dir.name,
"messages": text_parts,
"message_count": len(messages),
"char_count": char_count,
})
return sessions
def format_output(sessions: list[dict], target_date: str) -> str:
"""Format collected sessions into extraction input."""
lines = [
f"# Session Extraction Input — {target_date}",
f"# Collected: {datetime.now(timezone.utc).isoformat()[:19]}Z",
f"# Sessions: {len(sessions)} (after noise filtering)",
f"# Total conversations: {sum(s['message_count'] for s in sessions)}",
"",
]
total_chars = 0
for i, sess in enumerate(sessions, 1):
header = f"--- SESSION {i}/{len(sessions)} [{sess['agent']}:{sess['session_id'][:8]}] ({sess['message_count']} messages) ---"
lines.append(header)
for msg in sess["messages"]:
if total_chars + len(msg) > MAX_TOTAL_CHARS:
lines.append(f"[... total output truncated at {MAX_TOTAL_CHARS} chars ...]")
return "\n".join(lines)
lines.append(msg)
total_chars += len(msg)
lines.append("")
return "\n".join(lines)
def main():
parser = argparse.ArgumentParser(description="Collect sessions for dreaming extraction")
parser.add_argument("--date", default=None, help="Target date YYYY-MM-DD (default: yesterday)")
parser.add_argument("--agents-dir", default=str(AGENTS_DIR), help="Agents directory")
parser.add_argument("--output", default=None, help="Output file path")
parser.add_argument("--stats-only", action="store_true", help="Print stats without writing")
args = parser.parse_args()
if args.date:
target_date = args.date
else:
yesterday = datetime.now(timezone.utc) - timedelta(days=1)
target_date = yesterday.strftime("%Y-%m-%d")
agents_dir = Path(args.agents_dir)
if not agents_dir.exists():
print(f"ERROR: agents dir not found: {agents_dir}", file=sys.stderr)
sys.exit(1)
print(f"Collecting sessions for {target_date}...", file=sys.stderr)
sessions = collect_sessions(target_date, agents_dir)
if not sessions:
print(f"No real sessions found for {target_date}", file=sys.stderr)
sys.exit(0)
total_msgs = sum(s["message_count"] for s in sessions)
total_chars = sum(s["char_count"] for s in sessions)
print(f"Found {len(sessions)} sessions, {total_msgs} messages, ~{total_chars:,} chars", file=sys.stderr)
if args.stats_only:
for s in sessions:
print(f" {s['agent']}:{s['session_id'][:8]} — {s['message_count']} msgs, {s['char_count']:,} chars")
sys.exit(0)
output_text = format_output(sessions, target_date)
output_dir = OUTPUT_DIR
output_dir.mkdir(parents=True, exist_ok=True)
if args.output:
output_path = Path(args.output)
else:
output_path = output_dir / f"{target_date}.txt"
output_path.write_text(output_text)
print(f"Written to {output_path} ({len(output_text):,} chars)", file=sys.stderr)
print(str(output_path))
if __name__ == "__main__":
main()
Audits e-commerce sites for AI shopping agent discoverability, scoring product data, APIs, feeds, checkout compatibility, and description quality with priori...
# Skill: B2A Agent-Discoverability Audit
Version: 1.0 | Created: 2026-03-30
---
## Purpose
Audit any e-commerce site (WooCommerce/Shopify/custom) for AI agent discoverability and shopping readiness.
Outputs a scored report with prioritized fixes to make products visible to AI shopping agents (Google AI Shopping, Perplexity Shopping, AppFunctions agents).
## When to Use
- Before launching a new product line
- When traffic is low despite active inventory
- As part of B2A (Business-to-Agent) commerce infrastructure setup
- Quarterly storefront health check
## Inputs
- `SITE_URL` — the storefront URL to audit (e.g., https://example.com)
- `STORE_TYPE` — WooCommerce | Shopify | Custom (default: WooCommerce)
## Outputs
- Scored report appended to a notes file
- 5-dimension score card (0–10 each)
- Prioritized fix list: 🔴 HIGH / 🟡 MEDIUM / 🟢 LOW
## Audit Dimensions
### 1. Structured Data (schema.org/Product)
Check for:
- `@type: Product` with `name`, `sku`, `url`, `image`, `description`
- `Offer` block: `price`, `priceCurrency`, `availability`
- `brand` with `@type: Brand`
- `gtin` / `gtin13` / `gtin8` — required by Google Shopping and AI agent catalogs
- `mpn` (manufacturer part number)
- `aggregateRating` + `reviewCount`
- `color`, `material`, `weight` product attributes
Tool: Fetch any product page, inspect JSON-LD blocks.
### 2. Machine-Readable Catalog API
WooCommerce: Test `GET /wp-json/wc/store/v1/products` (public, no auth)
Shopify: Test `GET /products.json` (public, no auth)
Check: HTTP status, fields returned (id, name, sku, prices, images, in_stock)
### 3. Google Merchant Center / Shopping Feed
Test common feed URLs:
- `/feed=products`
- `/google-shopping-feed/`
- WooCommerce: Google Listings & Ads plugin
- Shopify: native GMC integration
### 4. Agent Checkout Compatibility
WooCommerce: Test `POST /wp-json/wc/store/v1/cart/add-item` availability
Check for headless cart API, programmatic checkout support
### 5. Product Descriptions (AI Intent Matching)
Evaluate description quality:
- Length > 100 words with specs
- Includes: connection type, battery life, dimensions, weight, use cases
- No tagline-only descriptions
## Scoring Rubric
| Score | Meaning |
|-------|---------|
| 9–10 | Production-ready for AI agents |
| 7–8 | Good, minor fixes needed |
| 5–6 | Partial — key gaps present |
| 3–4 | Visible but not optimized |
| 0–2 | Invisible to AI shopping agents |
## Report Format
```
## [YYYY-MM-DD] B2A Agent-Discoverability Audit — {SITE_URL}
### Score Card
| Dimension | Score | Notes |
...
### 🔴 HIGH Priority
...
### 🟡 MEDIUM Priority
...
### 🟢 LOW Priority (future)
...
Overall: X/10
```
## Key Insight (from production use)
The biggest gap most e-commerce sites have: WooCommerce/Shopify APIs are browsable by agents, but without GTIN identifiers and a Google Merchant Center feed, products are **invisible to Google AI Shopping, Perplexity Shopping, and Android AppFunctions agents**. The fix is a free WooCommerce plugin + 30 minutes of product data entry. High leverage, low effort.
Monitors new WooCommerce processing orders, auto-copies missing shipping addresses from billing, and emits one alert per new order for automation.
# skill-woocommerce-order-guard
## Description
Watches WooCommerce for new 'processing' orders and auto-fixes missing shipping addresses by copying billing → shipping. Deduplicates alerts so each new order triggers exactly once. Designed to run as a cron heartbeat or webhook trigger.
## Use case
WooCommerce sometimes receives orders where customers skip the shipping address form. This script ensures every processing order has a valid shipping address before fulfillment, and emits a signal for downstream automation (Telegram alert, CJ fulfillment, etc.)
## Usage
```bash
python3 scripts/order-guard.py
```
Optional args:
```bash
python3 scripts/order-guard.py \
--creds /path/to/woo-api.json \
--storage /path/to/fulfilled_orders.json
```
## Output
- `HEARTBEAT_OK` — no new orders
- `NEW_ORDER_ID: <id>` — one line per new order (pipe to Telegram alert, CJ submit, etc.)
## Configuration
### `woo-api.json`
```json
{
"url": "https://yourstore.com",
"consumerKey": "ck_...",
"consumerSecret": "cs_..."
}
```
Generate keys: WooCommerce → Settings → Advanced → REST API.
### Storage file
Local JSON at `--storage` path (default: `~/.openclaw/workspace/memory/fulfilled_orders.json`).
```json
{"alerted_orders": [12345, 12346]}
```
Prevents duplicate alerts across runs.
## Example cron (every 5 min)
```
*/5 * * * * python3 /path/to/skill-woocommerce-order-guard/scripts/order-guard.py >> /tmp/order-guard.log 2>&1
```
## Dependencies
- Python 3.x
- `requests` library (`pip install requests`)
## Logic flow
1. Fetch all `processing` orders (last 20)
2. Filter out already-alerted order IDs
3. For each new order: if shipping.address_1 is empty → PUT billing address to shipping
4. Print `NEW_ORDER_ID: <id>` and save to dedup store
FILE:scripts/order-guard.py
#!/usr/bin/env python3
"""
WooCommerce Order Guard
- Fetches all 'processing' orders
- Copies billing address → shipping when shipping address is empty
- Deduplicates alerts via a local JSON store
- Prints NEW_ORDER_ID: <id> for each new order (for cron/webhook upstream)
- Prints HEARTBEAT_OK if no new orders
Usage:
python3 order-guard.py [--creds /path/to/woo-api.json] [--storage /path/to/fulfilled_orders.json]
"""
import requests
import json
import os
import argparse
DEFAULT_CREDS = os.path.expanduser("~/woo-api.json")
DEFAULT_STORAGE = os.path.expanduser("~/.openclaw/workspace/memory/fulfilled_orders.json")
def load_creds(path):
with open(path) as f:
return json.load(f)
def load_storage(path):
if os.path.exists(path):
with open(path) as f:
return json.load(f).get("alerted_orders", [])
return []
def save_storage(path, alerted_orders):
with open(path, 'w') as f:
json.dump({"alerted_orders": alerted_orders}, f)
def copy_billing_to_shipping(order):
billing = order.get('billing', {})
return {
"first_name": billing.get('first_name'),
"last_name": billing.get('last_name'),
"company": billing.get('company'),
"address_1": billing.get('address_1'),
"address_2": billing.get('address_2'),
"city": billing.get('city'),
"state": billing.get('state'),
"postcode": billing.get('postcode'),
"country": billing.get('country'),
}
def main():
parser = argparse.ArgumentParser()
parser.add_argument('--creds', default=DEFAULT_CREDS)
parser.add_argument('--storage', default=DEFAULT_STORAGE)
args = parser.parse_args()
creds = load_creds(args.creds)
url = creds['url']
auth = (creds['consumerKey'], creds['consumerSecret'])
alerted_orders = load_storage(args.storage)
resp = requests.get(f"{url}/wp-json/wc/v3/orders?status=processing&per_page=20", auth=auth)
resp.raise_for_status()
orders = resp.json()
new_orders = [o for o in orders if o['id'] not in alerted_orders]
if not new_orders:
print("HEARTBEAT_OK")
return
for order in new_orders:
order_id = order['id']
shipping = order.get('shipping', {})
if not shipping.get('address_1'):
update_data = {"shipping": copy_billing_to_shipping(order)}
requests.put(f"{url}/wp-json/wc/v3/orders/{order_id}", auth=auth, json=update_data)
alerted_orders.append(order_id)
print(f"NEW_ORDER_ID: {order_id}")
save_storage(args.storage, alerted_orders)
if __name__ == "__main__":
main()
Paper trading monitors for SMC (Smart Money Concepts) + Macro Rotation strategies. Includes swing (4H BoS+FVG), day (1H BoS+FVG+CVD), coordinated 8D/2S orche...
---
name: skill-smc-multi-strategy-paper-trader
description: Paper trading monitors for SMC (Smart Money Concepts) + Macro Rotation strategies. Includes swing (4H BoS+FVG), day (1H BoS+FVG+CVD), coordinated 8D/2S orchestration, STPI/MTPI-gated monitors, macro rotation (LTPI/MTPI + RS Tournament), and multi-factor regime scorer. ATR-based SL/TP, z-score filters, orchestrator-lock. Binance public API only — no credentials needed.
author: Zero2Ai-hub
version: 2.0.0
emoji: 📈
tags: [crypto, trading, paper-trading, smc, smart-money, orchestrator, backtested, macro, regime-filter]
---
# SMC Multi-Strategy Paper Trader v2.0
Paper trading system implementing **Smart Money Concepts** across five strategy types plus a macro rotation overlay and regime filter. All data from Binance public API — no account or API key required.
## What's New in v2.0
- **Macro Rotation** (`macro-rotation.js`) — LTPI/MTPI + Relative Strength Tournament, daily rebalance, $10K capital
- **Regime Scorer** (`regime-scorer.js`) — 15 TA indicators + 5 FRED liquidity factors → STPI/MTPI/LTPI scores
- **v6 Day Monitor** (`paper-monitor-v6.js`) — SMC+TA≥5 gated by STPI ≥ 0.5
- **Swing v2** (`paper-monitor-swing-v2.js`) — Swing strategy gated by MTPI ≥ 0.5
## Architecture
```
Binance Public API (Kline, Ticker, Funding, OI)
↓
regime-scorer.js → regime.json (STPI, MTPI, LTPI)
↓
paper-monitor-[strategy].js
├── Fetch candles (1H/4H/1D)
├── Detect Break of Structure (BoS)
├── Find Fair Value Gaps (FVG)
├── Z-score: Volume, TA, FVG quality
├── ATR-based SL / TP / Trailing stop
├── Regime gate: STPI/MTPI threshold check
├── Orchestrator lock read/write
└── portfolio-[strategy].json (P&L tracking)
macro-rotation.js
├── Daily candles + 7d close for all tokens
├── LTPI: M2/liquidity proxy via BTC dominance + funding
├── MTPI: momentum + derivatives composite
├── RS Tournament: z-scored returns → top 1-3 tokens
└── portfolio-macro.json ($10K, daily rebalance)
paper-dashboard/index.html (5-curve equity chart)
```
## Strategies
| Strategy | File | Timeframe | Gate | Max Hold | Slots |
|----------|------|-----------|------|----------|-------|
| Swing | `paper-monitor-swing.js` | Daily BoS + 4H FVG | EMA | 72h | 5 |
| Swing v2 | `paper-monitor-swing-v2.js` | Daily BoS + 4H FVG | MTPI ≥ 0.5 | 72h | 5 |
| Day v5 | `paper-monitor-v5.js` | 1H BoS + FVG + CVD | None | 12h | 10 |
| Day v6 | `paper-monitor-v6.js` | 1H BoS + FVG | STPI ≥ 0.5 | 12h | 10 |
| Coordinated | `paper-monitor-coordinated.js` | 1H + 4H mixed | Lock | varies | 10 |
| Macro Rotation | `macro-rotation.js` | Daily candles | LTPI + MTPI | Daily | $10K |
## Regime Scorer
`regime-scorer.js` produces `regime.json` with three probability indicators:
- **STPI** (Short Term): 15m + 1h composite → fast regime
- **MTPI** (Medium Term): 4h composite → swing regime
- **LTPI** (Long Term): Daily composite → macro regime
Indicators per timeframe: EMA cross, ADX, MA200, RSI, MACD, BBwidth, ATR pct, OBV, CVD z-score, funding rate, long/short ratio, OI delta, Fear & Greed (macro only).
## Macro Rotation
LTPI/MTPI guide overall allocation:
- Both bearish → 100% CASH
- Mixed → reduced position count
- Both bullish → full RS Tournament deployment
RS Tournament: z-scored 7d returns across 30+ tokens → rotates into top 1-3 by strength.
## Orchestrator Lock
All SMC monitors share `orchestrator-lock.json` — prevents same-symbol entries across strategies:
```json
{
"BTCUSDT": { "strategy": "swing", "entryTime": 1742400000000 },
"ETHUSDT": { "strategy": "day", "entryTime": 1742405000000 }
}
```
## Portfolio JSON Structure
```json
{
"balance": 1000,
"positions": [],
"history": [],
"metrics": { "totalTrades": 0, "wins": 0, "losses": 0, "winRate": 0 }
}
```
Macro: starts at $10,000.
## Running
```bash
# Regime (every 15m)
node scripts/regime-scorer.js
# Day v5: every 1h (XX:02)
node scripts/paper-monitor-v5.js
# Day v6 (STPI-gated): every 1h (XX:03)
node scripts/paper-monitor-v6.js
# Swing (EMA-gated): every 4h (XX:30)
node scripts/paper-monitor-swing.js
# Swing v2 (MTPI-gated): every 4h (XX:35)
node scripts/paper-monitor-swing-v2.js
# Coordinated 8D/2S: every 1h (XX:05)
node scripts/paper-monitor-coordinated.js
# Macro Rotation: daily (00:15 UTC)
node scripts/macro-rotation.js
```
## Cron Setup (OpenClaw)
```
Regime: 50 * * * * (XX:50 UTC every 1h)
Day v5: 2 * * * * (XX:02 UTC every 1h)
Day v6: 3 * * * * (XX:03 UTC every 1h)
Swing: 30 */4 * * * (XX:30 UTC every 4h)
Swing v2: 35 */4 * * * (XX:35 UTC every 4h)
Coordinated: 5 * * * * (XX:05 UTC every 1h)
Macro Rotation: 15 0 * * * (00:15 UTC daily)
```
## Files
```
scripts/
paper-monitor-swing.js — swing (EMA-gated)
paper-monitor-swing-v2.js — swing (MTPI-gated, NEW v2.0)
paper-monitor-v5.js — day + CVD
paper-monitor-v6.js — day + STPI gate (NEW v2.0)
paper-monitor-coordinated.js — 8D/2S coordinated
macro-rotation.js — LTPI/MTPI + RS Tournament (NEW v2.0)
regime-scorer.js — Multi-factor regime scoring (NEW v2.0)
paper-dashboard/index.html — 5-curve equity chart
```
## Notes
- Uses Binance **public** API only — no credentials required
- All start balances configurable (default: $1,000 day/swing, $10,000 macro)
- Fear & Greed Index via alternative.me API (free, no key)
- Orchestrator lock: internal coordination only, files stored locally
FILE:public/SKILL.md
---
name: skill-smc-multi-strategy-paper-trader
description: Paper trading monitors for SMC (Smart Money Concepts) + Macro Rotation. Swing (4H BoS+FVG), day (1H+CVD), coordinated 8D/2S, STPI/MTPI-gated variants, macro rotation (LTPI/MTPI + RS Tournament), and multi-factor regime scorer. ATR-based SL/TP, z-score filters, orchestrator-lock. Binance public API only — no credentials needed.
author: Zero2Ai-hub
version: 2.0.0
emoji: 📈
tags: [crypto, trading, paper-trading, smc, smart-money, orchestrator, backtested, macro, regime-filter]
---
# SMC Multi-Strategy Paper Trader v2.0
Paper trading system implementing **Smart Money Concepts** across five strategy types plus a macro rotation overlay and regime filter. All data from Binance public API — no account or API key required.
## What's New in v2.0
- **Macro Rotation** — LTPI/MTPI + Relative Strength Tournament, daily rebalance, $10K capital
- **Regime Scorer** — 15 TA indicators + 5 liquidity factors → STPI/MTPI/LTPI scores
- **v6 Day Monitor** — SMC+TA≥5 gated by STPI ≥ 0.5
- **Swing v2** — Swing strategy gated by MTPI ≥ 0.5
## Architecture
```
Binance Public API
↓
regime-scorer.js → regime.json (STPI, MTPI, LTPI)
↓
paper-monitor-[strategy].js
├── BoS + FVG detection
├── Z-score filters (volume, TA, FVG quality)
├── ATR SL/TP + trailing stop
├── Regime gate (STPI/MTPI threshold)
└── portfolio-[strategy].json
macro-rotation.js
├── LTPI/MTPI composite scores
├── RS Tournament → top 1-3 tokens
└── portfolio-macro.json ($10K)
```
## Strategies
| Strategy | Timeframe | Gate | Max Hold | Slots |
|----------|-----------|------|----------|-------|
| Swing | Daily BoS + 4H FVG | EMA | 72h | 5 |
| Swing v2 | Daily BoS + 4H FVG | MTPI ≥ 0.5 | 72h | 5 |
| Day v5 | 1H BoS + FVG + CVD | None | 12h | 10 |
| Day v6 | 1H BoS + FVG | STPI ≥ 0.5 | 12h | 10 |
| Coordinated 8D/2S | 1H + 4H | Lock | varies | 10 |
| Macro Rotation | Daily | LTPI + MTPI | Daily | $10K |
## Regime Scorer
Produces `regime.json` with three probability indicators (0–1):
- **STPI** — short-term regime (15m + 1h)
- **MTPI** — medium-term regime (4h)
- **LTPI** — long-term macro regime (1d)
15 TA indicators (EMA, RSI, MACD, BBwidth, ATR, OBV, CVD, ADX...) + 5 derivatives factors (funding rate, long/short ratio, OI delta, Fear & Greed).
## Running
```bash
node scripts/regime-scorer.js # every 1h
node scripts/paper-monitor-v5.js # every 1h
node scripts/paper-monitor-v6.js # every 1h (STPI-gated)
node scripts/paper-monitor-swing.js # every 4h
node scripts/paper-monitor-swing-v2.js # every 4h (MTPI-gated)
node scripts/paper-monitor-coordinated.js # every 1h
node scripts/macro-rotation.js # daily 00:15 UTC
```
## Notes
- Binance public API only — no credentials needed
- Fear & Greed via alternative.me (free)
- Orchestrator lock prevents same-symbol conflicts across strategies
- All balances configurable ($1K default, $10K macro)
FILE:scripts/macro-rotation.js
#!/usr/bin/env node
/**
* macro-rotation.js — Long-Term Macro Rotation Strategy
*
* Inspired by institutional crypto macro rotation:
* - LTPI (Long Term Probability Indicator): 0-1, macro liquidity + on-chain + global M2
* - MTPI (Medium Term Probability Indicator): 0-1, market structure + derivatives + momentum
* - Relative Strength Tournament: z-scored cross-pair strength, rotates into top 1-3
* - Cash is a valid allocation when signals are weak
*
* Runs DAILY at candle close. Rebalances portfolio allocation.
* Starting capital: $10,000 (minimum for real gains)
*/
'use strict';
const https = require('https');
const fs = require('fs');
const path = require('path');
const { execSync } = require('child_process');
const PORTFOLIO_FILE = path.join(process.env.HOME, '.openclaw/workspace/trading/paper-dashboard/portfolio-macro.json');
const TOKENS = [
'BTCUSDT','ETHUSDT','SOLUSDT','BNBUSDT','XRPUSDT','DOGEUSDT','AVAXUSDT','LINKUSDT',
'ARBUSDT','OPUSDT','SUIUSDT','APTUSDT','INJUSDT','SEIUSDT','FETUSDT','RENDERUSDT',
'TAOUSDT','NEARUSDT','WIFUSDT','JUPUSDT','TIAUSDT','DYMUSDT','STRKUSDT','TONUSDT',
'NOTUSDT','EIGENUSDT','GRASSUSDT','VIRTUALUSDT','AKTUSDT',
'HYPEUSDT', // Hyperliquid — prof's pick
'PAXGUSDT', // Gold (Paxos) — defensive allocation
];
// Token categories for allocation logic
const DEFENSIVE_TOKENS = new Set(['PAXGUSDT']);
const BTC_TOKEN = 'BTCUSDT';
// ═══════════════════════════════════════════════════════════════════════════════
// HELPERS
// ═══════════════════════════════════════════════════════════════════════════════
function httpGet(url, timeout = 10000) {
return new Promise((res, rej) => {
const r = https.get(url, resp => {
let d = ''; resp.on('data', c => d += c);
resp.on('end', () => { try { res(JSON.parse(d)); } catch(e) { rej(e); } });
}); r.on('error', rej);
r.setTimeout(timeout, () => { r.destroy(); rej(new Error('timeout')); });
});
}
const sleep = ms => new Promise(r => setTimeout(r, ms));
function calcEMA(v,p){const k=2/(p+1);let e=v[0];return v.map((x,i)=>{if(i>0)e=x*k+e*(1-k);return e;});}
function calcRSI(c,p=14){const r=Array(c.length).fill(null);let ag=0,al=0;for(let i=1;i<=p;i++){const d=c[i]-c[i-1];if(d>0)ag+=d;else al-=d;}ag/=p;al/=p;r[p]=al===0?100:100-100/(1+ag/al);for(let i=p+1;i<c.length;i++){const d=c[i]-c[i-1];ag=(ag*(p-1)+(d>0?d:0))/p;al=(al*(p-1)+(d<0?-d:0))/p;r[i]=al===0?100:100-100/(1+ag/al);}return r;}
function clamp(v,min,max){return Math.max(min,Math.min(max,v));}
function zScoreArr(values) {
const valid = values.filter(v => v !== null && !isNaN(v));
if (valid.length < 3) return values.map(() => 0);
const mean = valid.reduce((s,v)=>s+v,0)/valid.length;
const std = Math.sqrt(valid.reduce((s,v)=>s+(v-mean)**2,0)/valid.length);
if (std === 0) return values.map(() => 0);
return values.map(v => v === null || isNaN(v) ? 0 : (v - mean) / std);
}
function sigmoid(x) { return 1 / (1 + Math.exp(-x)); } // maps any value to 0-1
// ═══════════════════════════════════════════════════════════════════════════════
// DATA FETCHERS
// ═══════════════════════════════════════════════════════════════════════════════
function fetchFRED(seriesId, startDate = '2024-01-01') {
try {
const csv = execSync(`curl -sL --max-time 12 "https://fred.stlouisfed.org/graph/fredgraph.csv?id=seriesId&cosd=startDate"`, { encoding: 'utf8', timeout: 15000 });
const lines = csv.trim().split('\n').slice(1);
return lines.map(l => { const [d,v] = l.split(','); const val = parseFloat(v); return { date: d, value: isNaN(val) ? null : val }; }).filter(d => d.value !== null);
} catch { return []; }
}
async function fetchCandles(sym, interval, limit) {
await sleep(60);
try {
const r = await httpGet(`https://fapi.binance.com/fapi/v1/klines?symbol=sym&interval=interval&limit=limit`);
return r.map(k => ({ ts:k[0], o:+k[1], h:+k[2], l:+k[3], c:+k[4], v:+k[5]*+k[4], tbv:+k[9]*+k[4] }));
} catch { return []; }
}
async function fetchFundingRate(sym = 'BTCUSDT') {
try { const r = await httpGet(`https://fapi.binance.com/fapi/v1/fundingRate?symbol=sym&limit=30`); return r.map(f => parseFloat(f.fundingRate)); } catch { return []; }
}
async function fetchLSRatio(sym = 'BTCUSDT') {
try { const r = await httpGet(`https://fapi.binance.com/futures/data/globalLongShortAccountRatio?symbol=sym&period=1d&limit=30`); return r.map(d => parseFloat(d.longShortRatio)); } catch { return []; }
}
async function fetchOI(sym = 'BTCUSDT') {
try { const r = await httpGet(`https://fapi.binance.com/futures/data/openInterestHist?symbol=sym&period=1d&limit=30`); return r.map(d => parseFloat(d.sumOpenInterestValue)); } catch { return []; }
}
async function fetchFearGreed() {
try { const r = await httpGet('https://api.alternative.me/fng/?limit=30'); return r.data ? r.data.map(d => parseInt(d.value)) : []; } catch { return []; }
}
// ═══════════════════════════════════════════════════════════════════════════════
// LTPI — Long Term Probability Indicator
// 30+ inputs → classified → weighted → z-scored → sigmoid → 0-1
// ═══════════════════════════════════════════════════════════════════════════════
async function computeLTPI() {
// ── Macro Liquidity (FRED) ──
const [walcl, tga, rrp, m2, dxy, dgs10, t10yie, dff] = await Promise.all([
fetchFRED('WALCL', '2024-01-01'), // Fed Balance Sheet
fetchFRED('WTREGEN', '2024-01-01'), // Treasury General Account
fetchFRED('RRPONTSYD', '2024-06-01'), // Reverse Repo
fetchFRED('M2SL', '2023-01-01'), // M2
fetchFRED('DTWEXBGS', '2024-06-01'), // Dollar Index
fetchFRED('DGS10', '2024-06-01'), // 10Y Yield
fetchFRED('T10YIE', '2024-06-01'), // Breakeven Inflation
fetchFRED('DFF', '2024-06-01'), // Fed Funds
]);
// ── BTC Daily Candles (trend) ──
const btcD = await fetchCandles('BTCUSDT', '1d', 250);
const btcCloses = btcD.map(c => c.c);
// ── Fear & Greed ──
const fng = await fetchFearGreed();
const inputs = {};
const categories = { liquidity: [], trend: [], macro_momentum: [], risk_appetite: [] };
// ─── LIQUIDITY INPUTS ───
// 1. Net Liquidity 4-week delta
if (walcl.length >= 5 && tga.length >= 5 && rrp.length >= 5) {
const nl = walcl[walcl.length-1].value - tga[tga.length-1].value - rrp[rrp.length-1].value * 1000;
const nl4 = walcl[walcl.length-5].value - tga[tga.length-5].value - (rrp.length >= 30 ? rrp[rrp.length-22].value : rrp[0].value) * 1000;
inputs.net_liq_delta = (nl - nl4) / Math.abs(nl4) * 100;
categories.liquidity.push(inputs.net_liq_delta * 2);
}
// 2. Net Liquidity 12-week delta (longer trend)
if (walcl.length >= 13 && tga.length >= 13) {
const nl = walcl[walcl.length-1].value - tga[tga.length-1].value;
const nl12 = walcl[walcl.length-13].value - tga[tga.length-13].value;
inputs.net_liq_12w = (nl - nl12) / Math.abs(nl12) * 100;
categories.liquidity.push(inputs.net_liq_12w);
}
// 3. M2 YoY growth
if (m2.length >= 13) {
inputs.m2_yoy = (m2[m2.length-1].value - m2[m2.length-13].value) / m2[m2.length-13].value * 100;
categories.liquidity.push((inputs.m2_yoy - 3) * 0.5); // 3% baseline
}
// 4. M2 3-month momentum
if (m2.length >= 4) {
inputs.m2_3m = (m2[m2.length-1].value - m2[m2.length-4].value) / m2[m2.length-4].value * 100;
categories.liquidity.push(inputs.m2_3m * 3);
}
// 5. RRP trend (declining RRP = liquidity release)
if (rrp.length >= 20) {
const rrpNow = rrp[rrp.length-1].value;
const rrp20 = rrp[rrp.length-20].value;
inputs.rrp_delta = rrp20 !== 0 ? -(rrpNow - rrp20) / Math.max(rrp20, 1) * 100 : 0; // declining = positive
categories.liquidity.push(inputs.rrp_delta * 0.5);
}
// ─── MACRO MOMENTUM ───
// 6. DXY 20-day change (inverse)
if (dxy.length >= 21) {
inputs.dxy_20d = -(dxy[dxy.length-1].value - dxy[dxy.length-21].value) / dxy[dxy.length-21].value * 100;
categories.macro_momentum.push(inputs.dxy_20d * 3);
}
// 7. DXY 60-day trend
if (dxy.length >= 61) {
inputs.dxy_60d = -(dxy[dxy.length-1].value - dxy[dxy.length-61].value) / dxy[dxy.length-61].value * 100;
categories.macro_momentum.push(inputs.dxy_60d * 2);
}
// 8. Real yields level
if (dgs10.length >= 1 && t10yie.length >= 1) {
inputs.real_yield = dgs10[dgs10.length-1].value - t10yie[t10yie.length-1].value;
categories.macro_momentum.push(-(inputs.real_yield - 1) * 1.5);
}
// 9. Real yields momentum (20d)
if (dgs10.length >= 21 && t10yie.length >= 21) {
const ryNow = dgs10[dgs10.length-1].value - t10yie[t10yie.length-1].value;
const ry20 = dgs10[dgs10.length-21].value - t10yie[t10yie.length-21].value;
inputs.real_yield_mom = -(ryNow - ry20) * 3;
categories.macro_momentum.push(inputs.real_yield_mom);
}
// 10. Fed rate trajectory
if (dff.length >= 30) {
inputs.fed_rate_30d = -(dff[dff.length-1].value - dff[dff.length-30].value) * 2;
categories.macro_momentum.push(inputs.fed_rate_30d);
}
// ─── BTC TREND ───
if (btcCloses.length >= 200) {
const n = btcCloses.length - 1;
// 11. Price vs 200 EMA
const ema200 = calcEMA(btcCloses, 200);
inputs.btc_vs_200 = (btcCloses[n] - ema200[n]) / ema200[n] * 100;
categories.trend.push(inputs.btc_vs_200 * 0.3);
// 12. 50 vs 200 EMA (golden/death cross)
const ema50 = calcEMA(btcCloses, 50);
inputs.btc_50_200 = (ema50[n] - ema200[n]) / ema200[n] * 100;
categories.trend.push(inputs.btc_50_200 * 0.5);
// 13. 20 vs 50 EMA
const ema20 = calcEMA(btcCloses, 20);
inputs.btc_20_50 = (ema20[n] - ema50[n]) / ema50[n] * 100;
categories.trend.push(inputs.btc_20_50);
// 14. BTC 90-day ROC
if (n >= 90) {
inputs.btc_roc_90 = (btcCloses[n] - btcCloses[n-90]) / btcCloses[n-90] * 100;
categories.trend.push(inputs.btc_roc_90 * 0.1);
}
// 15. BTC RSI 14 (daily)
const rsi = calcRSI(btcCloses);
if (rsi[n] !== null) {
inputs.btc_rsi = rsi[n];
categories.trend.push((rsi[n] - 50) * 0.05);
}
// 16. Higher highs count (last 60 days)
let hh = 0;
for (let i = n - 59; i <= n - 5; i++) {
if (i < 5) continue;
let isHigh = true;
for (let j = 1; j <= 5; j++) if (btcD[i-j].h >= btcD[i].h || btcD[i+j].h >= btcD[i].h) isHigh = false;
if (isHigh) hh++;
}
inputs.btc_hh_count = hh;
categories.trend.push(hh * 0.3);
}
// ─── RISK APPETITE ───
// 17. Fear & Greed (contrarian + confirmation)
if (fng.length >= 1) {
const fg = fng[0];
inputs.fear_greed = fg;
// Below 25: extreme fear → contrarian bullish
// 25-45: fear → mild bullish
// 45-55: neutral
// 55-75: greed → confirms trend
// 75+: extreme greed → contrarian bearish
if (fg < 25) categories.risk_appetite.push(2);
else if (fg < 40) categories.risk_appetite.push(0.5);
else if (fg <= 60) categories.risk_appetite.push(0);
else if (fg <= 75) categories.risk_appetite.push(0.5);
else categories.risk_appetite.push(-2);
}
// 18. F&G 7-day average trend
if (fng.length >= 7) {
const avg7 = fng.slice(0,7).reduce((s,v)=>s+v,0)/7;
const avg30 = fng.reduce((s,v)=>s+v,0)/fng.length;
inputs.fng_momentum = avg7 - avg30;
categories.risk_appetite.push(inputs.fng_momentum * 0.1);
}
// ── AGGREGATE ──
// Weight each category, z-score within, then aggregate
const weights = { liquidity: 3.0, macro_momentum: 2.0, trend: 2.0, risk_appetite: 1.0 };
let totalScore = 0, totalWeight = 0;
const breakdown = {};
for (const [cat, vals] of Object.entries(categories)) {
if (!vals.length) continue;
const avg = vals.reduce((s,v)=>s+v,0)/vals.length;
const w = weights[cat] || 1;
totalScore += avg * w;
totalWeight += w;
breakdown[cat] = { avg: parseFloat(avg.toFixed(3)), weight: w, count: vals.length };
}
const rawScore = totalWeight > 0 ? totalScore / totalWeight : 0;
// Sigmoid → 0-1 probability
// Scale factor: adjust so ±3 raw → ~0.95/0.05 probability
const ltpi = parseFloat(sigmoid(rawScore * 0.8).toFixed(4));
return { ltpi, rawScore: parseFloat(rawScore.toFixed(3)), inputs, breakdown };
}
// ═══════════════════════════════════════════════════════════════════════════════
// MTPI — Medium Term Probability Indicator
// Market structure + derivatives + faster momentum
// ═══════════════════════════════════════════════════════════════════════════════
async function computeMTPI() {
const btcD = await fetchCandles('BTCUSDT', '1d', 120);
const btc4h = await fetchCandles('BTCUSDT', '4h', 200);
if (btcD.length < 60 || btc4h.length < 100) return { mtpi: 0.5, rawScore: 0, inputs: {}, breakdown: {} };
const closes = btcD.map(c => c.c);
const closes4h = btc4h.map(c => c.c);
const n = closes.length - 1;
const n4 = closes4h.length - 1;
const [funding, lsRatio, oiData, fng] = await Promise.all([
fetchFundingRate(), fetchLSRatio(), fetchOI(), fetchFearGreed()
]);
const inputs = {};
const categories = { momentum: [], structure: [], derivatives: [], flow: [] };
// ─── MOMENTUM ───
// 1. Daily RSI
const rsi = calcRSI(closes);
if (rsi[n] !== null) {
inputs.rsi_14 = rsi[n];
categories.momentum.push((rsi[n] - 50) * 0.06);
}
// 2. 4h RSI (faster)
const rsi4h = calcRSI(closes4h);
if (rsi4h[n4] !== null) {
inputs.rsi_4h = rsi4h[n4];
categories.momentum.push((rsi4h[n4] - 50) * 0.04);
}
// 3. 7-day ROC
if (n >= 7) {
inputs.roc_7d = (closes[n] - closes[n-7]) / closes[n-7] * 100;
categories.momentum.push(inputs.roc_7d * 0.3);
}
// 4. 21-day ROC
if (n >= 21) {
inputs.roc_21d = (closes[n] - closes[n-21]) / closes[n-21] * 100;
categories.momentum.push(inputs.roc_21d * 0.15);
}
// 5. MACD histogram direction
const e12 = calcEMA(closes, 12), e26 = calcEMA(closes, 26);
const macdLine = closes.map((_,i) => e12[i] - e26[i]);
const macdSig = calcEMA(macdLine.slice(25), 9);
const hist = macdLine[n] - (macdSig[n-25] || 0);
const histPrev = n > 1 ? macdLine[n-1] - (macdSig[n-26] || 0) : hist;
inputs.macd_hist = hist;
inputs.macd_rising = hist > histPrev;
categories.momentum.push(hist > 0 ? 1 : -1);
categories.momentum.push(hist > histPrev ? 0.5 : -0.5);
// ─── STRUCTURE ───
// 6. EMA alignment (8 > 21 > 50)
const ema8 = calcEMA(closes, 8), ema21 = calcEMA(closes, 21), ema50 = calcEMA(closes, 50);
const aligned = ema8[n] > ema21[n] && ema21[n] > ema50[n];
const inverted = ema8[n] < ema21[n] && ema21[n] < ema50[n];
inputs.ema_aligned = aligned ? 'bullish' : inverted ? 'bearish' : 'mixed';
categories.structure.push(aligned ? 2 : inverted ? -2 : 0);
// 7. Price vs 50 EMA distance
inputs.dist_50ema = (closes[n] - ema50[n]) / ema50[n] * 100;
categories.structure.push(clamp(inputs.dist_50ema * 0.3, -2, 2));
// 8. 4h trend (20 vs 50 EMA)
const ema20_4h = calcEMA(closes4h, 20), ema50_4h = calcEMA(closes4h, 50);
inputs.trend_4h = (ema20_4h[n4] - ema50_4h[n4]) / ema50_4h[n4] * 100;
categories.structure.push(clamp(inputs.trend_4h * 0.5, -2, 2));
// ─── DERIVATIVES ───
// 9. Funding rate (contrarian)
if (funding.length >= 3) {
const avgF = funding.slice(-3).reduce((s,v)=>s+v,0)/3;
inputs.funding_avg = avgF;
categories.derivatives.push(clamp(-avgF * 3000, -2, 2));
}
// 10. L/S Ratio (contrarian)
if (lsRatio.length >= 5) {
const avgLS = lsRatio.slice(-5).reduce((s,v)=>s+v,0)/5;
inputs.ls_ratio = avgLS;
categories.derivatives.push(clamp(-(avgLS - 1) * 3, -2, 2));
}
// 11. OI delta 7-day
if (oiData.length >= 8) {
const oiDelta = (oiData[oiData.length-1] - oiData[oiData.length-8]) / oiData[oiData.length-8] * 100;
inputs.oi_7d_delta = oiDelta;
categories.derivatives.push(clamp(oiDelta / 5, -2, 2));
}
// ─── FLOW (Volume + CVD) ───
// 12. Daily volume trend
const vols = btcD.map(c => c.v);
if (n >= 20) {
const volAvg5 = vols.slice(n-4,n+1).reduce((s,v)=>s+v,0)/5;
const volAvg20 = vols.slice(n-19,n+1).reduce((s,v)=>s+v,0)/20;
inputs.vol_ratio = volAvg5 / volAvg20;
categories.flow.push(clamp((inputs.vol_ratio - 1) * 3, -2, 2));
}
// 13. CVD direction (daily)
const deltas = btcD.map(c => (c.tbv||0) - (c.v - (c.tbv||0)));
if (n >= 7) {
const cvd7 = deltas.slice(n-6,n+1).reduce((s,v)=>s+v,0);
const cvd14 = deltas.slice(Math.max(0,n-13),n+1).reduce((s,v)=>s+v,0);
inputs.cvd_direction = cvd7 > 0 ? 'buying' : 'selling';
categories.flow.push(cvd7 > 0 ? 1 : -1);
}
// ── AGGREGATE ──
const weights = { momentum: 2.0, structure: 2.0, derivatives: 1.5, flow: 1.5 };
let totalScore = 0, totalWeight = 0;
const breakdown = {};
for (const [cat, vals] of Object.entries(categories)) {
if (!vals.length) continue;
const avg = vals.reduce((s,v)=>s+v,0)/vals.length;
const w = weights[cat] || 1;
totalScore += avg * w;
totalWeight += w;
breakdown[cat] = { avg: parseFloat(avg.toFixed(3)), weight: w, count: vals.length };
}
const rawScore = totalWeight > 0 ? totalScore / totalWeight : 0;
const mtpi = parseFloat(sigmoid(rawScore * 0.8).toFixed(4));
return { mtpi, rawScore: parseFloat(rawScore.toFixed(3)), inputs, breakdown };
}
// ═══════════════════════════════════════════════════════════════════════════════
// RELATIVE STRENGTH TOURNAMENT — v2 (Full Cross-Pair)
//
// For each token: measure RS against EVERY other token in the pool.
// SOL: SOL/BTC, SOL/ETH, SOL/BNB, SOL/XRP... (N-1 pairs)
// Each pair: return ratio over 7d, 14d, 30d, 60d → average
// Pool-wide RS = average of all pair RS scores
// Then z-score pool-wide scores → true ranking
//
// A token wins only if it's beating the ENTIRE field, not just BTC.
// ═══════════════════════════════════════════════════════════════════════════════
async function runTournament() {
// Fetch daily candles for all tokens
const allCandles = {};
const returns = {}; // { symbol: { 7: pct, 14: pct, 30: pct, 60: pct } }
const PERIODS = [7, 14, 30, 60];
for (const sym of TOKENS) {
const cs = await fetchCandles(sym, '1d', 90);
if (cs.length < 30) continue;
allCandles[sym] = cs;
const closes = cs.map(c => c.c);
const n = closes.length - 1;
returns[sym] = {};
for (const p of PERIODS) {
if (n >= p) {
returns[sym][p] = (closes[n] - closes[n-p]) / closes[n-p] * 100;
}
}
}
const syms = Object.keys(returns);
if (syms.length < 5) return [];
// ── Cross-pair RS matrix ──
// For each token, compute RS against every other token across all periods
const poolRS = {}; // { symbol: average RS across all pairs and periods }
for (const sym of syms) {
const pairScores = [];
for (const other of syms) {
if (sym === other) continue;
for (const p of PERIODS) {
const symRet = returns[sym][p];
const otherRet = returns[other][p];
if (symRet === undefined || otherRet === undefined) continue;
// RS ratio: how much sym outperforms other
// Positive = sym winning, negative = sym losing
// Use difference rather than ratio to handle negative returns cleanly
pairScores.push(symRet - otherRet);
}
}
poolRS[sym] = pairScores.length > 0
? pairScores.reduce((s,v) => s+v, 0) / pairScores.length
: 0;
}
// ── Z-score the pool-wide RS ──
const poolValues = syms.map(s => poolRS[s]);
const poolZScores = zScoreArr(poolValues);
// ── Build results with additional metrics ──
const results = [];
for (let i = 0; i < syms.length; i++) {
const sym = syms[i];
const cs = allCandles[sym];
const closes = cs.map(c => c.c);
const n = closes.length - 1;
// Absolute momentum
const mom7 = returns[sym][7] || 0;
const mom30 = returns[sym][30] || 0;
// Volume trend
const vols = cs.map(c => c.v);
const volRatio = n >= 14 ? (vols.slice(n-6,n+1).reduce((s,v)=>s+v,0)/7) / (vols.slice(n-13,n+1).reduce((s,v)=>s+v,0)/14) : 1;
// RSI
const rsi = calcRSI(closes);
const rsiVal = rsi[n] || 50;
// RS vs BTC specifically (for display)
const btcRet7 = returns['BTCUSDT'] ? returns['BTCUSDT'][7] || 0 : 0;
const btcRet30 = returns['BTCUSDT'] ? returns['BTCUSDT'][30] || 0 : 0;
const rsVsBtc = ((mom7 - btcRet7) + (mom30 - btcRet30)) / 2;
// Pool-wide RS z-score (the main ranking metric)
const poolZ = poolZScores[i];
// Final composite: pool RS z-score (60%) + volume (20%) + RSI momentum (20%)
const volZ = zScoreArr(Object.values(allCandles).map(c => {
const v = c.map(x=>x.v); const nn = v.length-1;
return nn >= 14 ? (v.slice(nn-6,nn+1).reduce((s,x)=>s+x,0)/7)/(v.slice(nn-13,nn+1).reduce((s,x)=>s+x,0)/14) : 1;
}))[i] || 0;
const rsiZ = zScoreArr(syms.map(s => {
const c = allCandles[s].map(x=>x.c);
const r = calcRSI(c);
return r[r.length-1] || 50;
}))[i] || 0;
const composite = poolZ * 0.6 + volZ * 0.2 + rsiZ * 0.2;
results.push({
symbol: sym,
composite: parseFloat(composite.toFixed(4)),
pool_rs: parseFloat(poolRS[sym].toFixed(3)),
pool_z: parseFloat(poolZ.toFixed(3)),
rs_vs_btc: parseFloat(rsVsBtc.toFixed(3)),
mom_7d: parseFloat(mom7.toFixed(2)),
mom_30d: parseFloat(mom30.toFixed(2)),
vol_ratio: parseFloat(volRatio.toFixed(3)),
rsi: parseFloat(rsiVal.toFixed(1)),
price: closes[n],
n_pairs: syms.length - 1,
});
}
// Z-score the final composites
const composites = results.map(r => r.composite);
const finalZ = zScoreArr(composites);
results.forEach((r, i) => { r.z_score = parseFloat(finalZ[i].toFixed(3)); });
// Sort by z-score (best first)
results.sort((a, b) => b.z_score - a.z_score);
return results;
}
// ═══════════════════════════════════════════════════════════════════════════════
// ALLOCATION ENGINE — v2 (Prof-inspired LTPI×MTPI matrix)
//
// Key insight from prof:
// - LTPI bearish ≠ go to cash. It means: avoid BTC leverage, reduce BTC exposure
// - MTPI positive = deploy into RS winners REGARDLESS of LTPI
// - LTPI controls: BTC allocation + leverage permission + defensive hedge %
// - MTPI controls: overall deployment level + token count
// - Gold (PAXG) is a valid defensive allocation
// - Cash only when BOTH are deeply negative
// ═══════════════════════════════════════════════════════════════════════════════
function computeAllocation(ltpi, mtpi, tournament) {
// Convert our 0-1 scale to prof's -1 to +1 for matrix logic
const ltpiAdj = (ltpi - 0.5) * 2; // 0→-1, 0.5→0, 1→+1
const mtpiAdj = (mtpi - 0.5) * 2;
// ── LTPI determines: BTC permission, defensive hedge %, leverage permission ──
const btcAllowed = ltpiAdj > -0.2; // BTC only when LTPI isn't bearish
const btcLeverageAllowed = ltpiAdj > 0.3; // Leverage only when LTPI clearly bullish
const defensiveHedgePct = ltpiAdj < -0.3 ? 10 : ltpiAdj < 0 ? 5 : 0; // Gold allocation
// ── MTPI determines: deployment level, token count ──
let deployPct, maxTokens, riskLevel;
if (mtpiAdj >= 0.5) {
// MTPI strong positive — full deployment
deployPct = 95; maxTokens = 3; riskLevel = 'AGGRESSIVE';
} else if (mtpiAdj >= 0.2) {
// MTPI moderate positive — high deployment
deployPct = 80; maxTokens = 3; riskLevel = 'MODERATE';
} else if (mtpiAdj >= -0.1) {
// MTPI neutral — cautious deployment
deployPct = 50; maxTokens = 2; riskLevel = 'CAUTIOUS';
} else if (mtpiAdj >= -0.4) {
// MTPI mildly negative — light deployment
deployPct = 25; maxTokens = 1; riskLevel = 'DEFENSIVE';
} else {
// MTPI deeply negative — minimal to zero
deployPct = 5; maxTokens = 1; riskLevel = 'RISK_OFF';
}
// ── Override: both deeply negative → full cash ──
if (ltpiAdj < -0.5 && mtpiAdj < -0.3) {
deployPct = 0; maxTokens = 0; riskLevel = 'CASH';
}
// ── Macro environment modifier ──
// LTPI positive boosts deployment; LTPI negative reduces slightly but doesn't kill it
if (ltpiAdj > 0.3) deployPct = Math.min(100, deployPct + 10);
else if (ltpiAdj < -0.3) deployPct = Math.max(0, deployPct - 10);
// ── Build allocation from RS tournament ──
const allocations = [];
let usedPct = 0;
// Defensive hedge (gold) first — carved from total, not from deployment
if (defensiveHedgePct > 0) {
const goldToken = tournament.find(t => DEFENSIVE_TOKENS.has(t.symbol));
if (goldToken) {
allocations.push({
symbol: goldToken.symbol, pct: defensiveHedgePct,
z_score: goldToken.z_score, rs_vs_btc: goldToken.rs_vs_btc,
price: goldToken.price, role: 'HEDGE',
});
usedPct += defensiveHedgePct;
}
}
// RS winners (excluding defensive tokens and BTC if not allowed)
const candidates = tournament.filter(t => {
if (DEFENSIVE_TOKENS.has(t.symbol)) return false;
if (t.symbol === BTC_TOKEN && !btcAllowed) return false;
if (t.z_score < -0.5) return false; // skip weak tokens
return true;
});
const remainingDeploy = Math.max(0, deployPct - usedPct);
const selectedCount = Math.min(maxTokens, candidates.length);
const selected = candidates.slice(0, selectedCount);
if (selected.length > 0 && remainingDeploy > 0) {
// Weight by z-score (stronger tokens get more)
const totalZ = selected.reduce((s, t) => s + Math.max(t.z_score, 0.1), 0);
for (const t of selected) {
const weight = Math.max(t.z_score, 0.1) / totalZ;
const pct = parseFloat((weight * remainingDeploy).toFixed(1));
allocations.push({
symbol: t.symbol, pct,
z_score: t.z_score, rs_vs_btc: t.rs_vs_btc,
price: t.price, role: t.symbol === BTC_TOKEN ? 'BTC' : 'RS_WINNER',
});
usedPct += pct;
}
}
const cashPct = parseFloat((100 - usedPct).toFixed(1));
return {
risk_level: riskLevel,
ltpi_adj: parseFloat(ltpiAdj.toFixed(3)),
mtpi_adj: parseFloat(mtpiAdj.toFixed(3)),
btc_allowed: btcAllowed,
btc_leverage_allowed: btcLeverageAllowed,
defensive_hedge_pct: defensiveHedgePct,
deploy_pct: deployPct,
max_tokens: maxTokens,
allocations,
cash_pct: cashPct,
};
}
// ═══════════════════════════════════════════════════════════════════════════════
// PORTFOLIO
// ═══════════════════════════════════════════════════════════════════════════════
function loadPortfolio() {
try { return JSON.parse(fs.readFileSync(PORTFOLIO_FILE, 'utf8')); }
catch {
return {
version: '1.0', last_updated: new Date().toISOString(),
strategy: 'Macro Rotation — LTPI/MTPI + RS Tournament | Paper Trading',
paper_portfolio: {
starting_capital_usdc: 10000,
current_capital_usdc: 10000,
cash_usdc: 10000,
positions: [], // { symbol, entry_price, size_usdc, entry_time, pct }
closed_trades: [],
rebalance_log: [],
stats: { total_rebalances: 0, total_trades: 0, wins: 0, losses: 0, total_pnl_usdc: 0 }
}
};
}
}
function savePortfolio(data) {
data.last_updated = new Date().toISOString();
fs.mkdirSync(path.dirname(PORTFOLIO_FILE), { recursive: true });
fs.writeFileSync(PORTFOLIO_FILE, JSON.stringify(data, null, 2));
}
async function rebalance(portfolio, allocation, tournament) {
const p = portfolio.paper_portfolio;
const now = new Date().toISOString();
// Get current prices for all held positions
for (const pos of p.positions) {
try {
await sleep(60);
const ticker = await httpGet(`https://fapi.binance.com/fapi/v1/ticker/price?symbol=pos.symbol`);
pos.current_price = parseFloat(ticker.price);
} catch { pos.current_price = pos.entry_price; }
}
// Calculate current portfolio value
let totalValue = p.cash_usdc;
for (const pos of p.positions) {
const pnl = (pos.current_price - pos.entry_price) / pos.entry_price * pos.size_usdc;
totalValue += pos.size_usdc + pnl;
}
p.current_capital_usdc = parseFloat(totalValue.toFixed(2));
// Target allocations
const targetMap = {};
for (const a of allocation.allocations) {
targetMap[a.symbol] = { pct: a.pct, target_usdc: totalValue * a.pct / 100, price: a.price };
}
// Close positions not in target
const newPositions = [];
for (const pos of p.positions) {
if (!targetMap[pos.symbol]) {
// Close this position
const pnlPct = (pos.current_price - pos.entry_price) / pos.entry_price * 100;
const pnlUSD = pos.size_usdc * pnlPct / 100;
p.cash_usdc += pos.size_usdc + pnlUSD;
p.stats.total_trades++;
if (pnlUSD > 0) p.stats.wins++; else p.stats.losses++;
p.stats.total_pnl_usdc += pnlUSD;
p.closed_trades.push({
symbol: pos.symbol, entry_price: pos.entry_price, exit_price: pos.current_price,
entry_time: pos.entry_time, exit_time: now,
pnl_pct: parseFloat(pnlPct.toFixed(3)), pnl_usd: parseFloat(pnlUSD.toFixed(2)),
size_usdc: pos.size_usdc, exit_reason: 'rotation',
});
console.log(` 🔄 CLOSE pos.symbol: ''pnlPct.toFixed(2)% ($''pnlUSD.toFixed(2))`);
} else {
// Keep, but adjust size if needed
const target = targetMap[pos.symbol].target_usdc;
const currentVal = pos.size_usdc * (1 + (pos.current_price - pos.entry_price) / pos.entry_price);
const diff = target - currentVal;
if (Math.abs(diff) / totalValue > 0.05) { // rebalance if >5% drift
if (diff > 0) {
// Add to position
pos.size_usdc += diff;
p.cash_usdc -= diff;
console.log(` 📈 ADD pos.symbol: +$diff.toFixed(2)`);
} else {
// Reduce position
const reducePct = Math.abs(diff) / currentVal;
const reduceUSD = pos.size_usdc * reducePct;
const pnlPortion = reduceUSD * (pos.current_price - pos.entry_price) / pos.entry_price;
pos.size_usdc -= reduceUSD;
p.cash_usdc += reduceUSD + pnlPortion;
console.log(` 📉 REDUCE pos.symbol: -$reduceUSD.toFixed(2)`);
}
}
pos.target_pct = targetMap[pos.symbol].pct;
newPositions.push(pos);
delete targetMap[pos.symbol];
}
}
// Open new positions
for (const [sym, target] of Object.entries(targetMap)) {
if (target.target_usdc < 50) continue; // skip tiny positions
if (p.cash_usdc < target.target_usdc) continue;
const size = Math.min(target.target_usdc, p.cash_usdc);
p.cash_usdc -= size;
newPositions.push({
symbol: sym, entry_price: target.price, current_price: target.price,
size_usdc: parseFloat(size.toFixed(2)), entry_time: now, target_pct: target.pct,
});
console.log(` 🟢 OPEN sym: $size.toFixed(2) (target.pct%) @ $target.price.toFixed(4)`);
}
p.positions = newPositions;
p.stats.total_rebalances++;
p.rebalance_log.push({
time: now,
capital: p.current_capital_usdc,
positions: newPositions.map(pos => ({ symbol: pos.symbol, pct: pos.target_pct })),
cash_pct: allocation.cash_pct,
});
// Trim log to last 90 entries
if (p.rebalance_log.length > 90) p.rebalance_log = p.rebalance_log.slice(-90);
return portfolio;
}
// ═══════════════════════════════════════════════════════════════════════════════
// MAIN
// ═══════════════════════════════════════════════════════════════════════════════
async function run() {
const uaeTime = new Date().toLocaleString('en-AE', { timeZone: 'Asia/Dubai', hour12: false });
console.log(`\n[uaeTime UAE] Macro Rotation — Daily Rebalance\n`);
// 1. Compute LTPI
console.log('📊 Computing LTPI...');
const ltpiResult = await computeLTPI();
console.log(` LTPI: ltpiResult.ltpi.toFixed(4) (raw: ltpiResult.rawScore) '🔴 BEARISH'`);
// 2. Compute MTPI
console.log('📊 Computing MTPI...');
const mtpiResult = await computeMTPI();
console.log(` MTPI: mtpiResult.mtpi.toFixed(4) (raw: mtpiResult.rawScore) '🔴 BEARISH'`);
// 3. Run tournament
console.log('🏆 Running RS Tournament...');
const tournament = await runTournament();
console.log(` Top 5 (scored against tournament[0]?.n_pairs || '?' peers):`);
tournament.slice(0, 5).forEach((t, i) => {
console.log(` i+1. t.symbol.padEnd(12) z:t.z_score.toFixed(2).padStart(6) poolRS:t.pool_rs.toFixed(1).padStart(6) vsBTC:t.rs_vs_btc.toFixed(1).padStart(6) 7d:''t.mom_7d.toFixed(1)% RSI:t.rsi.toFixed(0)`);
});
// 4. Compute allocation
const allocation = computeAllocation(ltpiResult.ltpi, mtpiResult.mtpi, tournament);
console.log(`\n Risk Level: allocation.risk_level | Deploy: allocation.deploy_pct% | Cash: allocation.cash_pct%`);
console.log(` BTC: '❌ excluded' | Leverage: '❌' | Hedge: allocation.defensive_hedge_pct%`);
console.log(` Allocations:`);
for (const a of allocation.allocations) {
console.log(` a.symbol.padEnd(12) a.pct.toFixed(1)% [a.role] (z:a.z_score)`);
}
if (allocation.allocations.length === 0) console.log(` 100% CASH`);
// 5. Rebalance portfolio
console.log('\n🔄 Rebalancing...');
let portfolio = loadPortfolio();
// Store signals in portfolio for dashboard
portfolio.ltpi = ltpiResult.ltpi;
portfolio.ltpi_raw = ltpiResult.rawScore;
portfolio.ltpi_breakdown = ltpiResult.breakdown;
portfolio.mtpi = mtpiResult.mtpi;
portfolio.mtpi_raw = mtpiResult.rawScore;
portfolio.mtpi_breakdown = mtpiResult.breakdown;
portfolio.allocation = allocation;
portfolio.tournament_top10 = tournament.slice(0, 10);
portfolio = await rebalance(portfolio, allocation, tournament);
savePortfolio(portfolio);
// 6. GitHub push
try {
const ghToken = fs.readFileSync(path.join(process.env.HOME, '.github_token'), 'utf8').trim();
const content = Buffer.from(JSON.stringify(portfolio, null, 2)).toString('base64');
const { execSync } = require('child_process');
// Get SHA
const shaResp = JSON.parse(execSync(`curl -sL -H "Authorization: token ghToken" -H "Accept: application/vnd.github.v3+json" "https://api.github.com/repos/Zero2Ai-hub/Jarvis-Ops/contents/trading/portfolio-macro.json"`, { encoding: 'utf8', timeout: 10000 }).toString());
const sha = shaResp.sha || '';
const body = JSON.stringify({ message: 'trading: macro rotation update', content, sha: sha || undefined });
execSync(`curl -sL -X PUT -H "Authorization: token ghToken" -H "Accept: application/vnd.github.v3+json" -d 'body.replace(/'/g, "\\'")' "https://api.github.com/repos/Zero2Ai-hub/Jarvis-Ops/contents/trading/portfolio-macro.json"`, { timeout: 15000 });
console.log('GitHub updated ✅');
} catch(e) { console.log('GitHub push failed:', e.message); }
// Summary
const p = portfolio.paper_portfolio;
console.log(`\n'═'.repeat(60)`);
console.log(`LTPI: ltpiResult.ltpi.toFixed(3) (''allocation.ltpi_adj) | MTPI: mtpiResult.mtpi.toFixed(3) (''allocation.mtpi_adj) | Risk: allocation.risk_level`);
console.log(`BTC: 'EXCLUDED' | Deploy: allocation.deploy_pct% | Cash: allocation.cash_pct% | Hedge: allocation.defensive_hedge_pct%`);
console.log(`Portfolio: $p.current_capital_usdc.toFixed(2) | Cash: $p.cash_usdc.toFixed(2) | Positions: p.positions.length`);
console.log(`Trades: p.stats.total_trades | WR: '—'% | PnL: $p.stats.total_pnl_usdc.toFixed(2)`);
console.log('Done ✅');
}
run().catch(e => { console.error('Fatal:', e.message, e.stack); process.exit(1); });
FILE:scripts/paper-monitor-coordinated.js
#!/usr/bin/env node
/**
* paper-monitor-coordinated.js — 8D/2S Coordinated Paper Trading Monitor
*
* Runs both day trading (1h) and swing trading (4h) with shared capital.
* Optimal split: 8 day slots + 2 swing slots = 10 total concurrent.
* Orchestrator prevents same-symbol conflicts.
*
* Day: SMC v4.0 — 1h BoS+FVG+TA≥5, 12h max hold
* Swing: SMC — Daily BoS → 4h FVG+TA≥5, 72h max hold
*/
'use strict';
const https = require('https');
const fs = require('fs');
const path = require('path');
const PORTFOLIO_FILE = path.join(process.env.HOME, '.openclaw/workspace/trading/paper-dashboard/portfolio-coordinated.json');
const MAX_DAY = 8;
const MAX_SWING = 2;
const TOKENS = [
'BTCUSDT','ETHUSDT','SOLUSDT','BNBUSDT','XRPUSDT','DOGEUSDT','AVAXUSDT','LINKUSDT',
'ARBUSDT','OPUSDT','SUIUSDT','APTUSDT','INJUSDT','SEIUSDT','FETUSDT','RENDERUSDT',
'TAOUSDT','NEARUSDT','WIFUSDT','JUPUSDT','TIAUSDT','DYMUSDT','STRKUSDT','TONUSDT',
'NOTUSDT','EIGENUSDT','GRASSUSDT','VIRTUALUSDT','AKTUSDT',
];
function apiFetch(url) {
return new Promise((res, rej) => {
const r = https.get(url, resp => {
let d = ''; resp.on('data', c => d += c);
resp.on('end', () => { try { res(JSON.parse(d)); } catch(e) { rej(e); } });
}); r.on('error', rej);
r.setTimeout(15000, () => { r.destroy(); rej(new Error('timeout')); });
});
}
function apiFetch2(url, token) {
return new Promise((res, rej) => {
const r = https.get(url, { headers: { Authorization: `token token`, Accept: 'application/vnd.github.v3+json', 'User-Agent': 'Jarvis' } }, resp => {
let d = ''; resp.on('data', c => d += c);
resp.on('end', () => { try { res(JSON.parse(d)); } catch(e) { rej(e); } });
}); r.on('error', rej); r.setTimeout(10000, () => { r.destroy(); rej(new Error('timeout')); });
});
}
function apiFetch2Put(url, token, body) {
return new Promise((res, rej) => {
const opts = require('url').parse(url);
opts.method = 'PUT';
opts.headers = { Authorization: `token token`, Accept: 'application/vnd.github.v3+json', 'Content-Type': 'application/json', 'User-Agent': 'Jarvis', 'Content-Length': Buffer.byteLength(body) };
const r = require('https').request(opts, resp => {
let d = ''; resp.on('data', c => d += c);
resp.on('end', () => { try { res(JSON.parse(d)); } catch(e) { rej(e); } });
}); r.on('error', rej); r.write(body); r.end();
});
}
const sleep = ms => new Promise(r => setTimeout(r, ms));
// ─── Math ─────────────────────────────────────────────────────────────────────
function calcEMA(v,p){const k=2/(p+1);let e=v[0];return v.map((x,i)=>{if(i>0)e=x*k+e*(1-k);return e;});}
function calcATR(cs,period=14){const t=cs.map((c,i)=>{if(i===0)return c.h-c.l;const p2=cs[i-1].c;return Math.max(c.h-c.l,Math.abs(c.h-p2),Math.abs(c.l-p2));});const a=[];for(let i=0;i<cs.length;i++){if(i<period-1){a.push(null);continue;}if(i===period-1){a.push(t.slice(0,period).reduce((s,v2)=>s+v2,0)/period);}else{a.push((a[i-1]*(period-1)+t[i])/period);}}return a;}
function calcRSI(c,p=14){const r=Array(c.length).fill(null);let ag=0,al=0;for(let i=1;i<=p;i++){const d=c[i]-c[i-1];if(d>0)ag+=d;else al-=d;}ag/=p;al/=p;r[p]=al===0?100:100-100/(1+ag/al);for(let i=p+1;i<c.length;i++){const d=c[i]-c[i-1];ag=(ag*(p-1)+(d>0?d:0))/p;al=(al*(p-1)+(d<0?-d:0))/p;r[i]=al===0?100:100-100/(1+ag/al);}return r;}
function calcMACD(c){const e12=calcEMA(c,12),e26=calcEMA(c,26);const ml=c.map((_,i)=>e12[i]-e26[i]);const sig=calcEMA(ml.slice(25),9);const h=[];for(let i=0;i<c.length;i++){if(i<33){h.push(null);continue;}h.push(ml[i]-(sig[i-25]||0));}return{macdLine:ml,histogram:h};}
function calcOBV(cs){const o=[0];for(let i=1;i<cs.length;i++){if(cs[i].c>cs[i-1].c)o.push(o[i-1]+cs[i].v);else if(cs[i].c<cs[i-1].c)o.push(o[i-1]-cs[i].v);else o.push(o[i-1]);}return o;}
function calcSMA(v,p){return v.map((_,i)=>{if(i<p-1)return null;return v.slice(i-p+1,i+1).reduce((s,x)=>s+x,0)/p;});}
function zScore(value, arr) {
if (arr.length < 2) return 0;
const mean = arr.reduce((s,v)=>s+v,0)/arr.length;
const std = Math.sqrt(arr.reduce((s,v)=>s+(v-mean)**2,0)/arr.length);
return std === 0 ? 0 : (value - mean) / std;
}
function taConfluenceScore(cs, idx) {
const closes = cs.map(c => c.c);
const rsi = calcRSI(closes); const macd = calcMACD(closes); const obv = calcOBV(cs);
const ema9 = calcEMA(closes, 9); const ema21 = calcEMA(closes, 21); const bbSma = calcSMA(closes, 20);
const i = idx, ZLB = 30;
let score = 0;
if (rsi[i] !== null) { const w = rsi.slice(Math.max(0,i-ZLB), i).filter(v=>v!==null); const z = zScore(rsi[i], w); if (z > 0.3 && rsi[i] < 78) score += 2; if (z > 1.0 && rsi[i] < 75) score += 1; }
if (macd.histogram[i] !== null) { const w = macd.histogram.slice(Math.max(0,i-ZLB), i).filter(v=>v!==null); const z = zScore(macd.histogram[i], w); if (z > 0.3) score += 1; if (i > 0 && macd.histogram[i] > (macd.histogram[i-1]||0)) score += 1; }
if (ema9[i] > ema21[i]) score += 2;
if (bbSma[i] !== null && i >= 19) { const sl = closes.slice(i-19, i+1); const std = Math.sqrt(sl.reduce((s,v)=>s+(v-bbSma[i])**2,0)/20); const upper = bbSma[i]+2*std; const lower = bbSma[i]-2*std; const range = upper-lower; if(range>0){ const pB=(closes[i]-lower)/range; if(pB>0.5&&pB<0.85)score+=2; } }
if (i > 10) { const od = obv[i]-obv[i-5]; const ods = []; for(let j=Math.max(6,i-ZLB);j<i;j++) ods.push(obv[j]-obv[j-5]); const z = zScore(od, ods); if(z>0.5)score+=1; if(z>1.0)score+=1; }
return score;
}
// ─── Regime ───────────────────────────────────────────────────────────────────
async function getBTCRegime() {
try {
const r = await apiFetch('https://fapi.binance.com/fapi/v1/klines?symbol=BTCUSDT&interval=4h&limit=60');
const closes = r.map(k => parseFloat(k[4]));
const e20 = calcEMA(closes, 20), e50 = calcEMA(closes, 50);
return e20[e20.length-1] > e50[e50.length-1] ? 'bullish' : 'bearish';
} catch { return 'unknown'; }
}
async function getBTCFastRegime() {
try {
const r = await apiFetch('https://fapi.binance.com/fapi/v1/klines?symbol=BTCUSDT&interval=15m&limit=30');
const closes = r.map(k => parseFloat(k[4]));
const e8 = calcEMA(closes, 8), e21 = calcEMA(closes, 21);
const pct = (e8[e8.length-1] - e21[e21.length-1]) / e21[e21.length-1] * 100;
if (pct > 0.15) return { trend: 'bullish', spread: pct };
if (pct < -0.15) return { trend: 'bearish', spread: pct };
return { trend: 'neutral', spread: pct };
} catch { return { trend: 'unknown', spread: 0 }; }
}
async function getBTCDailyRegime() {
try {
const r = await apiFetch('https://fapi.binance.com/fapi/v1/klines?symbol=BTCUSDT&interval=1d&limit=60');
const closes = r.map(k => parseFloat(k[4]));
const e20 = calcEMA(closes, 20), e50 = calcEMA(closes, 50);
return e20[e20.length-1] > e50[e50.length-1] ? 'bullish' : 'bearish';
} catch { return 'unknown'; }
}
// ─── Data ─────────────────────────────────────────────────────────────────────
async function getCandles1h(sym, limit=80) {
await sleep(80);
try {
const r = await apiFetch(`https://fapi.binance.com/fapi/v1/klines?symbol=sym&interval=1h&limit=limit`);
return r.map(k => ({ ts:k[0], o:+k[1], h:+k[2], l:+k[3], c:+k[4], v:+k[5]*+k[4] }));
} catch { return []; }
}
async function getCandles4h(sym, limit=100) {
await sleep(80);
try {
const r = await apiFetch(`https://fapi.binance.com/fapi/v1/klines?symbol=sym&interval=4h&limit=limit`);
return r.map(k => ({ ts:k[0], o:+k[1], h:+k[2], l:+k[3], c:+k[4], v:+k[5]*+k[4] }));
} catch { return []; }
}
async function getDailyCandles(sym, limit=60) {
await sleep(80);
try {
const r = await apiFetch(`https://fapi.binance.com/fapi/v1/klines?symbol=sym&interval=1d&limit=limit`);
return r.map(k => ({ ts:k[0], o:+k[1], h:+k[2], l:+k[3], c:+k[4], v:+k[5]*+k[4] }));
} catch { return []; }
}
// ─── Day Trading Signal (1h SMC) ──────────────────────────────────────────────
function detectDaySignal(cs) {
const P = { swN:3, volZThreshold:1.0, fvgZThreshold:0.5, slAtrMult:1.5, tpAtrMult:3.0, taMinScore:5, volLookback:30, fvgLookback:50, atrPeriod:14 };
const hi = Array(cs.length).fill(null);
for (let i=P.swN; i<cs.length-P.swN; i++) {
let isH=true;
for (let j=1;j<=P.swN;j++) if(cs[i-j].h>=cs[i].h||cs[i+j].h>=cs[i].h) isH=false;
if(isH) hi[i]=cs[i].h;
}
const atrs = calcATR(cs, P.atrPeriod);
for (let i=cs.length-5; i<cs.length-1; i++) {
if (i < P.swN+2) continue;
let bosIdx = -1;
for (let j=i-1; j>=Math.max(0,i-60); j--) { if (hi[j]!==null && cs[i].c>hi[j]) { bosIdx=j; break; } }
if (bosIdx < 0) continue;
for (let j=Math.max(2,i-3); j<=i; j++) {
const gap = cs[j].l - cs[j-2].h;
if (gap <= 0) continue;
const gapPct = gap / cs[j].c;
const recentGaps = [];
for (let g=Math.max(2,j-P.fvgLookback); g<j; g++) { const pg=cs[g].l-cs[g-2].h; if(pg>0) recentGaps.push(pg/cs[g].c); }
const fvgZ = zScore(gapPct, recentGaps);
if (fvgZ < P.fvgZThreshold) continue;
const fvgLo=cs[j-2].h, fvgHi=cs[j].l, fvgMid=(fvgLo+fvgHi)/2;
const latest=cs[cs.length-1], lastIdx=cs.length-1;
if (!(latest.l<=fvgHi && latest.h>=fvgLo)) continue;
const volWindow = cs.slice(Math.max(0,lastIdx-P.volLookback),lastIdx).map(c=>c.v);
if (zScore(latest.v, volWindow) < P.volZThreshold) continue;
const atr = atrs[lastIdx]; if (!atr) continue;
const ep = Math.min(latest.c, fvgMid);
let obLo = ep - atr * P.slAtrMult;
for (let k=bosIdx-1; k>=Math.max(0,bosIdx-15); k--) { if (cs[k].c<cs[k].o) { obLo=cs[k].l; break; } }
const sl = Math.max(obLo*0.999, ep - atr * P.slAtrMult);
const tp = ep + atr * P.tpAtrMult;
if (sl >= ep) continue;
const ta = taConfluenceScore(cs, lastIdx);
if (ta < P.taMinScore) continue;
return { ep, sl, tp, ta, atr };
}
}
return null;
}
// ─── Swing Signal (Daily BoS → 4h FVG) ───────────────────────────────────────
function detectSwingSignal(k4h, kD) {
if (kD.length < 15 || k4h.length < 60) return null;
const SP = { swN:5, fvgAge:6, taMinScore:5, slAtrMult:2.5, tpAtrMult:5.0, atrPeriod:14 };
// Daily BoS check
const sh = Array(kD.length).fill(null);
for (let i=SP.swN; i<kD.length-SP.swN; i++) {
let ok=true;
for (let j=1;j<=SP.swN;j++) if(kD[i-j].h>=kD[i].h||kD[i+j].h>=kD[i].h){ok=false;break;}
if(ok) sh[i]=kD[i].h;
}
let hasBoS = false;
for (let i=kD.length-5; i<kD.length; i++) {
if (i<SP.swN+1) continue;
for (let j=i-1; j>=Math.max(0,i-30); j--) { if(sh[j]!==null && kD[i].c>sh[j]){hasBoS=true;break;} }
if (hasBoS) break;
}
if (!hasBoS) return null;
// 4h FVG entry
const atrs = calcATR(k4h, SP.atrPeriod);
for (let j=k4h.length-1; j>=Math.max(2,k4h.length-SP.fvgAge); j--) {
const gap = k4h[j].l - k4h[j-2].h;
if (gap<=0 || gap/k4h[j].c<0.0005) continue;
const fvgLo=k4h[j-2].h, fvgHi=k4h[j].l, fvgMid=(fvgLo+fvgHi)/2;
const latest = k4h[k4h.length-1], lastIdx = k4h.length-1;
if (!(latest.l<=fvgHi && latest.h>=fvgLo)) continue;
const atr = atrs[lastIdx]; if (!atr) continue;
const ep = Math.min(latest.c, fvgMid);
let obLo = ep - atr * SP.slAtrMult;
for (let k=j-1; k>=Math.max(0,j-15); k--) { if(k4h[k].c<k4h[k].o){obLo=k4h[k].l;break;} }
const sl = Math.min(obLo*0.999, ep - atr * SP.slAtrMult);
const tp = ep + atr * SP.tpAtrMult;
if (sl>=ep) continue;
const rr = (tp-ep)/(ep-sl); if (rr<1.5) continue;
const ta = taConfluenceScore(k4h, lastIdx);
if (ta < SP.taMinScore) continue;
return { ep, sl, tp, ta, atr };
}
return null;
}
// ─── Portfolio ────────────────────────────────────────────────────────────────
function loadPortfolio() {
try { return JSON.parse(fs.readFileSync(PORTFOLIO_FILE, 'utf8')); }
catch {
return {
version: '1.0', last_updated: new Date().toISOString(),
strategy: 'Coordinated 8D/2S — Day+Swing | Paper Trading',
btc_regime: 'unknown', btc_fast_regime: 'unknown', btc_daily_regime: 'unknown',
paper_portfolio: {
starting_capital_usdc: 1000, current_capital_usdc: 1000, deployed_usdc: 0,
open_positions: [], closed_trades: [],
stats: { total_trades:0, wins:0, losses:0, total_pnl_usdc:0 }
}
};
}
}
function savePortfolio(data) {
data.last_updated = new Date().toISOString();
fs.mkdirSync(path.dirname(PORTFOLIO_FILE), { recursive: true });
fs.writeFileSync(PORTFOLIO_FILE, JSON.stringify(data, null, 2));
}
// ─── Main ─────────────────────────────────────────────────────────────────────
async function run() {
const uaeTime = new Date().toLocaleString('en-AE', { timeZone: 'Asia/Dubai', hour12: false });
console.log(`\n[uaeTime UAE] Coordinated 8D/2S monitor running...`);
const portfolio = loadPortfolio();
const p = portfolio.paper_portfolio;
// Regimes
const btcRegime = await getBTCRegime();
const btcFast = await getBTCFastRegime();
const btcDaily = await getBTCDailyRegime();
portfolio.btc_regime = btcRegime;
portfolio.btc_fast_regime = btcFast.trend;
portfolio.btc_fast_spread = parseFloat(btcFast.spread.toFixed(3));
portfolio.btc_daily_regime = btcDaily;
const btcBull = btcRegime === 'bullish';
const sizeMultiplier = btcBull ? 1.0 : 0.5;
const fastAllowEntry = btcFast.trend !== 'bearish';
console.log(`BTC 4h: btcRegime | Daily: btcDaily | Fast 15m: btcFast.trend (btcFast.spread.toFixed(3)%)' ✅'`);
// Count positions by strategy
const dayPositions = p.open_positions.filter(pos => pos.strategy === 'day');
const swingPositions = p.open_positions.filter(pos => pos.strategy === 'swing');
console.log(`Positions: dayPositions.length/MAX_DAY day, swingPositions.length/MAX_SWING swing`);
// ── Manage existing positions ──
const now = Date.now();
const stillOpen = [];
for (const pos of p.open_positions) {
const ageH = (now - new Date(pos.entry_time).getTime()) / 3600000;
const maxH = pos.strategy === 'swing' ? 72 : 12;
const trail = pos.strategy === 'swing' ? 0.05 : 0.03; // 5% swing, 3% day
try {
await sleep(80);
const ticker = await apiFetch(`https://fapi.binance.com/fapi/v1/ticker/price?symbol=pos.symbol`);
const price = parseFloat(ticker.price);
pos.current_price = price;
let closed = false, exitType = '', exitPrice = price, pnlPct = 0;
if (price <= pos.sl_price) { exitType = 'SL'; exitPrice = pos.sl_price; closed = true; }
else if (!pos.tp1_hit && price >= pos.tp_price) {
pos.tp1_hit = true; pos.half_price = pos.tp_price; pos.trail_peak = price;
stillOpen.push(pos);
console.log(` 📈 pos.symbol [pos.strategy] TP1 hit at $price.toFixed(4)`);
continue;
}
else if (pos.tp1_hit) {
if (price > (pos.trail_peak || pos.tp_price)) pos.trail_peak = price;
const trailSL = (pos.trail_peak || price) * (1 - trail);
if (price <= trailSL) {
exitType = 'TRAIL'; exitPrice = trailSL; closed = true;
pnlPct = ((pos.half_price - pos.entry_price)*0.5 + (exitPrice - pos.entry_price)*0.5) / pos.entry_price * 100;
}
}
else if (ageH >= maxH) { exitType = 'TIME'; closed = true; }
if (closed) {
if (pnlPct === 0) pnlPct = (exitPrice - pos.entry_price) / pos.entry_price * 100;
const pnlUSD = pos.size_usdc * pnlPct / 100;
p.current_capital_usdc += pnlUSD;
p.stats.total_trades++;
if (pnlUSD > 0) p.stats.wins++; else p.stats.losses++;
p.stats.total_pnl_usdc += pnlUSD;
p.closed_trades.push({
symbol: pos.symbol, entry_price: pos.entry_price, exit_price: exitPrice,
entry_time: pos.entry_time, exit_time: new Date().toISOString(),
held_hours: parseFloat(ageH.toFixed(1)), pnl_pct: parseFloat(pnlPct.toFixed(3)),
pnl_usd: parseFloat(pnlUSD.toFixed(2)), exit_type: exitType,
size_usdc: pos.size_usdc, btc_bull: pos.btc_bull_at_entry, strategy: pos.strategy,
});
console.log(` '❌' pos.symbol [pos.strategy] exitType | ''pnlPct.toFixed(2)% | $''pnlUSD.toFixed(2) | ageH.toFixed(0)h`);
} else {
stillOpen.push(pos);
}
} catch { stillOpen.push(pos); }
}
p.open_positions = stillOpen;
// ── Scan for new day trading signals (1h) ──
const openDayCount = p.open_positions.filter(pos => pos.strategy === 'day').length;
const openSymbols = new Set(p.open_positions.map(pos => pos.symbol));
if (openDayCount < MAX_DAY && fastAllowEntry) {
for (const sym of TOKENS) {
if (p.open_positions.filter(pos => pos.strategy === 'day').length >= MAX_DAY) break;
if (openSymbols.has(sym)) continue;
const cs = await getCandles1h(sym);
if (cs.length < 60) continue;
const signal = detectDaySignal(cs);
if (!signal) continue;
const posSize = p.current_capital_usdc * 0.10 * sizeMultiplier;
p.open_positions.push({
symbol: sym, entry_price: signal.ep, sl_price: signal.sl, tp_price: signal.tp,
size_usdc: posSize, entry_time: new Date().toISOString(),
btc_bull_at_entry: btcBull, tp1_hit: false, trail_peak: null, half_price: null,
current_price: signal.ep, strategy: 'day',
});
openSymbols.add(sym);
console.log(` 🟢 DAY: sym | $signal.ep.toFixed(4) | SL:$signal.sl.toFixed(4) | TP:$signal.tp.toFixed(4) | TA:signal.ta/11`);
}
}
// ── Scan for new swing signals (4h + daily) ──
const openSwingCount = p.open_positions.filter(pos => pos.strategy === 'swing').length;
if (openSwingCount < MAX_SWING) {
for (const sym of TOKENS) {
if (p.open_positions.filter(pos => pos.strategy === 'swing').length >= MAX_SWING) break;
if (openSymbols.has(sym)) continue;
const [k4h, kD] = await Promise.all([getCandles4h(sym), getDailyCandles(sym)]);
const signal = detectSwingSignal(k4h, kD);
if (!signal) continue;
const posSize = p.current_capital_usdc * 0.10 * sizeMultiplier;
p.open_positions.push({
symbol: sym, entry_price: signal.ep, sl_price: signal.sl, tp_price: signal.tp,
size_usdc: posSize, entry_time: new Date().toISOString(),
btc_bull_at_entry: btcBull, tp1_hit: false, trail_peak: null, half_price: null,
current_price: signal.ep, atr_at_entry: signal.atr, strategy: 'swing',
});
openSymbols.add(sym);
console.log(` 🔵 SWING: sym | $signal.ep.toFixed(4) | SL:$signal.sl.toFixed(4) | TP:$signal.tp.toFixed(4) | TA:signal.ta/11`);
}
}
// ── Save ──
p.deployed_usdc = p.open_positions.reduce((s, pos) => s + pos.size_usdc, 0);
savePortfolio(portfolio);
// ── GitHub push ──
try {
const ghToken = fs.readFileSync(path.join(process.env.HOME, '.github_token'), 'utf8').trim();
const content = Buffer.from(JSON.stringify(portfolio, null, 2)).toString('base64');
const shaRes = await apiFetch2('https://api.github.com/repos/Zero2Ai-hub/Jarvis-Ops/contents/trading/portfolio-coordinated.json', ghToken);
const sha = shaRes.sha || '';
await apiFetch2Put('https://api.github.com/repos/Zero2Ai-hub/Jarvis-Ops/contents/trading/portfolio-coordinated.json', ghToken,
JSON.stringify({ message: 'trading: coordinated portfolio update', content, sha: sha || undefined }));
console.log('GitHub updated ✅');
} catch(e) { console.log('GitHub push failed:', e.message); }
// Summary
const totalPnl = p.current_capital_usdc - p.starting_capital_usdc;
const wr = p.stats.total_trades > 0 ? (p.stats.wins / p.stats.total_trades * 100).toFixed(1) : '—';
const dc = p.open_positions.filter(pos => pos.strategy === 'day').length;
const sc = p.open_positions.filter(pos => pos.strategy === 'swing').length;
console.log(`\nCoordinated: $p.current_capital_usdc.toFixed(2) | PnL: ''$totalPnl.toFixed(2) | WR: wr% | Day: dc/MAX_DAY | Swing: sc/MAX_SWING`);
console.log('Done ✅');
}
run().catch(e => { console.error('Error:', e.message); process.exit(1); });
FILE:scripts/paper-monitor-swing-v2.js
#!/usr/bin/env node
/**
* paper-monitor-swing-v2.js — Swing Trading with MTPI Gate
* Same SMC entry logic (Daily BoS → 4h FVG + TA≥5), but gated by MTPI
* probability instead of simple daily EMA cross.
* MTPI ≥ 0.5 → entries allowed. MTPI < 0.5 → no new positions.
* Runs every 4h (XX:35 UTC). Separate portfolio for A/B comparison.
*/
'use strict';
const https = require('https');
const fs = require('fs');
const path = require('path');
const PORTFOLIO_FILE = path.join(process.env.HOME, '.openclaw/workspace/trading/paper-dashboard/portfolio-swing-v2.json');
const REGIME_FILE = path.join(process.env.HOME, '.openclaw/workspace/trading/paper-dashboard/regime.json');
const TOKENS = [
'BTCUSDT','ETHUSDT','SOLUSDT','BNBUSDT','XRPUSDT','DOGEUSDT','AVAXUSDT','LINKUSDT',
'ARBUSDT','OPUSDT','SUIUSDT','APTUSDT','INJUSDT','SEIUSDT','FETUSDT','RENDERUSDT',
'TAOUSDT','NEARUSDT','WIFUSDT','JUPUSDT','TIAUSDT','DYMUSDT','STRKUSDT','TONUSDT',
'NOTUSDT','EIGENUSDT','GRASSUSDT','VIRTUALUSDT','AKTUSDT',
];
// Best swing config from Phase 3 backtest (row #9)
const P = {
swN: 5, // swing lookback for daily BoS
fvgAge: 6, // max FVG age in 4h candles (24h)
fvgSearch: 4, // candles to search for FVG after BoS
maxHold: 72, // max hold time in hours (3 days)
maxConcurrent: 5, // swing positions (orchestrator may override)
posPct: 0.10, // 10% position size
// ATR-based SL/TP
slAtrMult: 2.5,
tpAtrMult: 5.0,
trailAtrMult: 3.0,
atrPeriod: 14,
// TA z-score
taMinScore: 5,
taZLookback: 30,
// Volume z-score
volZThreshold: 0.0, // best config had volZ=0 (no filter)
volLookback: 30,
// FVG z-score
fvgZThreshold: 0.0,
};
function apiFetch(url) {
return new Promise((res, rej) => {
const r = https.get(url, resp => {
let d = '';
resp.on('data', c => d += c);
resp.on('end', () => { try { res(JSON.parse(d)); } catch(e) { rej(e); } });
});
r.on('error', rej);
r.setTimeout(15000, () => { r.destroy(); rej(new Error('timeout')); });
});
}
function apiFetch2(url, token) {
return new Promise((res, rej) => {
const r = https.get(url, { headers: { Authorization: `token token`, Accept: 'application/vnd.github.v3+json', 'User-Agent': 'Jarvis' } }, resp => {
let d = ''; resp.on('data', c => d += c);
resp.on('end', () => { try { res(JSON.parse(d)); } catch(e) { rej(e); } });
}); r.on('error', rej); r.setTimeout(10000, () => { r.destroy(); rej(new Error('timeout')); });
});
}
function apiFetch2Put(url, token, body) {
return new Promise((res, rej) => {
const opts = require('url').parse(url);
opts.method = 'PUT';
opts.headers = { Authorization: `token token`, Accept: 'application/vnd.github.v3+json', 'Content-Type': 'application/json', 'User-Agent': 'Jarvis', 'Content-Length': Buffer.byteLength(body) };
const r = require('https').request(opts, resp => {
let d = ''; resp.on('data', c => d += c);
resp.on('end', () => { try { res(JSON.parse(d)); } catch(e) { rej(e); } });
}); r.on('error', rej); r.write(body); r.end();
});
}
const sleep = ms => new Promise(r => setTimeout(r, ms));
// ─── Math helpers ─────────────────────────────────────────────────────────────
function calcEMA(v,p){const k=2/(p+1);let e=v[0];return v.map((x,i)=>{if(i>0)e=x*k+e*(1-k);return e;});}
function calcATR(cs,period=14){const t=cs.map((c,i)=>{if(i===0)return c.h-c.l;const p2=cs[i-1].c;return Math.max(c.h-c.l,Math.abs(c.h-p2),Math.abs(c.l-p2));});const a=[];for(let i=0;i<cs.length;i++){if(i<period-1){a.push(null);continue;}if(i===period-1){a.push(t.slice(0,period).reduce((s,v2)=>s+v2,0)/period);}else{a.push((a[i-1]*(period-1)+t[i])/period);}}return a;}
function calcRSI(c,p=14){const r=Array(c.length).fill(null);let ag=0,al=0;for(let i=1;i<=p;i++){const d=c[i]-c[i-1];if(d>0)ag+=d;else al-=d;}ag/=p;al/=p;r[p]=al===0?100:100-100/(1+ag/al);for(let i=p+1;i<c.length;i++){const d=c[i]-c[i-1];ag=(ag*(p-1)+(d>0?d:0))/p;al=(al*(p-1)+(d<0?-d:0))/p;r[i]=al===0?100:100-100/(1+ag/al);}return r;}
function calcMACD(c){const e12=calcEMA(c,12),e26=calcEMA(c,26);const ml=c.map((_,i)=>e12[i]-e26[i]);const sig=calcEMA(ml.slice(25),9);const h=[];for(let i=0;i<c.length;i++){if(i<33){h.push(null);continue;}h.push(ml[i]-(sig[i-25]||0));}return{macdLine:ml,histogram:h};}
function calcOBV(cs){const o=[0];for(let i=1;i<cs.length;i++){if(cs[i].c>cs[i-1].c)o.push(o[i-1]+cs[i].v);else if(cs[i].c<cs[i-1].c)o.push(o[i-1]-cs[i].v);else o.push(o[i-1]);}return o;}
function calcBB(c,p=20){const s=[],u=[],l=[];for(let i=0;i<c.length;i++){if(i<p-1){s.push(null);u.push(null);l.push(null);continue;}const sl=c.slice(i-p+1,i+1);const m=sl.reduce((a,b)=>a+b,0)/p;const st=Math.sqrt(sl.reduce((a,b)=>a+(b-m)**2,0)/p);s.push(m);u.push(m+2*st);l.push(m-2*st);}return{sma:s,upper:u,lower:l};}
function zScore(value, arr) {
if (arr.length < 2) return 0;
const mean = arr.reduce((s,v)=>s+v,0)/arr.length;
const std = Math.sqrt(arr.reduce((s,v)=>s+(v-mean)**2,0)/arr.length);
return std === 0 ? 0 : (value - mean) / std;
}
// ─── TA Confluence ────────────────────────────────────────────────────────────
function taConfluenceScore(cs, idx) {
const closes = cs.map(c => c.c);
const rsi = calcRSI(closes);
const macd = calcMACD(closes);
const obv = calcOBV(cs);
const ema9 = calcEMA(closes, 9);
const ema21 = calcEMA(closes, 21);
const bb = calcBB(closes);
const i = idx;
let score = 0;
if (rsi[i] !== null) {
const rsiW = rsi.slice(Math.max(0,i-P.taZLookback), i).filter(v=>v!==null);
const rsiZ = zScore(rsi[i], rsiW);
if (rsiZ > 0.3 && rsi[i] < 78) score += 2;
if (rsiZ > 1.0 && rsi[i] < 75) score += 1;
}
if (macd.histogram[i] !== null) {
const mW = macd.histogram.slice(Math.max(0,i-P.taZLookback), i).filter(v=>v!==null);
const mZ = zScore(macd.histogram[i], mW);
if (mZ > 0.3) score += 1;
if (i > 0 && macd.histogram[i] > (macd.histogram[i-1]||0)) score += 1;
}
if (ema9[i] > ema21[i]) score += 2;
if (bb.sma[i] !== null) {
const range = bb.upper[i] - bb.lower[i];
if (range > 0) {
const pB = (closes[i] - bb.lower[i]) / range;
if (pB > 0.5 && pB < 0.85) score += 2;
}
}
if (i > 10) {
const obvD = obv[i] - obv[i-5];
const obvDs = [];
for (let j = Math.max(6, i-P.taZLookback); j < i; j++) obvDs.push(obv[j] - obv[j-5]);
const oZ = zScore(obvD, obvDs);
if (oZ > 0.5) score += 1;
if (oZ > 1.0) score += 1;
}
return score;
}
// ─── BTC Regime ───────────────────────────────────────────────────────────────
async function getBTCRegime() {
try {
const r = await apiFetch('https://fapi.binance.com/fapi/v1/klines?symbol=BTCUSDT&interval=1d&limit=60');
const closes = r.map(k => parseFloat(k[4]));
const e20 = calcEMA(closes, 20), e50 = calcEMA(closes, 50);
return e20[e20.length-1] > e50[e50.length-1] ? 'bullish' : 'bearish';
} catch { return 'unknown'; }
}
// ─── Data ─────────────────────────────────────────────────────────────────────
async function getCandles4h(sym, limit=200) {
try {
await sleep(80);
const r = await apiFetch(`https://fapi.binance.com/fapi/v1/klines?symbol=sym&interval=4h&limit=limit`);
return r.map(k => ({
ts:k[0], o:+k[1], h:+k[2], l:+k[3], c:+k[4],
v:+k[5]*+k[4], tbv:+k[9]*+k[4]
}));
} catch { return []; }
}
async function getDailyCandles(sym, limit=120) {
try {
await sleep(80);
const r = await apiFetch(`https://fapi.binance.com/fapi/v1/klines?symbol=sym&interval=1d&limit=limit`);
return r.map(k => ({
ts:k[0], o:+k[1], h:+k[2], l:+k[3], c:+k[4], v:+k[5]*+k[4]
}));
} catch { return []; }
}
// ─── SMC on Daily → 4h Entry ─────────────────────────────────────────────────
function findDailySwingHighs(kD) {
const sh = Array(kD.length).fill(null);
const n = P.swN;
for (let i = n; i < kD.length - n; i++) {
let ok = true;
for (let j = 1; j <= n; j++) {
if (kD[i-j].h >= kD[i].h || kD[i+j].h >= kD[i].h) { ok = false; break; }
}
if (ok) sh[i] = kD[i].h;
}
return sh;
}
function hasDailyBoS(kD, sh) {
// Check if any of the last 5 daily candles broke a swing high
for (let i = kD.length - 5; i < kD.length; i++) {
if (i < P.swN + 1) continue;
for (let j = i - 1; j >= Math.max(0, i - 30); j--) {
if (sh[j] !== null && kD[i].c > sh[j]) return true;
}
}
return false;
}
function detect4hSignal(cs, hasBoS) {
if (!hasBoS) return null;
const atrs = calcATR(cs, P.atrPeriod);
// Look for FVG in recent candles (within fvgAge)
for (let j = cs.length - 1; j >= Math.max(2, cs.length - P.fvgAge); j--) {
const gap = cs[j].l - cs[j-2].h;
if (gap <= 0) continue;
const gapPct = gap / cs[j].c;
if (gapPct < 0.0005) continue;
// FVG z-score
if (P.fvgZThreshold > 0) {
const recentGaps = [];
for (let g = Math.max(2, j-50); g < j; g++) {
const pg = cs[g].l - cs[g-2].h;
if (pg > 0) recentGaps.push(pg / cs[g].c);
}
const fZ = zScore(gapPct, recentGaps);
if (fZ < P.fvgZThreshold) continue;
}
// Check if latest candle is retesting FVG
const fvgLo = cs[j-2].h, fvgHi = cs[j].l, fvgMid = (fvgLo + fvgHi) / 2;
const latest = cs[cs.length - 1];
const lastIdx = cs.length - 1;
if (!(latest.l <= fvgHi && latest.h >= fvgLo)) continue;
// Volume z-score
if (P.volZThreshold > 0) {
const vols = cs.slice(Math.max(0, lastIdx - P.volLookback), lastIdx).map(c => c.v);
const vZ = zScore(latest.v, vols);
if (vZ < P.volZThreshold) continue;
}
const atr = atrs[lastIdx];
if (!atr) continue;
const ep = Math.min(latest.c, fvgMid);
// OB low for SL
let obLo = ep - atr * P.slAtrMult;
for (let k = j - 1; k >= Math.max(0, j - 15); k--) {
if (cs[k].c < cs[k].o) { obLo = cs[k].l; break; }
}
const sl = Math.max(obLo * 0.999, ep - atr * P.slAtrMult);
const tp = ep + atr * P.tpAtrMult;
if (sl >= ep) continue;
const rr = (tp - ep) / (ep - sl);
if (rr < 1.5) continue;
return { ep, sl, tp, fvgMid, gap: gapPct, atr };
}
return null;
}
// ─── Portfolio ────────────────────────────────────────────────────────────────
function loadPortfolio() {
try {
return JSON.parse(fs.readFileSync(PORTFOLIO_FILE, 'utf8'));
} catch {
return {
version: '1.0',
last_updated: new Date().toISOString(),
strategy: 'Swing v2 — Daily BoS → 4h FVG + TA≥5 + MTPI Gate | Paper Trading',
btc_regime: 'unknown',
paper_portfolio: {
starting_capital_usdc: 1000,
current_capital_usdc: 1000,
deployed_usdc: 0,
open_positions: [],
closed_trades: [],
stats: { total_trades:0, wins:0, losses:0, total_pnl_usdc:0 }
}
};
}
}
function savePortfolio(data) {
data.last_updated = new Date().toISOString();
fs.mkdirSync(path.dirname(PORTFOLIO_FILE), { recursive: true });
fs.writeFileSync(PORTFOLIO_FILE, JSON.stringify(data, null, 2));
}
// ─── Main ─────────────────────────────────────────────────────────────────────
async function run() {
const uaeTime = new Date().toLocaleString('en-AE', { timeZone: 'Asia/Dubai', hour12: false });
console.log(`\n[uaeTime UAE] Swing paper monitor running...`);
const portfolio = loadPortfolio();
const p = portfolio.paper_portfolio;
// 1. Read MTPI from regime.json
let mtpi = 0.5, mtpiSignal = 'NO_DATA';
try {
const regime = JSON.parse(fs.readFileSync(REGIME_FILE, 'utf8'));
if (regime.mtpi) { mtpi = regime.mtpi.mtpi; mtpiSignal = regime.mtpi.signal; }
portfolio.btc_regime = regime.classification?.composite_signal || 'unknown';
portfolio.mtpi = mtpi;
portfolio.mtpi_signal = mtpiSignal;
} catch { console.log(' ⚠️ regime.json not found — using MTPI=0.5'); }
const mtpiAllowEntry = mtpi >= 0.5;
const sizeMultiplier = mtpi >= 0.6 ? 1.0 : 0.7;
console.log(`MTPI: (mtpi*100).toFixed(1)% mtpiSignal → '⛔ ENTRIES BLOCKED' | Size: (sizeMultiplier*100).toFixed(0)%`);
// 2. Manage open positions
const now = Date.now();
const stillOpen = [];
const atrs4h = {};
for (const pos of p.open_positions) {
const ageH = (now - new Date(pos.entry_time).getTime()) / 3600000;
try {
await sleep(80);
const ticker = await apiFetch(`https://fapi.binance.com/fapi/v1/ticker/price?symbol=pos.symbol`);
const price = parseFloat(ticker.price);
pos.current_price = price;
// Get current ATR for trailing
let currentATR = pos.atr_at_entry || 0;
if (!atrs4h[pos.symbol]) {
const cs = await getCandles4h(pos.symbol, 20);
if (cs.length >= 15) {
const atrs = calcATR(cs, 14);
currentATR = atrs[atrs.length - 1] || currentATR;
atrs4h[pos.symbol] = currentATR;
}
} else {
currentATR = atrs4h[pos.symbol];
}
let closed = false;
let exitType = '', exitPrice = price, pnlPct = 0;
// SL
if (price <= pos.sl_price) {
exitType = 'SL'; exitPrice = pos.sl_price; closed = true;
}
// TP1 (half position)
else if (!pos.tp1_hit && price >= pos.tp_price) {
pos.tp1_hit = true;
pos.half_price = pos.tp_price;
pos.trail_peak = price;
stillOpen.push(pos);
console.log(` 📈 pos.symbol TP1 hit at $price.toFixed(4) — trailing stop active`);
continue;
}
// Trailing stop after TP1
else if (pos.tp1_hit) {
if (price > (pos.trail_peak || pos.tp_price)) pos.trail_peak = price;
const trailSL = (pos.trail_peak || price) - currentATR * P.trailAtrMult;
if (price <= trailSL) {
exitType = 'TRAIL'; exitPrice = trailSL; closed = true;
pnlPct = ((pos.half_price - pos.entry_price)*0.5 + (exitPrice - pos.entry_price)*0.5) / pos.entry_price * 100;
}
}
// Time exit (72h = 3 days)
else if (ageH >= P.maxHold) {
exitType = 'TIME'; closed = true;
}
if (closed) {
if (pnlPct === 0) pnlPct = (exitPrice - pos.entry_price) / pos.entry_price * 100;
const pnlUSD = pos.size_usdc * pnlPct / 100;
p.current_capital_usdc += pnlUSD;
p.stats.total_trades++;
if (pnlUSD > 0) p.stats.wins++; else p.stats.losses++;
p.stats.total_pnl_usdc += pnlUSD;
p.closed_trades.push({
symbol: pos.symbol,
entry_price: pos.entry_price,
exit_price: exitPrice,
entry_time: pos.entry_time,
exit_time: new Date().toISOString(),
held_hours: parseFloat(ageH.toFixed(1)),
pnl_pct: parseFloat(pnlPct.toFixed(3)),
pnl_usd: parseFloat(pnlUSD.toFixed(2)),
exit_type: exitType,
size_usdc: pos.size_usdc,
btc_bull: pos.btc_bull_at_entry,
strategy: 'swing',
});
console.log(` '❌' pos.symbol CLOSED | exitType | ''pnlPct.toFixed(2)% | $''pnlUSD.toFixed(2) | held ageH.toFixed(0)h`);
} else {
stillOpen.push(pos);
}
} catch(e) {
stillOpen.push(pos);
}
}
p.open_positions = stillOpen;
// 3. Scan for new swing signals (gated by MTPI)
if (p.open_positions.length < P.maxConcurrent && mtpiAllowEntry) {
const openSymbols = new Set(p.open_positions.map(pos => pos.symbol));
for (const sym of TOKENS) {
if (p.open_positions.length >= P.maxConcurrent) break;
if (openSymbols.has(sym)) continue;
// Get daily candles for BoS detection
const kD = await getDailyCandles(sym, 60);
if (kD.length < 15) continue;
const dailySH = findDailySwingHighs(kD);
const hasBoS = hasDailyBoS(kD, dailySH);
if (!hasBoS) continue;
// Get 4h candles for entry
const k4h = await getCandles4h(sym, 100);
if (k4h.length < 60) continue;
const signal = detect4hSignal(k4h, hasBoS);
if (!signal) continue;
// TA confluence gate
const ta = taConfluenceScore(k4h, k4h.length - 1);
if (ta < P.taMinScore) continue;
const posSize = p.current_capital_usdc * P.posPct * sizeMultiplier;
const pos = {
symbol: sym,
entry_price: signal.ep,
sl_price: signal.sl,
tp_price: signal.tp,
size_usdc: posSize,
entry_time: new Date().toISOString(),
btc_bull_at_entry: mtpi >= 0.6,
tp1_hit: false,
trail_peak: null,
half_price: null,
current_price: signal.ep,
atr_at_entry: signal.atr,
strategy: 'swing',
};
p.open_positions.push(pos);
openSymbols.add(sym);
console.log(` 🟢 SWING SIGNAL: sym | Entry: $signal.ep.toFixed(4) | SL: $signal.sl.toFixed(4) | TP: $signal.tp.toFixed(4) | Size: $posSize.toFixed(2) | TA:ta/11`);
}
}
// 4. Update
p.deployed_usdc = p.open_positions.reduce((s, pos) => s + pos.size_usdc, 0);
savePortfolio(portfolio);
// 6. Push to GitHub
try {
const ghToken = fs.readFileSync(path.join(process.env.HOME, '.github_token'), 'utf8').trim();
const content = Buffer.from(JSON.stringify(portfolio, null, 2)).toString('base64');
const shaRes = await apiFetch2(`https://api.github.com/repos/Zero2Ai-hub/Jarvis-Ops/contents/trading/portfolio-swing-v2.json`, ghToken);
const sha = shaRes.sha || '';
const body = JSON.stringify({ message: 'trading: swing v2 portfolio update', content, sha: sha || undefined });
await apiFetch2Put(`https://api.github.com/repos/Zero2Ai-hub/Jarvis-Ops/contents/trading/portfolio-swing-v2.json`, ghToken, body);
console.log('GitHub portfolio-swing-v2.json updated ✅');
} catch(e) {
console.log('GitHub push failed (non-critical):', e.message);
}
// 7. Log observations
try {
const OBS_FILE = path.join(process.env.HOME, '.openclaw/workspace/trading/observations.md');
const closedThisRun = p.closed_trades.filter(t => {
return (now - new Date(t.exit_time).getTime()) / 60000 < 250; // within 4h window
});
if (closedThisRun.length > 0) {
const ts = new Date().toISOString().replace('T',' ').slice(0,19) + ' UTC';
let note = `\n## [ts] Swing Auto-Observation\n`;
note += `**BTC daily regime:** btcRegime\n`;
note += `**Closed this cycle:**\n`;
for (const t of closedThisRun) {
note += `- t.symbol: t.exit_type | ''t.pnl_pct% ($''t.pnl_usd) | held t.held_hoursh\n`;
}
note += `---\n`;
const existing = fs.existsSync(OBS_FILE) ? fs.readFileSync(OBS_FILE, 'utf8') : '# Trading Observations Log\n---\n';
const headerEnd = existing.indexOf('---\n');
if (headerEnd > -1) {
fs.writeFileSync(OBS_FILE, existing.slice(0, headerEnd + 4) + note + existing.slice(headerEnd + 4));
} else {
fs.writeFileSync(OBS_FILE, existing + note);
}
}
} catch {}
const totalPnl = p.current_capital_usdc - p.starting_capital_usdc;
const wr = p.stats.total_trades > 0 ? (p.stats.wins / p.stats.total_trades * 100).toFixed(1) : '—';
console.log(`\nSwing Portfolio: $p.current_capital_usdc.toFixed(2) | PnL: ''$totalPnl.toFixed(2) | WR: wr% | Open: p.open_positions.length/P.maxConcurrent`);
console.log('Done ✅');
}
run().catch(e => { console.error('Swing monitor error:', e.message); process.exit(1); });
FILE:scripts/paper-monitor-swing.js
#!/usr/bin/env node
/**
* paper-monitor-swing.js — Swing Paper Trading Monitor
* Best config from Phase 3 backtest: TA≥5, FVG6, srch4, SL2.5×ATR, TP5×ATR, trail3×ATR
* Uses 4H candles + Daily BoS detection. Runs every 4h.
* Separate portfolio from day trading for A/B comparison.
* Reads orchestrator lock file to prevent conflicting positions.
*/
'use strict';
const https = require('https');
const fs = require('fs');
const path = require('path');
const PORTFOLIO_FILE = path.join(process.env.HOME, '.openclaw/workspace/trading/paper-dashboard/portfolio-swing.json');
const LOCK_FILE = path.join(process.env.HOME, '.openclaw/workspace/trading/paper-dashboard/orchestrator-lock.json');
const TOKENS = [
'BTCUSDT','ETHUSDT','SOLUSDT','BNBUSDT','XRPUSDT','DOGEUSDT','AVAXUSDT','LINKUSDT',
'ARBUSDT','OPUSDT','SUIUSDT','APTUSDT','INJUSDT','SEIUSDT','FETUSDT','RENDERUSDT',
'TAOUSDT','NEARUSDT','WIFUSDT','JUPUSDT','TIAUSDT','DYMUSDT','STRKUSDT','TONUSDT',
'NOTUSDT','EIGENUSDT','GRASSUSDT','VIRTUALUSDT','AKTUSDT',
];
// Best swing config from Phase 3 backtest (row #9)
const P = {
swN: 5, // swing lookback for daily BoS
fvgAge: 6, // max FVG age in 4h candles (24h)
fvgSearch: 4, // candles to search for FVG after BoS
maxHold: 72, // max hold time in hours (3 days)
maxConcurrent: 5, // swing positions (orchestrator may override)
posPct: 0.10, // 10% position size
// ATR-based SL/TP
slAtrMult: 2.5,
tpAtrMult: 5.0,
trailAtrMult: 3.0,
atrPeriod: 14,
// TA z-score
taMinScore: 5,
taZLookback: 30,
// Volume z-score
volZThreshold: 0.0, // best config had volZ=0 (no filter)
volLookback: 30,
// FVG z-score
fvgZThreshold: 0.0,
};
function apiFetch(url) {
return new Promise((res, rej) => {
const r = https.get(url, resp => {
let d = '';
resp.on('data', c => d += c);
resp.on('end', () => { try { res(JSON.parse(d)); } catch(e) { rej(e); } });
});
r.on('error', rej);
r.setTimeout(15000, () => { r.destroy(); rej(new Error('timeout')); });
});
}
function apiFetch2(url, token) {
return new Promise((res, rej) => {
const r = https.get(url, { headers: { Authorization: `token token`, Accept: 'application/vnd.github.v3+json', 'User-Agent': 'Jarvis' } }, resp => {
let d = ''; resp.on('data', c => d += c);
resp.on('end', () => { try { res(JSON.parse(d)); } catch(e) { rej(e); } });
}); r.on('error', rej); r.setTimeout(10000, () => { r.destroy(); rej(new Error('timeout')); });
});
}
function apiFetch2Put(url, token, body) {
return new Promise((res, rej) => {
const opts = require('url').parse(url);
opts.method = 'PUT';
opts.headers = { Authorization: `token token`, Accept: 'application/vnd.github.v3+json', 'Content-Type': 'application/json', 'User-Agent': 'Jarvis', 'Content-Length': Buffer.byteLength(body) };
const r = require('https').request(opts, resp => {
let d = ''; resp.on('data', c => d += c);
resp.on('end', () => { try { res(JSON.parse(d)); } catch(e) { rej(e); } });
}); r.on('error', rej); r.write(body); r.end();
});
}
const sleep = ms => new Promise(r => setTimeout(r, ms));
// ─── Math helpers ─────────────────────────────────────────────────────────────
function calcEMA(v,p){const k=2/(p+1);let e=v[0];return v.map((x,i)=>{if(i>0)e=x*k+e*(1-k);return e;});}
function calcATR(cs,period=14){const t=cs.map((c,i)=>{if(i===0)return c.h-c.l;const p2=cs[i-1].c;return Math.max(c.h-c.l,Math.abs(c.h-p2),Math.abs(c.l-p2));});const a=[];for(let i=0;i<cs.length;i++){if(i<period-1){a.push(null);continue;}if(i===period-1){a.push(t.slice(0,period).reduce((s,v2)=>s+v2,0)/period);}else{a.push((a[i-1]*(period-1)+t[i])/period);}}return a;}
function calcRSI(c,p=14){const r=Array(c.length).fill(null);let ag=0,al=0;for(let i=1;i<=p;i++){const d=c[i]-c[i-1];if(d>0)ag+=d;else al-=d;}ag/=p;al/=p;r[p]=al===0?100:100-100/(1+ag/al);for(let i=p+1;i<c.length;i++){const d=c[i]-c[i-1];ag=(ag*(p-1)+(d>0?d:0))/p;al=(al*(p-1)+(d<0?-d:0))/p;r[i]=al===0?100:100-100/(1+ag/al);}return r;}
function calcMACD(c){const e12=calcEMA(c,12),e26=calcEMA(c,26);const ml=c.map((_,i)=>e12[i]-e26[i]);const sig=calcEMA(ml.slice(25),9);const h=[];for(let i=0;i<c.length;i++){if(i<33){h.push(null);continue;}h.push(ml[i]-(sig[i-25]||0));}return{macdLine:ml,histogram:h};}
function calcOBV(cs){const o=[0];for(let i=1;i<cs.length;i++){if(cs[i].c>cs[i-1].c)o.push(o[i-1]+cs[i].v);else if(cs[i].c<cs[i-1].c)o.push(o[i-1]-cs[i].v);else o.push(o[i-1]);}return o;}
function calcBB(c,p=20){const s=[],u=[],l=[];for(let i=0;i<c.length;i++){if(i<p-1){s.push(null);u.push(null);l.push(null);continue;}const sl=c.slice(i-p+1,i+1);const m=sl.reduce((a,b)=>a+b,0)/p;const st=Math.sqrt(sl.reduce((a,b)=>a+(b-m)**2,0)/p);s.push(m);u.push(m+2*st);l.push(m-2*st);}return{sma:s,upper:u,lower:l};}
function zScore(value, arr) {
if (arr.length < 2) return 0;
const mean = arr.reduce((s,v)=>s+v,0)/arr.length;
const std = Math.sqrt(arr.reduce((s,v)=>s+(v-mean)**2,0)/arr.length);
return std === 0 ? 0 : (value - mean) / std;
}
// ─── TA Confluence ────────────────────────────────────────────────────────────
function taConfluenceScore(cs, idx) {
const closes = cs.map(c => c.c);
const rsi = calcRSI(closes);
const macd = calcMACD(closes);
const obv = calcOBV(cs);
const ema9 = calcEMA(closes, 9);
const ema21 = calcEMA(closes, 21);
const bb = calcBB(closes);
const i = idx;
let score = 0;
if (rsi[i] !== null) {
const rsiW = rsi.slice(Math.max(0,i-P.taZLookback), i).filter(v=>v!==null);
const rsiZ = zScore(rsi[i], rsiW);
if (rsiZ > 0.3 && rsi[i] < 78) score += 2;
if (rsiZ > 1.0 && rsi[i] < 75) score += 1;
}
if (macd.histogram[i] !== null) {
const mW = macd.histogram.slice(Math.max(0,i-P.taZLookback), i).filter(v=>v!==null);
const mZ = zScore(macd.histogram[i], mW);
if (mZ > 0.3) score += 1;
if (i > 0 && macd.histogram[i] > (macd.histogram[i-1]||0)) score += 1;
}
if (ema9[i] > ema21[i]) score += 2;
if (bb.sma[i] !== null) {
const range = bb.upper[i] - bb.lower[i];
if (range > 0) {
const pB = (closes[i] - bb.lower[i]) / range;
if (pB > 0.5 && pB < 0.85) score += 2;
}
}
if (i > 10) {
const obvD = obv[i] - obv[i-5];
const obvDs = [];
for (let j = Math.max(6, i-P.taZLookback); j < i; j++) obvDs.push(obv[j] - obv[j-5]);
const oZ = zScore(obvD, obvDs);
if (oZ > 0.5) score += 1;
if (oZ > 1.0) score += 1;
}
return score;
}
// ─── BTC Regime ───────────────────────────────────────────────────────────────
async function getBTCRegime() {
try {
const r = await apiFetch('https://fapi.binance.com/fapi/v1/klines?symbol=BTCUSDT&interval=1d&limit=60');
const closes = r.map(k => parseFloat(k[4]));
const e20 = calcEMA(closes, 20), e50 = calcEMA(closes, 50);
return e20[e20.length-1] > e50[e50.length-1] ? 'bullish' : 'bearish';
} catch { return 'unknown'; }
}
// ─── Data ─────────────────────────────────────────────────────────────────────
async function getCandles4h(sym, limit=200) {
try {
await sleep(80);
const r = await apiFetch(`https://fapi.binance.com/fapi/v1/klines?symbol=sym&interval=4h&limit=limit`);
return r.map(k => ({
ts:k[0], o:+k[1], h:+k[2], l:+k[3], c:+k[4],
v:+k[5]*+k[4], tbv:+k[9]*+k[4]
}));
} catch { return []; }
}
async function getDailyCandles(sym, limit=120) {
try {
await sleep(80);
const r = await apiFetch(`https://fapi.binance.com/fapi/v1/klines?symbol=sym&interval=1d&limit=limit`);
return r.map(k => ({
ts:k[0], o:+k[1], h:+k[2], l:+k[3], c:+k[4], v:+k[5]*+k[4]
}));
} catch { return []; }
}
// ─── SMC on Daily → 4h Entry ─────────────────────────────────────────────────
function findDailySwingHighs(kD) {
const sh = Array(kD.length).fill(null);
const n = P.swN;
for (let i = n; i < kD.length - n; i++) {
let ok = true;
for (let j = 1; j <= n; j++) {
if (kD[i-j].h >= kD[i].h || kD[i+j].h >= kD[i].h) { ok = false; break; }
}
if (ok) sh[i] = kD[i].h;
}
return sh;
}
function hasDailyBoS(kD, sh) {
// Check if any of the last 5 daily candles broke a swing high
for (let i = kD.length - 5; i < kD.length; i++) {
if (i < P.swN + 1) continue;
for (let j = i - 1; j >= Math.max(0, i - 30); j--) {
if (sh[j] !== null && kD[i].c > sh[j]) return true;
}
}
return false;
}
function detect4hSignal(cs, hasBoS) {
if (!hasBoS) return null;
const atrs = calcATR(cs, P.atrPeriod);
// Look for FVG in recent candles (within fvgAge)
for (let j = cs.length - 1; j >= Math.max(2, cs.length - P.fvgAge); j--) {
const gap = cs[j].l - cs[j-2].h;
if (gap <= 0) continue;
const gapPct = gap / cs[j].c;
if (gapPct < 0.0005) continue;
// FVG z-score
if (P.fvgZThreshold > 0) {
const recentGaps = [];
for (let g = Math.max(2, j-50); g < j; g++) {
const pg = cs[g].l - cs[g-2].h;
if (pg > 0) recentGaps.push(pg / cs[g].c);
}
const fZ = zScore(gapPct, recentGaps);
if (fZ < P.fvgZThreshold) continue;
}
// Check if latest candle is retesting FVG
const fvgLo = cs[j-2].h, fvgHi = cs[j].l, fvgMid = (fvgLo + fvgHi) / 2;
const latest = cs[cs.length - 1];
const lastIdx = cs.length - 1;
if (!(latest.l <= fvgHi && latest.h >= fvgLo)) continue;
// Volume z-score
if (P.volZThreshold > 0) {
const vols = cs.slice(Math.max(0, lastIdx - P.volLookback), lastIdx).map(c => c.v);
const vZ = zScore(latest.v, vols);
if (vZ < P.volZThreshold) continue;
}
const atr = atrs[lastIdx];
if (!atr) continue;
const ep = Math.min(latest.c, fvgMid);
// OB low for SL
let obLo = ep - atr * P.slAtrMult;
for (let k = j - 1; k >= Math.max(0, j - 15); k--) {
if (cs[k].c < cs[k].o) { obLo = cs[k].l; break; }
}
const sl = Math.min(obLo * 0.999, ep - atr * P.slAtrMult);
const tp = ep + atr * P.tpAtrMult;
if (sl >= ep) continue;
const rr = (tp - ep) / (ep - sl);
if (rr < 1.5) continue;
return { ep, sl, tp, fvgMid, gap: gapPct, atr };
}
return null;
}
// ─── Orchestrator Lock ────────────────────────────────────────────────────────
function getLockedSymbols() {
try {
const lock = JSON.parse(fs.readFileSync(LOCK_FILE, 'utf8'));
return new Set(lock.day_trading_symbols || []);
} catch { return new Set(); }
}
function updateOrchestratorLock(openSymbols) {
try {
let lock = {};
try { lock = JSON.parse(fs.readFileSync(LOCK_FILE, 'utf8')); } catch {}
lock.swing_symbols = [...openSymbols];
lock.swing_updated = new Date().toISOString();
fs.writeFileSync(LOCK_FILE, JSON.stringify(lock, null, 2));
} catch {}
}
// ─── Portfolio ────────────────────────────────────────────────────────────────
function loadPortfolio() {
try {
return JSON.parse(fs.readFileSync(PORTFOLIO_FILE, 'utf8'));
} catch {
return {
version: '1.0',
last_updated: new Date().toISOString(),
strategy: 'Swing SMC — Daily BoS → 4h FVG + TA ≥5 | Paper Trading',
btc_regime: 'unknown',
paper_portfolio: {
starting_capital_usdc: 1000,
current_capital_usdc: 1000,
deployed_usdc: 0,
open_positions: [],
closed_trades: [],
stats: { total_trades:0, wins:0, losses:0, total_pnl_usdc:0 }
}
};
}
}
function savePortfolio(data) {
data.last_updated = new Date().toISOString();
fs.mkdirSync(path.dirname(PORTFOLIO_FILE), { recursive: true });
fs.writeFileSync(PORTFOLIO_FILE, JSON.stringify(data, null, 2));
}
// ─── Main ─────────────────────────────────────────────────────────────────────
async function run() {
const uaeTime = new Date().toLocaleString('en-AE', { timeZone: 'Asia/Dubai', hour12: false });
console.log(`\n[uaeTime UAE] Swing paper monitor running...`);
const portfolio = loadPortfolio();
const p = portfolio.paper_portfolio;
// 1. BTC daily regime
const btcRegime = await getBTCRegime();
portfolio.btc_regime = btcRegime;
const btcBull = btcRegime === 'bullish';
const sizeMultiplier = btcBull ? 1.0 : 0.5;
console.log(`BTC daily regime: btcRegime → size: '5%'`);
// 2. Check orchestrator lock for conflicting symbols
const lockedSymbols = getLockedSymbols();
if (lockedSymbols.size > 0) {
console.log(`Orchestrator lock: lockedSymbols.size symbols held by day trading: [...lockedSymbols].join(', ')`);
}
// 3. Manage open positions
const now = Date.now();
const stillOpen = [];
const atrs4h = {};
for (const pos of p.open_positions) {
const ageH = (now - new Date(pos.entry_time).getTime()) / 3600000;
try {
await sleep(80);
const ticker = await apiFetch(`https://fapi.binance.com/fapi/v1/ticker/price?symbol=pos.symbol`);
const price = parseFloat(ticker.price);
pos.current_price = price;
// Get current ATR for trailing
let currentATR = pos.atr_at_entry || 0;
if (!atrs4h[pos.symbol]) {
const cs = await getCandles4h(pos.symbol, 20);
if (cs.length >= 15) {
const atrs = calcATR(cs, 14);
currentATR = atrs[atrs.length - 1] || currentATR;
atrs4h[pos.symbol] = currentATR;
}
} else {
currentATR = atrs4h[pos.symbol];
}
let closed = false;
let exitType = '', exitPrice = price, pnlPct = 0;
// SL
if (price <= pos.sl_price) {
exitType = 'SL'; exitPrice = pos.sl_price; closed = true;
}
// TP1 (half position)
else if (!pos.tp1_hit && price >= pos.tp_price) {
pos.tp1_hit = true;
pos.half_price = pos.tp_price;
pos.trail_peak = price;
stillOpen.push(pos);
console.log(` 📈 pos.symbol TP1 hit at $price.toFixed(4) — trailing stop active`);
continue;
}
// Trailing stop after TP1
else if (pos.tp1_hit) {
if (price > (pos.trail_peak || pos.tp_price)) pos.trail_peak = price;
const trailSL = (pos.trail_peak || price) - currentATR * P.trailAtrMult;
if (price <= trailSL) {
exitType = 'TRAIL'; exitPrice = trailSL; closed = true;
pnlPct = ((pos.half_price - pos.entry_price)*0.5 + (exitPrice - pos.entry_price)*0.5) / pos.entry_price * 100;
}
}
// Time exit (72h = 3 days)
else if (ageH >= P.maxHold) {
exitType = 'TIME'; closed = true;
}
if (closed) {
if (pnlPct === 0) pnlPct = (exitPrice - pos.entry_price) / pos.entry_price * 100;
const pnlUSD = pos.size_usdc * pnlPct / 100;
p.current_capital_usdc += pnlUSD;
p.stats.total_trades++;
if (pnlUSD > 0) p.stats.wins++; else p.stats.losses++;
p.stats.total_pnl_usdc += pnlUSD;
p.closed_trades.push({
symbol: pos.symbol,
entry_price: pos.entry_price,
exit_price: exitPrice,
entry_time: pos.entry_time,
exit_time: new Date().toISOString(),
held_hours: parseFloat(ageH.toFixed(1)),
pnl_pct: parseFloat(pnlPct.toFixed(3)),
pnl_usd: parseFloat(pnlUSD.toFixed(2)),
exit_type: exitType,
size_usdc: pos.size_usdc,
btc_bull: pos.btc_bull_at_entry,
strategy: 'swing',
});
console.log(` '❌' pos.symbol CLOSED | exitType | ''pnlPct.toFixed(2)% | $''pnlUSD.toFixed(2) | held ageH.toFixed(0)h`);
} else {
stillOpen.push(pos);
}
} catch(e) {
stillOpen.push(pos);
}
}
p.open_positions = stillOpen;
// 4. Scan for new swing signals
if (p.open_positions.length < P.maxConcurrent) {
const openSymbols = new Set(p.open_positions.map(pos => pos.symbol));
for (const sym of TOKENS) {
if (p.open_positions.length >= P.maxConcurrent) break;
if (openSymbols.has(sym)) continue;
if (lockedSymbols.has(sym)) {
// Skip — day trading already has this symbol
continue;
}
// Get daily candles for BoS detection
const kD = await getDailyCandles(sym, 60);
if (kD.length < 15) continue;
const dailySH = findDailySwingHighs(kD);
const hasBoS = hasDailyBoS(kD, dailySH);
if (!hasBoS) continue;
// Get 4h candles for entry
const k4h = await getCandles4h(sym, 100);
if (k4h.length < 60) continue;
const signal = detect4hSignal(k4h, hasBoS);
if (!signal) continue;
// TA confluence gate
const ta = taConfluenceScore(k4h, k4h.length - 1);
if (ta < P.taMinScore) continue;
const posSize = p.current_capital_usdc * P.posPct * sizeMultiplier;
const pos = {
symbol: sym,
entry_price: signal.ep,
sl_price: signal.sl,
tp_price: signal.tp,
size_usdc: posSize,
entry_time: new Date().toISOString(),
btc_bull_at_entry: btcBull,
tp1_hit: false,
trail_peak: null,
half_price: null,
current_price: signal.ep,
atr_at_entry: signal.atr,
strategy: 'swing',
};
p.open_positions.push(pos);
openSymbols.add(sym);
console.log(` 🟢 SWING SIGNAL: sym | Entry: $signal.ep.toFixed(4) | SL: $signal.sl.toFixed(4) | TP: $signal.tp.toFixed(4) | Size: $posSize.toFixed(2) | TA:ta/11`);
}
}
// 5. Update
p.deployed_usdc = p.open_positions.reduce((s, pos) => s + pos.size_usdc, 0);
savePortfolio(portfolio);
// Update orchestrator lock
const swingSymbols = new Set(p.open_positions.map(pos => pos.symbol));
updateOrchestratorLock(swingSymbols);
// 6. Push to GitHub
try {
const ghToken = fs.readFileSync(path.join(process.env.HOME, '.github_token'), 'utf8').trim();
const content = Buffer.from(JSON.stringify(portfolio, null, 2)).toString('base64');
const shaRes = await apiFetch2(`https://api.github.com/repos/Zero2Ai-hub/Jarvis-Ops/contents/trading/portfolio-swing.json`, ghToken);
const sha = shaRes.sha || '';
const body = JSON.stringify({ message: 'trading: swing portfolio update', content, sha: sha || undefined });
await apiFetch2Put(`https://api.github.com/repos/Zero2Ai-hub/Jarvis-Ops/contents/trading/portfolio-swing.json`, ghToken, body);
console.log('GitHub portfolio-swing.json updated ✅');
} catch(e) {
console.log('GitHub push failed (non-critical):', e.message);
}
// 7. Log observations
try {
const OBS_FILE = path.join(process.env.HOME, '.openclaw/workspace/trading/observations.md');
const closedThisRun = p.closed_trades.filter(t => {
return (now - new Date(t.exit_time).getTime()) / 60000 < 250; // within 4h window
});
if (closedThisRun.length > 0) {
const ts = new Date().toISOString().replace('T',' ').slice(0,19) + ' UTC';
let note = `\n## [ts] Swing Auto-Observation\n`;
note += `**BTC daily regime:** btcRegime\n`;
note += `**Closed this cycle:**\n`;
for (const t of closedThisRun) {
note += `- t.symbol: t.exit_type | ''t.pnl_pct% ($''t.pnl_usd) | held t.held_hoursh\n`;
}
note += `---\n`;
const existing = fs.existsSync(OBS_FILE) ? fs.readFileSync(OBS_FILE, 'utf8') : '# Trading Observations Log\n---\n';
const headerEnd = existing.indexOf('---\n');
if (headerEnd > -1) {
fs.writeFileSync(OBS_FILE, existing.slice(0, headerEnd + 4) + note + existing.slice(headerEnd + 4));
} else {
fs.writeFileSync(OBS_FILE, existing + note);
}
}
} catch {}
const totalPnl = p.current_capital_usdc - p.starting_capital_usdc;
const wr = p.stats.total_trades > 0 ? (p.stats.wins / p.stats.total_trades * 100).toFixed(1) : '—';
console.log(`\nSwing Portfolio: $p.current_capital_usdc.toFixed(2) | PnL: ''$totalPnl.toFixed(2) | WR: wr% | Open: p.open_positions.length/P.maxConcurrent`);
console.log('Done ✅');
}
run().catch(e => { console.error('Swing monitor error:', e.message); process.exit(1); });
FILE:scripts/paper-monitor-v5.js
#!/usr/bin/env node
/**
* paper-monitor-v5.js — Live Paper Trading Monitor
* SMC v5.0 — Hybrid SMC + TA + CVD Confirmation
* Runs every 1h (XX:01 UTC). Scans all 29 tokens.
* SMC structural signal + z-scored TA (≥5) + strict CVD delta (z≥0.5).
* Separate portfolio from v4.0 for A/B comparison.
*/
'use strict';
const https = require('https');
const fs = require('fs');
const path = require('path');
const PORTFOLIO_FILE = path.join(process.env.HOME, '.openclaw/workspace/trading/paper-dashboard/portfolio-v5.json');
const JOURNAL_FILE = path.join(process.env.HOME, '.openclaw/workspace/trading/journal-v5.json');
const LOCK_FILE = path.join(process.env.HOME, '.openclaw/workspace/trading/paper-dashboard/orchestrator-lock.json');
const TOKENS = [
'BTCUSDT','ETHUSDT','SOLUSDT','BNBUSDT','XRPUSDT','DOGEUSDT','AVAXUSDT','LINKUSDT',
'ARBUSDT','OPUSDT','SUIUSDT','APTUSDT','INJUSDT','SEIUSDT','FETUSDT','RENDERUSDT',
'TAOUSDT','NEARUSDT','WIFUSDT','JUPUSDT','TIAUSDT','DYMUSDT','STRKUSDT','TONUSDT',
'NOTUSDT','EIGENUSDT','GRASSUSDT','VIRTUALUSDT','AKTUSDT',
];
const P = { swN:3, fvgAge:8, tr:0.03, maxH:12, maxConcurrent:10, posPct:0.10,
// v5.0 z-score params (Config H: SMC+TA≥5+CVD z≥0.5)
volZThreshold: 1.0, volLookback: 30,
fvgZThreshold: 0.5, fvgLookback: 50,
slAtrMult: 1.5, tpAtrMult: 3.0, atrPeriod: 14,
taMinScore: 5, taZLookback: 30,
cvdZThreshold: 0.5, cvdLookback: 30,
};
function apiFetch2(url, token) {
return new Promise((res, rej) => {
const r = require('https').get(url, { headers: { Authorization: `token token`, Accept: 'application/vnd.github.v3+json', 'User-Agent': 'Jarvis' } }, resp => {
let d = ''; resp.on('data', c => d += c);
resp.on('end', () => { try { res(JSON.parse(d)); } catch(e) { rej(e); } });
}); r.on('error', rej); r.setTimeout(10000, () => { r.destroy(); rej(new Error('timeout')); });
});
}
function apiFetch2Put(url, token, body) {
return new Promise((res, rej) => {
const opts = require('url').parse(url);
opts.method = 'PUT';
opts.headers = { Authorization: `token token`, Accept: 'application/vnd.github.v3+json', 'Content-Type': 'application/json', 'User-Agent': 'Jarvis', 'Content-Length': Buffer.byteLength(body) };
const r = require('https').request(opts, resp => {
let d = ''; resp.on('data', c => d += c);
resp.on('end', () => { try { res(JSON.parse(d)); } catch(e) { rej(e); } });
}); r.on('error', rej); r.write(body); r.end();
});
}
function apiFetch(url) {
return new Promise((res, rej) => {
const r = https.get(url, resp => {
let d = '';
resp.on('data', c => d += c);
resp.on('end', () => { try { res(JSON.parse(d)); } catch(e) { rej(e); } });
});
r.on('error', rej);
r.setTimeout(10000, () => { r.destroy(); rej(new Error('timeout')); });
});
}
const sleep = ms => new Promise(r => setTimeout(r, ms));
function calcEMA(v, p) {
const k = 2/(p+1); let e = v[0];
return v.map((x,i) => { if(i>0) e=x*k+e*(1-k); return e; });
}
function zScore(value, arr) {
if (arr.length < 2) return 0;
const mean = arr.reduce((s,v)=>s+v,0)/arr.length;
const std = Math.sqrt(arr.reduce((s,v)=>s+(v-mean)**2,0)/arr.length);
if (std === 0) return 0;
return (value - mean) / std;
}
function calcATR(cs, period=14) {
const trs = cs.map((c,i) => {
if (i===0) return c.h - c.l;
const prev = cs[i-1].c;
return Math.max(c.h - c.l, Math.abs(c.h - prev), Math.abs(c.l - prev));
});
const atrs = [];
for (let i=0; i<cs.length; i++) {
if (i < period-1) { atrs.push(null); continue; }
if (i === period-1) {
atrs.push(trs.slice(0, period).reduce((s,v)=>s+v,0)/period);
} else {
atrs.push((atrs[i-1]*(period-1)+trs[i])/period);
}
}
return atrs;
}
function calcSMA(v, p) {
return v.map((_, i) => {
if (i < p-1) return null;
return v.slice(i-p+1, i+1).reduce((s,x)=>s+x,0)/p;
});
}
function calcRSI(closes, period=14) {
const rsi = new Array(closes.length).fill(null);
let avgGain = 0, avgLoss = 0;
for (let i = 1; i <= period; i++) {
const d = closes[i] - closes[i-1];
if (d > 0) avgGain += d; else avgLoss += Math.abs(d);
}
avgGain /= period; avgLoss /= period;
rsi[period] = avgLoss === 0 ? 100 : 100 - 100/(1+avgGain/avgLoss);
for (let i = period+1; i < closes.length; i++) {
const d = closes[i] - closes[i-1];
avgGain = (avgGain*(period-1) + (d>0?d:0)) / period;
avgLoss = (avgLoss*(period-1) + (d<0?Math.abs(d):0)) / period;
rsi[i] = avgLoss === 0 ? 100 : 100 - 100/(1+avgGain/avgLoss);
}
return rsi;
}
function calcMACD(closes) {
const e12 = calcEMA(closes, 12), e26 = calcEMA(closes, 26);
const macdLine = closes.map((_, i) => e12[i] - e26[i]);
const signal = calcEMA(macdLine.slice(25), 9);
const histogram = [];
for (let i = 0; i < closes.length; i++) {
if (i < 33) { histogram.push(null); continue; }
histogram.push(macdLine[i] - (signal[i-25] || 0));
}
return { macdLine, histogram };
}
function calcOBV(cs) {
const obv = [0];
for (let i = 1; i < cs.length; i++) {
if (cs[i].c > cs[i-1].c) obv.push(obv[i-1] + cs[i].v);
else if (cs[i].c < cs[i-1].c) obv.push(obv[i-1] - cs[i].v);
else obv.push(obv[i-1]);
}
return obv;
}
function taConfluenceScore(cs, idx) {
const closes = cs.map(c => c.c);
const rsi = calcRSI(closes, 14);
const macd = calcMACD(closes);
const obv = calcOBV(cs);
const ema9 = calcEMA(closes, 9);
const ema21 = calcEMA(closes, 21);
const bbSma = calcSMA(closes, 20);
const i = idx;
const ZLB = P.taZLookback;
let score = 0;
// Z-scored RSI
if (rsi[i] !== null) {
const rsiWindow = rsi.slice(Math.max(0,i-ZLB), i).filter(v=>v!==null);
const rsiZ = zScore(rsi[i], rsiWindow);
if (rsiZ > 0.3 && rsi[i] < 78) score += 2;
if (rsiZ > 1.0 && rsi[i] < 75) score += 1;
}
// Z-scored MACD histogram
if (macd.histogram[i] !== null) {
const macdWindow = macd.histogram.slice(Math.max(0,i-ZLB), i).filter(v=>v!==null);
const macdZ = zScore(macd.histogram[i], macdWindow);
if (macdZ > 0.3) score += 1;
if (i > 0 && macd.histogram[i] !== null && macd.histogram[i-1] !== null && macd.histogram[i] > macd.histogram[i-1]) score += 1;
}
// EMA alignment (9 > 21)
if (ema9[i] > ema21[i]) score += 2;
// BB position (sweet spot)
if (bbSma[i] !== null && i >= 19) {
const slice = closes.slice(i-19, i+1);
const std = Math.sqrt(slice.reduce((s,v)=>s+(v-bbSma[i])**2,0)/20);
const upper = bbSma[i] + 2*std;
const lower = bbSma[i] - 2*std;
const range = upper - lower;
if (range > 0) {
const bbPct = (closes[i] - lower) / range;
if (bbPct > 0.5 && bbPct < 0.85) score += 2;
}
}
// Z-scored OBV
if (i > 10) {
const obvDelta = obv[i] - obv[i-5];
const obvDeltas = [];
for (let j = Math.max(6, i-ZLB); j < i; j++) {
obvDeltas.push(obv[j] - obv[j-5]);
}
const obvZ = zScore(obvDelta, obvDeltas);
if (obvZ > 0.5) score += 1;
if (obvZ > 1.0) score += 1;
}
return score; // max ~11
}
async function getBTCRegime() {
try {
const r = await apiFetch('https://fapi.binance.com/fapi/v1/klines?symbol=BTCUSDT&interval=4h&limit=60');
const closes = r.map(k => parseFloat(k[4]));
const e20 = calcEMA(closes, 20), e50 = calcEMA(closes, 50);
return e20[e20.length-1] > e50[e50.length-1] ? 'bullish' : 'bearish';
} catch { return 'unknown'; }
}
async function getBTCFastRegime() {
try {
const r = await apiFetch('https://fapi.binance.com/fapi/v1/klines?symbol=BTCUSDT&interval=15m&limit=30');
const closes = r.map(k => parseFloat(k[4]));
const e8 = calcEMA(closes, 8), e21 = calcEMA(closes, 21);
const fast = e8[e8.length-1], slow = e21[e21.length-1];
// Also check momentum: is price accelerating down?
const pctFromSlow = (fast - slow) / slow * 100;
// bullish: fast > slow, neutral: within 0.15%, bearish: fast < slow
if (pctFromSlow > 0.15) return { trend: 'bullish', spread: pctFromSlow };
if (pctFromSlow < -0.15) return { trend: 'bearish', spread: pctFromSlow };
return { trend: 'neutral', spread: pctFromSlow };
} catch { return { trend: 'unknown', spread: 0 }; }
}
async function getCandles(sym, limit=100) {
try {
await sleep(80);
const r = await apiFetch(`https://fapi.binance.com/fapi/v1/klines?symbol=sym&interval=1h&limit=limit`);
return r.map(k => ({
ts:k[0], o:parseFloat(k[1]), h:parseFloat(k[2]), l:parseFloat(k[3]), c:parseFloat(k[4]),
v:parseFloat(k[5])*parseFloat(k[4]),
tbv:parseFloat(k[9])*parseFloat(k[4]) // taker buy quote volume
}));
} catch { return []; }
}
function calcCVDDelta(cs) {
// Delta per candle = taker buy - taker sell (taker sell = total - taker buy)
return cs.map(c => {
const buyVol = c.tbv || 0;
const sellVol = (c.v || 0) - buyVol;
return buyVol - sellVol;
});
}
function detectSignal(cs) {
// Find swing highs
const hi = new Array(cs.length).fill(null);
const n = P.swN;
for (let i=n; i<cs.length-n; i++) {
let isH=true;
for (let j=1;j<=n;j++) if(cs[i-j].h>=cs[i].h||cs[i+j].h>=cs[i].h) isH=false;
if(isH) hi[i]=cs[i].h;
}
// Precompute ATR for SL/TP
const atrs = calcATR(cs, P.atrPeriod);
// Check last 5 candles for BoS + FVG
for (let i=cs.length-5; i<cs.length-1; i++) {
if (i < n+2) continue;
// BoS: close breaks prior swing high
let bosIdx = -1;
for (let j=i-1; j>=Math.max(0,i-60); j--) {
if (hi[j]!==null && cs[i].c>hi[j]) { bosIdx=j; break; }
}
if (bosIdx < 0) continue;
// FVG within 3 candles after BoS
for (let j=Math.max(2,i-3); j<=i; j++) {
const gap = cs[j].l - cs[j-2].h;
if (gap <= 0) continue;
const gapPct = gap / cs[j].c;
// v3.2: Z-scored FVG size (must be 0.5σ above recent average)
const recentGaps = [];
for (let g=Math.max(2, j-P.fvgLookback); g<j; g++) {
const pg = cs[g].l - cs[g-2].h;
if (pg > 0) recentGaps.push(pg / cs[g].c);
}
const fvgZ = zScore(gapPct, recentGaps);
if (fvgZ < P.fvgZThreshold) continue;
// Check if price is retesting the FVG right now (latest candle)
const fvgLo=cs[j-2].h, fvgHi=cs[j].l, fvgMid=(fvgLo+fvgHi)/2;
const latest=cs[cs.length-1];
const lastIdx = cs.length - 1;
if (!(latest.l<=fvgHi && latest.h>=fvgLo)) continue;
// v3.2: Z-scored volume (must be 1.0σ above recent average)
const volWindow = cs.slice(Math.max(0, lastIdx-P.volLookback), lastIdx).map(c=>c.v);
const volZ = zScore(latest.v, volWindow);
if (volZ < P.volZThreshold) continue;
// v3.2: ATR-based SL/TP
const atr = atrs[lastIdx];
if (!atr) continue;
const ep = Math.min(latest.c, fvgMid);
// SL: max of (OB low, entry - 1.5×ATR)
let obLo = ep - atr * P.slAtrMult;
for (let k=bosIdx-1; k>=Math.max(0,bosIdx-15); k--) {
if (cs[k].c<cs[k].o) { obLo=cs[k].l; break; }
}
const sl = Math.max(obLo*0.999, ep - atr * P.slAtrMult);
const tp = ep + atr * P.tpAtrMult;
if (sl >= ep) continue;
return { ep, sl, tp, fvgMid, gap: gapPct, volZ: volZ.toFixed(1), fvgZ: fvgZ.toFixed(1), atr };
}
}
return null;
}
function loadPortfolio() {
try {
return JSON.parse(fs.readFileSync(PORTFOLIO_FILE, 'utf8'));
} catch {
return {
version: '1.0',
last_updated: new Date().toISOString(),
strategy: 'SMC v5.0 — Hybrid SMC+TA+CVD | Paper Trading',
btc_regime: 'unknown',
paper_portfolio: {
starting_capital_usdc: 1000,
current_capital_usdc: 1000,
deployed_usdc: 0,
open_positions: [],
closed_trades: [],
stats: { total_trades:0, wins:0, losses:0, total_pnl_usdc:0 }
}
};
}
}
function savePortfolio(data) {
data.last_updated = new Date().toISOString();
fs.mkdirSync(path.dirname(PORTFOLIO_FILE), { recursive: true });
fs.writeFileSync(PORTFOLIO_FILE, JSON.stringify(data, null, 2));
}
async function run() {
const uaeTime = new Date().toLocaleString('en-AE', { timeZone: 'Asia/Dubai', hour12: false });
console.log(`\n[uaeTime UAE] Paper monitor running...`);
const portfolio = loadPortfolio();
const p = portfolio.paper_portfolio;
// 1. Check BTC regime (macro + fast)
const btcRegime = await getBTCRegime();
const btcFast = await getBTCFastRegime();
portfolio.btc_regime = btcRegime;
portfolio.btc_fast_regime = btcFast.trend;
portfolio.btc_fast_spread = parseFloat(btcFast.spread.toFixed(3));
const btcBull = btcRegime === 'bullish';
const sizeMultiplier = btcBull ? 1.0 : 0.5;
// Fast regime gates new entries — if bearish on 15m, skip new longs
const fastAllowEntry = btcFast.trend !== 'bearish';
console.log(`BTC regime: btcRegime → position size: '5%'`);
console.log(`BTC fast (15m): btcFast.trend (spread: btcFast.spread.toFixed(3)%)' ✅'`);
// 2. Close expired positions
const now = Date.now();
const stillOpen = [];
for (const pos of p.open_positions) {
const ageH = (now - new Date(pos.entry_time).getTime()) / 3600000;
// Get current price
try {
await sleep(80);
const ticker = await apiFetch(`https://fapi.binance.com/fapi/v1/ticker/price?symbol=pos.symbol`);
const price = parseFloat(ticker.price);
pos.current_price = price;
let closed = false;
let exitType = '', exitPrice = price, pnlPct = 0;
// Check SL
if (price <= pos.sl_price) {
exitType = 'SL'; exitPrice = pos.sl_price; closed = true;
}
// Check TP1 (if not yet hit)
else if (!pos.tp1_hit && price >= pos.tp_price) {
pos.tp1_hit = true;
pos.half_price = pos.tp_price;
pos.trail_peak = price;
stillOpen.push(pos);
console.log(` 📈 pos.symbol TP1 hit at $price.toFixed(4) — trailing stop active`);
continue;
}
// Check trailing stop after TP1
else if (pos.tp1_hit) {
if (price > (pos.trail_peak || pos.tp_price)) pos.trail_peak = price;
const trailSL = (pos.trail_peak || price) * (1 - P.tr);
if (price <= trailSL) {
exitType = 'TRAIL'; exitPrice = trailSL; closed = true;
pnlPct = ((pos.half_price - pos.entry_price)*0.5 + (exitPrice - pos.entry_price)*0.5) / pos.entry_price * 100;
}
}
// Time exit
else if (ageH >= P.maxH) {
exitType = 'TIME'; exitPrice = price; closed = true;
}
if (closed) {
if (pnlPct === 0) pnlPct = (exitPrice - pos.entry_price) / pos.entry_price * 100;
const pnlUSD = pos.size_usdc * pnlPct / 100;
p.current_capital_usdc += pnlUSD;
p.stats.total_trades++;
if (pnlUSD > 0) p.stats.wins++; else p.stats.losses++;
p.stats.total_pnl_usdc += pnlUSD;
const trade = {
symbol: pos.symbol,
entry_price: pos.entry_price,
exit_price: exitPrice,
entry_time: pos.entry_time,
exit_time: new Date().toISOString(),
held_hours: parseFloat(ageH.toFixed(1)),
pnl_pct: parseFloat(pnlPct.toFixed(3)),
pnl_usd: parseFloat(pnlUSD.toFixed(2)),
exit_type: exitType,
size_usdc: pos.size_usdc,
btc_bull: pos.btc_bull_at_entry,
};
p.closed_trades.push(trade);
console.log(` '❌' pos.symbol CLOSED | exitType | ''pnlPct.toFixed(2)% | $''pnlUSD.toFixed(2)`);
} else {
stillOpen.push(pos);
}
} catch(e) {
stillOpen.push(pos); // keep open on error
}
}
p.open_positions = stillOpen;
// 3. Scan for new signals (gated by fast regime + orchestrator lock)
let swingLockedSymbols = new Set();
try {
const lock = JSON.parse(fs.readFileSync(LOCK_FILE, 'utf8'));
swingLockedSymbols = new Set(lock.swing_symbols || []);
if (swingLockedSymbols.size > 0) console.log(`Orchestrator: swingLockedSymbols.size symbols locked by swing: [...swingLockedSymbols].join(', ')`);
} catch {}
if (p.open_positions.length < P.maxConcurrent && fastAllowEntry) {
const openSymbols = new Set(p.open_positions.map(pos => pos.symbol));
for (const sym of TOKENS) {
if (p.open_positions.length >= P.maxConcurrent) break;
if (openSymbols.has(sym)) continue; // already in position
if (swingLockedSymbols.has(sym)) continue; // swing holds this
const cs = await getCandles(sym, 80);
if (cs.length < 60) continue;
const signal = detectSignal(cs);
if (!signal) continue;
// v4.0: TA confluence confirmation
const ta = taConfluenceScore(cs, cs.length - 1);
if (ta < P.taMinScore) continue;
// v5.0: CVD delta confirmation (z ≥ 0.5 = strong buying aggression)
const deltas = calcCVDDelta(cs);
const lastIdx = cs.length - 1;
const deltaWindow = deltas.slice(Math.max(0, lastIdx - P.cvdLookback), lastIdx);
const cvdZ = zScore(deltas[lastIdx], deltaWindow);
if (cvdZ < P.cvdZThreshold) continue;
const posSize = p.current_capital_usdc * P.posPct * sizeMultiplier;
const pos = {
symbol: sym,
entry_price: signal.ep,
sl_price: signal.sl,
tp_price: signal.tp,
size_usdc: posSize,
entry_time: new Date().toISOString(),
btc_bull_at_entry: btcBull,
tp1_hit: false,
trail_peak: null,
half_price: null,
current_price: signal.ep,
};
p.open_positions.push(pos);
console.log(` 🟢 NEW SIGNAL: sym | Entry: $signal.ep.toFixed(4) | SL: $signal.sl.toFixed(4) | TP: $signal.tp.toFixed(4) | Size: $posSize.toFixed(2) | volZ:signal.volZ fvgZ:signal.fvgZ TA:ta/11 CVD:cvdZ.toFixed(1)`);
}
}
// 4. Update deployed
p.deployed_usdc = p.open_positions.reduce((s, pos) => s + pos.size_usdc, 0);
// 5. Save locally + update orchestrator lock
savePortfolio(portfolio);
try {
let lock = {};
try { lock = JSON.parse(fs.readFileSync(LOCK_FILE, 'utf8')); } catch {}
lock.day_trading_symbols = p.open_positions.map(pos => pos.symbol);
lock.day_trading_updated = new Date().toISOString();
fs.writeFileSync(LOCK_FILE, JSON.stringify(lock, null, 2));
} catch {}
// 6. Push to GitHub (feeds Vercel dashboard)
try {
const fs2 = require('fs'), path2 = require('path');
const ghToken = fs2.readFileSync(path2.join(process.env.HOME, '.github_token'), 'utf8').trim();
const content = Buffer.from(JSON.stringify(portfolio, null, 2)).toString('base64');
// Get current SHA
const shaRes = await apiFetch2(`https://api.github.com/repos/Zero2Ai-hub/Jarvis-Ops/contents/trading/portfolio-v5.json`, ghToken);
const sha = shaRes.sha || '';
const body = JSON.stringify({ message: 'trading: v5 portfolio update', content, sha: sha || undefined });
await apiFetch2Put(`https://api.github.com/repos/Zero2Ai-hub/Jarvis-Ops/contents/trading/portfolio-v5.json`, ghToken, body);
console.log('GitHub portfolio.json updated ✅');
} catch(e) {
console.log('GitHub push failed (non-critical):', e.message);
}
const totalPnl = p.current_capital_usdc - p.starting_capital_usdc;
const wr = p.stats.total_trades > 0 ? (p.stats.wins / p.stats.total_trades * 100).toFixed(1) : '—';
console.log(`Portfolio: $p.current_capital_usdc.toFixed(2) | PnL: ''$totalPnl.toFixed(2) | WR: wr% | Open: p.open_positions.length/10`);
// 7. Log observations (append to trading/observations.md for any notable events)
try {
const OBS_FILE = path.join(process.env.HOME, '.openclaw/workspace/trading/observations.md');
const closedThisRun = p.closed_trades.filter(t => {
const exitAge = (now - new Date(t.exit_time).getTime()) / 60000;
return exitAge < 35; // closed in last 35 min (within this cycle)
});
if (closedThisRun.length > 0 || !fastAllowEntry) {
const ts = new Date().toISOString().replace('T',' ').slice(0,19) + ' UTC';
let note = `\n## [ts] Auto-Observation\n`;
note += `**Macro regime:** btcRegime | **Fast regime (15m):** btcFast.trend (btcFast.spread.toFixed(3)%)\n`;
if (!fastAllowEntry) note += `**⛔ Fast filter blocked new entries** — BTC 15m trend bearish\n`;
if (closedThisRun.length > 0) {
note += `**Closed this cycle:**\n`;
for (const t of closedThisRun) {
note += `- t.symbol: t.exit_type | ''t.pnl_pct% ($''t.pnl_usd) | held t.held_hoursh | BTC bull at entry: t.btc_bull\n`;
}
}
note += `**Portfolio:** $p.current_capital_usdc.toFixed(2) | WR: wr% | Open: p.open_positions.length/10\n---\n`;
const existing = fs.existsSync(OBS_FILE) ? fs.readFileSync(OBS_FILE, 'utf8') : '# Trading Observations Log\n---\n';
// Insert after the header (after first ---)
const headerEnd = existing.indexOf('---\n');
if (headerEnd > -1) {
const before = existing.slice(0, headerEnd + 4);
const after = existing.slice(headerEnd + 4);
fs.writeFileSync(OBS_FILE, before + note + after);
} else {
fs.writeFileSync(OBS_FILE, existing + note);
}
}
} catch(e) {
console.log('Observation log failed (non-critical):', e.message);
}
console.log('Done ✅');
}
run().catch(e => { console.error('Monitor error:', e.message); process.exit(1); });
FILE:scripts/paper-monitor-v6.js
#!/usr/bin/env node
/**
* paper-monitor-v6.js — Day Trading with STPI Gate
* SMC v6.0 — Same SMC+TA≥5 entry logic, but gated by STPI probability
* instead of simple EMA crosses. Reads STPI from regime.json.
* STPI ≥ 0.5 → entries allowed. STPI < 0.5 → no new positions.
* Runs every 1h (XX:03 UTC). Separate portfolio for A/B comparison.
*/
'use strict';
const https = require('https');
const fs = require('fs');
const path = require('path');
const PORTFOLIO_FILE = path.join(process.env.HOME, '.openclaw/workspace/trading/paper-dashboard/portfolio-v6.json');
const REGIME_FILE = path.join(process.env.HOME, '.openclaw/workspace/trading/paper-dashboard/regime.json');
const TOKENS = [
'BTCUSDT','ETHUSDT','SOLUSDT','BNBUSDT','XRPUSDT','DOGEUSDT','AVAXUSDT','LINKUSDT',
'ARBUSDT','OPUSDT','SUIUSDT','APTUSDT','INJUSDT','SEIUSDT','FETUSDT','RENDERUSDT',
'TAOUSDT','NEARUSDT','WIFUSDT','JUPUSDT','TIAUSDT','DYMUSDT','STRKUSDT','TONUSDT',
'NOTUSDT','EIGENUSDT','GRASSUSDT','VIRTUALUSDT','AKTUSDT',
];
const P = { swN:3, fvgAge:8, tr:0.03, maxH:12, maxConcurrent:10, posPct:0.10,
// v4.0 z-score params (Config C: SMC+TA≥5)
volZThreshold: 1.0, volLookback: 30,
fvgZThreshold: 0.5, fvgLookback: 50,
slAtrMult: 1.5, tpAtrMult: 3.0, atrPeriod: 14,
taMinScore: 5, taZLookback: 30,
};
function apiFetch2(url, token) {
return new Promise((res, rej) => {
const r = require('https').get(url, { headers: { Authorization: `token token`, Accept: 'application/vnd.github.v3+json', 'User-Agent': 'Jarvis' } }, resp => {
let d = ''; resp.on('data', c => d += c);
resp.on('end', () => { try { res(JSON.parse(d)); } catch(e) { rej(e); } });
}); r.on('error', rej); r.setTimeout(10000, () => { r.destroy(); rej(new Error('timeout')); });
});
}
function apiFetch2Put(url, token, body) {
return new Promise((res, rej) => {
const opts = require('url').parse(url);
opts.method = 'PUT';
opts.headers = { Authorization: `token token`, Accept: 'application/vnd.github.v3+json', 'Content-Type': 'application/json', 'User-Agent': 'Jarvis', 'Content-Length': Buffer.byteLength(body) };
const r = require('https').request(opts, resp => {
let d = ''; resp.on('data', c => d += c);
resp.on('end', () => { try { res(JSON.parse(d)); } catch(e) { rej(e); } });
}); r.on('error', rej); r.write(body); r.end();
});
}
function apiFetch(url) {
return new Promise((res, rej) => {
const r = https.get(url, resp => {
let d = '';
resp.on('data', c => d += c);
resp.on('end', () => { try { res(JSON.parse(d)); } catch(e) { rej(e); } });
});
r.on('error', rej);
r.setTimeout(10000, () => { r.destroy(); rej(new Error('timeout')); });
});
}
const sleep = ms => new Promise(r => setTimeout(r, ms));
function calcEMA(v, p) {
const k = 2/(p+1); let e = v[0];
return v.map((x,i) => { if(i>0) e=x*k+e*(1-k); return e; });
}
function zScore(value, arr) {
if (arr.length < 2) return 0;
const mean = arr.reduce((s,v)=>s+v,0)/arr.length;
const std = Math.sqrt(arr.reduce((s,v)=>s+(v-mean)**2,0)/arr.length);
if (std === 0) return 0;
return (value - mean) / std;
}
function calcATR(cs, period=14) {
const trs = cs.map((c,i) => {
if (i===0) return c.h - c.l;
const prev = cs[i-1].c;
return Math.max(c.h - c.l, Math.abs(c.h - prev), Math.abs(c.l - prev));
});
const atrs = [];
for (let i=0; i<cs.length; i++) {
if (i < period-1) { atrs.push(null); continue; }
if (i === period-1) {
atrs.push(trs.slice(0, period).reduce((s,v)=>s+v,0)/period);
} else {
atrs.push((atrs[i-1]*(period-1)+trs[i])/period);
}
}
return atrs;
}
function calcSMA(v, p) {
return v.map((_, i) => {
if (i < p-1) return null;
return v.slice(i-p+1, i+1).reduce((s,x)=>s+x,0)/p;
});
}
function calcRSI(closes, period=14) {
const rsi = new Array(closes.length).fill(null);
let avgGain = 0, avgLoss = 0;
for (let i = 1; i <= period; i++) {
const d = closes[i] - closes[i-1];
if (d > 0) avgGain += d; else avgLoss += Math.abs(d);
}
avgGain /= period; avgLoss /= period;
rsi[period] = avgLoss === 0 ? 100 : 100 - 100/(1+avgGain/avgLoss);
for (let i = period+1; i < closes.length; i++) {
const d = closes[i] - closes[i-1];
avgGain = (avgGain*(period-1) + (d>0?d:0)) / period;
avgLoss = (avgLoss*(period-1) + (d<0?Math.abs(d):0)) / period;
rsi[i] = avgLoss === 0 ? 100 : 100 - 100/(1+avgGain/avgLoss);
}
return rsi;
}
function calcMACD(closes) {
const e12 = calcEMA(closes, 12), e26 = calcEMA(closes, 26);
const macdLine = closes.map((_, i) => e12[i] - e26[i]);
const signal = calcEMA(macdLine.slice(25), 9);
const histogram = [];
for (let i = 0; i < closes.length; i++) {
if (i < 33) { histogram.push(null); continue; }
histogram.push(macdLine[i] - (signal[i-25] || 0));
}
return { macdLine, histogram };
}
function calcOBV(cs) {
const obv = [0];
for (let i = 1; i < cs.length; i++) {
if (cs[i].c > cs[i-1].c) obv.push(obv[i-1] + cs[i].v);
else if (cs[i].c < cs[i-1].c) obv.push(obv[i-1] - cs[i].v);
else obv.push(obv[i-1]);
}
return obv;
}
function taConfluenceScore(cs, idx) {
const closes = cs.map(c => c.c);
const rsi = calcRSI(closes, 14);
const macd = calcMACD(closes);
const obv = calcOBV(cs);
const ema9 = calcEMA(closes, 9);
const ema21 = calcEMA(closes, 21);
const bbSma = calcSMA(closes, 20);
const i = idx;
const ZLB = P.taZLookback;
let score = 0;
// Z-scored RSI
if (rsi[i] !== null) {
const rsiWindow = rsi.slice(Math.max(0,i-ZLB), i).filter(v=>v!==null);
const rsiZ = zScore(rsi[i], rsiWindow);
if (rsiZ > 0.3 && rsi[i] < 78) score += 2;
if (rsiZ > 1.0 && rsi[i] < 75) score += 1;
}
// Z-scored MACD histogram
if (macd.histogram[i] !== null) {
const macdWindow = macd.histogram.slice(Math.max(0,i-ZLB), i).filter(v=>v!==null);
const macdZ = zScore(macd.histogram[i], macdWindow);
if (macdZ > 0.3) score += 1;
if (i > 0 && macd.histogram[i] !== null && macd.histogram[i-1] !== null && macd.histogram[i] > macd.histogram[i-1]) score += 1;
}
// EMA alignment (9 > 21)
if (ema9[i] > ema21[i]) score += 2;
// BB position (sweet spot)
if (bbSma[i] !== null && i >= 19) {
const slice = closes.slice(i-19, i+1);
const std = Math.sqrt(slice.reduce((s,v)=>s+(v-bbSma[i])**2,0)/20);
const upper = bbSma[i] + 2*std;
const lower = bbSma[i] - 2*std;
const range = upper - lower;
if (range > 0) {
const bbPct = (closes[i] - lower) / range;
if (bbPct > 0.5 && bbPct < 0.85) score += 2;
}
}
// Z-scored OBV
if (i > 10) {
const obvDelta = obv[i] - obv[i-5];
const obvDeltas = [];
for (let j = Math.max(6, i-ZLB); j < i; j++) {
obvDeltas.push(obv[j] - obv[j-5]);
}
const obvZ = zScore(obvDelta, obvDeltas);
if (obvZ > 0.5) score += 1;
if (obvZ > 1.0) score += 1;
}
return score; // max ~11
}
async function getBTCRegime() {
try {
const r = await apiFetch('https://fapi.binance.com/fapi/v1/klines?symbol=BTCUSDT&interval=4h&limit=60');
const closes = r.map(k => parseFloat(k[4]));
const e20 = calcEMA(closes, 20), e50 = calcEMA(closes, 50);
return e20[e20.length-1] > e50[e50.length-1] ? 'bullish' : 'bearish';
} catch { return 'unknown'; }
}
async function getBTCFastRegime() {
try {
const r = await apiFetch('https://fapi.binance.com/fapi/v1/klines?symbol=BTCUSDT&interval=15m&limit=30');
const closes = r.map(k => parseFloat(k[4]));
const e8 = calcEMA(closes, 8), e21 = calcEMA(closes, 21);
const fast = e8[e8.length-1], slow = e21[e21.length-1];
// Also check momentum: is price accelerating down?
const pctFromSlow = (fast - slow) / slow * 100;
// bullish: fast > slow, neutral: within 0.15%, bearish: fast < slow
if (pctFromSlow > 0.15) return { trend: 'bullish', spread: pctFromSlow };
if (pctFromSlow < -0.15) return { trend: 'bearish', spread: pctFromSlow };
return { trend: 'neutral', spread: pctFromSlow };
} catch { return { trend: 'unknown', spread: 0 }; }
}
async function getCandles(sym, limit=100) {
try {
await sleep(80);
const r = await apiFetch(`https://fapi.binance.com/fapi/v1/klines?symbol=sym&interval=1h&limit=limit`);
return r.map(k => ({ ts:k[0], o:parseFloat(k[1]), h:parseFloat(k[2]), l:parseFloat(k[3]), c:parseFloat(k[4]), v:parseFloat(k[5])*parseFloat(k[4]) }));
} catch { return []; }
}
function detectSignal(cs) {
// Find swing highs
const hi = new Array(cs.length).fill(null);
const n = P.swN;
for (let i=n; i<cs.length-n; i++) {
let isH=true;
for (let j=1;j<=n;j++) if(cs[i-j].h>=cs[i].h||cs[i+j].h>=cs[i].h) isH=false;
if(isH) hi[i]=cs[i].h;
}
// Precompute ATR for SL/TP
const atrs = calcATR(cs, P.atrPeriod);
// Check last 5 candles for BoS + FVG
for (let i=cs.length-5; i<cs.length-1; i++) {
if (i < n+2) continue;
// BoS: close breaks prior swing high
let bosIdx = -1;
for (let j=i-1; j>=Math.max(0,i-60); j--) {
if (hi[j]!==null && cs[i].c>hi[j]) { bosIdx=j; break; }
}
if (bosIdx < 0) continue;
// FVG within 3 candles after BoS
for (let j=Math.max(2,i-3); j<=i; j++) {
const gap = cs[j].l - cs[j-2].h;
if (gap <= 0) continue;
const gapPct = gap / cs[j].c;
// v3.2: Z-scored FVG size (must be 0.5σ above recent average)
const recentGaps = [];
for (let g=Math.max(2, j-P.fvgLookback); g<j; g++) {
const pg = cs[g].l - cs[g-2].h;
if (pg > 0) recentGaps.push(pg / cs[g].c);
}
const fvgZ = zScore(gapPct, recentGaps);
if (fvgZ < P.fvgZThreshold) continue;
// Check if price is retesting the FVG right now (latest candle)
const fvgLo=cs[j-2].h, fvgHi=cs[j].l, fvgMid=(fvgLo+fvgHi)/2;
const latest=cs[cs.length-1];
const lastIdx = cs.length - 1;
if (!(latest.l<=fvgHi && latest.h>=fvgLo)) continue;
// v3.2: Z-scored volume (must be 1.0σ above recent average)
const volWindow = cs.slice(Math.max(0, lastIdx-P.volLookback), lastIdx).map(c=>c.v);
const volZ = zScore(latest.v, volWindow);
if (volZ < P.volZThreshold) continue;
// v3.2: ATR-based SL/TP
const atr = atrs[lastIdx];
if (!atr) continue;
const ep = Math.min(latest.c, fvgMid);
// SL: max of (OB low, entry - 1.5×ATR)
let obLo = ep - atr * P.slAtrMult;
for (let k=bosIdx-1; k>=Math.max(0,bosIdx-15); k--) {
if (cs[k].c<cs[k].o) { obLo=cs[k].l; break; }
}
const sl = Math.max(obLo*0.999, ep - atr * P.slAtrMult);
const tp = ep + atr * P.tpAtrMult;
if (sl >= ep) continue;
return { ep, sl, tp, fvgMid, gap: gapPct, volZ: volZ.toFixed(1), fvgZ: fvgZ.toFixed(1), atr };
}
}
return null;
}
function loadPortfolio() {
try {
return JSON.parse(fs.readFileSync(PORTFOLIO_FILE, 'utf8'));
} catch {
return {
version: '1.0',
last_updated: new Date().toISOString(),
strategy: 'SMC v6.0 — BoS+FVG+TA≥5 + STPI Gate | Paper Trading',
btc_regime: 'unknown',
paper_portfolio: {
starting_capital_usdc: 1000,
current_capital_usdc: 1000,
deployed_usdc: 0,
open_positions: [],
closed_trades: [],
stats: { total_trades:0, wins:0, losses:0, total_pnl_usdc:0 }
}
};
}
}
function savePortfolio(data) {
data.last_updated = new Date().toISOString();
fs.mkdirSync(path.dirname(PORTFOLIO_FILE), { recursive: true });
fs.writeFileSync(PORTFOLIO_FILE, JSON.stringify(data, null, 2));
}
async function run() {
const uaeTime = new Date().toLocaleString('en-AE', { timeZone: 'Asia/Dubai', hour12: false });
console.log(`\n[uaeTime UAE] Paper monitor running...`);
const portfolio = loadPortfolio();
const p = portfolio.paper_portfolio;
// 1. Read STPI from regime.json (computed by regime-scorer.js at XX:50)
let stpi = 0.5, stpiSignal = 'NO_DATA';
try {
const regime = JSON.parse(fs.readFileSync(REGIME_FILE, 'utf8'));
if (regime.stpi) {
stpi = regime.stpi.stpi;
stpiSignal = regime.stpi.signal;
}
portfolio.btc_regime = regime.classification?.composite_signal || 'unknown';
portfolio.stpi = stpi;
portfolio.stpi_signal = stpiSignal;
} catch { console.log(' ⚠️ regime.json not found — using STPI=0.5'); }
const stpiAllowEntry = stpi >= 0.5;
const sizeMultiplier = stpi >= 0.6 ? 1.0 : 0.7; // confident STPI → full size, marginal → 70%
console.log(`STPI: (stpi*100).toFixed(1)% stpiSignal → '⛔ ENTRIES BLOCKED' | Size: (sizeMultiplier*100).toFixed(0)%`);
// 2. Close expired positions
const now = Date.now();
const stillOpen = [];
for (const pos of p.open_positions) {
const ageH = (now - new Date(pos.entry_time).getTime()) / 3600000;
// Get current price
try {
await sleep(80);
const ticker = await apiFetch(`https://fapi.binance.com/fapi/v1/ticker/price?symbol=pos.symbol`);
const price = parseFloat(ticker.price);
pos.current_price = price;
let closed = false;
let exitType = '', exitPrice = price, pnlPct = 0;
// Check SL
if (price <= pos.sl_price) {
exitType = 'SL'; exitPrice = pos.sl_price; closed = true;
}
// Check TP1 (if not yet hit)
else if (!pos.tp1_hit && price >= pos.tp_price) {
pos.tp1_hit = true;
pos.half_price = pos.tp_price;
pos.trail_peak = price;
stillOpen.push(pos);
console.log(` 📈 pos.symbol TP1 hit at $price.toFixed(4) — trailing stop active`);
continue;
}
// Check trailing stop after TP1
else if (pos.tp1_hit) {
if (price > (pos.trail_peak || pos.tp_price)) pos.trail_peak = price;
const trailSL = (pos.trail_peak || price) * (1 - P.tr);
if (price <= trailSL) {
exitType = 'TRAIL'; exitPrice = trailSL; closed = true;
pnlPct = ((pos.half_price - pos.entry_price)*0.5 + (exitPrice - pos.entry_price)*0.5) / pos.entry_price * 100;
}
}
// Time exit
else if (ageH >= P.maxH) {
exitType = 'TIME'; exitPrice = price; closed = true;
}
if (closed) {
if (pnlPct === 0) pnlPct = (exitPrice - pos.entry_price) / pos.entry_price * 100;
const pnlUSD = pos.size_usdc * pnlPct / 100;
p.current_capital_usdc += pnlUSD;
p.stats.total_trades++;
if (pnlUSD > 0) p.stats.wins++; else p.stats.losses++;
p.stats.total_pnl_usdc += pnlUSD;
const trade = {
symbol: pos.symbol,
entry_price: pos.entry_price,
exit_price: exitPrice,
entry_time: pos.entry_time,
exit_time: new Date().toISOString(),
held_hours: parseFloat(ageH.toFixed(1)),
pnl_pct: parseFloat(pnlPct.toFixed(3)),
pnl_usd: parseFloat(pnlUSD.toFixed(2)),
exit_type: exitType,
size_usdc: pos.size_usdc,
btc_bull: pos.btc_bull_at_entry,
};
p.closed_trades.push(trade);
console.log(` '❌' pos.symbol CLOSED | exitType | ''pnlPct.toFixed(2)% | $''pnlUSD.toFixed(2)`);
} else {
stillOpen.push(pos);
}
} catch(e) {
stillOpen.push(pos); // keep open on error
}
}
p.open_positions = stillOpen;
// 3. Scan for new signals (gated by STPI)
if (p.open_positions.length < P.maxConcurrent && stpiAllowEntry) {
const openSymbols = new Set(p.open_positions.map(pos => pos.symbol));
for (const sym of TOKENS) {
if (p.open_positions.length >= P.maxConcurrent) break;
if (openSymbols.has(sym)) continue; // already in position
const cs = await getCandles(sym, 80);
if (cs.length < 60) continue;
const signal = detectSignal(cs);
if (!signal) continue;
// v4.0: TA confluence confirmation
const ta = taConfluenceScore(cs, cs.length - 1);
if (ta < P.taMinScore) continue;
const posSize = p.current_capital_usdc * P.posPct * sizeMultiplier;
const pos = {
symbol: sym,
entry_price: signal.ep,
sl_price: signal.sl,
tp_price: signal.tp,
size_usdc: posSize,
entry_time: new Date().toISOString(),
btc_bull_at_entry: stpi >= 0.6,
tp1_hit: false,
trail_peak: null,
half_price: null,
current_price: signal.ep,
};
p.open_positions.push(pos);
console.log(` 🟢 NEW SIGNAL: sym | Entry: $signal.ep.toFixed(4) | SL: $signal.sl.toFixed(4) | TP: $signal.tp.toFixed(4) | Size: $posSize.toFixed(2) | volZ:signal.volZ fvgZ:signal.fvgZ TA:ta/11`);
}
}
// 4. Update deployed
p.deployed_usdc = p.open_positions.reduce((s, pos) => s + pos.size_usdc, 0);
// 5. Save locally
savePortfolio(portfolio);
// 6. Push to GitHub (feeds Vercel dashboard)
try {
const fs2 = require('fs'), path2 = require('path');
const ghToken = fs2.readFileSync(path2.join(process.env.HOME, '.github_token'), 'utf8').trim();
const content = Buffer.from(JSON.stringify(portfolio, null, 2)).toString('base64');
// Get current SHA
const shaRes = await apiFetch2(`https://api.github.com/repos/Zero2Ai-hub/Jarvis-Ops/contents/trading/portfolio-v6.json`, ghToken);
const sha = shaRes.sha || '';
const body = JSON.stringify({ message: 'trading: v6 portfolio update', content, sha: sha || undefined });
await apiFetch2Put(`https://api.github.com/repos/Zero2Ai-hub/Jarvis-Ops/contents/trading/portfolio-v6.json`, ghToken, body);
console.log('GitHub portfolio.json updated ✅');
} catch(e) {
console.log('GitHub push failed (non-critical):', e.message);
}
const totalPnl = p.current_capital_usdc - p.starting_capital_usdc;
const wr = p.stats.total_trades > 0 ? (p.stats.wins / p.stats.total_trades * 100).toFixed(1) : '—';
console.log(`Portfolio: $p.current_capital_usdc.toFixed(2) | PnL: ''$totalPnl.toFixed(2) | WR: wr% | Open: p.open_positions.length/10`);
// 7. Log observations (append to trading/observations.md for any notable events)
try {
const OBS_FILE = path.join(process.env.HOME, '.openclaw/workspace/trading/observations.md');
const closedThisRun = p.closed_trades.filter(t => {
const exitAge = (now - new Date(t.exit_time).getTime()) / 60000;
return exitAge < 35; // closed in last 35 min (within this cycle)
});
if (closedThisRun.length > 0 || !stpiAllowEntry) {
const ts = new Date().toISOString().replace('T',' ').slice(0,19) + ' UTC';
let note = `\n## [ts] Auto-Observation\n`;
note += `**Macro regime:** btcRegime | **Fast regime (15m):** btcFast.trend (btcFast.spread.toFixed(3)%)\n`;
if (!stpiAllowEntry) note += `**⛔ STPI blocked new entries** — STPI: (stpi*100).toFixed(1)%\n`;
if (closedThisRun.length > 0) {
note += `**Closed this cycle:**\n`;
for (const t of closedThisRun) {
note += `- t.symbol: t.exit_type | ''t.pnl_pct% ($''t.pnl_usd) | held t.held_hoursh | BTC bull at entry: t.btc_bull\n`;
}
}
note += `**Portfolio:** $p.current_capital_usdc.toFixed(2) | WR: wr% | Open: p.open_positions.length/10\n---\n`;
const existing = fs.existsSync(OBS_FILE) ? fs.readFileSync(OBS_FILE, 'utf8') : '# Trading Observations Log\n---\n';
// Insert after the header (after first ---)
const headerEnd = existing.indexOf('---\n');
if (headerEnd > -1) {
const before = existing.slice(0, headerEnd + 4);
const after = existing.slice(headerEnd + 4);
fs.writeFileSync(OBS_FILE, before + note + after);
} else {
fs.writeFileSync(OBS_FILE, existing + note);
}
}
} catch(e) {
console.log('Observation log failed (non-critical):', e.message);
}
console.log('Done ✅');
}
run().catch(e => { console.error('Monitor error:', e.message); process.exit(1); });
FILE:scripts/regime-scorer.js
#!/usr/bin/env node
/**
* regime-scorer.js — Multi-Factor BTC Regime Scoring System
*
* Three timeframes: Fast (15m), Medium (4h), Macro (1d)
* Each timeframe aggregates multiple z-scored indicators into a composite score (-10 to +10)
*
* INDICATORS PER TIMEFRAME:
* Trend: EMA cross, ADX, price vs MA200, higher highs/lows
* Momentum: RSI, MACD histogram, rate of change
* Volatility: BB width, ATR percentile, Keltner squeeze
* Volume: OBV trend, volume z-score, CVD
* Derivatives: Funding rate, long/short ratio, open interest delta
* Sentiment: Fear & Greed Index (macro only)
*
* OUTPUT: regime.json consumed by all monitors + dashboard
*/
'use strict';
const https = require('https');
const fs = require('fs');
const path = require('path');
const REGIME_FILE = path.join(process.env.HOME, '.openclaw/workspace/trading/paper-dashboard/regime.json');
function httpGet(url, timeout = 10000) {
return new Promise((res, rej) => {
const r = https.get(url, resp => {
let d = ''; resp.on('data', c => d += c);
resp.on('end', () => { try { res(JSON.parse(d)); } catch(e) { rej(e); } });
});
r.on('error', rej);
r.setTimeout(timeout, () => { r.destroy(); rej(new Error('timeout')); });
});
}
const sleep = ms => new Promise(r => setTimeout(r, ms));
// ═══════════════════════════════════════════════════════════════════════════════
// MATH HELPERS
// ═══════════════════════════════════════════════════════════════════════════════
function calcEMA(v, p) { const k = 2/(p+1); let e = v[0]; return v.map((x,i) => { if(i>0) e=x*k+e*(1-k); return e; }); }
function calcSMA(v, p) { return v.map((_,i) => i < p-1 ? null : v.slice(i-p+1,i+1).reduce((s,x)=>s+x,0)/p); }
function calcATR(cs, p=14) {
const tr = cs.map((c,i) => i===0 ? c.h-c.l : Math.max(c.h-c.l, Math.abs(c.h-cs[i-1].c), Math.abs(c.l-cs[i-1].c)));
const a = []; for (let i=0;i<cs.length;i++) { if(i<p-1){a.push(null);continue;} if(i===p-1){a.push(tr.slice(0,p).reduce((s,v)=>s+v,0)/p);} else{a.push((a[i-1]*(p-1)+tr[i])/p);} } return a;
}
function calcRSI(c, p=14) {
const r = Array(c.length).fill(null); let ag=0,al=0;
for(let i=1;i<=p;i++){const d=c[i]-c[i-1];if(d>0)ag+=d;else al-=d;} ag/=p;al/=p;
r[p]=al===0?100:100-100/(1+ag/al);
for(let i=p+1;i<c.length;i++){const d=c[i]-c[i-1];ag=(ag*(p-1)+(d>0?d:0))/p;al=(al*(p-1)+(d<0?-d:0))/p;r[i]=al===0?100:100-100/(1+ag/al);}
return r;
}
function calcMACD(c) {
const e12=calcEMA(c,12),e26=calcEMA(c,26);
const ml=c.map((_,i)=>e12[i]-e26[i]);
const sig=calcEMA(ml.slice(25),9);
const hist=[];for(let i=0;i<c.length;i++){if(i<33){hist.push(null);continue;}hist.push(ml[i]-(sig[i-25]||0));}
return { line: ml, signal: sig, histogram: hist };
}
function calcOBV(cs) {
const o=[0];for(let i=1;i<cs.length;i++){if(cs[i].c>cs[i-1].c)o.push(o[i-1]+cs[i].v);else if(cs[i].c<cs[i-1].c)o.push(o[i-1]-cs[i].v);else o.push(o[i-1]);}return o;
}
function calcADX(cs, p=14) {
if (cs.length < p * 3) return Array(cs.length).fill(null);
const pdm=[0],ndm=[0],tr=[cs[0].h-cs[0].l];
for(let i=1;i<cs.length;i++){
const upMove=cs[i].h-cs[i-1].h, downMove=cs[i-1].l-cs[i].l;
pdm.push(upMove>downMove&&upMove>0?upMove:0);
ndm.push(downMove>upMove&&downMove>0?downMove:0);
tr.push(Math.max(cs[i].h-cs[i].l,Math.abs(cs[i].h-cs[i-1].c),Math.abs(cs[i].l-cs[i-1].c)));
}
const atr=calcEMA(tr,p),spdm=calcEMA(pdm,p),sndm=calcEMA(ndm,p);
const pdi=[],ndi=[],dx=[];
for(let i=0;i<cs.length;i++){
if(!atr[i]||atr[i]===0){pdi.push(null);ndi.push(null);dx.push(null);continue;}
const pd=spdm[i]/atr[i]*100,nd=sndm[i]/atr[i]*100;
pdi.push(pd);ndi.push(nd);
const sum=pd+nd;dx.push(sum===0?0:Math.abs(pd-nd)/sum*100);
}
const adx=calcEMA(dx.map(v=>v||0),p);
return adx.map((v,i)=>dx[i]!==null?v:null);
}
function calcBBWidth(closes, p=20) {
return closes.map((_,i) => {
if (i < p-1) return null;
const sl = closes.slice(i-p+1,i+1);
const m = sl.reduce((a,b)=>a+b,0)/p;
const std = Math.sqrt(sl.reduce((a,b)=>a+(b-m)**2,0)/p);
return m === 0 ? 0 : (4 * std) / m * 100; // BB width as % of price
});
}
function calcROC(closes, p=10) {
return closes.map((v,i) => i < p ? null : (v - closes[i-p]) / closes[i-p] * 100);
}
function zScore(value, arr) {
const valid = arr.filter(v => v !== null && v !== undefined && !isNaN(v));
if (valid.length < 5) return 0;
const mean = valid.reduce((s,v)=>s+v,0)/valid.length;
const std = Math.sqrt(valid.reduce((s,v)=>s+(v-mean)**2,0)/valid.length);
return std === 0 ? 0 : (value - mean) / std;
}
function percentile(value, arr) {
const valid = arr.filter(v => v !== null && !isNaN(v)).sort((a,b) => a - b);
if (!valid.length) return 50;
let below = 0; for (const v of valid) if (v < value) below++;
return below / valid.length * 100;
}
function clamp(v, min, max) { return Math.max(min, Math.min(max, v)); }
function sigmoid(x) { return 1 / (1 + Math.exp(-x)); }
function higherHighsLows(cs, lookback = 20) {
if (cs.length < lookback * 2) return 0;
const recent = cs.slice(-lookback), prior = cs.slice(-lookback*2, -lookback);
const rHighs = recent.map(c=>c.h), rLows = recent.map(c=>c.l);
const pHighs = prior.map(c=>c.h), pLows = prior.map(c=>c.l);
const hh = Math.max(...rHighs) > Math.max(...pHighs) ? 1 : -1;
const hl = Math.min(...rLows) > Math.min(...pLows) ? 1 : -1;
return (hh + hl) / 2; // -1 to +1
}
// ═══════════════════════════════════════════════════════════════════════════════
// DATA FETCHERS
// ═══════════════════════════════════════════════════════════════════════════════
async function fetchCandles(sym, interval, limit) {
await sleep(50);
try {
const r = await httpGet(`https://fapi.binance.com/fapi/v1/klines?symbol=sym&interval=interval&limit=limit`);
return r.map(k => ({ ts:k[0], o:+k[1], h:+k[2], l:+k[3], c:+k[4], v:+k[5]*+k[4], tbv:+k[9]*+k[4] }));
} catch { return []; }
}
async function fetchFundingRate(sym = 'BTCUSDT') {
try {
const r = await httpGet(`https://fapi.binance.com/fapi/v1/fundingRate?symbol=sym&limit=30`);
return r.map(f => ({ ts: f.fundingTime, rate: parseFloat(f.fundingRate) }));
} catch { return []; }
}
async function fetchLongShortRatio(sym = 'BTCUSDT', period = '4h') {
try {
const r = await httpGet(`https://fapi.binance.com/futures/data/globalLongShortAccountRatio?symbol=sym&period=period&limit=30`);
return r.map(d => ({ ts: d.timestamp, ratio: parseFloat(d.longShortRatio) }));
} catch { return []; }
}
async function fetchOpenInterest(sym = 'BTCUSDT', period = '4h') {
try {
const r = await httpGet(`https://fapi.binance.com/futures/data/openInterestHist?symbol=sym&period=period&limit=30`);
return r.map(d => ({ ts: d.timestamp, oi: parseFloat(d.sumOpenInterest), oiVal: parseFloat(d.sumOpenInterestValue) }));
} catch { return []; }
}
// ─── FRED Macro Data (no API key — CSV via curl for reliability) ───────────────
function fetchFRED(seriesId, startDate = '2023-01-01') {
return new Promise((res) => {
const url = `https://fred.stlouisfed.org/graph/fredgraph.csv?id=seriesId&cosd=startDate`;
const { execSync } = require('child_process');
try {
const csv = execSync(`curl -sL --max-time 12 "url"`, { encoding: 'utf8', timeout: 15000 });
const lines = csv.trim().split('\n').slice(1);
const data = lines.map(l => {
const [date, val] = l.split(',');
const v = parseFloat(val);
return { date, ts: new Date(date).getTime(), value: isNaN(v) ? null : v };
}).filter(d => d.value !== null);
res(data);
} catch(e) { res([]); }
});
}
async function fetchMacroLiquidity() {
const [walcl, tga, rrp, m2, dxy, dgs10, t10yie, dff] = await Promise.all([
fetchFRED('WALCL', '2024-01-01'), // Fed Balance Sheet (weekly, millions)
fetchFRED('WTREGEN', '2024-01-01'), // Treasury General Account (weekly, millions)
fetchFRED('RRPONTSYD', '2024-01-01'), // Reverse Repo (daily, billions)
fetchFRED('M2SL', '2023-01-01'), // M2 Money Supply (monthly, billions)
fetchFRED('DTWEXBGS', '2024-01-01'), // Trade-Weighted Dollar Index
fetchFRED('DGS10', '2024-01-01'), // 10Y Treasury Yield
fetchFRED('T10YIE', '2024-01-01'), // 10Y Breakeven Inflation
fetchFRED('DFF', '2024-01-01'), // Fed Funds Rate
]);
return { walcl, tga, rrp, m2, dxy, dgs10, t10yie, dff };
}
function scoreMacroLiquidity(macro) {
const scores = {};
// 1. Net Liquidity = Fed Balance Sheet - TGA - RRP
// WALCL in millions, TGA in millions, RRP in billions (convert to millions)
if (macro.walcl.length >= 4 && macro.tga.length >= 4 && macro.rrp.length >= 4) {
const latestBS = macro.walcl[macro.walcl.length - 1].value; // millions
const latestTGA = macro.tga[macro.tga.length - 1].value; // millions
const latestRRP = macro.rrp[macro.rrp.length - 1].value * 1000; // billions→millions
const netLiq = latestBS - latestTGA - latestRRP;
// Compare to 4 weeks ago
const bs4w = macro.walcl.length >= 5 ? macro.walcl[macro.walcl.length - 5].value : latestBS;
const tga4w = macro.tga.length >= 5 ? macro.tga[macro.tga.length - 5].value : latestTGA;
const rrp4w = macro.rrp.length >= 5 ? macro.rrp[macro.rrp.length - 5].value * 1000 : latestRRP;
const netLiq4w = bs4w - tga4w - rrp4w;
const liqChange = (netLiq - netLiq4w) / netLiq4w * 100;
// Expanding liquidity = bullish, contracting = bearish
scores.net_liquidity = clamp(liqChange * 3, -2, 2); // 0.67% change = +2
scores.net_liquidity_raw = parseFloat((netLiq / 1e6).toFixed(3)); // in trillions
scores.net_liquidity_delta = parseFloat(liqChange.toFixed(3));
} else { scores.net_liquidity = 0; }
// 2. M2 Money Supply YoY growth rate
if (macro.m2.length >= 13) {
const latest = macro.m2[macro.m2.length - 1].value;
const yoyAgo = macro.m2[macro.m2.length - 13].value; // ~12 months prior
const yoyGrowth = (latest - yoyAgo) / yoyAgo * 100;
// Positive M2 growth = bullish, negative = bearish
// Historical: M2 growth 5-10% = normal, >10% = very bullish, <0% = bearish
scores.m2_yoy = clamp((yoyGrowth - 3) / 4, -2, 2); // 3% = neutral, 7%+ = +1, 11%+ = +2
scores.m2_yoy_raw = parseFloat(yoyGrowth.toFixed(2));
// M2 momentum: 3-month change direction
if (macro.m2.length >= 4) {
const m3ago = macro.m2[macro.m2.length - 4].value;
const m3delta = (latest - m3ago) / m3ago * 100;
scores.m2_momentum = clamp(m3delta * 2, -1, 1); // 0.5% in 3mo = +1
} else { scores.m2_momentum = 0; }
} else { scores.m2_yoy = 0; scores.m2_momentum = 0; }
// 3. DXY (Dollar Strength) — INVERSE correlation with BTC
if (macro.dxy.length >= 20) {
const latestDXY = macro.dxy[macro.dxy.length - 1].value;
const dxy20ago = macro.dxy[Math.max(0, macro.dxy.length - 21)].value;
const dxyChange = (latestDXY - dxy20ago) / dxy20ago * 100;
// Rising dollar = bearish for BTC, falling = bullish
scores.dxy = clamp(-dxyChange * 2, -2, 2); // 1% dollar rise = -2
scores.dxy_raw = latestDXY;
scores.dxy_delta = parseFloat(dxyChange.toFixed(3));
} else { scores.dxy = 0; }
// 4. Real Yields (10Y - breakeven inflation)
// High real yields = bad for BTC (money flows to bonds), low/negative = good
if (macro.dgs10.length >= 5 && macro.t10yie.length >= 5) {
const nominal = macro.dgs10[macro.dgs10.length - 1].value;
const breakeven = macro.t10yie[macro.t10yie.length - 1].value;
const realYield = nominal - breakeven;
// Real yield > 2% = very bearish, < 0 = very bullish, 1% = neutral
scores.real_yields = clamp(-(realYield - 1) * 1.5, -2, 2);
scores.real_yields_raw = parseFloat(realYield.toFixed(3));
// Direction matters too
const nom5 = macro.dgs10[macro.dgs10.length - 6].value;
const be5 = macro.t10yie[macro.t10yie.length - 6].value;
const realYield5 = nom5 - be5;
const ryDelta = realYield - realYield5;
scores.real_yields_momentum = clamp(-ryDelta * 3, -1, 1); // rising real yields = bearish
} else { scores.real_yields = 0; scores.real_yields_momentum = 0; }
// 5. Fed Funds Rate trajectory
if (macro.dff.length >= 30) {
const latest = macro.dff[macro.dff.length - 1].value;
const dff30 = macro.dff[Math.max(0, macro.dff.length - 31)].value;
const rateChange = latest - dff30;
// Cutting rates = bullish, hiking = bearish
scores.fed_rate = clamp(-rateChange * 2, -2, 2); // 1% cut = +2
scores.fed_rate_raw = latest;
scores.fed_rate_delta = parseFloat(rateChange.toFixed(3));
} else { scores.fed_rate = 0; }
return scores;
}
async function fetchFearGreed() {
try {
const r = await httpGet('https://api.alternative.me/fng/?limit=30');
if (r && r.data) return r.data.map(d => ({ ts: d.timestamp * 1000, value: parseInt(d.value), label: d.value_classification }));
} catch {}
return [];
}
// ═══════════════════════════════════════════════════════════════════════════════
// INDICATOR SCORING (each returns -2 to +2, z-scored where applicable)
// ═══════════════════════════════════════════════════════════════════════════════
function scoreTrend(cs, closes) {
const scores = {};
const n = cs.length - 1;
// 1. EMA Cross (fast vs slow)
const ema20 = calcEMA(closes, 20), ema50 = calcEMA(closes, 50);
const emaDiff = (ema20[n] - ema50[n]) / closes[n] * 100;
scores.ema_cross = clamp(emaDiff * 5, -2, 2); // scale: 0.4% diff = +2
// 2. ADX (trend strength, direction-agnostic)
const adx = calcADX(cs);
const adxVal = adx[n] || 20;
scores.adx = adxVal > 25 ? clamp((adxVal - 25) / 15, 0, 2) : clamp(-(25 - adxVal) / 15, -1, 0);
// ADX only measures strength; combine with direction
if (ema20[n] < ema50[n]) scores.adx *= -1; // strong downtrend = negative
// 3. Price vs MA200
const ema200 = calcEMA(closes, Math.min(200, closes.length - 1));
const distMA200 = (closes[n] - ema200[n]) / ema200[n] * 100;
scores.price_vs_ma200 = clamp(distMA200 / 5, -2, 2);
// 4. Higher highs / higher lows
const hhhl = higherHighsLows(cs, 20);
scores.structure = hhhl * 2; // -2 to +2
return scores;
}
function scoreMomentum(closes) {
const scores = {};
const n = closes.length - 1;
// 1. RSI position + z-score
const rsi = calcRSI(closes);
if (rsi[n] !== null) {
const rsiCentered = (rsi[n] - 50) / 25; // 50=neutral, 75=+1, 25=-1
const rsiWindow = rsi.slice(Math.max(0, n-30), n).filter(v => v !== null);
const rsiZ = zScore(rsi[n], rsiWindow);
scores.rsi = clamp((rsiCentered + rsiZ * 0.3) * 1.5, -2, 2);
} else { scores.rsi = 0; }
// 2. MACD histogram z-score + direction
const macd = calcMACD(closes);
if (macd.histogram[n] !== null) {
const macdWindow = macd.histogram.slice(Math.max(0, n-30), n).filter(v => v !== null);
const macdZ = zScore(macd.histogram[n], macdWindow);
const rising = n > 0 && macd.histogram[n] > (macd.histogram[n-1] || 0) ? 0.5 : -0.5;
scores.macd = clamp((macdZ + rising) * 0.8, -2, 2);
} else { scores.macd = 0; }
// 3. Rate of Change (10-period)
const roc = calcROC(closes, 10);
if (roc[n] !== null) {
const rocWindow = roc.slice(Math.max(0, n-30), n).filter(v => v !== null);
const rocZ = zScore(roc[n], rocWindow);
scores.roc = clamp(rocZ * 0.8, -2, 2);
} else { scores.roc = 0; }
return scores;
}
function scoreVolatility(cs, closes) {
const scores = {};
const n = cs.length - 1;
// 1. BB Width percentile (high = trending, low = squeeze)
const bbw = calcBBWidth(closes, 20);
if (bbw[n] !== null) {
const bbwWindow = bbw.slice(Math.max(0, n-60), n+1).filter(v => v !== null);
const pct = percentile(bbw[n], bbwWindow);
// Low BB width (squeeze) = mean reversion likely. High = trend following
scores.bb_width = clamp((pct - 50) / 25, -2, 2); // >75th = trending, <25th = squeeze
} else { scores.bb_width = 0; }
// 2. ATR percentile
const atrs = calcATR(cs, 14);
if (atrs[n] !== null) {
const atrPct = atrs[n] / closes[n] * 100; // ATR as % of price
const atrWindow = atrs.slice(Math.max(0, n-60), n+1).filter(v => v !== null).map((v,i) => v / closes[Math.max(0, n-60)+i] * 100);
const pct = percentile(atrPct, atrWindow);
scores.atr_pctile = clamp((pct - 50) / 25, -2, 2);
} else { scores.atr_pctile = 0; }
// 3. Keltner squeeze detection (BB inside Keltner = compression)
// Simplified: compare BB width to ATR ratio
if (bbw[n] !== null && atrs[n] !== null) {
const keltnerWidth = atrs[n] * 3 / closes[n] * 100; // ~3×ATR bands
const squeeze = bbw[n] < keltnerWidth; // BB inside Keltner = squeeze
scores.squeeze = squeeze ? -1 : 1; // squeeze = expect breakout (direction unknown)
} else { scores.squeeze = 0; }
return scores;
}
function scoreVolume(cs) {
const scores = {};
const n = cs.length - 1;
// 1. OBV trend (z-scored 5-period OBV delta)
const obv = calcOBV(cs);
if (n > 10) {
const obvDelta = obv[n] - obv[n-5];
const obvDeltas = [];
for (let j = Math.max(6, n-30); j < n; j++) obvDeltas.push(obv[j] - obv[j-5]);
scores.obv = clamp(zScore(obvDelta, obvDeltas) * 0.8, -2, 2);
} else { scores.obv = 0; }
// 2. Volume z-score (current vs recent average)
const vols = cs.map(c => c.v);
const volWindow = vols.slice(Math.max(0, n-30), n);
const volZ = zScore(vols[n], volWindow);
scores.volume_z = clamp(volZ * 0.5, -2, 2); // high volume = confirms move
// 3. CVD (taker buy pressure)
const deltas = cs.map(c => { const buy = c.tbv || 0; return buy - (c.v - buy); });
const deltaWindow = deltas.slice(Math.max(0, n-30), n);
const cvdZ = zScore(deltas[n], deltaWindow);
scores.cvd = clamp(cvdZ * 0.8, -2, 2);
return scores;
}
function scoreDerivatives(funding, lsRatio, oiData) {
const scores = {};
// 1. Funding rate (positive = longs pay, crowded long = bearish contrarian)
if (funding.length >= 3) {
const recent = funding.slice(-3).map(f => f.rate);
const avgFunding = recent.reduce((s,v) => s+v, 0) / recent.length;
// Extreme positive funding = bearish (crowded long), extreme negative = bullish
scores.funding = clamp(-avgFunding * 5000, -2, 2); // 0.01% = -0.5 (slightly bearish)
} else { scores.funding = 0; }
// 2. Long/Short ratio
if (lsRatio.length >= 5) {
const recent = lsRatio.slice(-5).map(d => d.ratio);
const avgRatio = recent.reduce((s,v) => s+v, 0) / recent.length;
// >1 = more longs (contrarian bearish), <1 = more shorts (contrarian bullish)
scores.ls_ratio = clamp(-(avgRatio - 1) * 3, -2, 2);
} else { scores.ls_ratio = 0; }
// 3. Open Interest delta (rising OI + rising price = strong trend)
if (oiData.length >= 5) {
const oiVals = oiData.map(d => d.oiVal);
const oiDelta = (oiVals[oiVals.length-1] - oiVals[0]) / oiVals[0] * 100;
scores.oi_delta = clamp(oiDelta / 5, -2, 2); // 5% OI increase = +1
} else { scores.oi_delta = 0; }
return scores;
}
function scoreSentiment(fng) {
const scores = {};
if (fng.length >= 1) {
const latest = fng[0].value; // 0-100
// Contrarian: extreme fear = bullish, extreme greed = bearish
// But also direct: moderate greed = healthy trend
const centered = (latest - 50) / 50; // -1 to +1
// Below 25 = extreme fear (contrarian bullish)
// 25-45 = fear (mildly bullish)
// 45-55 = neutral
// 55-75 = greed (mildly bullish trend confirmation)
// Above 75 = extreme greed (contrarian bearish)
if (latest < 25) scores.fear_greed = 1.5; // extreme fear = buy
else if (latest < 45) scores.fear_greed = 0.5;
else if (latest <= 55) scores.fear_greed = 0;
else if (latest <= 75) scores.fear_greed = 0.5; // greed confirms trend
else scores.fear_greed = -1.5; // extreme greed = caution
scores.fear_greed_raw = latest;
scores.fear_greed_label = fng[0].label;
} else { scores.fear_greed = 0; }
return scores;
}
// ═══════════════════════════════════════════════════════════════════════════════
// COMPOSITE SCORING
// ═══════════════════════════════════════════════════════════════════════════════
function computeComposite(categories) {
// Weights per category
const weights = {
trend: 2.0,
momentum: 1.5,
volatility: 1.0,
volume: 1.5,
derivatives: 1.5,
sentiment: 1.0,
liquidity: 3.0, // Heaviest weight — "trade the tide, not the waves"
};
let totalScore = 0, totalWeight = 0;
const breakdown = {};
for (const [cat, scores] of Object.entries(categories)) {
const vals = Object.values(scores).filter(v => typeof v === 'number');
if (!vals.length) continue;
const catAvg = vals.reduce((s,v) => s+v, 0) / vals.length;
const w = weights[cat] || 1;
totalScore += catAvg * w;
totalWeight += w;
breakdown[cat] = { avg: parseFloat(catAvg.toFixed(2)), weight: w, indicators: scores };
}
const composite = totalWeight > 0 ? totalScore / totalWeight : 0;
// Scale to -10 to +10
const scaled = clamp(composite * 5, -10, 10);
return {
score: parseFloat(scaled.toFixed(1)),
signal: scaled > 3 ? 'BULLISH' : scaled > 1 ? 'LEAN_BULL' : scaled < -3 ? 'BEARISH' : scaled < -1 ? 'LEAN_BEAR' : 'NEUTRAL',
breakdown,
};
}
// ═══════════════════════════════════════════════════════════════════════════════
// TIMEFRAME SCORERS
// ═══════════════════════════════════════════════════════════════════════════════
async function scoreFast() {
const cs = await fetchCandles('BTCUSDT', '15m', 200);
if (cs.length < 100) return { score: 0, signal: 'NO_DATA', breakdown: {} };
const closes = cs.map(c => c.c);
return computeComposite({
trend: scoreTrend(cs, closes),
momentum: scoreMomentum(closes),
volatility: scoreVolatility(cs, closes),
volume: scoreVolume(cs),
});
}
// ═══════════════════════════════════════════════════════════════════════════════
// STPI — Short Term Probability Indicator (for day trading)
// 15m + 1h indicators + fast derivatives → 0-1 probability
// Updated hourly. Gates day trading entries at 0.5 threshold.
// ═══════════════════════════════════════════════════════════════════════════════
async function computeSTPI() {
const [cs15m, cs1h] = await Promise.all([
fetchCandles('BTCUSDT', '15m', 200),
fetchCandles('BTCUSDT', '1h', 100),
]);
if (cs15m.length < 100 || cs1h.length < 60) return { stpi: 0.5, rawScore: 0, breakdown: {} };
const closes15 = cs15m.map(c => c.c);
const closes1h = cs1h.map(c => c.c);
const n15 = closes15.length - 1;
const n1h = closes1h.length - 1;
const categories = { trend_15m: [], momentum_15m: [], structure_1h: [], volatility: [], flow: [] };
// ─── 15m TREND ───
const ema8 = calcEMA(closes15, 8), ema21 = calcEMA(closes15, 21), ema50_15 = calcEMA(closes15, 50);
// 1. EMA8 vs EMA21 spread
const spread821 = (ema8[n15] - ema21[n15]) / closes15[n15] * 100;
categories.trend_15m.push(clamp(spread821 * 10, -2, 2));
// 2. EMA alignment (8 > 21 > 50)
const aligned = ema8[n15] > ema21[n15] && ema21[n15] > ema50_15[n15];
const inverted = ema8[n15] < ema21[n15] && ema21[n15] < ema50_15[n15];
categories.trend_15m.push(aligned ? 2 : inverted ? -2 : 0);
// 3. Price vs 50 EMA (short term trend)
const dist50 = (closes15[n15] - ema50_15[n15]) / ema50_15[n15] * 100;
categories.trend_15m.push(clamp(dist50 * 3, -2, 2));
// ─── 15m MOMENTUM ───
const rsi15 = calcRSI(closes15, 14);
if (rsi15[n15] !== null) {
const rsiCentered = (rsi15[n15] - 50) / 25;
categories.momentum_15m.push(clamp(rsiCentered * 2, -2, 2));
}
const macd15 = calcMACD(closes15);
if (macd15.histogram[n15] !== null) {
const macdWindow = macd15.histogram.slice(Math.max(0, n15-30), n15).filter(v => v !== null);
const mz = zScore(macd15.histogram[n15], macdWindow);
categories.momentum_15m.push(clamp(mz * 0.8, -2, 2));
// MACD rising/falling
const rising = n15 > 0 && macd15.histogram[n15] > (macd15.histogram[n15-1] || 0);
categories.momentum_15m.push(rising ? 1 : -1);
}
// ROC 5-period on 15m (~1.25 hours)
if (n15 >= 5) {
const roc5 = (closes15[n15] - closes15[n15-5]) / closes15[n15-5] * 100;
categories.momentum_15m.push(clamp(roc5 * 5, -2, 2));
}
// ─── 1h STRUCTURE ───
const ema9_1h = calcEMA(closes1h, 9), ema21_1h = calcEMA(closes1h, 21), ema50_1h = calcEMA(closes1h, 50);
// 1h EMA alignment
const aligned1h = ema9_1h[n1h] > ema21_1h[n1h] && ema21_1h[n1h] > ema50_1h[n1h];
const inverted1h = ema9_1h[n1h] < ema21_1h[n1h] && ema21_1h[n1h] < ema50_1h[n1h];
categories.structure_1h.push(aligned1h ? 2 : inverted1h ? -2 : 0);
// 1h RSI
const rsi1h = calcRSI(closes1h);
if (rsi1h[n1h] !== null) {
categories.structure_1h.push(clamp((rsi1h[n1h] - 50) / 25 * 1.5, -2, 2));
}
// 1h MACD direction
const macd1h = calcMACD(closes1h);
if (macd1h.histogram[n1h] !== null) {
categories.structure_1h.push(macd1h.histogram[n1h] > 0 ? 1 : -1);
const rising1h = n1h > 0 && macd1h.histogram[n1h] > (macd1h.histogram[n1h-1] || 0);
categories.structure_1h.push(rising1h ? 0.5 : -0.5);
}
// ─── VOLATILITY (15m) ───
const bbw = calcBBWidth(closes15, 20);
if (bbw[n15] !== null) {
const bbwWindow = bbw.slice(Math.max(0, n15-60), n15+1).filter(v => v !== null);
const pct = percentile(bbw[n15], bbwWindow);
categories.volatility.push(clamp((pct - 50) / 25, -2, 2));
}
// ATR percentile
const atrs15 = calcATR(cs15m, 14);
if (atrs15[n15] !== null) {
const atrPct = atrs15[n15] / closes15[n15] * 100;
categories.volatility.push(clamp((atrPct - 0.3) * 5, -2, 2)); // 0.3% = neutral for 15m
}
// ─── FLOW (volume + CVD on 15m) ───
const vols15 = cs15m.map(c => c.v);
const volWindow = vols15.slice(Math.max(0, n15-30), n15);
const volZ = zScore(vols15[n15], volWindow);
categories.flow.push(clamp(volZ * 0.5, -2, 2));
// CVD
const deltas = cs15m.map(c => { const buy = c.tbv || 0; return buy - (c.v - buy); });
const deltaWindow = deltas.slice(Math.max(0, n15-30), n15);
const cvdZ = zScore(deltas[n15], deltaWindow);
categories.flow.push(clamp(cvdZ * 0.8, -2, 2));
// OBV trend
const obv = calcOBV(cs15m);
if (n15 > 10) {
const obvDelta = obv[n15] - obv[n15-5];
const obvDs = [];
for (let j = Math.max(6, n15-30); j < n15; j++) obvDs.push(obv[j] - obv[j-5]);
categories.flow.push(clamp(zScore(obvDelta, obvDs) * 0.6, -2, 2));
}
// ── AGGREGATE ──
const weights = { trend_15m: 2.5, momentum_15m: 2.0, structure_1h: 2.0, volatility: 1.0, flow: 1.5 };
let totalScore = 0, totalWeight = 0;
const breakdown = {};
for (const [cat, vals] of Object.entries(categories)) {
if (!vals.length) continue;
const avg = vals.reduce((s,v) => s+v, 0) / vals.length;
const w = weights[cat] || 1;
totalScore += avg * w;
totalWeight += w;
breakdown[cat] = { avg: parseFloat(avg.toFixed(3)), weight: w, count: vals.length };
}
const rawScore = totalWeight > 0 ? totalScore / totalWeight : 0;
const stpi = parseFloat(sigmoid(rawScore * 0.8).toFixed(4));
return { stpi, rawScore: parseFloat(rawScore.toFixed(3)), signal: stpi >= 0.5 ? 'BULLISH' : 'BEARISH', breakdown };
}
async function scoreMedium() {
const cs = await fetchCandles('BTCUSDT', '4h', 200);
if (cs.length < 100) return { score: 0, signal: 'NO_DATA', breakdown: {} };
const closes = cs.map(c => c.c);
const [funding, lsRatio, oiData] = await Promise.all([
fetchFundingRate(),
fetchLongShortRatio('BTCUSDT', '4h'),
fetchOpenInterest('BTCUSDT', '4h'),
]);
return computeComposite({
trend: scoreTrend(cs, closes),
momentum: scoreMomentum(closes),
volatility: scoreVolatility(cs, closes),
volume: scoreVolume(cs),
derivatives: scoreDerivatives(funding, lsRatio, oiData),
});
}
async function scoreMacro() {
const cs = await fetchCandles('BTCUSDT', '1d', 250);
if (cs.length < 60) return { score: 0, signal: 'NO_DATA', breakdown: {} };
const closes = cs.map(c => c.c);
const [funding, lsRatio, oiData, fng, macroData] = await Promise.all([
fetchFundingRate(),
fetchLongShortRatio('BTCUSDT', '1d'),
fetchOpenInterest('BTCUSDT', '1d'),
fetchFearGreed(),
fetchMacroLiquidity(),
]);
return computeComposite({
trend: scoreTrend(cs, closes),
momentum: scoreMomentum(closes),
volatility: scoreVolatility(cs, closes),
volume: scoreVolume(cs),
derivatives: scoreDerivatives(funding, lsRatio, oiData),
sentiment: scoreSentiment(fng),
liquidity: scoreMacroLiquidity(macroData),
});
}
// ═══════════════════════════════════════════════════════════════════════════════
// REGIME CLASSIFICATION
// ═══════════════════════════════════════════════════════════════════════════════
function classifyRegime(fast, medium, macro) {
// Determine overall market type
const avgScore = (fast.score * 0.2 + medium.score * 0.4 + macro.score * 0.4);
// Volatility regime from medium timeframe
const medVol = medium.breakdown.volatility;
const isSqueezing = medVol && medVol.indicators && medVol.indicators.squeeze < 0;
const isHighVol = medVol && medVol.avg > 1;
let marketType = 'MIXED';
if (isSqueezing) marketType = 'COMPRESSION'; // expect breakout
else if (isHighVol && Math.abs(avgScore) > 3) marketType = 'TRENDING';
else if (!isHighVol && Math.abs(avgScore) < 2) marketType = 'RANGE_BOUND';
else if (Math.abs(avgScore) > 5) marketType = 'STRONG_TREND';
// Strategy recommendation
let recommendation = 'NEUTRAL';
if (marketType === 'TRENDING' || marketType === 'STRONG_TREND') {
recommendation = avgScore > 0 ? 'TREND_FOLLOW_LONG' : 'TREND_FOLLOW_SHORT';
} else if (marketType === 'RANGE_BOUND') {
recommendation = 'MEAN_REVERSION';
} else if (marketType === 'COMPRESSION') {
recommendation = 'WAIT_BREAKOUT';
}
// Position sizing suggestion
const confidence = Math.min(Math.abs(avgScore) / 7, 1); // 0-1
const sizeMult = avgScore > 1 ? (0.5 + confidence * 0.5) : avgScore < -1 ? 0.5 * (1 - confidence * 0.3) : 0.5;
return {
composite_score: parseFloat(avgScore.toFixed(1)),
composite_signal: avgScore > 3 ? 'BULLISH' : avgScore > 1 ? 'LEAN_BULL' : avgScore < -3 ? 'BEARISH' : avgScore < -1 ? 'LEAN_BEAR' : 'NEUTRAL',
market_type: marketType,
recommendation,
confidence: parseFloat(confidence.toFixed(2)),
size_multiplier: parseFloat(sizeMult.toFixed(2)),
allow_new_longs: avgScore > -3, // block longs in strong downtrend
};
}
// ═══════════════════════════════════════════════════════════════════════════════
// MAIN
// ═══════════════════════════════════════════════════════════════════════════════
async function computeMTPIProbability() {
// MTPI for swing trading — mirrors macro-rotation MTPI logic
// but outputs as 0-1 probability for regime.json consumption
const cs = await fetchCandles('BTCUSDT', '4h', 200);
if (cs.length < 100) return { mtpi: 0.5, rawScore: 0, signal: 'NO_DATA', breakdown: {} };
const closes = cs.map(c => c.c);
const n = closes.length - 1;
let funding = [], lsRatio = [], oiData = [];
try { const r = await httpGet('https://fapi.binance.com/fapi/v1/fundingRate?symbol=BTCUSDT&limit=30'); funding = r.map(f => parseFloat(f.fundingRate)); } catch {}
try { const r = await httpGet('https://fapi.binance.com/futures/data/globalLongShortAccountRatio?symbol=BTCUSDT&period=4h&limit=30'); lsRatio = r.map(d => parseFloat(d.longShortRatio)); } catch {}
try { const r = await httpGet('https://fapi.binance.com/futures/data/openInterestHist?symbol=BTCUSDT&period=4h&limit=30'); oiData = r.map(d => parseFloat(d.sumOpenInterestValue)); } catch {}
const categories = { momentum: [], structure: [], derivatives: [], flow: [] };
// Momentum
const rsi = calcRSI(closes);
if (rsi[n] !== null) categories.momentum.push(clamp((rsi[n] - 50) / 20, -2, 2));
const roc7 = n >= 42 ? (closes[n] - closes[n-42]) / closes[n-42] * 100 : 0; // 42×4h = 7 days
categories.momentum.push(clamp(roc7 * 0.3, -2, 2));
const macd = calcMACD(closes);
if (macd.histogram[n] !== null) {
categories.momentum.push(macd.histogram[n] > 0 ? 1 : -1);
if (n > 0) categories.momentum.push(macd.histogram[n] > (macd.histogram[n-1]||0) ? 0.5 : -0.5);
}
// Structure
const ema20 = calcEMA(closes, 20), ema50 = calcEMA(closes, 50);
categories.structure.push(ema20[n] > ema50[n] ? 1.5 : -1.5);
categories.structure.push(clamp((closes[n] - ema50[n]) / ema50[n] * 100 * 0.5, -2, 2));
const hhhl = higherHighsLows(cs, 30);
categories.structure.push(hhhl * 2);
// Derivatives (data is already parsed to numbers)
if (funding.length >= 3) {
const valid = funding.slice(-3).filter(v => !isNaN(v));
if (valid.length) { const avgF = valid.reduce((s,v)=>s+v,0)/valid.length; categories.derivatives.push(clamp(-avgF * 3000, -2, 2)); }
}
if (lsRatio.length >= 5) {
const valid = lsRatio.slice(-5).filter(v => !isNaN(v));
if (valid.length) { const avgLS = valid.reduce((s,v)=>s+v,0)/valid.length; categories.derivatives.push(clamp(-(avgLS - 1) * 3, -2, 2)); }
}
if (oiData.length >= 5) {
const valid = oiData.filter(v => !isNaN(v));
if (valid.length >= 2) { const oiDelta = (valid[valid.length-1] - valid[0]) / valid[0] * 100; categories.derivatives.push(clamp(oiDelta / 5, -2, 2)); }
}
// Flow
const obv = calcOBV(cs);
if (n > 10) {
const obvDelta = obv[n] - obv[n-5];
const obvDs = []; for (let j = Math.max(6,n-30); j < n; j++) obvDs.push(obv[j]-obv[j-5]);
categories.flow.push(clamp(zScore(obvDelta, obvDs) * 0.8, -2, 2));
}
const deltas = cs.map(c => { const buy = c.tbv || 0; return buy - (c.v - buy); });
const dw = deltas.slice(Math.max(0,n-30),n);
categories.flow.push(clamp(zScore(deltas[n], dw) * 0.8, -2, 2));
const weights = { momentum: 2.0, structure: 2.0, derivatives: 1.5, flow: 1.5 };
let totalScore = 0, totalWeight = 0;
const breakdown = {};
for (const [cat, vals] of Object.entries(categories)) {
if (!vals.length) continue;
const avg = vals.reduce((s,v)=>s+v,0)/vals.length;
const w = weights[cat] || 1;
totalScore += avg * w;
totalWeight += w;
breakdown[cat] = { avg: parseFloat(avg.toFixed(3)), weight: w, count: vals.length };
}
const rawScore = totalWeight > 0 ? totalScore / totalWeight : 0;
const mtpi = parseFloat(sigmoid(rawScore * 0.8).toFixed(4));
return { mtpi, rawScore: parseFloat(rawScore.toFixed(3)), signal: mtpi >= 0.5 ? 'BULLISH' : 'BEARISH', breakdown };
}
async function run() {
const uaeTime = new Date().toLocaleString('en-AE', { timeZone: 'Asia/Dubai', hour12: false });
console.log(`[uaeTime UAE] Regime scorer running...`);
const [fast, medium, macro, stpiResult, mtpiResult] = await Promise.all([
scoreFast(),
scoreMedium(),
scoreMacro(),
computeSTPI(),
computeMTPIProbability(),
]);
const classification = classifyRegime(fast, medium, macro);
const regime = {
updated: new Date().toISOString(),
fast: { timeframe: '15m', ...fast },
medium: { timeframe: '4h', ...medium },
macro: { timeframe: '1d', ...macro },
stpi: stpiResult, // for day trading v6
mtpi: mtpiResult, // for swing v2
classification,
};
// Save
fs.mkdirSync(path.dirname(REGIME_FILE), { recursive: true });
fs.writeFileSync(REGIME_FILE, JSON.stringify(regime, null, 2));
// Console summary
const arrow = s => s > 3 ? '🟢↑' : s > 1 ? '🟡↗' : s < -3 ? '🔴↓' : s < -1 ? '🟠↘' : '⚪→';
console.log(`\n Fast (15m): arrow(fast.score) fast.score.toFixed(1).padStart(5) | fast.signal`);
console.log(` Medium (4h): arrow(medium.score) medium.score.toFixed(1).padStart(5) | medium.signal`);
console.log(` Macro (1d): arrow(macro.score) macro.score.toFixed(1).padStart(5) | macro.signal`);
console.log(`\n STPI (day): '🔴' (stpiResult.stpi*100).toFixed(1).padStart(5)% | stpiResult.signal (raw: stpiResult.rawScore)`);
console.log(` MTPI (swing):'🔴' (mtpiResult.mtpi*100).toFixed(1).padStart(5)% | mtpiResult.signal (raw: mtpiResult.rawScore)`);
console.log(`\n Composite: arrow(classification.composite_score) classification.composite_score.toFixed(1).padStart(5) | classification.composite_signal`);
console.log(` Market: classification.market_type | Rec: classification.recommendation`);
console.log(` Confidence: (classification.confidence * 100).toFixed(0)% | Size mult: classification.size_multiplierx`);
console.log(` Longs OK: '⛔'`);
// Show breakdown
for (const [tf, data] of [['Fast', fast], ['Medium', medium], ['Macro', macro]]) {
console.log(`\n tf Breakdown:`);
for (const [cat, info] of Object.entries(data.breakdown)) {
const indicators = Object.entries(info.indicators || {}).map(([k,v]) => `k:v`).join(' ');
console.log(` cat.padEnd(12) avg:info.avg.toFixed(1).padStart(5) w:info.weight | indicators`);
}
}
console.log('\nDone ✅');
}
// Export for use as module
module.exports = { run, scoreFast, scoreMedium, scoreMacro, classifyRegime };
// Run if called directly
if (require.main === module) {
run().catch(e => { console.error('Error:', e.message); process.exit(1); });
}
High-frequency paper trading framework for crypto. Multi-indicator TA scoring (RSI/MACD/EMA/BB/OBV/StochRSI), dual-regime filter (15m fast + 4h macro), posit...
---
name: hft-paper-trader
version: 1.1.0
description: High-frequency paper trading framework for crypto. Multi-indicator TA scoring (RSI/MACD/EMA/BB/OBV/StochRSI), dual-regime filter (15m fast + 4h macro), position sizing (Kelly criterion), correct stop-loss management (3% max risk cap), auto-observation logging, and trade ledger. Use for paper trading, backtesting trading logic, HFT simulation, or building an autonomous trading agent.
author: JamieRossouw
tags: [trading, paper-trading, hft, crypto, kelly, backtesting, autonomous-agent, regime-filter]
---
# HFT Paper Trader — Autonomous Crypto Trading Framework
A complete high-frequency paper trading system for building and testing autonomous crypto trading agents.
## Architecture
```
Market Data (Binance public API)
↓
TA Engine (RSI + MACD + EMA + BB + OBV + StochRSI)
↓
Signal Score (-10 to +10)
↓
Dual Regime Filter (15m EMA8/21 fast + 4h EMA20/50 macro)
↓
Kelly Position Sizer (3% max risk per trade)
↓
Paper Portfolio Manager (portfolio.json)
↓
Trade Ledger (journal.json) + Auto-Observation Logger (observations.md)
```
## Features
- **Multi-indicator confluence**: 7 indicators combined into one score
- **Dual regime filter**: 15m EMA8/21 (fast) gates entries alongside 4h EMA20/50 (macro) — prevents trading against short-term trend
- **OBV divergence detection**: hidden accumulation/distribution
- **Quarter-Kelly sizing**: conservative risk management
- **Correct SL placement**: Math.max caps risk at 3% — fixes bug where SL ran 5–11% instead of 3%
- **Drawdown controls**: auto-pause at 2% daily NAV
- **Full audit trail**: every trade logged with entry/stop/target/outcome
- **Auto-observation logging**: losses and SL hits automatically appended to `observations.md` for strategy learning
- **Self-improvement loop**: lessons captured after each loss
## Usage
```
Use hft-paper-trader to run TA on BTC and place a paper trade
Use hft-paper-trader to check portfolio performance
Use hft-paper-trader to scan the watchlist and trade all signals
Use hft-paper-trader to check today's observations and lessons
```
## Regime Filter Logic
Entries are only allowed when BOTH regimes are bullish:
- **Fast (15m)**: EMA8 > EMA21
- **Macro (4h)**: EMA20 > EMA50
This gates out counter-trend entries that otherwise pass signal scoring. When either regime is bearish, new positions are blocked regardless of score.
## Stop-Loss Placement (Fixed v1.1.0)
```js
// CORRECT — caps SL at 3% below entry
stopLoss = entry * (1 - MAX_RISK); // Math.max not used here — direct cap
// Bug in v1.0.0 — Math.min let SL run to 5-11%
// Fix: use direct percentage cap on entry price, not min of wick distance
```
## Watchlist
BTC ETH SOL XRP TRX DOGE ADA AVAX BNB LINK LTC SUI ARB OP NEAR DOT ATOM UNI MATIC
## File Layout
```
trading/
paper-dashboard/portfolio.json ← live portfolio state
journal.json ← full trade ledger
observations.md ← auto-logged trade lessons
```
## Performance
Starting capital: $1,000. Runs hourly (XX:01 UTC). Max 5 concurrent positions at 10% each.
FILE:_meta.json
{
"ownerId": "kn7151x9kdhkgx502kcrdqkgqd81gpt4",
"slug": "hft-paper-trader",
"version": "1.0.0",
"publishedAt": 1771685237755
}Track upcoming market-moving events (macro, crypto protocol, exchange listings, regulatory decisions, conference keynotes, ETF approvals) and pre-flag releva...
---
name: skill-catalyst-calendar
description: Track upcoming market-moving events (macro, crypto protocol, exchange listings, regulatory decisions, conference keynotes, ETF approvals) and pre-flag relevant assets 7-14 days before the event. Use when building a forward-looking trading calendar, identifying pre-event positioning opportunities, or reviewing upcoming catalysts for any asset class.
---
# Catalyst Calendar
Forward-looking event tracker. Identifies upcoming catalysts and surfaces pre-positioning opportunities before the market prices them in.
## What Counts as a Catalyst
- **Macro:** Fed decisions, CPI prints, GDP data, regulatory announcements
- **Crypto-specific:** Protocol upgrades, halving events, token unlocks, mainnet launches
- **Exchange:** Binance/Coinbase/Kraken new listings, futures launches
- **Regulatory:** ETF approvals/rejections, SEC/CFTC rulings, country-level bans or legalization
- **Conferences:** Major industry events (ETHDenver, Consensus, Binance Blockchain Week, NVIDIA GTC, etc.)
- **Earnings/Partnerships:** Public company earnings with crypto exposure (Coinbase, MicroStrategy, Marathon)
## Calendar Storage
Stored at: `~/.openclaw/workspace/trading/catalyst-calendar.json`
```json
{
"events": [
{
"id": "evt-001",
"date": "2026-03-19",
"event": "FOMC Rate Decision",
"category": "macro",
"impact": "high",
"affected_assets": ["BTC", "ETH", "all"],
"pre_position_days": 3,
"notes": "Rate hold expected — risk-on if confirmed",
"source": "federalreserve.gov"
},
{
"id": "evt-002",
"date": "2026-04-10",
"event": "Ethereum Pectra Upgrade",
"category": "protocol",
"impact": "high",
"affected_assets": ["ETH", "staking tokens"],
"pre_position_days": 14,
"notes": "EIP-7251 — raises validator limit, reduces sell pressure",
"source": "ethereum.org"
}
]
}
```
## Usage
### View upcoming catalysts (next 14 days)
```
List upcoming catalysts from trading/catalyst-calendar.json for the next 14 days. Flag any where pre_position_days window is now open.
```
### Add new event
```
Add to catalyst-calendar.json: [event details]
```
### Weekly scan (find new catalysts)
```
Search for upcoming crypto and macro events this week. Update catalyst-calendar.json with any new high-impact events in the next 30 days.
```
## Alert Logic
When today's date is within `pre_position_days` of an event:
```
📅 CATALYST ALERT — 7 days to Ethereum Pectra Upgrade
Date: 2026-04-10
Impact: HIGH
Affected: ETH, staking tokens
Pre-position window: OPEN NOW
Notes: EIP-7251 — raises validator limit, reduces sell pressure
Action: Review ETH position vs threshold-watcher signal
```
## Cron Integration
- **Weekly scan** (Monday 07:00 UTC): scrape upcoming events, update calendar
- **Daily check** (07:00 UTC): flag events where pre-position window opens today
## Integration with Trading Pipeline
- Outputs feed `skill-crypto-threshold-watcher` (set tighter thresholds near high-impact events)
- Logged to `skill-trading-journal` as context for trade decisions
- Informs `backtest-expert` on regime conditions (pre/post catalyst)
Log every trade with full context (thesis, entry, exit, PnL, emotion, lesson). Generate weekly and monthly performance reports. Identify patterns in wins/los...
---
name: skill-trading-journal
description: Log every trade with full context (thesis, entry, exit, PnL, emotion, lesson). Generate weekly and monthly performance reports. Identify patterns in wins/losses. Use when recording a new trade, reviewing performance, running a weekly debrief, or updating the trading strategy based on results.
---
# Trading Journal
Systematic trade logging and performance review. Every trade logged = strategy gets smarter over time.
## Trade Log Storage
Stored at: `~/.openclaw/workspace/trading/journal.json`
## Log a New Trade
```
Log trade to trading/journal.json:
- Symbol: GRASSUSDT
- Direction: LONG
- Entry: $0.36
- Entry date: 2026-03-13
- Size: $50 USDC
- Thesis: Price broke above $0.30 threshold + GTC keynote catalyst in 3 days + volume rising
- Catalyst: NVIDIA GTC 2026
- Signal source: skill-crypto-threshold-watcher
- Stop loss: $0.28 (22% below entry)
- Take profit: $0.50 (39% above entry)
```
## Log Trade Exit
```
Update journal entry [trade-id] with exit:
- Exit price: $0.46
- Exit date: 2026-03-17
- Exit reason: Take profit not hit, manual exit (Aladdin decision)
- PnL: +$12.78 (+27.7%)
- Lesson: Entry thesis was correct. Exit was early — could have held to $0.50 TP.
```
## Trade Record Schema
```json
{
"id": "trade-001",
"symbol": "GRASSUSDT",
"direction": "LONG",
"status": "CLOSED",
"entry_price": 0.36,
"entry_date": "2026-03-13",
"entry_size_usdc": 50,
"thesis": "Price broke above $0.30 threshold + GTC keynote catalyst",
"catalyst": "NVIDIA GTC 2026",
"signal_source": "skill-crypto-threshold-watcher",
"stop_loss": 0.28,
"take_profit": 0.50,
"exit_price": 0.46,
"exit_date": "2026-03-17",
"exit_reason": "Manual exit — pre-TP",
"pnl_usdc": 12.78,
"pnl_pct": 27.7,
"lesson": "Exit was early. Hold to TP next time unless thesis breaks.",
"emotion": "neutral",
"thesis_correct": true,
"execution_correct": false
}
```
## Weekly Performance Report
```
Generate weekly trading report from trading/journal.json for the past 7 days.
Include: total PnL, win rate, avg win vs avg loss, best/worst trade, lessons summary, strategy adjustments.
```
**Sample output:**
```
📊 WEEKLY TRADING REPORT — Mar 10–17, 2026
Trades: 2 | Wins: 2 | Losses: 0 | Win Rate: 100%
Total PnL: +$18.43 (+23.4% on deployed capital)
Avg Win: +$9.21 | Avg Loss: N/A
Best Trade: GRASSUSDT +$12.78 (+27.7%)
LESSONS THIS WEEK:
- Thesis-correct but execution left early on GRASS — hold to TP when thesis intact
- Volume spike threshold ($100M) on FET called the move correctly
STRATEGY ADJUSTMENTS:
- Raise FET volume threshold to $120M (reduce false positives)
- Add trailing stop at +20% to avoid leaving gains on table
NEXT WEEK CATALYSTS:
- [from catalyst-calendar]
```
## Monthly Strategy Review
```
Generate monthly strategy review from trading/journal.json.
Identify: which signal types worked, which failed, regime conditions, recommended rule updates.
```
## Integration with Trading Pipeline
- **Inputs from:** `skill-crypto-threshold-watcher` (signal), `skill-catalyst-calendar` (context), `binance-pro` (execution confirmation)
- **Outputs to:** `backtest-expert` (validated signals for re-testing), `quant-trading-system` (parameter updates)
- **Review cadence:** Weekly debrief every Sunday 20:00 UTC
## Emotion Tracking
Log emotion at trade entry: `calm | excited | fearful | greedy | neutral`
Over time, identify if emotional state correlates with win/loss rate. This is where most traders lose edge.
## Rules Enforcement
Before logging a new trade, verify:
- [ ] Thesis stated in one sentence
- [ ] Stop-loss defined
- [ ] Take-profit defined
- [ ] Position size within rulebook limits
- [ ] Catalyst identified (or "none — pure technical")
If any field missing → trade is NOT valid per rulebook.
Monitor any crypto token against configurable price/volume thresholds. Fires alerts when entry conditions are met. Use when you need proactive notification t...
---
name: skill-crypto-threshold-watcher
description: Monitor any crypto token against configurable price/volume thresholds. Fires alerts when entry conditions are met. Use when you need proactive notification that a watchlist token has crossed its threshold — not reactive price checks. Supports multiple tokens, multiple threshold types (price above/below, volume spike), and custom alert messages.
---
# Crypto Threshold Watcher
Proactive alert engine for any token on any exchange. Checks watchlist against configurable thresholds and fires signals.
## Watchlist Config
Stored at: `~/.openclaw/workspace/trading/watchlist.json`
```json
{
"tokens": [
{
"symbol": "GRASSUSDT",
"exchange": "binance",
"thresholds": {
"price_above": 0.30,
"price_below": 0.20,
"volume_24h_above": 50000000
},
"notes": "AI data network — entry above $0.30"
},
{
"symbol": "FETUSDT",
"exchange": "binance",
"thresholds": {
"price_above": 0.20,
"volume_24h_above": 100000000
},
"notes": "ASI Alliance token — volume spike = breakout signal"
}
]
}
```
## Usage
### Check all watchlist tokens
```bash
node ~/.openclaw/workspace/scripts/trading/threshold-watcher.js
```
### Add a token to watchlist
```bash
node ~/.openclaw/workspace/scripts/trading/threshold-watcher.js --add --symbol BTCUSDT --price-above 90000
```
### Check single token
```bash
node ~/.openclaw/workspace/scripts/trading/threshold-watcher.js --symbol ETHUSDT
```
## Output
When threshold is crossed:
```
🚨 THRESHOLD ALERT — GRASSUSDT
Price: $0.3245 (threshold: $0.30 ↑)
Volume 24h: $62.3M
Signal: ENTRY — price above threshold
Time: 2026-03-17 18:30 UTC
Notes: AI data network — entry above $0.30
```
When no threshold crossed:
```
✅ GRASSUSDT — $0.28 (below $0.30 threshold, watching)
```
## Data Sources
- Primary: Binance API (no auth required for market data)
- Fallback: CoinGecko API (free tier)
## Cron Integration
Add to TASKS.md cron:
```
Every 1h: node scripts/trading/threshold-watcher.js
```
Alerts delivered to Telegram DM automatically.
## Threshold Types
| Type | Field | Description |
|------|-------|-------------|
| Price breakout | `price_above` | Price crosses above level |
| Price breakdown | `price_below` | Price drops below level |
| Volume spike | `volume_24h_above` | 24h volume exceeds threshold |
| RSI overbought | `rsi_above` | RSI > value (requires OHLC data) |
| RSI oversold | `rsi_below` | RSI < value |
## Integration with Trading Pipeline
This skill feeds signals to:
- `backtest-expert` — validate signal before acting
- `skill-trading-journal` — log signal + decision
- `binance-pro` — execute if approved
Create and launch an Instantly.ai cold email campaign with D0/D3/D8 sequences and bulk-import leads via API, no dashboard needed.
# skill-instantly-campaign-launcher
Programmatically create an Instantly.ai cold email campaign with D0/D3/D8 sequences and bulk-import leads — all via Instantly API v2. One script, zero dashboard clicking.
## What it does
1. Creates a new Instantly campaign (or finds an existing one by name)
2. Adds a 3-step D0/D3/D8 email sequence to the campaign
3. Imports leads from a JSON file (with dedup/skip for existing leads)
4. Reports imported/skipped/failed counts
## Use cases
- Launching any cold email campaign via code (no UI required)
- B2B agency or service business outreach
- Quickly adapting a campaign template for new ICPs or industries
## Inputs
- `INSTANTLY_KEY` — Instantly API v2 Bearer token (set as env var)
- `scripts/campaign.config.js` — campaign name, schedule, D0/D3/D8 email bodies
- `leads.json` — array of lead objects: `{email, firstName, lastName, companyName, website}`
## Outputs
- Instantly campaign created (or found) with sequences attached
- Leads imported; console report: total / imported / skipped / failed
## Scripts
- `scripts/campaign-launcher.js` — main entry point
- `scripts/campaign.config.js` — config template (edit before running)
## Usage
```bash
# 1. Get your Instantly API v2 token from app.instantly.ai → Settings → API Keys
export INSTANTLY_KEY=your_token_here
# 2. Edit scripts/campaign.config.js — set campaign name, schedule, email copy
# 3. Create leads.json (array of lead objects)
# 4. Run:
node scripts/campaign-launcher.js --config scripts/campaign.config.js --leads leads.json
```
## leads.json format
```json
[
{ "email": "[email protected]", "firstName": "John", "lastName": "Smith", "companyName": "Acme Inc", "website": "acme.com" },
{ "email": "[email protected]", "firstName": "Jane", "companyName": "Corp IO" }
]
```
## campaign.config.js format
```js
module.exports = {
campaignName: 'My Outreach Campaign',
schedule: {
name: 'Business Hours',
timing: { from: '09:00', to: '17:00' },
days: { monday: true, tuesday: true, wednesday: true, thursday: true, friday: true, saturday: false, sunday: false },
timezone: 'America/New_York', // or Asia/Dubai, Europe/London, etc.
},
sequences: [
{ step: 1, delay: 0, subject: 'Your subject', body: 'Hi {{firstName}}, ...' },
{ step: 2, delay: 3, subject: 'Re: Your subject', body: 'Follow-up body...' },
{ step: 3, delay: 8, subject: 'Last note, {{firstName}}', body: 'Closing body...' },
],
};
```
## Notes
- Instantly API v2 — sequences endpoint: `POST /campaigns/:id/sequences`
- Dedup: 409 responses = lead already in campaign (counted as skipped, not error)
- Rate limit: 200ms sleep between lead imports to avoid 429s
- Known issue: Instantly API v2 `/sequences` endpoint occasionally returns 404 → add sequences manually in dashboard if this occurs
- Instantly free plan supports unlimited campaigns; warming up inboxes recommended before launch
FILE:scripts/campaign-launcher.js
#!/usr/bin/env node
/**
* skill-instantly-campaign-launcher
* Creates an Instantly campaign with D0/D3/D8 sequences and imports leads.
*
* Usage:
* node campaign-launcher.js --config campaign.config.js --leads leads.json
* INSTANTLY_KEY=<token> node campaign-launcher.js --config campaign.config.js --leads leads.json
*/
const https = require('https');
const fs = require('fs');
const path = require('path');
// ── Config ──────────────────────────────────────────────────────────────────
const args = process.argv.slice(2);
const configPath = args[args.indexOf('--config') + 1] || path.join(__dirname, 'campaign.config.js');
const leadsPath = args[args.indexOf('--leads') + 1] || path.join(process.cwd(), 'leads.json');
const config = require(path.resolve(configPath));
const INSTANTLY_KEY = process.env.INSTANTLY_KEY || config.instantlyKey;
if (!INSTANTLY_KEY) {
console.error('❌ Missing INSTANTLY_KEY — set env var or add instantlyKey to campaign.config.js');
process.exit(1);
}
// ── HTTP helper ──────────────────────────────────────────────────────────────
function httpsRequest(options, body = null) {
return new Promise((resolve, reject) => {
const req = https.request(options, (res) => {
let data = '';
res.on('data', chunk => (data += chunk));
res.on('end', () => {
try { resolve({ status: res.statusCode, body: JSON.parse(data) }); }
catch { resolve({ status: res.statusCode, body: data }); }
});
});
req.on('error', reject);
if (body) req.write(JSON.stringify(body));
req.end();
});
}
function instantly(method, path_, body = null) {
return httpsRequest({
hostname: 'api.instantly.ai',
path: `/api/v2path_`,
method,
headers: {
Authorization: `Bearer INSTANTLY_KEY`,
'Content-Type': 'application/json',
},
}, body);
}
function sleep(ms) { return new Promise(r => setTimeout(r, ms)); }
// ── Main ─────────────────────────────────────────────────────────────────────
async function main() {
const { campaignName, schedule, sequences } = config;
console.log('='.repeat(56));
console.log(`📧 Instantly Campaign Launcher`);
console.log(` Campaign: campaignName`);
console.log('='.repeat(56));
// ── Step 1: Find or create campaign ────────────────────────────────────────
console.log('\n🔍 Step 1: Find or create campaign...');
let campaignId = null;
const listRes = await instantly('GET', '/campaigns?limit=100');
if (listRes.status === 200 && listRes.body?.items) {
const found = listRes.body.items.find(c => c.name === campaignName);
if (found) {
campaignId = found.id;
console.log(` ✅ Found existing campaign: campaignId`);
}
}
if (!campaignId) {
const createRes = await instantly('POST', '/campaigns', {
name: campaignName,
campaign_schedule: {
schedules: [
{
name: schedule.name,
timing: schedule.timing,
days: schedule.days,
timezone: schedule.timezone,
},
],
},
});
if (createRes.status === 200 || createRes.status === 201) {
campaignId = createRes.body?.id;
console.log(` ✅ Campaign created: campaignId`);
} else {
console.error(` ❌ Create failed: createRes.status`, JSON.stringify(createRes.body));
process.exit(1);
}
}
// ── Step 2: Add sequences ───────────────────────────────────────────────────
console.log('\n✉️ Step 2: Adding D0/D3/D8 sequences...');
const seqPayload = sequences.map(s => ({
step: s.step,
type: 'email',
delay: s.delay,
variants: [{ subject: s.subject, body: s.body }],
}));
const seqRes = await instantly('POST', `/campaigns/campaignId/sequences`, {
campaign_id: campaignId,
sequences: seqPayload,
});
if (seqRes.status === 200 || seqRes.status === 201) {
console.log(` ✅ Sequences added (sequences.length steps)`);
} else {
console.warn(` ⚠️ Sequences: seqRes.status — JSON.stringify(seqRes.body).substring(0, 120)`);
console.warn(' ↳ Add sequences manually in Instantly dashboard if this persists (known API v2 quirk)');
}
// ── Step 3: Import leads ────────────────────────────────────────────────────
if (!fs.existsSync(leadsPath)) {
console.log(`\n⚠️ No leads file found at leadsPath. Campaign created but no leads imported.`);
console.log(` To import leads later, create leads.json and re-run with --leads leads.json`);
return { campaignId, imported: 0, skipped: 0, failed: 0 };
}
const leadsRaw = JSON.parse(fs.readFileSync(leadsPath, 'utf8'));
const leads = Array.isArray(leadsRaw) ? leadsRaw : leadsRaw.leads || [];
console.log(`\n📥 Step 3: Importing leads.length leads...`);
let imported = 0, skipped = 0, failed = 0;
for (const lead of leads) {
const res = await instantly('POST', '/leads', {
campaign_id: campaignId,
email: lead.email,
first_name: lead.firstName || lead.first_name || 'there',
last_name: lead.lastName || lead.last_name || '',
company_name: lead.companyName || lead.company_name || '',
website: lead.website || '',
});
if (res.status === 200 || res.status === 201) {
imported++;
process.stdout.write(` ✅ lead.email\n`);
} else if (res.status === 409) {
skipped++;
process.stdout.write(` ⏭️ lead.email (already exists)\n`);
} else {
failed++;
process.stdout.write(` ❌ lead.email — res.status\n`);
}
await sleep(200); // Avoid rate limits
}
// ── Summary ─────────────────────────────────────────────────────────────────
console.log('\n' + '='.repeat(56));
console.log('✅ DONE');
console.log(` Campaign: campaignName (campaignId)`);
console.log(` Leads: leads.length total | imported imported | skipped skipped | failed failed`);
console.log('='.repeat(56));
return { campaignId, imported, skipped, failed };
}
main().catch(e => { console.error('Fatal:', e.message); process.exit(1); });
FILE:scripts/campaign.config.js
/**
* Instantly Campaign Config Template
* Customize this for each new outreach campaign.
*/
module.exports = {
// Campaign name — if already exists, the script reuses it (no duplicate created)
campaignName: 'My Cold Email Campaign',
// Schedule — when emails are sent
schedule: {
name: 'UAE Business Hours',
timing: { from: '09:00', to: '17:00' },
days: {
monday: true, tuesday: true, wednesday: true,
thursday: true, friday: true, saturday: false, sunday: false,
},
timezone: 'Asia/Dubai',
},
// 3-step D0 / D3 / D8 email sequence
// Instantly variables: {{firstName}}, {{companyName}}, {{website}}
sequences: [
{
step: 1,
delay: 0, // days after campaign start (D0)
subject: 'Quick question, {{firstName}}',
body: `Hi {{firstName}},
[D0 email body here — introduce yourself, state the pain, offer the value prop]
[CTA — link or reply ask]
— [Your Name]
[Company]`,
},
{
step: 2,
delay: 3, // D3 follow-up
subject: 'Re: Quick question, {{firstName}}',
body: `Hi {{firstName}},
[D3 follow-up — address the most common hesitation, reinforce ROI]
[Soft CTA]
— [Your Name]`,
},
{
step: 3,
delay: 8, // D8 closing email
subject: 'Last note, {{firstName}}',
body: `Hi {{firstName}},
[D8 close — create urgency or give a soft out]
[Final CTA]
— [Your Name]`,
},
],
};
Calculates Amazon UAE FBA fees, net margin, and DG risk for CJ Dropshipping products, ranking by margin % and generating detailed markdown and JSON reports.
# skill-fba-margin-calculator
Version: 1.0.0 | Owner: GitHub Ops | Created: 2026-03-09
## What It Does
Calculates Amazon UAE FBA fees, net margin, and DG risk for CJ Dropshipping product candidates. Ranks results by margin % and outputs a markdown report + JSON.
Extracted from the Tech1Mart FBA pilot selection workflow that ran 2026-03-08 and 2026-03-09.
## When to Use
- After running skill-dropshipping-sourcing (CJ product lookup)
- Before ordering samples — confirm margin viability
- When comparing multiple SKUs for Amazon UAE FBA launch
## Prerequisites
- Node.js (no npm installs required — stdlib only)
- CJ product data: SKU, price (USD), weight (grams), target AED price
## Usage
### Single product
```bash
node skills/skill-fba-margin-calculator/calc.js \
--sku CJLE2170569 \
--name "Ring Light Phone Stand" \
--price 12.96 \
--weight 400 \
--target 129 \
--shipping 4
```
### Batch (JSON file)
```bash
node skills/skill-fba-margin-calculator/calc.js \
--input products.json \
--output output/fba-report
```
### Stdin pipe
```bash
cat products.json | node skills/skill-fba-margin-calculator/calc.js
```
### With DG flag
```bash
node skills/skill-fba-margin-calculator/calc.js \
--sku CJYS1240831 --price 12.72 --weight 150 --target 99 --dg
```
## Input JSON Format
```json
[
{
"sku": "CJLE2170569",
"name": "Ring Light + Phone Stand",
"cj_price_usd": 12.96,
"weight_g": 400,
"target_aed": 129,
"shipping_usd": 4,
"dg_risk": false,
"referral_pct": 8
}
]
```
Optional fields: `shipping_usd` (default 3.5), `dg_risk` (default false), `referral_pct` (default 8%).
## Output
- Ranked markdown table by margin %
- Detail breakdown per SKU (landed cost, FBA fee tier, referral fee, net margin)
- Verdicts: ✅ FBA-safe / 🟡 Marginal / ❌ Too thin / ⚠️ DG Risk (WooCommerce-only)
- `--output` flag: writes `.md` + `.json` files
## FBA Fee Tiers (Amazon UAE, Q1 2026 estimates)
| Size Class | Max Weight | Est. Fee |
|------------|-----------|----------|
| Small Standard | 150g | 10 AED |
| Standard S | 350g | 13.5 AED |
| Standard M | 700g | 16.5 AED |
| Standard L | 1kg | 20 AED |
| Large Standard | 2kg | 26 AED |
| Oversize | 2kg+ | 38 AED |
Plus: 8% referral fee (electronics) + ~0.75 AED/unit storage.
## Flags
| Flag | Description |
|------|-------------|
| `--input FILE` | JSON file with product array |
| `--output BASE` | Write BASE.md + BASE.json |
| `--exchange RATE` | USD→AED rate (default 3.67) |
| `--sku` | Single SKU mode (with --price, --weight, --target) |
| `--price USD` | CJ price in USD |
| `--weight GRAMS` | Product weight in grams |
| `--target AED` | Target sell price in AED |
| `--shipping USD` | CJ shipping cost in USD (default 3.5) |
| `--referral PCT` | Referral fee % (default 8) |
| `--dg` | Flag product as Dangerous Goods (recommends WooCommerce-only) |
## Notes
- FBA fees are estimates — verify final numbers via Amazon Seller Central FBA Revenue Calculator before ordering samples
- Exchange rate defaults to 3.67 (AED/USD peg) — adjust if needed
- DG risk applies to: Li-ion batteries, aerosols, liquids. Always check before sending to FBA.
- Minimum viable margin: >25% AND >30 AED net for FBA to be worthwhile
FILE:_meta.json
{
"name": "skill-fba-margin-calculator",
"version": "1.0.0",
"description": "Amazon UAE FBA fee estimator + margin ranker for CJ Dropshipping product candidates. Ranks SKUs by net margin and flags DG risk.",
"author": "Zero2Ai-hub",
"license": "MIT"
}
FILE:calc.js
#!/usr/bin/env node
/**
* skill-fba-margin-calculator
* Amazon UAE FBA fee estimator + margin ranker for CJ Dropshipping candidates.
*
* Usage:
* node calc.js --input products.json [--exchange 3.67] [--output report.md]
* node calc.js --sku CJLE2170569 --price 12.96 --weight 400 --target 129 --shipping 4
* cat products.json | node calc.js
*
* Input JSON (array):
* [{ sku, name, cj_price_usd, weight_g, target_aed, shipping_usd?, dg_risk?, referral_pct? }]
*
* Output: ranked markdown table + JSON
*/
const fs = require('fs');
// --- FBA fee tiers for Amazon UAE (Q1 2026 estimates) ---
const FBA_TIERS = [
{ label: 'Small Standard', maxWeight: 150, maxDims: [30, 20, 5], fee: 10 },
{ label: 'Standard S', maxWeight: 350, fee: 13.5 },
{ label: 'Standard M', maxWeight: 700, fee: 16.5 },
{ label: 'Standard L', maxWeight: 1000, fee: 20 },
{ label: 'Large Standard', maxWeight: 2000, fee: 26 },
{ label: 'Oversize', maxWeight: Infinity, fee: 38 },
];
const STORAGE_PER_UNIT = 0.75; // AED/unit/month avg
function getFbaFee(weightG) {
for (const tier of FBA_TIERS) {
if (weightG <= tier.maxWeight) return { fee: tier.fee, tier: tier.label };
}
return { fee: 38, tier: 'Oversize' };
}
function calcMargin(product, exchangeRate = 3.67) {
const {
sku,
name,
cj_price_usd,
weight_g,
target_aed,
shipping_usd = 3.5,
dg_risk = false,
referral_pct = 8,
} = product;
const cj_aed = (parseFloat(cj_price_usd) + parseFloat(shipping_usd)) * exchangeRate;
const { fee: fba_fee, tier } = getFbaFee(parseFloat(weight_g));
const referral_fee = (target_aed * referral_pct) / 100;
const total_fees = fba_fee + referral_fee + STORAGE_PER_UNIT;
const net_margin_aed = target_aed - cj_aed - total_fees;
const margin_pct = ((net_margin_aed / target_aed) * 100).toFixed(1);
const recommendation =
dg_risk
? '⚠️ DG RISK — WooCommerce-only (no FBA until DG cleared)'
: net_margin_aed >= 30 && parseFloat(margin_pct) >= 25
? '✅ FBA-safe — READY'
: net_margin_aed >= 15
? '🟡 MARGINAL — consider WooCommerce-only'
: '❌ Too thin for FBA';
return {
sku,
name: name || sku,
cj_price_usd,
shipping_usd,
cj_landed_aed: +cj_aed.toFixed(1),
fba_tier: tier,
fba_fee,
referral_fee: +referral_fee.toFixed(1),
storage_fee: STORAGE_PER_UNIT,
total_fees: +total_fees.toFixed(1),
target_aed,
net_margin_aed: +net_margin_aed.toFixed(1),
margin_pct: +margin_pct,
dg_risk: !!dg_risk,
recommendation,
};
}
function toMarkdown(results, exchangeRate) {
const date = new Date().toISOString().split('T')[0];
let md = `# Amazon UAE FBA Margin Report\n_Generated: date | Exchange rate: 1 USD = exchangeRate AED_\n\n`;
md += `## Ranked Results (by margin %)\n\n`;
md += `| Rank | SKU | Name | CJ+Ship (AED) | FBA Tier | Net Margin | Margin % | DG | Verdict |\n`;
md += `|------|-----|------|--------------|----------|------------|----------|----|---------|\n`;
results.forEach((r, i) => {
md += `| i + 1 | r.sku | r.name.substring(0, 35) | r.cj_landed_aed AED | r.fba_tier | r.net_margin_aed AED | r.margin_pct% | '✅' | r.recommendation |\n`;
});
md += `\n## Detail\n\n`;
results.forEach((r, i) => {
md += `### i + 1. r.name (r.sku)\n`;
md += `- CJ price: $r.cj_price_usd + $r.shipping_usd shipping = **r.cj_landed_aed AED** landed\n`;
md += `- FBA fee (r.fba_tier): r.fba_fee AED\n`;
md += `- Referral fee: r.referral_fee AED\n`;
md += `- Storage (avg): r.storage_fee AED\n`;
md += `- Total fees: r.total_fees AED\n`;
md += `- Target price: r.target_aed AED → **Net margin: r.net_margin_aed AED (r.margin_pct%)**\n`;
md += `- **r.recommendation**\n\n`;
});
return md;
}
// --- CLI entrypoint ---
async function main() {
const args = process.argv.slice(2);
const exchangeRate = parseFloat(getArg(args, '--exchange') || '3.67');
const outputFile = getArg(args, '--output');
let products = [];
// Single product via flags
if (hasArg(args, '--sku')) {
products = [{
sku: getArg(args, '--sku'),
name: getArg(args, '--name') || getArg(args, '--sku'),
cj_price_usd: parseFloat(getArg(args, '--price')),
weight_g: parseFloat(getArg(args, '--weight')),
target_aed: parseFloat(getArg(args, '--target')),
shipping_usd: parseFloat(getArg(args, '--shipping') || '3.5'),
dg_risk: hasArg(args, '--dg'),
referral_pct: parseFloat(getArg(args, '--referral') || '8'),
}];
} else {
// JSON from --input file or stdin
const inputFile = getArg(args, '--input');
let raw = '';
if (inputFile) {
raw = fs.readFileSync(inputFile, 'utf8');
} else if (!process.stdin.isTTY) {
raw = fs.readFileSync('/dev/stdin', 'utf8');
} else {
console.error('Usage: node calc.js --input products.json OR pipe JSON via stdin');
process.exit(1);
}
products = JSON.parse(raw);
}
const results = products
.map(p => calcMargin(p, exchangeRate))
.sort((a, b) => b.margin_pct - a.margin_pct);
const md = toMarkdown(results, exchangeRate);
const json = JSON.stringify(results, null, 2);
if (outputFile) {
const base = outputFile.replace(/\.(md|json)$/, '');
fs.writeFileSync(base + '.md', md);
fs.writeFileSync(base + '.json', json);
console.log(`✅ Wrote base.md + base.json`);
} else {
console.log(md);
console.log('\n--- JSON ---\n' + json);
}
}
function getArg(args, flag) {
const i = args.indexOf(flag);
return i >= 0 && args[i + 1] ? args[i + 1] : null;
}
function hasArg(args, flag) {
return args.includes(flag);
}
main().catch(e => { console.error(e); process.exit(1); });
FILE:example-products.json
[
{
"sku": "CJLE2170569",
"name": "Ring Light + Phone Stand Combo",
"cj_price_usd": 12.96,
"weight_g": 400,
"target_aed": 129,
"shipping_usd": 4,
"dg_risk": false,
"referral_pct": 8
},
{
"sku": "CJTY1053448",
"name": "Portable Mini Projector",
"cj_price_usd": 19.72,
"weight_g": 500,
"target_aed": 159,
"shipping_usd": 5,
"dg_risk": false,
"referral_pct": 8
},
{
"sku": "CJYS1240831",
"name": "Wearable Neck BT Speaker",
"cj_price_usd": 12.72,
"weight_g": 150,
"target_aed": 99,
"shipping_usd": 3.5,
"dg_risk": true,
"referral_pct": 8
}
]
Google Workspace CLI (official Google release) for Drive, Gmail, Calendar, Sheets, Docs, Chat, Admin, and every Workspace API. Includes native MCP server mod...
---
name: gws
description: Google Workspace CLI (official Google release) for Drive, Gmail, Calendar, Sheets, Docs, Chat, Admin, and every Workspace API. Includes native MCP server mode for AI agents. Use when you need to manage Google Workspace services via CLI or expose them as MCP tools.
license: MIT
homepage: https://github.com/googleworkspace/cli
metadata:
{
"openclaw":
{
"emoji": "🏢",
"requires": { "bins": ["gws"] },
"install":
[
{
"id": "npm",
"kind": "npm",
"package": "@googleworkspace/cli",
"global": true,
"bins": ["gws"],
"label": "Install gws (npm)",
},
],
},
}
---
# gws — Google Workspace CLI
Official Google-published CLI for all Workspace APIs. Dynamically built from Google Discovery Service — covers every API endpoint automatically as Google adds them.
**Note:** This is the official Google-org CLI (`googleworkspace/cli`), distinct from third-party alternatives. Prefer `gws` for new integrations — it has native MCP mode and active development from Google.
---
## APIs Covered
- **Drive** — files, folders, sharing, permissions
- **Gmail** — messages, labels, drafts, send
- **Calendar** — events, calendars, invites
- **Sheets** — spreadsheets, values, formatting
- **Docs** — documents read/write
- **Chat** — spaces, messages
- **Admin** — users, groups, org units
- **Tasks** — task lists, tasks
- **Meet** — meeting resources
- *...and every other Workspace API via Discovery Service*
---
## Installation
```bash
npm install -g @googleworkspace/cli
# or: cargo install --git https://github.com/googleworkspace/cli --locked
# or: nix run github:googleworkspace/cli
```
Verify:
```bash
gws --version
```
---
## Authentication (One-time Setup Required)
⚠️ **Auth requires manual action** — OAuth credentials must be set up once per account.
### Option A: With gcloud (fastest)
```bash
gws auth setup # creates GCP project, enables APIs, logs in
gws auth login # subsequent logins / scope changes
```
### Option B: Without gcloud (manual GCP console)
1. Go to console.cloud.google.com → create project
2. Enable Workspace APIs (Drive, Gmail, Calendar, etc.)
3. Create OAuth 2.0 Client ID (Desktop app type)
4. Download credentials JSON
5. Set: `export GOOGLE_WORKSPACE_CLI_CREDENTIALS_FILE=/path/to/credentials.json`
6. Run: `gws auth login`
### Option C: Service Account (server/headless)
```bash
export GOOGLE_WORKSPACE_CLI_CREDENTIALS_FILE=/path/to/service-account.json
```
Credentials are encrypted at rest (AES-256-GCM) with OS keyring.
---
## Basic Usage
```bash
# List 10 most recent Drive files
gws drive files list --params '{"pageSize": 10}'
# Create a spreadsheet
gws sheets spreadsheets create --json '{"properties": {"title": "Q1 Budget"}}'
# List Gmail messages
gws gmail users messages list --params '{"userId": "me", "maxResults": 5}'
# Send a Gmail message
gws gmail users messages send --params '{"userId": "me"}' --json '{"raw": "<base64>"}'
# Inspect any method schema
gws schema drive.files.list
# Stream paginated results
gws drive files list --params '{"pageSize": 100}' --page-all | jq -r '.files[].name'
# Dry-run (preview request without executing)
gws chat spaces messages create --params '{"parent": "spaces/xyz"}' --json '{"text": "test"}' --dry-run
```
---
## MCP Server Mode (AI Agent Use)
`gws` can act as an MCP server, exposing all Workspace APIs as structured tools for Claude, Cursor, VS Code, etc.
```bash
# Start MCP server (all services)
gws mcp
# Start MCP server (specific services only — recommended)
gws mcp -s drive,gmail,calendar
# Compact mode — reduces from 200-400 tools to ~26 meta-tools (saves context)
gws mcp -s drive,gmail,calendar --tool-mode compact
```
### Add to your agent config (e.g. OpenClaw / mcporter):
```json
{
"mcpServers": {
"google-workspace": {
"command": "gws",
"args": ["mcp", "-s", "drive,gmail,calendar,sheets,docs", "--tool-mode", "compact"]
}
}
}
```
---
## OpenClaw Agent Usage
After auth setup, the agent can:
- Read/send Gmail → automate email workflows
- Read/write Calendar → schedule meetings, parse availability
- Read/write Sheets → log data, pull reports
- Manage Drive → organize files, share docs
- Chat → send notifications to Google Chat spaces
Example — list recent emails:
```bash
gws gmail users messages list --params '{"userId": "me", "maxResults": 10, "q": "is:unread"}'
```
---
## Notes
- **Auth is manual (one-time)** — must complete `gws auth setup` before first use
- **Active development** — pre-v1.0, expect breaking changes; check GitHub for latest
- **Official Google org** — published by `googleworkspace` on GitHub, not a third-party
- **No boilerplate** — structured JSON output, works with `jq` and scripts
- **MCP-native** — expose any Workspace API as an MCP tool with a single command
---
## Status
- **Viability:** ✅ HIGH — official Google org, MCP-native, npm installable
- **Auth blocker:** ⚠️ Manual one-time `gws auth setup` required (GCP project needed)
- **Replacement:** Better long-term choice vs. gcloud scripting — auto-updates with Google API surface
Transforms supplier or CJ source videos into 1080×1920 TikTok/Instagram Reels ads with clean zone detection, Pillow text overlays, CTA card, and trending audio.
---
name: skill-supplier-video-ad
version: 1.0.0
description: Build polished 1080×1920 TikTok/Instagram Reels product ads from supplier or CJ Dropshipping source videos. Handles: text-free zone detection, slow-motion cuts, Montserrat Bold text overlays (Pillow-rendered, no ffmpeg drawtext), branded CTA end card, and trending TikTok audio. Use when you have raw supplier/CJ footage and need a ready-to-post short-form ad.
---
# Supplier Video Ad Builder
Turns raw supplier source footage into polished 1080×1920 TikTok/IG Reels ads.
**Pipeline:** detect clean zones → cut + slow-mo → Pillow text overlays → CTA end card → trending audio → final MP4.
## Requirements
- `ffmpeg` + `ffprobe`
- Python 3.10+: `Pillow`
- Montserrat-Bold font (or any TTF — configurable in scripts)
- Your brand logo as PNG (with transparency)
- Source video from supplier/CJ
## File Structure
```
skill-supplier-video-ad/
├── scripts/
│ ├── detect_clean_zones.py # Step 1: extract 1fps frames for text analysis
│ └── make_product_ad.py # Step 2: full pipeline from config JSON
├── example/
│ └── product-config.json # Reference config template
└── SKILL.md
```
## Workflow
### Step 1 — Detect Clean Zones
Run this on each source video to extract frames for visual inspection:
```bash
python3 scripts/detect_clean_zones.py path/to/source.mp4 --output-dir ./frames
```
Then analyze the frames with an image model:
> "For each frame (f_01=t1s, f_02=t2s, ...), is there baked-in text? YES/NO"
Map YES/NO → timestamps → define `clean_zones` in your config.
### Step 2 — Create Product Config
```json
{
"product_name": "Cool Gadget",
"price": "29.99 USD",
"output_name": "cool-gadget-ad-v1.mp4",
"source_videos": {
"main": "path/to/source.mp4"
},
"clean_zones": [
{ "source": "main", "ss": 2.5, "dur": 2.5, "slowmo": true, "label": "hero-shot" },
{ "source": "main", "ss": 8.0, "dur": 2.0, "slowmo": false, "label": "lifestyle" },
{ "source": "main", "ss": 14.0,"dur": 3.0, "slowmo": false, "label": "detail" }
],
"text_overlays": [
{ "lines": ["Line 1", "Line 2"], "start": 0.5, "end": 3.5, "fontsize": 70, "y_pct": 0.62 },
{ "lines": ["Feature Text"], "start": 3.8, "end": 6.5, "fontsize": 70, "y_pct": 0.62 },
{ "lines": ["SHOP NOW"], "start": 6.8, "end": 9.5, "fontsize": 92, "y_pct": 0.66 },
{ "lines": ["yourstore.com"], "start": 6.8, "end": 9.5, "fontsize": 58, "y_pct": 0.76 }
],
"font_path": "/path/to/Montserrat-Bold.ttf",
"audio": "path/to/trending-sound.mp3",
"audio_start_offset": 2,
"audio_volume": 0.88,
"cta": {
"duration": 3,
"price": "29.99 USD",
"line1": "Link in bio 🛒",
"line2": "yourstore.com",
"bg_color": [8, 8, 14],
"logo_path": "path/to/logo.png"
}
}
```
**Config tips:**
- `y_pct`: vertical position as fraction of frame height. `0.62` = product copy, `0.66` = CTA verb, `0.76` = URL.
- `fontsize`: 70–74 for body, 92 for "Shop Now", 58 for URL.
- Lines ≤ 20 chars each for clean wrapping.
- `slowmo: true` → 0.8× speed (PTS=1.25×). Use on hero/dark cinematic shots.
- Total duration: ~10–13s product footage + 3s CTA = 13–16s ideal ad length.
### Step 3 — Run Pipeline
```bash
python3 scripts/make_product_ad.py example/product-config.json
```
Output: `output/cool-gadget-ad-v1.mp4` — ready to post.
## Text Rendering Note
**This skill does NOT use `ffmpeg drawtext`** — it renders text as Pillow PNGs and overlays them via `overlay` filter. This avoids ffmpeg font/emoji compatibility issues and works on any Linux server.
## Output Spec
- Resolution: 1080×1920 (9:16 portrait)
- FPS: 30
- Codec: H.264 + AAC 128k
- Typical size: 8–20 MB
## Proven TikTok Sounds (March 2026)
| Track | Vibe | TikTok Videos |
|-------|------|---------------|
| "Bounce (i just wanna dance)" — фрози & joyful | Upbeat instrumental | 8M+ ✅ |
| "warm nights" — Xori | Lofi/chill lifestyle | Trending |
| "Break Me" — Blake Whiten | Cinematic product demo | Trending |
## CTA Card
The pipeline auto-generates a branded end card with:
- Your logo (PNG with alpha) centered, scaled to 400px wide
- Glowing ambient light effect behind logo
- Price in large bold type
- CTA line + URL in muted gray
- Dark background (configurable `bg_color`)
FILE:example/product-config.json
{
"product_name": "Cool Gadget",
"price": "29.99 USD",
"output_name": "cool-gadget-ad-v1.mp4",
"font_path": "/path/to/Montserrat-Bold.ttf",
"source_videos": {
"main": "path/to/supplier-source.mp4",
"alt": "path/to/supplier-source-2.mp4"
},
"clean_zones": [
{ "source": "main", "ss": 2.5, "dur": 2.5, "slowmo": true, "label": "hero-dark" },
{ "source": "main", "ss": 8.0, "dur": 2.0, "slowmo": false, "label": "lifestyle" },
{ "source": "alt", "ss": 3.0, "dur": 3.0, "slowmo": false, "label": "detail-shot" }
],
"text_overlays": [
{
"lines": ["Your Hook Here", "Second Line"],
"start": 0.5, "end": 3.5,
"fontsize": 70,
"y_pct": 0.62
},
{
"lines": ["Key Feature"],
"start": 3.8, "end": 6.5,
"fontsize": 70,
"y_pct": 0.62
},
{
"lines": ["SHOP NOW"],
"start": 6.8, "end": 9.5,
"fontsize": 92,
"y_pct": 0.66
},
{
"lines": ["yourstore.com"],
"start": 6.8, "end": 9.5,
"fontsize": 58,
"y_pct": 0.76
}
],
"audio": "path/to/trending-sound.mp3",
"audio_start_offset": 2,
"audio_volume": 0.88,
"cta": {
"duration": 3,
"price": "29.99 USD",
"line1": "Link in bio 🛒",
"line2": "yourstore.com",
"bg_color": [8, 8, 14],
"logo_path": "path/to/logo.png"
}
}
FILE:scripts/detect_clean_zones.py
#!/usr/bin/env python3
"""
detect_clean_zones.py — Step 1: Supplier Video Ad Builder
Extracts 1fps frames from source video so you can visually identify text-free zones.
Usage:
python3 detect_clean_zones.py <video_path> [--output-dir ./frames]
Output:
frames/<video_stem>/f_01.jpg, f_02.jpg, ... (one frame per second)
frames/<video_stem>/summary.txt (frame list with timestamps)
After running: analyze frames with an image model or manually.
Text-free frames define clean_zones in your product config JSON.
"""
import subprocess
import argparse
from pathlib import Path
def main():
parser = argparse.ArgumentParser()
parser.add_argument('video', help='Path to source video')
parser.add_argument('--output-dir', default='./frames', help='Output directory for frames')
args = parser.parse_args()
video = Path(args.video)
out_dir = Path(args.output_dir) / video.stem
out_dir.mkdir(parents=True, exist_ok=True)
# Get video duration
r = subprocess.run(
['ffprobe', '-v', 'quiet', '-show_entries', 'format=duration',
'-of', 'csv=p=0', str(video)],
capture_output=True, text=True
)
duration = float(r.stdout.strip())
n_frames = int(duration)
print(f'📹 {video.name} — {duration:.1f}s → extracting {n_frames} frames at 1fps...')
subprocess.run([
'ffmpeg', '-y', '-i', str(video),
'-vf', 'fps=1,scale=480:-1',
str(out_dir / 'f_%02d.jpg'),
'-loglevel', 'error'
], check=True)
# Write summary
summary_lines = [f'# Frame Summary — {video.name}', f'Duration: {duration:.1f}s', '']
for i in range(1, n_frames + 1):
summary_lines.append(f'f_{i:02d}.jpg → t={i}s (ss={i-1} to {i})')
summary = out_dir / 'summary.txt'
summary.write_text('\n'.join(summary_lines))
print(f'✅ {n_frames} frames → {out_dir}/')
print(f' Review: inspect each frame for baked-in supplier text/watermarks.')
print(f' Prompt for image model: "For each frame (f_01=t1s, ...), is there baked-in text? YES/NO"')
print(f' Text-free frames → use ss/dur values as clean_zones in your config.')
if __name__ == '__main__':
main()
FILE:scripts/make_product_ad.py
#!/usr/bin/env python3
"""
make_product_ad.py — Supplier Video Ad Builder (Main Pipeline)
Assembles a 1080×1920 TikTok/IG Reels ad from clean video zones + text + CTA card + music.
Usage:
python3 make_product_ad.py <product_config.json>
Output:
output/<output_name> (final ready-to-post MP4)
Requirements:
pip install Pillow
ffmpeg + ffprobe in PATH
Montserrat-Bold.ttf (or any TTF font — set font_path in config)
"""
import subprocess
import json
import argparse
import shutil
from pathlib import Path
from PIL import Image, ImageDraw, ImageFont
W, H = 1080, 1920
# Default font — override via "font_path" in product config
DEFAULT_FONT = 'Montserrat-Bold.ttf'
def run(cmd, label=''):
r = subprocess.run(cmd, capture_output=True, text=True)
if r.returncode != 0:
print(f'ERROR [{label}]:\n{r.stderr[-800:]}')
raise SystemExit(1)
if label:
print(f' ✅ {label}')
return r
def get_duration(path):
r = subprocess.run(
['ffprobe', '-v', 'quiet', '-show_entries', 'format=duration', '-of', 'csv=p=0', str(path)],
capture_output=True, text=True
)
return float(r.stdout.strip())
def make_text_png(lines, fontsize, y_pct, out_path, font_path):
img = Image.new('RGBA', (W, H), (0, 0, 0, 0))
draw = ImageDraw.Draw(img)
font = ImageFont.truetype(font_path, fontsize)
gap = 16
bbs = [draw.textbbox((0, 0), l, font=font) for l in lines]
ws = [b[2] - b[0] for b in bbs]
hs = [b[3] - b[1] for b in bbs]
tot = sum(hs) + gap * (len(lines) - 1)
y = int(H * y_pct) - tot // 2
for i, line in enumerate(lines):
x = (W - ws[i]) // 2
# Shadow layers
for dx, dy, alpha in [(3, 3, 180), (2, 2, 140), (1, 1, 100), (4, 4, 120)]:
draw.text((x + dx, y + dy), line, font=font, fill=(0, 0, 0, alpha))
draw.text((x, y), line, font=font, fill=(255, 255, 255, 255))
y += hs[i] + gap
img.save(str(out_path), 'PNG')
def make_cta_card(cfg, font_path, out_path):
cta = cfg['cta']
bg = tuple(cta.get('bg_color', [8, 8, 14]))
logo_path = cta.get('logo_path')
img = Image.new('RGB', (W, H), bg)
# Glow effect
glow = Image.new('RGBA', (W, H), (0, 0, 0, 0))
gd = ImageDraw.Draw(glow)
for r in range(260, 0, -20):
alpha = int(12 * (1 - r / 260))
gd.ellipse([W // 2 - r * 2, 480 - r, W // 2 + r * 2, 480 + r], fill=(60, 80, 160, alpha))
img = Image.alpha_composite(img.convert('RGBA'), glow).convert('RGB')
draw = ImageDraw.Draw(img)
# Logo (optional)
logo_bottom = 560
if logo_path and Path(logo_path).exists():
logo_raw = Image.open(logo_path).convert('RGBA')
scale = 400 / logo_raw.width
logo = logo_raw.resize((400, int(logo_raw.height * scale)), Image.LANCZOS)
img.paste(logo, ((W - logo.width) // 2, 560), logo)
logo_bottom = 560 + logo.height
# Divider
line_y = logo_bottom + 50
draw.line([(W // 2 - 180, line_y), (W // 2 + 180, line_y)], fill=(80, 80, 100), width=2)
# Price
font_price = ImageFont.truetype(font_path, 110)
price_text = cta['price']
pb = draw.textbbox((0, 0), price_text, font=font_price)
px = (W - (pb[2] - pb[0])) // 2
py = line_y + 60
draw.text((px + 3, py + 3), price_text, font=font_price, fill=(0, 0, 0, 180))
draw.text((px, py), price_text, font=font_price, fill=(255, 255, 255))
# CTA line 1
font_l1 = ImageFont.truetype(font_path, 58)
l1 = cta.get('line1', 'Link in bio 🛒')
bb = draw.textbbox((0, 0), l1, font=font_l1)
bx = (W - (bb[2] - bb[0])) // 2
by = py + (pb[3] - pb[1]) + 40
draw.text((bx + 2, by + 2), l1, font=font_l1, fill=(0, 0, 0, 150))
draw.text((bx, by), l1, font=font_l1, fill=(220, 220, 230))
# CTA line 2 (URL)
font_l2 = ImageFont.truetype(font_path, 46)
l2 = cta.get('line2', '')
if l2:
ub = draw.textbbox((0, 0), l2, font=font_l2)
ux = (W - (ub[2] - ub[0])) // 2
uy = by + (bb[3] - bb[1]) + 28
draw.text((ux, uy), l2, font=font_l2, fill=(140, 140, 180))
img.save(str(out_path))
def main():
parser = argparse.ArgumentParser(description='Build a supplier video product ad')
parser.add_argument('config', help='Product config JSON path')
parser.add_argument('--output-dir', default='output', help='Output directory (default: ./output)')
args = parser.parse_args()
cfg = json.loads(Path(args.config).read_text())
font_path = cfg.get('font_path', DEFAULT_FONT)
out_dir = Path(args.output_dir)
out_dir.mkdir(parents=True, exist_ok=True)
tmp = out_dir / f'_tmp_{cfg["output_name"].replace(".mp4","")}'
tmp.mkdir(parents=True, exist_ok=True)
out_final = out_dir / cfg['output_name']
print(f'\n🎬 Building: {cfg["product_name"]}')
print(f' Output: {out_final}')
# ── 1. Cut clean segments ──
seg_files = []
for i, zone in enumerate(cfg['clean_zones']):
src = Path(cfg['source_videos'][zone['source']])
pts = '1.25*PTS' if zone.get('slowmo') else 'PTS'
seg = tmp / f's{i}.mp4'
vf = (
f'scale={W}:{H}:force_original_aspect_ratio=decrease,'
f'pad={W}:{H}:(ow-iw)/2:(oh-ih)/2,setsar=1,fps=30,'
f'setpts={pts}'
)
run([
'ffmpeg', '-y', '-i', str(src),
'-ss', str(zone['ss']), '-t', str(zone['dur']),
'-vf', vf, '-an',
'-c:v', 'libx264', '-preset', 'fast', '-crf', '17',
str(seg)
], f's{i} ({zone["label"]})')
seg_files.append(seg)
# ── 2. Concat segments ──
list_txt = tmp / 'list.txt'
list_txt.write_text('\n'.join(f"file '{f}'" for f in seg_files))
raw = tmp / 'raw.mp4'
run(['ffmpeg', '-y', '-f', 'concat', '-safe', '0', '-i', str(list_txt), '-c', 'copy', str(raw)], 'concat')
total_dur = get_duration(raw)
print(f' Duration: {total_dur:.1f}s')
# ── 3. Text overlays ──
text_pngs = []
for i, ov in enumerate(cfg.get('text_overlays', [])):
png = tmp / f'txt_{i}.png'
make_text_png(ov['lines'], ov['fontsize'], ov['y_pct'], png, font_path)
text_pngs.append((str(png), ov['start'], ov['end']))
if text_pngs:
print(f' ✅ {len(text_pngs)} text overlays rendered')
# ── 4. CTA end card ──
cta_dur = cfg['cta']['duration']
cta_png = tmp / 'cta.png'
make_cta_card(cfg, font_path, cta_png)
cta_clip = tmp / 'cta.mp4'
run([
'ffmpeg', '-y', '-loop', '1', '-i', str(cta_png),
'-vf', 'fps=30,format=yuv420p', '-t', str(cta_dur),
'-an', '-c:v', 'libx264', '-preset', 'fast', '-crf', '18',
str(cta_clip)
], f'CTA clip ({cta_dur}s)')
# ── 5. Concat video + CTA ──
full_list = tmp / 'full_list.txt'
full_list.write_text(f"file '{raw}'\nfile '{cta_clip}'")
full_raw = tmp / 'full_raw.mp4'
run(['ffmpeg', '-y', '-f', 'concat', '-safe', '0', '-i', str(full_list), '-c', 'copy', str(full_raw)], 'video+CTA')
full_dur = get_duration(full_raw)
# ── 6. Overlay text ──
if text_pngs:
inputs = ['-i', str(full_raw)]
for png, _, _ in text_pngs:
inputs += ['-i', png]
filter_parts = []
cur = '[0:v]'
for i, (png, ts, te) in enumerate(text_pngs):
vi = f'[{i+1}:v]'
out_label = f'[ot{i}]' if i < len(text_pngs) - 1 else '[vfinal]'
filter_parts.append(f"{cur}{vi}overlay=0:0:enable='between(t,{ts},{te})'{out_label}")
cur = f'[ot{i}]'
text_overlaid = tmp / 'text_overlaid.mp4'
run([
'ffmpeg', '-y', *inputs,
'-filter_complex', '; '.join(filter_parts),
'-map', '[vfinal]', '-an',
'-c:v', 'libx264', '-preset', 'fast', '-crf', '17',
str(text_overlaid)
], 'text overlay')
else:
text_overlaid = full_raw
# ── 7. Mux audio ──
audio_path = Path(cfg['audio'])
audio_offset = cfg.get('audio_start_offset', 2)
vol = cfg.get('audio_volume', 0.88)
fade_out_start = max(0, full_dur - 1.2)
run([
'ffmpeg', '-y',
'-i', str(text_overlaid),
'-ss', str(audio_offset), '-i', str(audio_path),
'-filter_complex',
f'[1:a]atrim=0:{full_dur},afade=t=in:st=0:d=0.5,'
f'afade=t=out:st={fade_out_start}:d=1.2,'
f'asetpts=PTS-STARTPTS,volume={vol}[music]',
'-map', '0:v', '-map', '[music]',
'-c:v', 'copy', '-c:a', 'aac', '-b:a', '128k',
'-t', str(full_dur),
str(out_final)
], 'final mux')
# Cleanup
shutil.rmtree(tmp)
mb = out_final.stat().st_size // 1024 // 1024
print(f'\n🎬 Done: {out_final.name} — {mb}MB, {full_dur:.1f}s')
print(f' Ready to post on TikTok / Instagram Reels')
if __name__ == '__main__':
main()
Fetch Amazon Ads Sponsored Products campaign reports asynchronously by requesting and polling separately to avoid API timeouts, with no npm dependencies.
# skill-amazon-ads-reporter
## Description
Fetch Amazon Ads Sponsored Products campaign performance reports using a decoupled async pattern. Avoids timeout issues with the v3 Reporting API (2–10 min generation time) by splitting request and poll into separate steps. Also includes keyword-level winner/dead analysis and a quick bid inspector.
## Why two steps?
Amazon's Reporting API v3 is async — you request a report, get a `reportId`, and poll until it's ready. Doing this inline in a cron causes timeouts. The correct pattern:
```
request → save reportId → (wait 1-2 min) → poll + download
```
## Usage
### Campaign-level report (step-by-step, recommended for crons)
```bash
# Step 1: Request report — exits immediately with reportId
node scripts/request-report.js --days 7
# Step 2: Poll + download (run 1-2 min later, or from a separate cron)
node scripts/poll-report.js
```
### Campaign-level report (all-in-one, for manual runs)
```bash
node scripts/get-report.js --days 7
```
### Keyword-level winner/dead analysis (14-day async report)
```bash
node scripts/keyword-report.js
```
Output: table of all ENABLED keywords with clicks > 0 OR impressions ≥ 50 (winners), plus count of dead keywords (0 clicks, <50 imp).
### Quick bid inspector (live, across campaigns)
```bash
node scripts/get-bids.js
```
Output: all ENABLED + PAUSED keywords per campaign with current bids. Reads live data (no report needed).
## Arguments
| Arg | Default | Description |
|-----|---------|-------------|
| `--days N` | `7` | Number of days to include in report (campaign and keyword reports) |
## Configuration
Reads credentials from `AMAZON_ADS_PATH` env var, defaulting to `~/amazon-ads-api.json`.
### `amazon-ads-api.json` format
```json
{
"refreshToken": "...",
"lwaClientId": "...",
"lwaClientSecret": "...",
"profileId": "...",
"region": "EU"
}
```
Regions: `EU` (default, includes UAE), `NA` (North America), `FE` (Far East).
## Output
- `~/.openclaw/workspace/tmp/amazon-report-pending.json` — created by request-report.js
- `~/.openclaw/workspace/tmp/amazon-report-latest.json` — created by poll-report.js after success
- Console table: Campaign | Impressions | Clicks | CTR% | Spend | Sales | ACOS%
## Report columns (campaign-level)
`campaignName`, `campaignId`, `impressions`, `clicks`, `spend`, `purchases7d`, `sales7d`
Paused campaigns are automatically filtered out by cross-referencing `GET /sp/campaigns/list`.
## Report columns (keyword-level — keyword-report.js)
`keywordId`, `keywordText`, `matchType`, `impressions`, `clicks`, `cost`, `purchases7d`, `sales7d`
## Dependencies
Node.js built-ins only (`https`, `zlib`, `fs`, `path`). No npm install required.
## Notes
- Access tokens expire — refresh via Amazon Login with Advertising if needed
- The `GZIP_JSON` format is gunzipped automatically by poll-report.js
- Reports are only available for the previous day and earlier (endDate = yesterday)
- `get-bids.js` uses the live v3 keyword list endpoint — no async report needed, instant response
- keyword-report.js uses the same async pattern as campaign reports (30s poll intervals, up to 10 min)
FILE:scripts/get-bids.js
#!/usr/bin/env node
'use strict';
/**
* Amazon Ads — Quick Bid Inspector
* Lists all ENABLED + PAUSED keywords with current bids for specified campaigns.
* Uses the live v3 keyword list endpoint (no async report needed).
*
* Usage: node get-bids.js
* Config: AMAZON_ADS_PATH env var (default: ~/amazon-ads-api.json)
*/
const https = require('https');
const fs = require('fs');
const ADS_PATH = process.env.AMAZON_ADS_PATH || require('os').homedir() + '/amazon-ads-api.json';
const creds = JSON.parse(fs.readFileSync(ADS_PATH, 'utf8'));
// Set your campaign IDs here
const CAMPAIGN_IDS = process.env.CAMPAIGN_IDS
? process.env.CAMPAIGN_IDS.split(',').map(Number)
: [];
if (CAMPAIGN_IDS.length === 0) {
console.error('Set CAMPAIGN_IDS env var (comma-separated) or edit the array in this script.');
process.exit(1);
}
function apiReq(hostname, path, method, headers, body) {
const payload = body ? JSON.stringify(body) : null;
if (payload) headers['Content-Length'] = Buffer.byteLength(payload);
return new Promise((resolve, reject) => {
const req = https.request({ hostname, path, method, headers }, res => {
let d = ''; res.on('data', c => d += c);
res.on('end', () => {
try { resolve({ status: res.statusCode, body: JSON.parse(d) }); }
catch { resolve({ status: res.statusCode, body: d }); }
});
});
req.on('error', reject);
if (payload) req.write(payload);
req.end();
});
}
async function getToken() {
const params = `grant_type=refresh_token&refresh_token=encodeURIComponent(creds.refreshToken)&client_id=encodeURIComponent(creds.lwaClientId)&client_secret=encodeURIComponent(creds.lwaClientSecret)`;
return new Promise((resolve, reject) => {
const req = https.request({
hostname: 'api.amazon.com', path: '/auth/o2/token', method: 'POST',
headers: { 'Content-Type': 'application/x-www-form-urlencoded', 'Content-Length': Buffer.byteLength(params) }
}, res => {
let d = ''; res.on('data', c => d += c);
res.on('end', () => {
const t = JSON.parse(d);
if (!t.access_token) reject(new Error('LWA failed: ' + d));
else resolve(t.access_token);
});
});
req.on('error', reject); req.write(params); req.end();
});
}
(async () => {
const token = await getToken();
const endpoint = 'advertising-api-eu.amazon.com';
const headers = {
Authorization: `Bearer token`,
'Amazon-Advertising-API-ClientId': creds.lwaClientId,
'Amazon-Advertising-API-Scope': String(creds.profileId),
'Content-Type': 'application/json',
};
for (const campaignId of CAMPAIGN_IDS) {
const r = await apiReq(endpoint, '/sp/keywords/list', 'POST', { ...headers }, {
campaignIdFilter: { include: [String(campaignId)] },
stateFilter: { include: ['ENABLED', 'PAUSED'] },
maxResults: 50
});
console.log(`\nCampaign campaignId keywords (HTTP r.status):`);
if (r.status === 200 && r.body.keywords) {
for (const kw of r.body.keywords) {
console.log(` keywordId=kw.keywordId text="kw.keywordText" matchType=kw.matchType bid=kw.bid state=kw.state`);
}
} else {
console.log(' Response:', JSON.stringify(r.body).substring(0, 400));
}
}
})().catch(e => { console.error(e.message); process.exit(1); });
FILE:scripts/keyword-report.js
/**
* Fetch keyword-level performance report across all active campaigns
* Filters: only ENABLED keywords with meaningful data
* "Winning" = clicks > 0 OR impressions > 50
*/
const zlib = require('zlib');
const fs = require('fs');
const CREDS_PATH = process.env.AMAZON_ADS_PATH || `process.env.HOME/amazon-ads-api.json`;
const API_BASE = 'https://advertising-api-eu.amazon.com';
function getCreds() { return JSON.parse(fs.readFileSync(CREDS_PATH, 'utf8')); }
async function getAccessToken() {
const creds = getCreds();
const res = await fetch('https://api.amazon.com/auth/o2/token', {
method: 'POST',
headers: { 'Content-Type': 'application/x-www-form-urlencoded' },
body: new URLSearchParams({
grant_type: 'refresh_token',
refresh_token: creds.refreshToken,
client_id: creds.lwaClientId,
client_secret: creds.lwaClientSecret,
}),
});
const t = await res.json();
if (!t.access_token) throw new Error('Ads auth failed: ' + JSON.stringify(t));
return t.access_token;
}
async function apiCall(path, method = 'GET', body = null, contentType = 'application/json') {
const creds = getCreds();
const token = await getAccessToken();
const opts = {
method,
headers: {
'Authorization': `Bearer token`,
'Amazon-Advertising-API-ClientId': creds.lwaClientId,
'Amazon-Advertising-API-Scope': creds.profileId,
'Content-Type': contentType,
'Accept': contentType,
}
};
if (body) opts.body = JSON.stringify(body);
const res = await fetch(API_BASE + path, opts);
const text = await res.text();
try { return JSON.parse(text); } catch(e) { return text; }
}
async function downloadGzip(url) {
const res = await fetch(url);
const buf = Buffer.from(await res.arrayBuffer());
return new Promise((resolve, reject) => {
zlib.gunzip(buf, (err, decoded) => {
if (err) reject(err);
else resolve(JSON.parse(decoded.toString()));
});
});
}
async function sleep(ms) { return new Promise(r => setTimeout(r, ms)); }
async function main() {
// Date range: last 14 days for more signal
const end = new Date(); end.setDate(end.getDate() - 1);
const start = new Date(); start.setDate(start.getDate() - 14);
const fmt = d => d.toISOString().split('T')[0];
console.error(`Requesting keyword report fmt(start)–fmt(end)...`);
const created = await apiCall('/reporting/reports', 'POST', {
name: `kw-perf-fmt(start)`,
startDate: fmt(start),
endDate: fmt(end),
configuration: {
adProduct: 'SPONSORED_PRODUCTS',
groupBy: ['adGroup'],
columns: ['keywordId', 'keywordText', 'matchType', 'impressions', 'clicks', 'cost', 'purchases7d', 'sales7d', 'startDate', 'endDate'],
reportTypeId: 'spKeywords',
timeUnit: 'SUMMARY',
format: 'GZIP_JSON'
}
}, 'application/vnd.createasyncreportrequest.v3+json');
if (!created.reportId) {
console.error('Failed:', JSON.stringify(created));
process.exit(1);
}
const reportId = created.reportId;
console.error(`Report ID: reportId — polling...`);
// Poll up to 20 times (10 min)
let status;
for (let i = 0; i < 20; i++) {
await sleep(30000);
status = await apiCall(`/reporting/reports/reportId`, 'GET', null, 'application/json');
console.error(` [i+1/20] Status: status.status`);
if (status.status === 'COMPLETED') break;
if (status.status === 'FAILED') { console.error('Report failed:', JSON.stringify(status)); process.exit(1); }
}
if (status.status !== 'COMPLETED') {
console.error('Timed out waiting for report');
process.exit(1);
}
const rows = await downloadGzip(status.url);
console.error(`Downloaded rows.length keyword rows`);
// Filter: only ENABLED keywords
const enabled = rows.filter(r => r.state === 'ENABLED');
// Winning = clicks > 0 OR impressions >= 50
const winners = enabled.filter(r => (r.clicks || 0) > 0 || (r.impressions || 0) >= 50);
// Sort by clicks desc, then impressions desc
winners.sort((a, b) => {
const clickDiff = (b.clicks || 0) - (a.clicks || 0);
if (clickDiff !== 0) return clickDiff;
return (b.impressions || 0) - (a.impressions || 0);
});
console.log('\n🏆 WINNING KEYWORDS (14-day, ENABLED only, clicks>0 OR imp≥50)\n');
console.log(`'Campaign'.padEnd(25) 'Ad Group'.padEnd(18) 'Keyword'.padEnd(35) 'Match'.padEnd(7) 'Bid'.padStart(6) 'Imp'.padStart(6) 'Clk'.padStart(4) 'CTR'.padStart(6) 'Spend'.padStart(8) 'Sales'.padStart(8) 'ACoS'.padStart(6)`);
console.log('-'.repeat(140));
for (const r of winners) {
const imp = r.impressions || 0;
const clk = r.clicks || 0;
const spend = r.spend || 0;
const sales = r.sales7d || 0;
const ctr = imp > 0 ? ((clk / imp) * 100).toFixed(2) + '%' : '0.00%';
const acos = sales > 0 ? ((spend / sales) * 100).toFixed(0) + '%' : '—';
const camp = (r.campaignName || '').slice(0, 24).padEnd(25);
const ag = (r.adGroupName || '').slice(0, 17).padEnd(18);
const kw = (r.keywordText || '').slice(0, 34).padEnd(35);
const match = (r.matchType || '').slice(0, 6).padEnd(7);
const bid = (r.keywordBid || 0).toFixed(2).padStart(6);
console.log(`camp ag kw match bid String(imp).padStart(6) String(clk).padStart(4) ctr.padStart(6) spend.toFixed(2).padStart(8) sales.toFixed(2).padStart(8) acos.padStart(6)`);
}
console.log(`\nTotal winning keywords: winners.length / enabled.length enabled`);
// Also show zero-performers for awareness
const zeros = enabled.filter(r => (r.clicks || 0) === 0 && (r.impressions || 0) < 50);
console.log(`Dead keywords (0 clicks, <50 imp): zeros.length`);
}
main().catch(e => { console.error(e); process.exit(1); });
End-to-end dropship product lifecycle pipeline. CJ Dropshipping sourcing → margin check → Flux Kontext AI hero image → WooCommerce publish → CJ supplier mapp...
---
name: skill-dropship-product-pipeline
version: 1.0.0
description: >
End-to-end dropship product lifecycle pipeline. CJ Dropshipping sourcing → margin check
→ Flux Kontext AI hero image → WooCommerce publish → CJ supplier mapping for auto-fulfillment.
requires:
env:
- FAL_KEY
- OPENAI_API_KEY
- CJ_ACCESS_TOKEN
- WOO_URL
- WOO_KEY
- WOO_SECRET
- WP_URL
- WP_USER
- WP_APP_PASS
bins:
- node
---
# skill-dropship-product-pipeline v1.0.0
Full end-to-end dropship product lifecycle — from CJ Dropshipping search to a live WooCommerce listing with an AI-generated hero image.
## Pipeline Steps
1. **CJ Sourcing** — Keyword search or direct product ID. Margin check (min 40%). Variant extraction.
2. **Hero Image** — Flux Kontext Dev (`fal-ai/flux-kontext/dev`) using the real CJ product photo as reference. Lifestyle background, product in active use, warm mood, 1:1 square.
3. **WooCommerce Publish** — Upload hero + gallery images, create product, set price/SKU.
4. **CJ Mapping** — Add product to your `cj-supplier-selection.json` for auto-fulfillment via `skill-dropshipping-fulfillment`.
> **Pipeline ends at WooCommerce publish.** Video creation is a separate step — use [skill-tiktok-video-pipeline](https://clawhub.com/skills/skill-tiktok-video-pipeline).
## Usage
```bash
# Source by keyword — finds best margin product
node scripts/pipeline.js --keyword "ring light" --sell-price 89
# Source by CJ product ID — skip sourcing step
node scripts/pipeline.js --cj-pid 2603020206551636100 --sell-price 69
# Dry run — skip WooCommerce publish (test mode)
node scripts/pipeline.js --keyword "desk lamp" --sell-price 99 --dry-run
```
## Options
| Flag | Required | Description |
|---|---|---|
| `--keyword` | ✅ (or `--cj-pid`) | CJ search keyword |
| `--cj-pid` | ✅ (or `--keyword`) | Known CJ product ID, skips search |
| `--sell-price` | ✅ | Selling price in your local currency |
| `--dry-run` | ❌ | Skip WooCommerce publish |
| `--lang` | ❌ | Language: `en`, `ar`, `both` (default: `en`) |
| `--min-margin` | ❌ | Minimum margin % (default: 40) |
## Credentials Setup
Create credential files or use environment variables:
```bash
# CJ Dropshipping
export CJ_ACCESS_TOKEN="your-cj-token"
# WooCommerce
export WOO_URL="https://yourstore.com"
export WOO_KEY="ck_..."
export WOO_SECRET="cs_..."
# WordPress media upload
export WP_URL="https://yourstore.com"
export WP_USER="your-wp-username"
export WP_APP_PASS="your-app-password"
# AI services
export FAL_KEY="your-fal-key" # Flux Kontext hero image
export OPENAI_API_KEY="your-key" # GPT-4o fallback for hero
```
## Hero Image Standard
- **Model:** Flux Kontext Dev (`fal-ai/flux-kontext/dev`)
- **Method:** Real CJ product photo as `image_url` input — product appearance locked from frame 1
- **Style:** Lifestyle background, product in active use, shallow DOF, warm mood, 1:1 square
- **Fallback:** GPT-4o `images/edits` if Flux fails
## Output Files
- `hero-{slug}.jpg` — Product hero (Flux Kontext or GPT-4o fallback)
- `pipeline-result-{slug}.json` — WooCommerce product ID, CJ mapping, cost/margin breakdown
## Economics
- Min margin default: 40%
- Hero image cost: ~$0.05–0.10 per product (Flux Kontext)
- Total pipeline cost per product: under $0.20
## Recommended Stack
For the full dropship automation stack:
1. **This skill** — source + list products
2. [skill-tiktok-video-pipeline](https://clawhub.com/skills/skill-tiktok-video-pipeline) — create video ads
3. [skill-dropshipping-fulfillment](https://clawhub.com/skills/skill-dropshipping-fulfillment) — auto-fulfill orders via CJ
4. [skill-woocommerce-stock-monitor](https://clawhub.com/skills/skill-woocommerce-stock-monitor) — OOS alerts
Generate polished 1080×1920 TikTok/Reels/Shorts video ads from product clips and images. Three viral styles: Clean, Meme, UGC. Python + ffmpeg, no cloud requ...
---
name: skill-ad-creative-engine
version: 1.0.0
description: >
Generate polished 1080×1920 TikTok/Reels/Shorts video ads from product clips and images.
Three viral styles: Clean, Meme, UGC. Python + ffmpeg, no cloud required.
requires:
bins:
- python3
- ffmpeg
pip:
- moviepy
- pillow
- librosa (optional, for beat_sync)
---
# skill-ad-creative-engine v1.0.0
Render polished short-form video ads (TikTok / Reels / Shorts) from product clips and images.
Three viral styles built-in: Clean, Meme, UGC. Runs locally — no cloud API required.
## Quick Start
```bash
cd skill-ad-creative-engine
pip3 install -r requirements.txt
python3 scripts/render.py --config examples/config_example.json
```
## Styles
| Style | Font | Overlay | Best for |
|---|---|---|---|
| `clean` | Montserrat ExtraBold | White text, drop shadow, upper-center | Product launch, brand ads |
| `meme` | Anton (ALL CAPS) | White + 8px black stroke, top-center | Viral hooks, humor ads |
| `ugc` | — | TikTok username pill only (no hook text) | Authentic creator-style |
## Config Format
```json
{
"style": "clean",
"hook_text": "You need this →",
"username": "@yourstore",
"scenes": [
{ "path": "clips/product_clip.mp4", "duration": 3.0 },
{ "path": "images/product_hero.jpg", "duration": 2.5 }
],
"transitions": ["cut", "dissolve"],
"music": "luts/background_track.mp3",
"beat_sync": false,
"output": "output/ad_final.mp4"
}
```
See `examples/config_example.json` for full reference.
## Dependencies
```bash
pip3 install -r requirements.txt
```
System requirements:
- `ffmpeg` (install via `brew install ffmpeg` or `apt install ffmpeg`)
- Fonts bundled: `fonts/Anton-Regular.ttf` + `fonts/Montserrat-ExtraBold.ttf`
## Output Spec
- Resolution: 1080×1920 (9:16 vertical)
- FPS: 30
- Codec: H.264 + AAC audio
- Color grade: Warm tone applied automatically
## Beat Sync (optional)
Set `"beat_sync": true` in config + provide a music track. Requires `librosa`:
```bash
pip3 install librosa
```
Cuts will snap to detected beat timestamps for a professional music-video feel.
End-to-end AI UGC video pipeline. Product info → GPT-4o-mini script → ElevenLabs voiceover → Aurora talking head (fal-ai/creatify/aurora) → Kling 2.6 Pro pro...
---
name: skill-ugc-pipeline
version: 1.2.0
description: >
End-to-end AI UGC video pipeline. Product info → GPT-4o-mini script → ElevenLabs voiceover
→ Aurora talking head (fal-ai/creatify/aurora) → Kling 2.6 Pro product B-roll →
Whisper-synced captions → UGC post-processing filter (grain + handheld shake on avatar,
clean product shot) → final MP4. Full pipeline ~$1.75/video.
requires:
env:
- FAL_KEY
- ELEVENLABS_API_KEY
- OPENAI_API_KEY
bins:
- node
- ffmpeg
- uv
---
# skill-ugc-pipeline v1.2.0
Build your own MakeUGC pipeline. Direct API access — no $300/mo enterprise tier.
**Default model: Aurora** (`fal-ai/creatify/aurora`) — locked after A/B test.
Aurora produces significantly more realistic lip sync and narration vs alternatives.
## What's new in v1.2.0
- **B-roll splicing** — Kling 2.6 Pro image-to-video generates a cinematic product shot, spliced into the avatar video at a configurable timecode
- **UGC filter** — grain + handheld shake applied to avatar segments ONLY; product B-roll stays clean and cinematic
- Continuous audio across splice points (no audio gap)
## Architecture
```
product info → [GPT-4o-mini] → script
→ [ElevenLabs] → audio.mp3
avatar image + audio → [fal.ai Aurora] → avatar.mp4
product image → [Kling 2.6 Pro] → broll.mp4
avatar + broll + ugc-filter → [ffmpeg] → final.mp4
audio.mp3 → [OpenAI Whisper] → captions overlay
```
## Full Pipeline (6 steps)
```
1. Script GPT-4o-mini → spoken script
2. Voice ElevenLabs → audio.mp3
3. Avatar fal-ai/creatify/aurora → talking head MP4
4. B-roll Kling 2.6 Pro image-to-video → product shot
5. Splice ffmpeg: avatar(hook) + broll + avatar(resume) + continuous audio
6. Captions Whisper word-level → overlay.py → final MP4
7. UGC filter grain + handheld shake on avatar ONLY (product shot stays clean)
```
## Quick Start — Full Pipeline
```bash
cd skill-ugc-pipeline
npm install
# Step 1-3: Script + voice + avatar
node scripts/generate.js \
--product "Rain Cloud Humidifier" \
--product-desc "USB cool mist humidifier. 300ml tank, LED glow, silent mode." \
--avatar avatars/my_avatar.png \
--output output/ad_raincloud.mp4
# Step 4-5: Add B-roll + UGC filter
node scripts/broll.js \
--avatar-video output/ad_raincloud_aurora.mp4 \
--audio output/ad_raincloud_audio.mp3 \
--product-image https://example.com/product.jpg \
--product-name "Rain Cloud Humidifier" \
--splice-at 4.5 \
--broll-duration 5 \
--ugc-filter \
--output output/final.mp4
# Step 6: Whisper captions
node scripts/transcribe_captions.js \
--audio output/ad_raincloud_audio.mp3 \
--video output/final.mp4 \
--output output/final_captioned.mp4
```
## Scripts
| Script | Description |
|--------|-------------|
| `generate.js` | Main pipeline: script → voice → Aurora talking head |
| `broll.js` | B-roll splice + optional UGC filter (grain + shake on avatar) |
| `transcribe_captions.js` | Whisper word-level caption overlay |
| `aurora_only.js` | Generate Aurora talking head only (skip script/voice) |
| `batch.js` | Run pipeline for multiple products |
| `product_in_hand.js` | Generate product-in-hand composite image |
## broll.js Options
| Flag | Default | Description |
|------|---------|-------------|
| `--avatar-video` | required | Path to Aurora talking head MP4 |
| `--audio` | required | Original voiceover MP3 (plays continuously) |
| `--product-image` | required | URL or local path to product image |
| `--product-name` | required | Product name (used in Kling prompt) |
| `--splice-at` | `4.5` | Seconds into avatar video where B-roll inserts |
| `--broll-duration` | `5` | B-roll duration: 5 or 10 seconds |
| `--ugc-filter` | off | Add grain + handheld shake to avatar (product stays clean) |
| `--output` | required | Output MP4 path |
## Cost Estimate
| Step | Model | Cost/video |
|------|-------|-----------|
| Script | GPT-4o-mini | ~$0.01 |
| Voice | ElevenLabs | ~$0.05 |
| Avatar | Aurora (fal.ai) | ~$1.00 |
| B-roll | Kling 2.6 Pro (fal.ai) | ~$0.40 |
| Captions | Whisper API | ~$0.01 |
| **Total** | | **~$1.47–1.75** |
## Requirements
- `FAL_KEY` — fal.ai API key (Aurora + Kling)
- `ELEVENLABS_API_KEY` — ElevenLabs API key
- `OPENAI_API_KEY` — OpenAI API key (GPT-4o-mini + Whisper)
- `ffmpeg` — for video splicing and UGC filter
- `uv` — for Python caption overlay (via skill-tiktok-ads-video)
## Avatar Requirements
- Portrait photo, face visible (no heavy makeup or accessories that obscure mouth)
- Resolution: 512×512 minimum, 1024×1024 recommended
- File format: PNG or JPG
Automate B2B cold email outreach by sourcing leads from Apollo, verifying with Hunter.io, and uploading to Instantly for a 3-email drip campaign in one command.
# skill-cold-email-outreach
Apollo → Hunter → Instantly automated cold email pipeline.
Scrape leads, verify emails, upload to campaign — fully automated B2B outreach in one command.
## What it does
1. **Source leads** — Apollo CSV export (free tier) or Apollo API scrape (paid)
2. **Verify emails** — Hunter.io filters invalid/risky addresses
3. **Upload & personalize** — pushes verified leads to Instantly v2 with dynamic first lines
4. **Sequence** — 3-email drip: D0 (ROI pitch) → D3 (pain question) → D8 (soft close)
## Requirements
- Node.js 18+
- Apollo.io account (free = CSV export, paid = API)
- Hunter.io API key (free = 25/mo, paid for scale)
- Instantly.ai account + campaign pre-created
- Instantly v2 Bearer token
## Setup
```bash
# 1. Edit config.js — add your keys
cp scripts/config.example.js scripts/config.js
# Fill in: instantly.apiKey, hunter.apiKey, apollo.apiKey, target ICP
# 2. Run
node scripts/import-csv.js your-apollo-export.csv
```
## Scripts
| File | Purpose |
|------|---------|
| `import-csv.js` | Apollo CSV → Hunter verify → Instantly upload |
| `pipeline.js` | Apollo API scrape → Hunter verify → Instantly upload |
| `config.example.js` | Config template (copy to config.js) |
| `emails.js` | 3-email sequence content (customize subject/body) |
## Target Config
```js
target: {
industries: ["ecommerce", "retail"],
countries: ["AE", "SA", "EG"], // ISO-2 codes
titles: ["Founder", "CEO", "Owner"],
perPage: 25,
maxLeads: 200,
}
```
## Output
```
📋 Parsed 18 rows from CSV
🔍 Verifying emails with Hunter...
✓ [email protected] — John @ Acme Store
✗ [email protected] — invalid
✅ Verified: 17 / 18 leads
📤 Uploading to Instantly...
╔══════════════════════════════════╗
║ IMPORT COMPLETE ⚡ ║
║ Uploaded: 17 ║
║ Failed: 0 ║
╚══════════════════════════════════╝
```
## After running
1. Open app.instantly.ai → your campaign
2. Connect sending inboxes (warm them first)
3. Set daily limit: 40/inbox
4. Hit Launch 🚀
## Tips
- Warm inboxes 2–3 weeks before launching cold campaigns
- Apollo free: 50 contacts/month via manual CSV export
- Keep sequences short (3 emails max for cold)
- Personalize first line by industry/location for higher reply rates
FILE:README.md
# skill-cold-email-outreach
Apollo → Hunter → Instantly automated cold email pipeline for B2B outreach
FILE:scripts/config.example.js
// Copy this to config.js and fill in your keys
module.exports = {
instantly: {
apiKey: "YOUR_INSTANTLY_V2_BEARER_TOKEN",
},
hunter: {
apiKey: "YOUR_HUNTER_API_KEY",
},
apollo: {
apiKey: "YOUR_APOLLO_API_KEY", // Only needed for pipeline.js (API scrape mode)
},
target: {
industries: ["ecommerce", "retail", "online retail"],
countries: ["AE", "SA", "EG", "KW", "QA", "BH", "OM"], // ISO-2 codes
titles: [
"Founder", "Co-Founder", "CEO", "Owner", "Director",
"Head of E-commerce", "E-commerce Manager", "Operations Manager"
],
perPage: 25,
maxLeads: 200,
},
sequence: {
name: "Your Campaign Name",
campaignId: "YOUR_INSTANTLY_CAMPAIGN_ID",
dailyLimit: 40, // per inbox
delayBetweenEmails: { min: 5, max: 15 }, // minutes
}
};
FILE:scripts/emails.js
// Cold email sequence — customize for your ICP
// Lead variables available: {{firstName}}, {{companyName}}
module.exports = {
email1: {
subject: "your store is leaving hours on the table every week",
body: (lead) => `Hi {{firstName}},
I looked at {{companyName}} — you're running a real store, which means someone on your team is probably spending 3-5 hours a day on things that should be automatic: order routing, supplier submissions, tracking updates, stock alerts.
We fix that. [Your Company] builds e-commerce automation systems that take those exact workflows off your plate in under 30 days.
Worth a 30-min call to see if your store has the same wins available?
→ [YOUR BOOKING LINK]
— [Your Name]
[Your Company]
`,
},
email2: {
subject: "Re: your store is leaving hours on the table every week",
delayDays: 3,
body: (lead) => `Hi {{firstName}},
Just following up — wanted to make sure this didn't get buried.
Quick question: what's the most manual, repetitive task your team deals with right now? Order processing? Inventory? Supplier coordination?
Happy to share how we've handled it in a 30-min call.
→ [YOUR BOOKING LINK]
— [Your Name]
`,
},
email3: {
subject: "last one from me",
delayDays: 5,
body: (lead) => `Hi {{firstName}},
Won't keep following up after this — I know your inbox is busy.
But if {{companyName}} ever hits a point where the manual ops are slowing growth, we're the team to call.
The free audit link stays open: [YOUR BOOKING LINK]
Good luck.
— [Your Name]
`,
},
};
FILE:scripts/import-csv.js
#!/usr/bin/env node
/**
* Import Apollo CSV export → Hunter verify → Instantly campaign
* Usage: node import-csv.js leads.csv
*/
const https = require("https");
const fs = require("fs");
const path = require("path");
const config = require("./config");
const CAMPAIGN_ID = config.sequence.campaignId; // already created
function sleep(ms) { return new Promise(r => setTimeout(r, ms)); }
function request(options, body) {
return new Promise((resolve, reject) => {
const req = https.request(options, (res) => {
let data = "";
res.on("data", c => data += c);
res.on("end", () => {
try { resolve({ status: res.statusCode, body: JSON.parse(data) }); }
catch { resolve({ status: res.statusCode, body: data }); }
});
});
req.on("error", reject);
if (body) req.write(typeof body === "string" ? body : JSON.stringify(body));
req.end();
});
}
// Parse Apollo CSV (handles quoted fields)
function parseCSV(content) {
const lines = content.trim().split("\n");
const headers = lines[0].split(",").map(h => h.replace(/"/g, "").trim().toLowerCase());
return lines.slice(1).map(line => {
const values = [];
let current = "", inQuotes = false;
for (const char of line) {
if (char === '"') { inQuotes = !inQuotes; continue; }
if (char === "," && !inQuotes) { values.push(current.trim()); current = ""; continue; }
current += char;
}
values.push(current.trim());
const obj = {};
headers.forEach((h, i) => obj[h] = values[i] || "");
return obj;
}).filter(r => r["email"] || r["work email"] || r["email address"]);
}
function getEmail(row) {
return row["email"] || row["work email"] || row["email address"] || row["corporate email"] || "";
}
function getField(row, ...keys) {
for (const k of keys) if (row[k]) return row[k];
return "";
}
async function verifyEmail(email) {
if (!email || !email.includes("@")) return false;
const res = await request({
hostname: "api.hunter.io",
path: `/v2/email-verifier?email=encodeURIComponent(email)&api_key=config.hunter.apiKey`,
method: "GET",
});
if (res.status !== 200) return true;
const status = res.body?.data?.status;
return ["valid", "accept_all"].includes(status);
}
function personalizeFirstLine(row) {
const company = getField(row, "company", "company name", "organization name", "account name") || "your store";
const country = getField(row, "country", "location") || "UAE";
const lines = [
`Noticed company is scaling in country — curious if the ops side is keeping up with growth.`,
`company caught my eye — you're building something real in the GCC e-commerce space.`,
`Saw company in the country market — stores at your stage usually have processes ready to automate.`,
`Running company in the GCC — you're probably the one feeling the manual ops pain most.`,
];
return lines[Math.floor(Math.random() * lines.length)];
}
async function uploadLead(lead) {
const body = JSON.stringify({
email: lead.email,
first_name: lead.firstName,
last_name: lead.lastName,
company_name: lead.company,
personalization: lead.firstLine,
campaign_id: CAMPAIGN_ID,
});
return request(
{
hostname: "api.instantly.ai",
path: `/api/v2/leads`,
method: "POST",
headers: {
"Content-Type": "application/json",
"Authorization": `Bearer config.instantly.apiKey`,
"Content-Length": Buffer.byteLength(body),
},
},
body
);
}
async function run() {
const csvPath = process.argv[2];
if (!csvPath) {
console.error("Usage: node import-csv.js <path-to-apollo-export.csv>");
process.exit(1);
}
if (!fs.existsSync(csvPath)) {
console.error(`File not found: csvPath`);
process.exit(1);
}
console.log(`⚡ Importing path.basename(csvPath) into Instantly...\n`);
const content = fs.readFileSync(csvPath, "utf8");
const rows = parseCSV(content);
console.log(`📋 Parsed rows.length rows from CSV`);
if (rows.length > 0) {
console.log(` Columns detected: Object.keys(rows[0]).slice(0, 8).join(", ")...\n`);
}
const verifiedLeads = [];
console.log("🔍 Verifying emails with Hunter...");
for (const row of rows) {
const email = getEmail(row);
if (!email) continue;
const valid = await verifyEmail(email);
if (!valid) {
console.log(` ✗ email — invalid`);
continue;
}
const firstName = getField(row, "first name", "firstname", "first_name") || "there";
const lastName = getField(row, "last name", "lastname", "last_name") || "";
const company = getField(row, "company", "company name", "organization name", "account name") || "";
verifiedLeads.push({ email, firstName, lastName, company, firstLine: personalizeFirstLine(row) });
console.log(` ✓ email — firstName @ company`);
await sleep(250);
}
console.log(`\n✅ Verified: verifiedLeads.length / rows.length leads\n`);
if (!verifiedLeads.length) {
console.log("No valid leads to upload.");
return;
}
console.log(`📤 Uploading leads to Instantly campaign...`);
let uploaded = 0;
let failed = 0;
for (const lead of verifiedLeads) {
try {
const res = await uploadLead(lead);
if (res.status === 200 || res.status === 201) {
uploaded++;
console.log(` ✓ lead.email`);
} else {
failed++;
console.log(` ✗ lead.email — res.status JSON.stringify(res.body).slice(0, 80)`);
}
} catch (e) {
failed++;
console.log(` ✗ lead.email — e.message`);
}
await sleep(300);
}
console.log(`\n╔══════════════════════════════════╗`);
console.log(`║ IMPORT COMPLETE ⚡ ║`);
console.log(`║ Uploaded: String(uploaded).padEnd(23)║`);
console.log(`║ Failed: String(failed).padEnd(23)║`);
console.log(`║ Campaign: Zeerotoai Outreach ║`);
console.log(`║ ║`);
console.log(`║ NEXT: Connect inboxes in ║`);
console.log(`║ Instantly → hit Launch 🚀 ║`);
console.log(`╚══════════════════════════════════╝`);
}
run().catch(console.error);
Monitor WooCommerce products for out-of-stock changes and send Telegram alerts. Run daily via cron.
---
name: skill-woocommerce-stock-monitor
version: 1.0.0
description: Monitor WooCommerce products for out-of-stock changes and send Telegram alerts. Run daily via cron.
metadata:
openclaw:
requires: { bins: ["node"] }
---
# skill-woocommerce-stock-monitor v1.0.0
Monitor WooCommerce products for out-of-stock changes and send Telegram alerts. Tracks instock → outofstock transitions and alerts your team daily.
## Usage
```bash
node scripts/stock-monitor.js
```
## Configuration
Set via environment variables:
| Variable | Default | Description |
|---|---|---|
| `WOO_API_PATH` | `~/woo-api.json` | Path to WooCommerce API credentials JSON |
| `TELEGRAM_BOT_TOKEN` | — | Telegram bot token for alerts |
| `TELEGRAM_CHAT_ID` | — | Telegram chat/group ID to send alerts to |
### woo-api.json format
```json
{
"url": "https://your-store.com",
"consumer_key": "ck_...",
"consumer_secret": "cs_..."
}
```
## Cron setup
```bash
# Run daily at 07:00 UTC
0 7 * * * TELEGRAM_BOT_TOKEN=xxx TELEGRAM_CHAT_ID=yyy node /path/to/scripts/stock-monitor.js
```
## Behavior
- **First run:** Sends a baseline report of all currently OOS products
- **Subsequent runs:** Only alerts on new instock → outofstock transitions
- **State file:** Saved to `memory/stock-state.json` (tracks previous run)
## Alerts
- `📦 Stock Monitor — Baseline Report` — first run summary
- `⚠️ Stock Alert — Out of Stock` — when products go OOS
FILE:README.md
# skill-woocommerce-stock-monitor — WooCommerce Out-of-Stock Alerts
Monitor WooCommerce products for out-of-stock changes and send Telegram alerts.
## Use When
- Monitoring product stock levels daily
- Getting notified when products go out of stock
- Tracking restocks after a product was OOS
- Running automated inventory health checks
## Key Features
- Fetches all WooCommerce products via REST API
- Detects out-of-stock status changes
- Sends Telegram alert with product name and link
- Designed to run daily via OpenClaw cron
## Requirements
- WooCommerce REST API key (read access)
- Telegram bot token and chat ID
## Scheduling
Recommended: daily cron at 08:00 local time.
Load the SKILL.md for cron setup instructions.
## Version
1.0.0
FILE:_meta.json
{
"name": "skill-woocommerce-stock-monitor",
"version": "1.0.0",
"description": "Monitor WooCommerce products for out-of-stock changes and send Telegram alerts. Run daily via cron.",
"author": "Zero2Ai-hub",
"license": "MIT"
}
FILE:scripts/stock-monitor.js
#!/usr/bin/env node
/**
* Stock Monitor — checks WooCommerce products for out-of-stock status
* Sends Telegram alert if any product changes from instock → outofstock
*
* Usage: node scripts/stock-monitor.js
* Cron: daily at 07:00 UTC
*/
const fs = require('fs');
const path = require('path');
const https = require('https');
const http = require('http');
const WOO_API_PATH = process.env.WOO_API_PATH || path.join(require('os').homedir(), 'woo-api.json');
const STATE_FILE = path.join(__dirname, '../memory/stock-state.json');
const TELEGRAM_TOKEN = process.env.TELEGRAM_BOT_TOKEN;
const TELEGRAM_CHAT = process.env.TELEGRAM_CHAT_ID;
function wooCfg() {
const cfg = JSON.parse(fs.readFileSync(WOO_API_PATH, 'utf8'));
const auth = Buffer.from(`cfg.consumer_key:cfg.consumer_secret`).toString('base64');
return { base: cfg.url, auth };
}
function httpGet(url, headers = {}) {
return new Promise((resolve, reject) => {
const mod = url.startsWith('https') ? https : http;
mod.get(url, { headers }, (res) => {
let data = '';
res.on('data', d => data += d);
res.on('end', () => {
try { resolve(JSON.parse(data)); }
catch(e) { reject(new Error(`Parse error: data.slice(0,200)`)); }
});
}).on('error', reject);
});
}
async function fetchAllProducts(base, auth) {
const products = [];
let page = 1;
while (true) {
const url = `base/wp-json/wc/v3/products?per_page=100&page=page&status=publish`;
const batch = await httpGet(url, { 'Authorization': `Basic auth` });
if (!Array.isArray(batch) || batch.length === 0) break;
products.push(...batch);
if (batch.length < 100) break;
page++;
}
return products;
}
async function sendTelegramAlert(message) {
if (!TELEGRAM_TOKEN || !TELEGRAM_CHAT) {
console.log('[Telegram] No token/chat configured, skipping alert');
return;
}
const body = JSON.stringify({ chat_id: TELEGRAM_CHAT, text: message, parse_mode: 'Markdown' });
return new Promise((resolve) => {
const req = https.request({
hostname: 'api.telegram.org',
path: `/botTELEGRAM_TOKEN/sendMessage`,
method: 'POST',
headers: { 'Content-Type': 'application/json', 'Content-Length': body.length }
}, (res) => { res.on('data', () => {}); res.on('end', resolve); });
req.on('error', (e) => console.error('Telegram error:', e.message));
req.write(body);
req.end();
});
}
async function main() {
console.log(`[new Date().toISOString()] Stock monitor starting...`);
let cfg;
try { cfg = wooCfg(); }
catch(e) {
console.error('WooCommerce config not found:', e.message);
process.exit(1);
}
// Load previous state
let prevState = {};
if (fs.existsSync(STATE_FILE)) {
prevState = JSON.parse(fs.readFileSync(STATE_FILE, 'utf8'));
}
// Fetch current products
const products = await fetchAllProducts(cfg.base, cfg.auth);
console.log(`Fetched products.length products`);
const currentState = {};
const newlyOOS = [];
const allOOS = [];
for (const p of products) {
const status = p.stock_status; // 'instock', 'outofstock', 'onbackorder'
const key = String(p.id);
currentState[key] = { name: p.name, status, sku: p.sku };
if (status === 'outofstock') {
allOOS.push({ id: p.id, name: p.name, sku: p.sku });
}
// Check for instock → outofstock transition
if (prevState[key] && prevState[key].status === 'instock' && status === 'outofstock') {
newlyOOS.push({ id: p.id, name: p.name, sku: p.sku });
}
}
// Save new state
fs.mkdirSync(path.dirname(STATE_FILE), { recursive: true });
fs.writeFileSync(STATE_FILE, JSON.stringify(currentState, null, 2));
const isFirstRun = Object.keys(prevState).length === 0;
if (isFirstRun) {
console.log(`First run baseline — allOOS.length products currently out of stock`);
if (allOOS.length > 0) {
const list = allOOS.map(p => `• p.name (SKU: p.sku || p.id)`).join('\n');
const msg = `📦 *Stock Monitor — Baseline Report*\n\nallOOS.length products currently OOS:\nlist`;
await sendTelegramAlert(msg);
console.log('Baseline alert sent');
} else {
console.log('All products in stock ✅');
}
} else if (newlyOOS.length > 0) {
console.log(`⚠️ newlyOOS.length product(s) just went out of stock!`);
const list = newlyOOS.map(p => `• p.name (SKU: p.sku || p.id)`).join('\n');
const msg = `⚠️ *Stock Alert — Out of Stock*\n\nnewlyOOS.length product(s) went OOS:\nlist\n\nTotal OOS: allOOS.length`;
await sendTelegramAlert(msg);
console.log('Alert sent');
} else {
console.log(`No new OOS products. Total OOS: allOOS.length / products.length`);
}
// Summary
console.log(`Done. Products: products.length, OOS: allOOS.length, New OOS: newlyOOS.length`);
}
main().catch(e => { console.error('Fatal:', e); process.exit(1); });
Broadcast a message to multiple OpenClaw group sessions simultaneously. Use for cross-agent coordination, alerts, and announcements.
---
name: skill-agent-broadcast
version: 1.0.0
description: Broadcast a message to multiple OpenClaw group sessions simultaneously. Use for cross-agent coordination, alerts, and announcements.
metadata:
openclaw:
requires: { bins: ["node"] }
---
# skill-agent-broadcast
Cross-group signal router. Send one message to multiple OpenClaw Telegram/Discord groups simultaneously.
## Usage
```bash
# Broadcast to named groups
node scripts/broadcast.js --message "Deploy complete!" --groups "github-ops,amazon-ops"
# Broadcast to all registered groups
node scripts/broadcast.js --message "⚡ System alert" --groups all
# Use raw Telegram chat IDs
node scripts/broadcast.js --message "Hello" --groups "-1003871838436,-1003578613620"
# Custom delay between sends
node scripts/broadcast.js --message "Update" --groups all --delay 1000
```
## Arguments
| Arg | Default | Description |
|---|---|---|
| `--message` / `-m` | required | Message text to broadcast |
| `--groups` / `-g` | all | Comma-separated group names or IDs. Use `all` for all registered groups |
| `--channel` | `telegram` | Channel type: `telegram` or `discord` |
| `--delay` | `500` | Milliseconds between sends (rate limiting) |
## Environment Variables
| Var | Default | Description |
|---|---|---|
| `OPENCLAW_PORT` | `3000` | OpenClaw gateway port |
| `OPENCLAW_TOKEN` | — | Gateway auth token |
| `GROUPS_CONFIG_PATH` | `config/groups.json` | Path to group registry JSON |
## Group Registry
Edit `config/groups.json` to add/remove groups:
```json
{
"github-ops": "-1003871838436",
"social-media": "-1003578613620",
"amazon-ops": "-1003898064257"
}
```
## Output
Delivery receipt per group:
```
✅ github-ops (-1003871838436)
✅ amazon-ops (-1003898064257)
❌ social-media (-1003578613620) (status: 500)
✅ Broadcast complete: 2 sent, 1 failed
```
FILE:_meta.json
{
"name": "skill-agent-broadcast",
"version": "1.0.0",
"description": "Broadcast a message to multiple OpenClaw group sessions simultaneously. Use for cross-agent coordination, alerts, and announcements.",
"author": "Zero2Ai-hub",
"license": "MIT"
}
FILE:config/groups.json
{
"github-ops": "-1003871838436",
"social-media": "-1003578613620",
"amazon-ops": "-1003898064257",
"dropship-ops": "-1003577202952",
"reporting": "-1003475582700"
}
FILE:scripts/broadcast.js
#!/usr/bin/env node
/**
* broadcast.js — Cross-group signal router for OpenClaw
*
* Usage:
* node broadcast.js --message "Hello!" --groups "github-ops,social-media"
* node broadcast.js --message "Alert!" --groups "-1003871838436,-1003578613620"
* node broadcast.js --message "All hands!" --groups all --channel telegram
*
* Env vars:
* OPENCLAW_PORT — Gateway port (default: 3000)
* OPENCLAW_TOKEN — Gateway auth token
* GROUPS_CONFIG_PATH — Path to groups registry JSON
*/
'use strict';
const https = require('https');
const http = require('http');
const fs = require('fs');
const path = require('path');
const os = require('os');
// ── Config ────────────────────────────────────────────────────────────────────
const GATEWAY_PORT = parseInt(process.env.OPENCLAW_PORT || '3000', 10);
const GATEWAY_TOKEN = process.env.OPENCLAW_TOKEN || '';
const GROUPS_CONFIG_PATH = process.env.GROUPS_CONFIG_PATH ||
path.join(__dirname, '..', 'config', 'groups.json');
function loadGroups() {
try {
return JSON.parse(fs.readFileSync(GROUPS_CONFIG_PATH, 'utf8'));
} catch {
return {};
}
}
function parseArgs() {
const args = process.argv.slice(2);
const opts = { message: null, groups: null, channel: 'telegram', delay: 500 };
for (let i = 0; i < args.length; i++) {
if (args[i] === '--message' || args[i] === '-m') opts.message = args[++i];
else if (args[i] === '--groups' || args[i] === '-g') opts.groups = args[++i];
else if (args[i] === '--channel') opts.channel = args[++i];
else if (args[i] === '--delay') opts.delay = parseInt(args[++i], 10);
}
return opts;
}
function resolveGroups(groupsArg, registry) {
if (!groupsArg) return Object.values(registry);
if (groupsArg === 'all') return Object.values(registry);
return groupsArg.split(',').map(g => {
g = g.trim();
// If it's a negative number (Telegram chat ID), use as-is
if (/^-?\d+$/.test(g)) return g;
// Otherwise resolve by name
const resolved = registry[g] || registry[g.replace(/-/g, '_')];
if (!resolved) {
console.warn(`⚠️ Unknown group name: "g" — skipping`);
return null;
}
return resolved;
}).filter(Boolean);
}
function sleep(ms) {
return new Promise(r => setTimeout(r, ms));
}
function sendToGateway(channel, to, message, token) {
return new Promise((resolve) => {
const body = JSON.stringify({ channel, to, message });
const options = {
hostname: 'localhost',
port: GATEWAY_PORT,
path: '/api/send',
method: 'POST',
headers: {
'Content-Type': 'application/json',
'Content-Length': Buffer.byteLength(body),
...(token ? { Authorization: `Bearer token` } : {}),
},
};
const req = http.request(options, (res) => {
let data = '';
res.on('data', c => data += c);
res.on('end', () => {
resolve({ status: res.statusCode, body: data });
});
});
req.on('error', (err) => {
resolve({ status: 0, body: err.message });
});
req.write(body);
req.end();
});
}
async function main() {
const opts = parseArgs();
if (!opts.message) {
console.error('❌ --message is required');
console.error(' Usage: node broadcast.js --message "Hello!" --groups "github-ops,amazon-ops"');
process.exit(1);
}
const registry = loadGroups();
const targets = resolveGroups(opts.groups, registry);
if (targets.length === 0) {
console.error('❌ No valid targets resolved. Check --groups and config/groups.json');
process.exit(1);
}
console.log(`\n📡 Broadcasting to targets.length group(s) via opts.channel`);
console.log(` Message: opts.message.slice(0, 80)''\n`);
const results = [];
for (const target of targets) {
const { status, body } = await sendToGateway(opts.channel, target, opts.message, GATEWAY_TOKEN);
const ok = status === 200 || status === 201;
const label = Object.entries(registry).find(([, v]) => v === target)?.[0] || target;
const receipt = ok ? '✅' : `❌ (status)`;
console.log(` receipt label (target)`);
results.push({ target, label, status, ok });
if (opts.delay > 0 && targets.indexOf(target) < targets.length - 1) {
await sleep(opts.delay);
}
}
const succeeded = results.filter(r => r.ok).length;
const failed = results.filter(r => !r.ok).length;
console.log(`\n✅ Broadcast complete: succeeded sent, failed failed`);
if (failed > 0) process.exit(1);
}
main().catch(err => {
console.error(`FATAL: err.message`);
process.exit(1);
});
FILE:scripts/formatters.js
'use strict';
const STRENGTH_BADGE = {
low: '🟡',
medium: '🟠',
high: '🔴',
};
function strengthLine(strength) {
const badge = STRENGTH_BADGE[strength] || '⚪';
return `badge Signal Strength: strength.toUpperCase()`;
}
function ts() {
return new Date().toISOString().replace('T', ' ').slice(0, 16) + ' UTC';
}
const formatters = {
product_launch(content, strength) {
const { name = 'Unknown', price = '—', sku = '', url = '', description = '' } = content;
return [
`🚀 *PRODUCT LAUNCH*`,
``,
`📦 *name*`,
sku ? `🔑 SKU: \`sku\`` : null,
`💰 Price: *$price*`,
description ? `📝 description` : null,
url ? `🔗 url` : null,
``,
strengthLine(strength),
`🕐 ts()`,
].filter(l => l !== null).join('\n');
},
price_alert(content, strength) {
const { name = 'Unknown', sku = '', old_price, new_price, change_pct, marketplace = '' } = content;
const arrow = (old_price && new_price && new_price < old_price) ? '📉' : '📈';
return [
`arrow *PRICE ALERT*`,
``,
`📦 *name*`,
sku ? `🔑 SKU: \`sku\`` : null,
marketplace ? `🏪 Marketplace: marketplace` : null,
old_price != null ? `💵 Old Price: $old_price` : null,
new_price != null ? `💰 New Price: *$new_price*` : null,
change_pct != null ? `📊 Change: ''change_pct%` : null,
``,
strengthLine(strength),
`🕐 ts()`,
].filter(l => l !== null).join('\n');
},
ad_signal(content, strength) {
const { campaign = '', action = '', budget, acos, roas, note = '' } = content;
return [
`📣 *AD SIGNAL*`,
``,
campaign ? `🎯 Campaign: *campaign*` : null,
action ? `⚡ Action: action` : null,
budget != null ? `💰 Budget: $budget` : null,
acos != null ? `📊 ACoS: acos%` : null,
roas != null ? `📈 ROAS: roasx` : null,
note ? `📝 note` : null,
``,
strengthLine(strength),
`🕐 ts()`,
].filter(l => l !== null).join('\n');
},
stock_alert(content, strength) {
const { name = 'Unknown', sku = '', units, warehouse = '', status = '', threshold } = content;
const icon = (units != null && threshold != null && units < threshold) ? '⚠️' : '📦';
return [
`icon *STOCK ALERT*`,
``,
`📦 *name*`,
sku ? `🔑 SKU: \`sku\`` : null,
warehouse ? `🏭 Warehouse: warehouse` : null,
units != null ? `📊 Units: *units*` : null,
threshold != null ? `🚨 Threshold: threshold` : null,
status ? `📋 Status: status` : null,
``,
strengthLine(strength),
`🕐 ts()`,
].filter(l => l !== null).join('\n');
},
custom(content, strength) {
const { title = 'Signal', body = '', emoji = '📡' } = content;
const extra = Object.entries(content)
.filter(([k]) => !['title', 'body', 'emoji'].includes(k))
.map(([k, v]) => `• *k*: v`)
.join('\n');
return [
`emoji *title.toUpperCase()*`,
``,
body || null,
extra || null,
``,
strengthLine(strength),
`🕐 ts()`,
].filter(l => l !== null).join('\n');
},
};
module.exports = { formatters };
Sends Amazon review requests for eligible shipped orders using SP-API with retry, deduplication, eligibility checks, and optional dry-run mode.
# skill-amazon-review-request
Sends Amazon review requests for eligible Shipped orders via SP-API Messaging API.
Hardened with retry logic, deduplication, eligibility window enforcement, and dry-run mode.
## Prerequisites
- SP-API credentials in `~/amazon-sp-api.json`:
```json
{
"refreshToken": "...",
"clientId": "...",
"clientSecret": "...",
"marketplaceId": "A2VIGQ35RCS4UG"
}
```
- Or set env vars: `SP_API_REFRESH_TOKEN`, `SP_API_CLIENT_ID`, `SP_API_CLIENT_SECRET`, `SP_API_MARKETPLACE_ID`
- SP-API app must have **Messaging** permission granted
## Usage
```bash
# Dry run — see what would be sent (no requests made)
node scripts/request-reviews.js --dry-run
# Live run
node scripts/request-reviews.js
```
## Behavior
| Feature | Detail |
|---|---|
| **Eligibility window** | Orders 5–30 days old only (Amazon's allowed window) |
| **Deduplication** | Skips orders already logged as `sent` in the tracking log |
| **Retry logic** | Up to 3 attempts with 5s delay on 5xx / 429 responses |
| **Rate limiting** | 1.1s pause between requests |
| **Dry-run** | `--dry-run` flag — logs what would be sent, no API calls |
| **Tracking log** | `data/review-requests-log.json` — per-order status, sentAt, attempts |
| **Text log** | `data/review-requests.log` — timestamped human-readable run log |
## Tracking Log Schema
`data/review-requests-log.json`:
```json
[
{
"orderId": "123-4567890-1234567",
"sentAt": "2026-03-01T10:00:00.000Z",
"status": "sent", // "sent" | "failed" | "skipped"
"attempts": 1,
"reason": "optional error string for failed/skipped"
}
]
```
## Summary Output
```
=== DONE | Sent: 12 | Skipped: 4 | Failed: 1 ===
```
Dry-run:
```
=== DONE [DRY RUN] | Would send: 15 | Skipped: 4 ===
```
## Scheduling (Recommended)
Run daily via cron:
```bash
# 9am UAE time (UTC+4) = 5am UTC
0 5 * * * cd $HOME/.openclaw/workspace && node skills/skill-amazon-review-request/scripts/request-reviews.js >> data/review-requests-cron.log 2>&1
```
## Region Note
Script targets `sellingpartnerapi-eu.amazon.com` (EU endpoint, covers UAE marketplace).
Change to `sellingpartnerapi-na.amazon.com` or `sellingpartnerapi-fe.amazon.com` for other regions.
FILE:_meta.json
{
"name": "skill-amazon-review-request",
"version": "1.0.0",
"description": "Send Amazon review solicitation requests via SP-API with retry logic and Supabase deduplication logging.",
"author": "Zero2Ai-hub",
"license": "MIT"
}
FILE:config/internal.md
# Internal — Jarvis Ops Context (gitignored)
## Credentials
- SP-API: ~/amazon-sp-api.json (or SP_API_PATH env)
- Supabase: ~/supabase-api.json (or SUPABASE_API_PATH env)
## Supabase
- Table: review_requests
- Fields: order_id, asin, status, attempted_at, error
## Eligibility
- Orders delivered >5 days, not already in Supabase
- Max window: 30 days (Amazon requirement)
FILE:scripts/request-reviews.js
#!/usr/bin/env node
/**
* request-reviews.js
* Hardened Amazon Review Request script (skill version)
*
* Enhancements over original:
* - Eligibility window: only orders 5–30 days old (Amazon's allowed window)
* - Deduplication: skips orders already logged as 'sent'
* - Retry logic: up to 3 attempts with 5s delay on failure
* - Tracking: persists results to data/review-requests-log.json
* - Dry-run: --dry-run flag shows what would be sent without sending
* - Summary: sent / skipped / failed / dry-run counts
*
* Usage:
* node request-reviews.js # Live run
* node request-reviews.js --dry-run # Dry run
*
* Credentials loaded from ~/amazon-sp-api.json (or SP_API_PATH env):
* { refreshToken, clientId, clientSecret, marketplaceId }
*
* Or set env vars:
* SP_API_REFRESH_TOKEN, SP_API_CLIENT_ID, SP_API_CLIENT_SECRET, SP_API_MARKETPLACE_ID
*/
const https = require('https');
const fs = require('fs');
const path = require('path');
// ── Config ──────────────────────────────────────────────────────────────────
const DRY_RUN = process.argv.includes('--dry-run');
const RATE_LIMIT_MS = 1100; // 1 req/sec (safe margin)
const RETRY_MAX = 3;
const RETRY_DELAY_MS = 5000;
const ELIGIBILITY_MIN_DAYS = 5;
const ELIGIBILITY_MAX_DAYS = 30;
const CREDS_FILE = process.env.SP_API_PATH || require('os').homedir() + '/amazon-sp-api.json';
const LOG_FILE = path.join(__dirname, '../../data/review-requests-log.json');
const TEXT_LOG = path.join(__dirname, '../../data/review-requests.log');
// ── Load credentials ─────────────────────────────────────────────────────────
let creds = {};
if (fs.existsSync(CREDS_FILE)) {
creds = JSON.parse(fs.readFileSync(CREDS_FILE, 'utf8'));
}
const SP_API_REFRESH_TOKEN = process.env.SP_API_REFRESH_TOKEN || creds.refreshToken;
const SP_API_CLIENT_ID = process.env.SP_API_CLIENT_ID || creds.clientId;
const SP_API_CLIENT_SECRET = process.env.SP_API_CLIENT_SECRET || creds.clientSecret;
const SP_API_MARKETPLACE_ID = process.env.SP_API_MARKETPLACE_ID || creds.marketplaceId;
// ── Supabase ──────────────────────────────────────────────────────────────────
const SUPABASE_CREDS_FILE = process.env.SUPABASE_API_PATH || require('os').homedir() + '/supabase-api.json';
let supabaseCfg = null;
try {
supabaseCfg = JSON.parse(fs.readFileSync(SUPABASE_CREDS_FILE, 'utf8'));
} catch {
// Supabase optional — fall back to local JSON log only
}
async function supabaseInsert(table, row) {
if (!supabaseCfg) return;
const { url, key } = supabaseCfg;
const apiUrl = new URL(`/rest/v1/table`, url);
return new Promise((resolve) => {
const body = JSON.stringify(row);
const req = require('https').request({
hostname: apiUrl.hostname,
path: apiUrl.pathname + apiUrl.search,
method: 'POST',
headers: {
apikey: key,
Authorization: `Bearer key`,
'Content-Type': 'application/json',
Prefer: 'return=minimal',
'Content-Length': Buffer.byteLength(body),
},
}, (res) => {
res.resume();
resolve(res.statusCode);
});
req.on('error', () => resolve(null));
req.write(body);
req.end();
});
}
async function supabaseAlreadySent(orderId) {
if (!supabaseCfg) return false;
const { url, key } = supabaseCfg;
const apiUrl = new URL(`/rest/v1/review_requests?order_id=eq.orderId&status=eq.sent&select=order_id`, url);
return new Promise((resolve) => {
const req = require('https').request({
hostname: apiUrl.hostname,
path: apiUrl.pathname + apiUrl.search,
method: 'GET',
headers: {
apikey: key,
Authorization: `Bearer key`,
Accept: 'application/json',
},
}, (res) => {
let data = '';
res.on('data', c => data += c);
res.on('end', () => {
try {
const rows = JSON.parse(data);
resolve(Array.isArray(rows) && rows.length > 0);
} catch {
resolve(false);
}
});
});
req.on('error', () => resolve(false));
req.end();
});
}
async function logToSupabase(orderId, asin, status, error = null) {
await supabaseInsert('review_requests', {
order_id: orderId,
asin: asin || null,
status,
attempted_at: new Date().toISOString(),
error: error || null,
});
}
// ── Logging ──────────────────────────────────────────────────────────────────
function log(msg) {
const line = `[new Date().toISOString()] msg`;
console.log(line);
fs.appendFileSync(TEXT_LOG, line + '\n');
}
function sleep(ms) {
return new Promise(resolve => setTimeout(resolve, ms));
}
// ── JSON tracking log ────────────────────────────────────────────────────────
function loadLog() {
if (!fs.existsSync(LOG_FILE)) return [];
try {
return JSON.parse(fs.readFileSync(LOG_FILE, 'utf8'));
} catch {
return [];
}
}
function saveLog(entries) {
fs.mkdirSync(path.dirname(LOG_FILE), { recursive: true });
fs.writeFileSync(LOG_FILE, JSON.stringify(entries, null, 2));
}
function isAlreadySent(entries, orderId) {
return entries.some(e => e.orderId === orderId && e.status === 'sent');
}
function upsertEntry(entries, entry) {
const idx = entries.findIndex(e => e.orderId === entry.orderId);
if (idx >= 0) entries[idx] = entry;
else entries.push(entry);
}
// ── Eligibility window ───────────────────────────────────────────────────────
function isEligible(order) {
const purchaseDate = new Date(order.PurchaseDate);
const now = Date.now();
const ageMs = now - purchaseDate.getTime();
const ageDays = ageMs / (1000 * 60 * 60 * 24);
return ageDays >= ELIGIBILITY_MIN_DAYS && ageDays <= ELIGIBILITY_MAX_DAYS;
}
// ── HTTP helper ──────────────────────────────────────────────────────────────
function httpsRequest(options, body = null) {
return new Promise((resolve, reject) => {
const req = https.request(options, (res) => {
let data = '';
res.on('data', chunk => data += chunk);
res.on('end', () => {
try {
resolve({ status: res.statusCode, body: data ? JSON.parse(data) : {} });
} catch {
resolve({ status: res.statusCode, body: data });
}
});
});
req.on('error', reject);
if (body !== null) req.write(JSON.stringify(body));
req.end();
});
}
// ── SP-API: access token ─────────────────────────────────────────────────────
async function getAccessToken() {
const body = new URLSearchParams({
grant_type: 'refresh_token',
refresh_token: SP_API_REFRESH_TOKEN,
client_id: SP_API_CLIENT_ID,
client_secret: SP_API_CLIENT_SECRET,
}).toString();
return new Promise((resolve, reject) => {
const req = https.request({
hostname: 'api.amazon.com',
path: '/auth/o2/token',
method: 'POST',
headers: {
'Content-Type': 'application/x-www-form-urlencoded',
'Content-Length': Buffer.byteLength(body),
},
}, (res) => {
let data = '';
res.on('data', chunk => data += chunk);
res.on('end', () => {
const parsed = JSON.parse(data);
if (parsed.access_token) resolve(parsed.access_token);
else reject(new Error(`Token error: data`));
});
});
req.on('error', reject);
req.write(body);
req.end();
});
}
// ── SP-API: fetch orders ─────────────────────────────────────────────────────
async function getShippedOrders(accessToken) {
const createdAfter = new Date(Date.now() - ELIGIBILITY_MAX_DAYS * 24 * 60 * 60 * 1000).toISOString();
const query = new URLSearchParams({
MarketplaceIds: SP_API_MARKETPLACE_ID,
OrderStatuses: 'Shipped',
CreatedAfter: createdAfter,
});
const result = await httpsRequest({
hostname: 'sellingpartnerapi-eu.amazon.com',
path: `/orders/v0/orders?query.toString()`,
method: 'GET',
headers: {
'x-amz-access-token': accessToken,
'Content-Type': 'application/json',
},
});
if (result.status !== 200) throw new Error(`Orders fetch failed: JSON.stringify(result.body)`);
return result.body?.payload?.Orders || [];
}
// ── SP-API: request review (single attempt) ──────────────────────────────────
async function requestReviewOnce(accessToken, orderId) {
return httpsRequest({
hostname: 'sellingpartnerapi-eu.amazon.com',
path: `/messaging/v1/orders/orderId/messages/requestReview?marketplaceIds=SP_API_MARKETPLACE_ID`,
method: 'POST',
headers: {
'x-amz-access-token': accessToken,
'Content-Type': 'application/json',
},
}, {});
}
// ── SP-API: request review with retry ───────────────────────────────────────
async function requestReviewWithRetry(accessToken, orderId) {
let lastError = null;
for (let attempt = 1; attempt <= RETRY_MAX; attempt++) {
try {
const result = await requestReviewOnce(accessToken, orderId);
// 4xx (except 429) are permanent — don't retry
if (result.status === 429 || result.status >= 500) {
log(` ⚠️ Attempt attempt/RETRY_MAX failed (result.status) for orderId. Retrying in RETRY_DELAY_MS / 1000s…`);
lastError = result;
if (attempt < RETRY_MAX) await sleep(RETRY_DELAY_MS);
continue;
}
return { result, attempts: attempt };
} catch (err) {
log(` ⚠️ Attempt attempt/RETRY_MAX threw for orderId: err.message`);
lastError = err;
if (attempt < RETRY_MAX) await sleep(RETRY_DELAY_MS);
}
}
throw lastError instanceof Error ? lastError : new Error(`HTTP lastError?.status: JSON.stringify(lastError?.body)`);
}
// ── Main ─────────────────────────────────────────────────────────────────────
async function main() {
log(`=== Amazon Review Request START'' ===`);
if (!SP_API_REFRESH_TOKEN || !SP_API_CLIENT_ID || !SP_API_CLIENT_SECRET || !SP_API_MARKETPLACE_ID) {
log('ERROR: Missing credentials. Provide ~/amazon-sp-api.json or set SP_API_PATH / env vars.');
process.exit(1);
}
// Load existing log for deduplication
const logEntries = loadLog();
log(`Loaded logEntries.length existing log entries.`);
// Get access token
log('Fetching access token…');
const accessToken = await getAccessToken();
log('Access token obtained.');
// Fetch orders
log(`Fetching Shipped orders (last ELIGIBILITY_MAX_DAYS days)…`);
const orders = await getShippedOrders(accessToken);
log(`Found orders.length shipped orders.`);
let sent = 0, failed = 0, skipped = 0, dryRun = 0;
for (const order of orders) {
const orderId = order.AmazonOrderId;
await sleep(RATE_LIMIT_MS);
// Deduplication — check Supabase first, then local log
const inSupabase = await supabaseAlreadySent(orderId);
if (inSupabase || isAlreadySent(logEntries, orderId)) {
log(`⏭️ Skip (already sent): orderId`);
skipped++;
continue;
}
// Eligibility window
if (!isEligible(order)) {
const purchaseDate = new Date(order.PurchaseDate);
const ageDays = ((Date.now() - purchaseDate.getTime()) / 86400000).toFixed(1);
log(`⏭️ Skip (age ageDaysd, outside 5–30d window): orderId`);
skipped++;
continue;
}
// Dry-run
if (DRY_RUN) {
log(`[DRY RUN] Would send review request for: orderId`);
dryRun++;
continue;
}
// Send with retry
try {
const { result, attempts } = await requestReviewWithRetry(accessToken, orderId);
const asin = order.OrderItems?.[0]?.ASIN || null;
if (result.status === 201 || result.status === 200) {
log(`✅ Sent (attempts attempt''): orderId`);
upsertEntry(logEntries, { orderId, sentAt: new Date().toISOString(), status: 'sent', attempts });
await logToSupabase(orderId, asin, 'sent');
sent++;
} else if (result.status === 400 || result.status === 403) {
const reason = `HTTP result.status`;
log(`⚠️ Skipped orderId (result.status): JSON.stringify(result.body)`);
upsertEntry(logEntries, { orderId, sentAt: new Date().toISOString(), status: 'skipped', attempts: 1, reason });
await logToSupabase(orderId, asin, 'skipped', reason);
skipped++;
} else {
const reason = `HTTP result.status`;
log(`❌ Failed orderId (result.status): JSON.stringify(result.body)`);
upsertEntry(logEntries, { orderId, sentAt: new Date().toISOString(), status: 'failed', attempts: RETRY_MAX, reason });
await logToSupabase(orderId, asin, 'failed', reason);
failed++;
}
} catch (err) {
log(`❌ Error for orderId: err.message`);
upsertEntry(logEntries, { orderId, sentAt: new Date().toISOString(), status: 'failed', attempts: RETRY_MAX, reason: err.message });
await logToSupabase(orderId, null, 'failed', err.message);
failed++;
}
// Persist after each order (safe against mid-run crashes)
saveLog(logEntries);
}
// Final summary
if (DRY_RUN) {
log(`=== DONE [DRY RUN] | Would send: dryRun | Skipped: skipped ===`);
} else {
log(`=== DONE | Sent: sent | Skipped: skipped | Failed: failed ===`);
}
}
main().catch(err => {
log(`FATAL: err.message`);
process.exit(1);
});
Daily GitHub repo health check + safe Dependabot auto-merge. Outputs markdown report.
---
name: skill-github-daily-ops
version: 1.0.0
description: Daily GitHub repo health check + safe Dependabot auto-merge. Outputs markdown report.
metadata:
openclaw:
requires: { bins: ["node"] }
---
# skill-github-daily-ops
Daily GitHub ops: health report per repo + safe auto-merge of Dependabot PRs (medium/low CVEs only).
## Usage
```bash
# Health report for all org repos
node scripts/daily-ops.js --org Zero2Ai-hub --report
# Auto-merge safe Dependabot PRs
node scripts/daily-ops.js --org Zero2Ai-hub --merge-dependabot
# Both — specific repos
node scripts/daily-ops.js --org Zero2Ai-hub --repos "repo1,repo2" --report --merge-dependabot
```
## Arguments
| Arg | Default | Description |
|---|---|---|
| `--org` | `$GITHUB_ORG` env | GitHub organization |
| `--repos` | all | Comma-separated repo names |
| `--report` | false | Output markdown health report |
| `--merge-dependabot` | false | Auto-merge safe Dependabot PRs |
## Environment Variables
| Var | Description |
|---|---|
| `GITHUB_TOKEN` | GitHub PAT (or reads from `~/.github_token`) |
| `GITHUB_ORG` | Default org |
## Auto-Merge Rules
- ✅ Merges: Dependabot PRs where severity is LOW or MEDIUM **and** CI passes
- ⛔ Skips: HIGH or CRITICAL CVE PRs (require human review)
- ⏳ Skips: PRs with failing or in-progress CI
## Report Output
Markdown table per repo with: open PRs, open issues, last commit date, Dependabot PR count.
## Cron Example
```bash
# Daily at 08:00 Dubai time (04:00 UTC)
0 4 * * * cd /path/to/workspace && node skills/skill-github-daily-ops/scripts/daily-ops.js --org Zero2Ai-hub --report --merge-dependabot >> /var/log/github-daily-ops.log 2>&1
```
FILE:_meta.json
{
"name": "skill-github-daily-ops",
"version": "1.0.0",
"description": "Daily GitHub repo health check + safe Dependabot auto-merge. Outputs markdown report.",
"author": "Zero2Ai-hub",
"license": "MIT"
}
FILE:scripts/auto-merge.sh
#!/usr/bin/env bash
# auto-merge.sh — Find open Dependabot PRs and merge if CI is green
# Usage: bash auto-merge.sh [repo1 repo2 ...]
set -euo pipefail
export GH_TOKEN="-$(cat ~/.github_token 2>/dev/null || echo '')"
if [[ -z "$GH_TOKEN" ]]; then
echo "❌ GH_TOKEN not set and ~/.github_token not found" >&2
exit 1
fi
REPOS=("-Zero2Ai-hub/Jarvis-Ops Zero2Ai-hub/Zeerotoai.com Zero2Ai-hub/openclaw-skills")
if [[ $# -eq 0 ]]; then
REPOS=(
"Zero2Ai-hub/Jarvis-Ops"
"Zero2Ai-hub/Zeerotoai.com"
"Zero2Ai-hub/openclaw-skills"
)
fi
MERGED=()
SKIPPED=()
FAILED=()
check_and_merge() {
local repo="$1"
local pr_number="$2"
local pr_title="$3"
local head_sha="$4"
echo " 🔍 Checking CI for PR #pr_number: pr_title"
# Get check runs for the head SHA
local checks
checks=$(gh api "repos/repo/commits/head_sha/check-runs" \
--jq '.check_runs[] | {name: .name, status: .status, conclusion: .conclusion}' 2>/dev/null || echo "")
if [[ -z "$checks" ]]; then
echo " ⚠️ No CI checks found for PR #pr_number — skipping"
SKIPPED+=("repo#pr_number (no CI checks)")
return
fi
# Check if all completed checks have passed
local total incomplete failed_count
total=$(echo "$checks" | grep -c '"status"' || true)
incomplete=$(echo "$checks" | grep '"status": "in_progress"\|"status": "queued"' | wc -l || echo 0)
failed_count=$(echo "$checks" | grep '"conclusion": "failure"\|"conclusion": "cancelled"\|"conclusion": "timed_out"' | wc -l || echo 0)
if [[ "$incomplete" -gt 0 ]]; then
echo " ⏳ PR #pr_number — CI still running (incomplete pending) — skipping"
SKIPPED+=("repo#pr_number (CI pending)")
return
fi
if [[ "$failed_count" -gt 0 ]]; then
echo " ❌ PR #pr_number — CI failed (failed_count failure(s)) — skipping"
SKIPPED+=("repo#pr_number (CI failed)")
return
fi
# All checks passed — merge
echo " ✅ CI green — merging PR #pr_number..."
if gh pr merge "pr_number" --repo "repo" --squash --auto 2>/dev/null || \
gh pr merge "pr_number" --repo "repo" --merge 2>/dev/null; then
echo " 🎉 Merged PR #pr_number: pr_title"
MERGED+=("repo#pr_number: pr_title")
else
echo " ⚠️ Could not merge PR #pr_number (may need approval or branch protection)"
FAILED+=("repo#pr_number: pr_title")
fi
}
for REPO in "REPOS[@]"; do
echo ""
echo "🔎 Checking Dependabot PRs in REPO..."
# Fetch open Dependabot PRs
prs_json=$(gh pr list \
--repo "REPO" \
--author "app/dependabot" \
--state open \
--json number,title,headRefOid,headRefName \
2>/dev/null || echo "[]")
pr_count=$(echo "$prs_json" | python3 -c "import sys,json; print(len(json.load(sys.stdin)))" 2>/dev/null || echo 0)
if [[ "$pr_count" -eq 0 ]]; then
echo " ✓ No open Dependabot PRs"
continue
fi
echo " Found pr_count Dependabot PR(s)"
# Process each PR
echo "$prs_json" | python3 -c "
import sys, json
prs = json.load(sys.stdin)
for pr in prs:
print(f\"{pr['number']}|{pr['title']}|{pr['headRefOid']}\")
" | while IFS='|' read -r number title sha; do
check_and_merge "REPO" "number" "title" "sha"
done
done
echo ""
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
echo "📊 Auto-merge Summary"
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
echo "✅ Merged: #MERGED[@]"
for m in "-"; do [[ -n "$m" ]] && echo " • $m"; done
echo "⏭️ Skipped: #SKIPPED[@]"
for s in "-"; do [[ -n "$s" ]] && echo " • $s"; done
echo "⚠️ Could not merge: #FAILED[@]"
for f in "-"; do [[ -n "$f" ]] && echo " • $f"; done
# Export results for daily-ops.sh to consume
export AUTOMERGE_MERGED="-"
export AUTOMERGE_SKIPPED="-"
export AUTOMERGE_FAILED="-"
FILE:scripts/daily-ops.js
#!/usr/bin/env node
/**
* daily-ops.js — GitHub Daily Health Check + Dependabot Auto-Merge
*
* Usage:
* node daily-ops.js --org Zero2Ai-hub --report
* node daily-ops.js --org Zero2Ai-hub --merge-dependabot
* node daily-ops.js --repos "repo1,repo2" --report --merge-dependabot
*
* Env vars:
* GITHUB_TOKEN — GitHub PAT (or reads from ~/.github_token)
* GITHUB_ORG — default org if --org not passed
*/
const https = require('https');
const fs = require('fs');
const os = require('os');
const path = require('path');
// ── Config ───────────────────────────────────────────────────────────────────
function getToken() {
if (process.env.GITHUB_TOKEN) return process.env.GITHUB_TOKEN;
const tokenFile = path.join(os.homedir(), '.github_token');
if (fs.existsSync(tokenFile)) return fs.readFileSync(tokenFile, 'utf8').trim();
throw new Error('No GitHub token. Set GITHUB_TOKEN env or create ~/.github_token');
}
function parseArgs() {
const args = process.argv.slice(2);
const opts = { repos: null, mergeDependabot: false, report: false };
for (let i = 0; i < args.length; i++) {
if (args[i] === '--org') opts.org = args[++i];
else if (args[i] === '--repos') opts.repos = args[++i].split(',').map(r => r.trim());
else if (args[i] === '--merge-dependabot') opts.mergeDependabot = true;
else if (args[i] === '--report') opts.report = true;
}
opts.org = opts.org || process.env.GITHUB_ORG;
return opts;
}
// ── GitHub API ────────────────────────────────────────────────────────────────
function ghRequest(token, path, method = 'GET', body = null) {
return new Promise((resolve, reject) => {
const options = {
hostname: 'api.github.com',
path,
method,
headers: {
Authorization: `Bearer token`,
Accept: 'application/vnd.github+json',
'X-GitHub-Api-Version': '2022-11-28',
'User-Agent': 'jarvis-daily-ops/1.0',
'Content-Type': 'application/json',
},
};
const req = https.request(options, (res) => {
let data = '';
res.on('data', c => data += c);
res.on('end', () => {
try {
resolve({ status: res.statusCode, body: data ? JSON.parse(data) : {} });
} catch {
resolve({ status: res.statusCode, body: data });
}
});
});
req.on('error', reject);
if (body) req.write(JSON.stringify(body));
req.end();
});
}
async function listRepos(token, org) {
const all = [];
let page = 1;
while (true) {
const { body } = await ghRequest(token, `/orgs/org/repos?per_page=100&page=page`);
if (!Array.isArray(body) || body.length === 0) break;
all.push(...body.map(r => r.name));
page++;
}
return all;
}
async function getRepoInfo(token, org, repo) {
const [prs, issues, commits, alerts] = await Promise.all([
ghRequest(token, `/repos/org/repo/pulls?state=open&per_page=100`),
ghRequest(token, `/repos/org/repo/issues?state=open&per_page=100&filter=all`),
ghRequest(token, `/repos/org/repo/commits?per_page=1`),
ghRequest(token, `/repos/org/repo/vulnerability-alerts`).catch(() => ({ status: 404 })),
]);
const openPRs = Array.isArray(prs.body) ? prs.body : [];
const openIssues = Array.isArray(issues.body)
? issues.body.filter(i => !i.pull_request)
: [];
const lastCommit = Array.isArray(commits.body) && commits.body[0]
? commits.body[0].commit?.committer?.date || 'unknown'
: 'unknown';
const dependabotPRs = openPRs.filter(pr =>
pr.user?.login === 'dependabot[bot]' || pr.user?.login === 'dependabot'
);
return { openPRs, openIssues, lastCommit, dependabotPRs };
}
async function getPRChecks(token, org, repo, pr) {
const ref = pr.head?.sha;
if (!ref) return { passing: false };
const { body } = await ghRequest(token, `/repos/org/repo/commits/ref/check-runs`);
const runs = body?.check_runs || [];
if (runs.length === 0) return { passing: true }; // No CI = assume ok
const passing = runs.every(r => r.conclusion === 'success' || r.conclusion === 'skipped' || r.status === 'in_progress');
return { passing };
}
function getSeverityFromTitle(title) {
const low = /low/i.test(title);
const medium = /moderate|medium/i.test(title);
const high = /high|critical/i.test(title);
if (high) return 'HIGH';
if (medium) return 'MEDIUM';
if (low) return 'LOW';
return 'UNKNOWN';
}
async function mergePR(token, org, repo, pr) {
const { status } = await ghRequest(token,
`/repos/org/repo/pulls/pr.number/merge`,
'PUT',
{ merge_method: 'squash', commit_title: `chore: pr.title (auto-merge)` }
);
return status === 200;
}
// ── Main ─────────────────────────────────────────────────────────────────────
async function main() {
const opts = parseArgs();
const token = getToken();
if (!opts.org && !opts.repos) {
console.error('❌ Provide --org or --repos');
process.exit(1);
}
let repos = opts.repos;
if (!repos) {
console.log(`📋 Fetching repos for org: opts.org ...`);
repos = await listRepos(token, opts.org);
console.log(` Found repos.length repos\n`);
}
const org = opts.org || 'unknown';
const results = [];
let totalMerged = 0, totalSkipped = 0;
for (const repo of repos) {
process.stdout.write(`🔍 repo ... `);
try {
const info = await getRepoInfo(token, org, repo);
process.stdout.write(`✅\n`);
results.push({ repo, ...info });
if (opts.mergeDependabot && info.dependabotPRs.length > 0) {
for (const pr of info.dependabotPRs) {
const severity = getSeverityFromTitle(pr.title);
if (severity === 'HIGH') {
console.log(` ⛔ Skip HIGH CVE PR #pr.number: pr.title (human review required)`);
totalSkipped++;
continue;
}
const { passing } = await getPRChecks(token, org, repo, pr);
if (!passing) {
console.log(` ⏳ Skip PR #pr.number: CI not passing yet`);
totalSkipped++;
continue;
}
console.log(` 🔀 Merging PR #pr.number: pr.title`);
const merged = await mergePR(token, org, repo, pr);
if (merged) {
console.log(` ✅ Merged`);
totalMerged++;
} else {
console.log(` ❌ Merge failed`);
totalSkipped++;
}
}
}
} catch (e) {
process.stdout.write(`❌ e.message\n`);
results.push({ repo, error: e.message });
}
}
// ── Report ─────────────────────────────────────────────────────────────────
if (opts.report) {
const now = new Date().toISOString().slice(0, 10);
let md = `# GitHub Daily Ops Report — now\n\n`;
md += `**Org:** org | **Repos:** repos.length\n\n`;
md += `| Repo | Open PRs | Open Issues | Last Commit | Dependabot PRs |\n`;
md += `|---|---|---|---|---|\n`;
for (const r of results) {
if (r.error) {
md += `| r.repo | ❌ error | — | — | — |\n`;
} else {
const lastCommit = r.lastCommit?.slice(0, 10) || '—';
md += `| r.repo | r.openPRs.length | r.openIssues.length | lastCommit | r.dependabotPRs.length |\n`;
}
}
if (opts.mergeDependabot) {
md += `\n## Dependabot Auto-Merge\n`;
md += `- Merged: totalMerged\n`;
md += `- Skipped (HIGH/CI): totalSkipped\n`;
}
console.log('\n' + md);
}
if (opts.mergeDependabot) {
console.log(`\n✅ Dependabot: merged totalMerged, skipped totalSkipped`);
}
}
main().catch(err => {
console.error(`FATAL: err.message`);
process.exit(1);
});
FILE:scripts/daily-ops.sh
#!/usr/bin/env bash
# daily-ops.sh — GitHub daily health check for Zero2Ai-hub
# Runs: Dependabot auto-merge, CI failure check, open PR/issue review, workspace push
# Output: /tmp/github-daily-ops-report.md
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "BASH_SOURCE[0]")" && pwd)"
REPORT="/tmp/github-daily-ops-report.md"
WORKSPACE="-$HOME/.openclaw/workspace"
TODAY=$(date -u +"%Y-%m-%d %H:%M UTC")
REPOS=(
"Zero2Ai-hub/Jarvis-Ops"
"Zero2Ai-hub/Zeerotoai.com"
"Zero2Ai-hub/openclaw-skills"
)
# ── Auth ──────────────────────────────────────────────────────────────────────
export GH_TOKEN="-$(cat ~/.github_token 2>/dev/null || echo '')"
if [[ -z "$GH_TOKEN" ]]; then
echo "❌ GH_TOKEN not set and ~/.github_token not found" >&2
exit 1
fi
echo "⚡ Jarvis GitHub Daily Ops — TODAY"
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
# ── Report init ───────────────────────────────────────────────────────────────
cat > "$REPORT" <<EOF
# ⚡ GitHub Daily Ops Report
**Generated:** TODAY
**Repos:** REPOS[*]
---
EOF
# ── Section helpers ───────────────────────────────────────────────────────────
report_section() {
echo "" >> "$REPORT"
echo "## $1" >> "$REPORT"
}
report_line() {
echo "$1" >> "$REPORT"
}
# ── 1. Dependabot Auto-merge ──────────────────────────────────────────────────
echo ""
echo "🤖 STEP 1: Dependabot Auto-merge"
echo "─────────────────────────────────"
report_section "🤖 Dependabot Auto-merge"
MERGED_PRS=()
SKIPPED_PRS=()
FAILED_MERGE=()
for REPO in "REPOS[@]"; do
echo " 📦 REPO"
report_line ""
report_line "### REPO"
prs_json=$(gh pr list \
--repo "REPO" \
--author "app/dependabot" \
--state open \
--json number,title,headRefOid,url \
2>/dev/null || echo "[]")
pr_count=$(echo "$prs_json" | python3 -c "import sys,json; print(len(json.load(sys.stdin)))" 2>/dev/null || echo 0)
if [[ "$pr_count" -eq 0 ]]; then
echo " ✓ No open Dependabot PRs"
report_line "✓ No open Dependabot PRs"
continue
fi
echo " Found pr_count Dependabot PR(s)"
while IFS='|' read -r number title sha url; do
echo " 🔍 PR #number: title"
# Get check runs
checks_json=$(gh api "repos/REPO/commits/sha/check-runs" 2>/dev/null || echo '{"check_runs":[]}')
total=$(echo "$checks_json" | python3 -c "import sys,json; d=json.load(sys.stdin); print(len(d.get('check_runs',[])))")
pending=$(echo "$checks_json" | python3 -c "
import sys,json
d=json.load(sys.stdin)
print(sum(1 for r in d.get('check_runs',[]) if r['status'] in ('in_progress','queued')))
")
failed=$(echo "$checks_json" | python3 -c "
import sys,json
d=json.load(sys.stdin)
print(sum(1 for r in d.get('check_runs',[]) if r.get('conclusion') in ('failure','cancelled','timed_out')))
")
if [[ "$total" -eq 0 ]]; then
echo " ⚠️ No CI checks — skipping"
report_line "- ⚠️ PR #number [title](url) — no CI checks, skipped"
SKIPPED_PRS+=("REPO#number")
elif [[ "$pending" -gt 0 ]]; then
echo " ⏳ CI pending (pending/total) — skipping"
report_line "- ⏳ PR #number [title](url) — CI pending (pending/total)"
SKIPPED_PRS+=("REPO#number")
elif [[ "$failed" -gt 0 ]]; then
echo " ❌ CI failed (failed failure(s)) — skipping"
report_line "- ❌ PR #number [title](url) — CI failed, skipped"
SKIPPED_PRS+=("REPO#number")
else
echo " ✅ CI green — merging..."
if gh pr merge "number" --repo "REPO" --squash --delete-branch 2>/dev/null; then
echo " 🎉 Merged!"
report_line "- ✅ PR #number [title](url) — **MERGED** 🎉"
MERGED_PRS+=("REPO#number: title")
else
echo " ⚠️ Merge failed (branch protection?)"
report_line "- ⚠️ PR #number [title](url) — merge failed (branch protection?)"
FAILED_MERGE+=("REPO#number: title")
fi
fi
done < <(echo "$prs_json" | python3 -c "
import sys, json
for pr in json.load(sys.stdin):
print(f\"{pr['number']}|{pr['title']}|{pr['headRefOid']}|{pr['url']}\")
")
done
# ── 2. Failed CI Runs ─────────────────────────────────────────────────────────
echo ""
echo "🚨 STEP 2: Failed CI Runs (last 24h)"
echo "─────────────────────────────────────"
report_section "🚨 Failed CI Runs (last 24h)"
FAILED_CI=()
for REPO in "REPOS[@]"; do
echo " 📦 REPO"
report_line ""
report_line "### REPO"
# Get workflow runs that failed in last 24h
since=$(date -u -d '24 hours ago' +"%Y-%m-%dT%H:%M:%SZ" 2>/dev/null || date -u -v-24H +"%Y-%m-%dT%H:%M:%SZ" 2>/dev/null || echo "")
failed_runs=$(gh run list \
--repo "REPO" \
--status failure \
--limit 10 \
--json databaseId,name,headBranch,createdAt,url \
2>/dev/null || echo "[]")
run_count=$(echo "$failed_runs" | python3 -c "import sys,json; print(len(json.load(sys.stdin)))" 2>/dev/null || echo 0)
if [[ "$run_count" -eq 0 ]]; then
echo " ✓ No failed runs"
report_line "✓ No failed CI runs"
else
echo " ⚠️ run_count failed run(s):"
echo "$failed_runs" | python3 -c "
import sys, json
for r in json.load(sys.stdin):
print(f\" ❌ [{r['name']}] branch: {r['headBranch']} — {r['url']}\")
" | while read -r line; do
echo "$line"
report_line "- $line"
FAILED_CI+=("$line")
done
fi
done
# ── 3. Open Issues & PRs needing review ───────────────────────────────────────
echo ""
echo "📋 STEP 3: Open Issues & PRs Needing Review"
echo "─────────────────────────────────────────────"
report_section "📋 Open Issues & PRs Needing Review"
for REPO in "REPOS[@]"; do
echo " 📦 REPO"
report_line ""
report_line "### REPO"
# Open PRs (non-Dependabot)
open_prs=$(gh pr list \
--repo "REPO" \
--state open \
--json number,title,author,url,reviewDecision,isDraft \
2>/dev/null || echo "[]")
human_prs=$(echo "$open_prs" | python3 -c "
import sys, json
prs = [p for p in json.load(sys.stdin) if p['author']['login'] != 'dependabot[bot]' and not p.get('isDraft')]
for p in prs:
decision = p.get('reviewDecision') or 'NEEDS_REVIEW'
print(f\"PR #{p['number']}: {p['title']} [{decision}] — {p['url']}\")
print(f'__COUNT__{len(prs)}')
" 2>/dev/null || echo "__COUNT__0")
pr_count=$(echo "$human_prs" | grep '__COUNT__' | sed 's/__COUNT__//')
pr_lines=$(echo "$human_prs" | grep -v '__COUNT__')
if [[ "$pr_count" -gt 0 ]]; then
echo " 📝 pr_count open PR(s):"
while IFS= read -r line; do
[[ -z "$line" ]] && continue
echo " • line"
report_line "- 📝 line"
done <<< "$pr_lines"
else
report_line "✓ No open human PRs"
fi
# Open Issues
open_issues=$(gh issue list \
--repo "REPO" \
--state open \
--limit 10 \
--json number,title,author,url,labels \
2>/dev/null || echo "[]")
issue_count=$(echo "$open_issues" | python3 -c "import sys,json; print(len(json.load(sys.stdin)))" 2>/dev/null || echo 0)
if [[ "$issue_count" -gt 0 ]]; then
echo " 🐛 issue_count open issue(s):"
echo "$open_issues" | python3 -c "
import sys, json
for i in json.load(sys.stdin):
labels = ', '.join(l['name'] for l in i.get('labels', []))
label_str = f' [{labels}]' if labels else ''
print(f\" • Issue #{i['number']}: {i['title']}{label_str} — {i['url']}\")
" | while IFS= read -r line; do
echo "$line"
report_line "- 🐛 line"
done
else
report_line "✓ No open issues"
fi
if [[ "$pr_count" -eq 0 ]] && [[ "$issue_count" -eq 0 ]]; then
echo " ✓ All clear"
fi
done
# ── 4. Push workspace commits ─────────────────────────────────────────────────
echo ""
echo "📤 STEP 4: Push Workspace Commits"
echo "──────────────────────────────────"
report_section "📤 Workspace Push"
if [[ -d "WORKSPACE/.git" ]]; then
cd "WORKSPACE"
# Check for unpushed commits
unpushed=$(git log @{u}..HEAD --oneline 2>/dev/null || echo "")
uncommitted=$(git status --porcelain 2>/dev/null || echo "")
if [[ -n "$uncommitted" ]]; then
echo " ⚠️ Uncommitted changes in workspace (not auto-committing)"
report_line "⚠️ Uncommitted changes present — manual commit required"
git status --short
fi
if [[ -n "$unpushed" ]]; then
commit_count=$(echo "$unpushed" | wc -l)
echo " 📤 Pushing commit_count unpushed commit(s)..."
echo "$unpushed" | while read -r line; do echo " • $line"; done
if git push 2>&1; then
echo " ✅ Push successful"
report_line "✅ Pushed commit_count commit(s):"
echo "$unpushed" | while IFS= read -r line; do
report_line "- \`line\`"
done
else
echo " ❌ Push failed"
report_line "❌ Push failed — check remote/auth"
fi
else
echo " ✓ Workspace is up to date"
report_line "✓ No unpushed commits"
fi
else
echo " ⚠️ Workspace is not a git repo"
report_line "⚠️ Workspace at WORKSPACE is not a git repo"
fi
# ── 5. Summary ────────────────────────────────────────────────────────────────
echo ""
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
echo "📊 DAILY OPS SUMMARY"
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
echo "✅ Dependabot PRs merged : #MERGED_PRS[@]"
echo "⏭️ PRs skipped : #SKIPPED_PRS[@]"
echo "⚠️ Merge failures : #FAILED_MERGE[@]"
echo "🚨 Failed CI runs : #FAILED_CI[@]"
echo ""
echo "📄 Full report: REPORT"
report_section "📊 Summary"
report_line ""
report_line "| Metric | Count |"
report_line "|--------|-------|"
report_line "| Dependabot PRs merged | #MERGED_PRS[@] |"
report_line "| PRs skipped (CI not ready) | #SKIPPED_PRS[@] |"
report_line "| Merge failures | #FAILED_MERGE[@] |"
report_line "| Failed CI runs | #FAILED_CI[@] |"
report_line ""
report_line "---"
report_line "*Generated by skill-github-daily-ops · Jarvis ⚡*"
echo ""
cat "$REPORT"
Creates WooCommerce draft product listings with images, variants, and margin calculation from CJ Dropshipping products using product ID and sell price.
# skill-dropshipping-product-launcher
**One command: CJ Dropshipping product → WooCommerce draft listing.**
Fetches product data from CJ, downloads images, uploads to WordPress media library, calculates margin, and creates a WooCommerce draft product — with variants if CJ has them.
---
## Inputs
| Argument | Required | Description |
|---|---|---|
| `--product` / `-p` | ✅ | CJ Dropshipping product ID (e.g. `CJA123456789`) |
| `--price` | ✅ | Your sell price in **AED** |
| `--category` / `-c` | ❌ | Category name or WooCommerce category ID |
| `--dry-run` | ❌ | Preview without creating anything in WooCommerce |
---
## Outputs
```
✅ Product created!
WooCommerce ID : 4821
Margin : 42.3%
Admin URL : https://tech1mart.com/wp-admin/post.php?post=4821&action=edit
Product URL : https://tech1mart.com/?p=4821
```
Machine-readable JSON block at end of stdout (key: `__RESULT__`):
```json
{
"wooProductId": 4821,
"margin": "42.3",
"productUrl": "https://tech1mart.com/?p=4821",
"adminUrl": "https://tech1mart.com/wp-admin/post.php?post=4821&action=edit",
"title": "Product Name",
"cjProductId": "CJA123456789",
"sellPriceAED": 149.99,
"cjPriceUSD": 22.5
}
```
---
## Auth Config
- **CJ API**: `~/cj-api.json` — fields: `apiKey`, `accessToken`, `tokenExpiry`
- **WooCommerce**: `~/woo-api.json` — fields: `url`, `consumerKey`, `consumerSecret`
Tokens are refreshed automatically.
---
## Pipeline
1. Fetch product from CJ (`/product/query` + `/product/variant/query`)
2. Download images → `/tmp/product-images/<product_id>/`
3. Upload images → WooCommerce media library (`/wp-json/wp/v2/media`)
4. Calculate margin: `((sellAED - cjUSD × 3.67) / sellAED × 100)` — warns if < 30%
5. Create WooCommerce draft product (`/wp-json/wc/v3/products`) with:
- Title, description, images
- `regular_price` (AED)
- `type: variable` + variations if CJ has multiple variants
- `meta_data: _cj_product_id`
6. Output: product ID, margin %, admin + storefront URLs
---
## Usage Examples
```bash
# Basic launch
node scripts/launch.js --product CJA123456789 --price 149.99
# With category
node scripts/launch.js --product CJA123456789 --price 149.99 --category Electronics
# Dry run (preview only, no WooCommerce writes)
node scripts/launch.js --product CJA123456789 --price 149.99 --dry-run
# Check CJ product data only
node scripts/cj-fetch.js CJA123456789
```
---
## Dependencies
```bash
cd scripts
npm install axios form-data
```
Or from skill root:
```bash
npm install
```
---
## Files
```
skill-dropshipping-product-launcher/
├── SKILL.md ← this file
├── package.json
└── scripts/
├── launch.js ← main entry point
├── cj-fetch.js ← CJ API fetcher + token refresh
└── woo-create.js ← WooCommerce uploader + product creator
```
---
## Margin Formula
```js
const cjPriceAED = cjPriceUSD * 3.67;
const margin = ((sellPriceAED - cjPriceAED) / sellPriceAED * 100).toFixed(1);
```
⚠️ Warning shown if margin < 30%.
FILE:_meta.json
{
"name": "skill-dropshipping-product-launcher",
"version": "1.0.0",
"description": "Launch a CJ Dropshipping product to WooCommerce in one command. Fetches details, uploads images, creates listing.",
"author": "Zero2Ai-hub",
"license": "MIT"
}
FILE:config/internal.md
# Internal — Jarvis Ops Context (gitignored)
## Credentials
- CJ API: ~/cj-api.json (or CJ_API_PATH env)
- WooCommerce: ~/woo-api.json (or WOO_API_PATH env)
## Store
- Active store: tech1mart.com
- Base URL: https://tech1mart.com/wp-json/wc/v3
## Notes
- FBA product IDs (exclude from WooCommerce dropship): env var FBA_PRODUCT_IDS
- Default status: draft
- Default images: 4
FILE:package-lock.json
{
"name": "skill-dropshipping-product-launcher",
"version": "1.0.0",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "skill-dropshipping-product-launcher",
"version": "1.0.0",
"dependencies": {
"axios": "^1.6.0",
"form-data": "^4.0.0"
}
},
"node_modules/asynckit": {
"version": "0.4.0",
"resolved": "https://registry.npmjs.org/asynckit/-/asynckit-0.4.0.tgz",
"integrity": "sha512-Oei9OH4tRh0YqU3GxhX79dM/mwVgvbZJaSNaRk+bshkj0S5cfHcgYakreBjrHwatXKbz+IoIdYLxrKim2MjW0Q==",
"license": "MIT"
},
"node_modules/axios": {
"version": "1.13.6",
"resolved": "https://registry.npmjs.org/axios/-/axios-1.13.6.tgz",
"integrity": "sha512-ChTCHMouEe2kn713WHbQGcuYrr6fXTBiu460OTwWrWob16g1bXn4vtz07Ope7ewMozJAnEquLk5lWQWtBig9DQ==",
"license": "MIT",
"dependencies": {
"follow-redirects": "^1.15.11",
"form-data": "^4.0.5",
"proxy-from-env": "^1.1.0"
}
},
"node_modules/call-bind-apply-helpers": {
"version": "1.0.2",
"resolved": "https://registry.npmjs.org/call-bind-apply-helpers/-/call-bind-apply-helpers-1.0.2.tgz",
"integrity": "sha512-Sp1ablJ0ivDkSzjcaJdxEunN5/XvksFJ2sMBFfq6x0ryhQV/2b/KwFe21cMpmHtPOSij8K99/wSfoEuTObmuMQ==",
"license": "MIT",
"dependencies": {
"es-errors": "^1.3.0",
"function-bind": "^1.1.2"
},
"engines": {
"node": ">= 0.4"
}
},
"node_modules/combined-stream": {
"version": "1.0.8",
"resolved": "https://registry.npmjs.org/combined-stream/-/combined-stream-1.0.8.tgz",
"integrity": "sha512-FQN4MRfuJeHf7cBbBMJFXhKSDq+2kAArBlmRBvcvFE5BB1HZKXtSFASDhdlz9zOYwxh8lDdnvmMOe/+5cdoEdg==",
"license": "MIT",
"dependencies": {
"delayed-stream": "~1.0.0"
},
"engines": {
"node": ">= 0.8"
}
},
"node_modules/delayed-stream": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/delayed-stream/-/delayed-stream-1.0.0.tgz",
"integrity": "sha512-ZySD7Nf91aLB0RxL4KGrKHBXl7Eds1DAmEdcoVawXnLD7SDhpNgtuII2aAkg7a7QS41jxPSZ17p4VdGnMHk3MQ==",
"license": "MIT",
"engines": {
"node": ">=0.4.0"
}
},
"node_modules/dunder-proto": {
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/dunder-proto/-/dunder-proto-1.0.1.tgz",
"integrity": "sha512-KIN/nDJBQRcXw0MLVhZE9iQHmG68qAVIBg9CqmUYjmQIhgij9U5MFvrqkUL5FbtyyzZuOeOt0zdeRe4UY7ct+A==",
"license": "MIT",
"dependencies": {
"call-bind-apply-helpers": "^1.0.1",
"es-errors": "^1.3.0",
"gopd": "^1.2.0"
},
"engines": {
"node": ">= 0.4"
}
},
"node_modules/es-define-property": {
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/es-define-property/-/es-define-property-1.0.1.tgz",
"integrity": "sha512-e3nRfgfUZ4rNGL232gUgX06QNyyez04KdjFrF+LTRoOXmrOgFKDg4BCdsjW8EnT69eqdYGmRpJwiPVYNrCaW3g==",
"license": "MIT",
"engines": {
"node": ">= 0.4"
}
},
"node_modules/es-errors": {
"version": "1.3.0",
"resolved": "https://registry.npmjs.org/es-errors/-/es-errors-1.3.0.tgz",
"integrity": "sha512-Zf5H2Kxt2xjTvbJvP2ZWLEICxA6j+hAmMzIlypy4xcBg1vKVnx89Wy0GbS+kf5cwCVFFzdCFh2XSCFNULS6csw==",
"license": "MIT",
"engines": {
"node": ">= 0.4"
}
},
"node_modules/es-object-atoms": {
"version": "1.1.1",
"resolved": "https://registry.npmjs.org/es-object-atoms/-/es-object-atoms-1.1.1.tgz",
"integrity": "sha512-FGgH2h8zKNim9ljj7dankFPcICIK9Cp5bm+c2gQSYePhpaG5+esrLODihIorn+Pe6FGJzWhXQotPv73jTaldXA==",
"license": "MIT",
"dependencies": {
"es-errors": "^1.3.0"
},
"engines": {
"node": ">= 0.4"
}
},
"node_modules/es-set-tostringtag": {
"version": "2.1.0",
"resolved": "https://registry.npmjs.org/es-set-tostringtag/-/es-set-tostringtag-2.1.0.tgz",
"integrity": "sha512-j6vWzfrGVfyXxge+O0x5sh6cvxAog0a/4Rdd2K36zCMV5eJ+/+tOAngRO8cODMNWbVRdVlmGZQL2YS3yR8bIUA==",
"license": "MIT",
"dependencies": {
"es-errors": "^1.3.0",
"get-intrinsic": "^1.2.6",
"has-tostringtag": "^1.0.2",
"hasown": "^2.0.2"
},
"engines": {
"node": ">= 0.4"
}
},
"node_modules/follow-redirects": {
"version": "1.15.11",
"resolved": "https://registry.npmjs.org/follow-redirects/-/follow-redirects-1.15.11.tgz",
"integrity": "sha512-deG2P0JfjrTxl50XGCDyfI97ZGVCxIpfKYmfyrQ54n5FO/0gfIES8C/Psl6kWVDolizcaaxZJnTS0QSMxvnsBQ==",
"funding": [
{
"type": "individual",
"url": "https://github.com/sponsors/RubenVerborgh"
}
],
"license": "MIT",
"engines": {
"node": ">=4.0"
},
"peerDependenciesMeta": {
"debug": {
"optional": true
}
}
},
"node_modules/form-data": {
"version": "4.0.5",
"resolved": "https://registry.npmjs.org/form-data/-/form-data-4.0.5.tgz",
"integrity": "sha512-8RipRLol37bNs2bhoV67fiTEvdTrbMUYcFTiy3+wuuOnUog2QBHCZWXDRijWQfAkhBj2Uf5UnVaiWwA5vdd82w==",
"license": "MIT",
"dependencies": {
"asynckit": "^0.4.0",
"combined-stream": "^1.0.8",
"es-set-tostringtag": "^2.1.0",
"hasown": "^2.0.2",
"mime-types": "^2.1.12"
},
"engines": {
"node": ">= 6"
}
},
"node_modules/function-bind": {
"version": "1.1.2",
"resolved": "https://registry.npmjs.org/function-bind/-/function-bind-1.1.2.tgz",
"integrity": "sha512-7XHNxH7qX9xG5mIwxkhumTox/MIRNcOgDrxWsMt2pAr23WHp6MrRlN7FBSFpCpr+oVO0F744iUgR82nJMfG2SA==",
"license": "MIT",
"funding": {
"url": "https://github.com/sponsors/ljharb"
}
},
"node_modules/get-intrinsic": {
"version": "1.3.0",
"resolved": "https://registry.npmjs.org/get-intrinsic/-/get-intrinsic-1.3.0.tgz",
"integrity": "sha512-9fSjSaos/fRIVIp+xSJlE6lfwhES7LNtKaCBIamHsjr2na1BiABJPo0mOjjz8GJDURarmCPGqaiVg5mfjb98CQ==",
"license": "MIT",
"dependencies": {
"call-bind-apply-helpers": "^1.0.2",
"es-define-property": "^1.0.1",
"es-errors": "^1.3.0",
"es-object-atoms": "^1.1.1",
"function-bind": "^1.1.2",
"get-proto": "^1.0.1",
"gopd": "^1.2.0",
"has-symbols": "^1.1.0",
"hasown": "^2.0.2",
"math-intrinsics": "^1.1.0"
},
"engines": {
"node": ">= 0.4"
},
"funding": {
"url": "https://github.com/sponsors/ljharb"
}
},
"node_modules/get-proto": {
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/get-proto/-/get-proto-1.0.1.tgz",
"integrity": "sha512-sTSfBjoXBp89JvIKIefqw7U2CCebsc74kiY6awiGogKtoSGbgjYE/G/+l9sF3MWFPNc9IcoOC4ODfKHfxFmp0g==",
"license": "MIT",
"dependencies": {
"dunder-proto": "^1.0.1",
"es-object-atoms": "^1.0.0"
},
"engines": {
"node": ">= 0.4"
}
},
"node_modules/gopd": {
"version": "1.2.0",
"resolved": "https://registry.npmjs.org/gopd/-/gopd-1.2.0.tgz",
"integrity": "sha512-ZUKRh6/kUFoAiTAtTYPZJ3hw9wNxx+BIBOijnlG9PnrJsCcSjs1wyyD6vJpaYtgnzDrKYRSqf3OO6Rfa93xsRg==",
"license": "MIT",
"engines": {
"node": ">= 0.4"
},
"funding": {
"url": "https://github.com/sponsors/ljharb"
}
},
"node_modules/has-symbols": {
"version": "1.1.0",
"resolved": "https://registry.npmjs.org/has-symbols/-/has-symbols-1.1.0.tgz",
"integrity": "sha512-1cDNdwJ2Jaohmb3sg4OmKaMBwuC48sYni5HUw2DvsC8LjGTLK9h+eb1X6RyuOHe4hT0ULCW68iomhjUoKUqlPQ==",
"license": "MIT",
"engines": {
"node": ">= 0.4"
},
"funding": {
"url": "https://github.com/sponsors/ljharb"
}
},
"node_modules/has-tostringtag": {
"version": "1.0.2",
"resolved": "https://registry.npmjs.org/has-tostringtag/-/has-tostringtag-1.0.2.tgz",
"integrity": "sha512-NqADB8VjPFLM2V0VvHUewwwsw0ZWBaIdgo+ieHtK3hasLz4qeCRjYcqfB6AQrBggRKppKF8L52/VqdVsO47Dlw==",
"license": "MIT",
"dependencies": {
"has-symbols": "^1.0.3"
},
"engines": {
"node": ">= 0.4"
},
"funding": {
"url": "https://github.com/sponsors/ljharb"
}
},
"node_modules/hasown": {
"version": "2.0.2",
"resolved": "https://registry.npmjs.org/hasown/-/hasown-2.0.2.tgz",
"integrity": "sha512-0hJU9SCPvmMzIBdZFqNPXWa6dqh7WdH0cII9y+CyS8rG3nL48Bclra9HmKhVVUHyPWNH5Y7xDwAB7bfgSjkUMQ==",
"license": "MIT",
"dependencies": {
"function-bind": "^1.1.2"
},
"engines": {
"node": ">= 0.4"
}
},
"node_modules/math-intrinsics": {
"version": "1.1.0",
"resolved": "https://registry.npmjs.org/math-intrinsics/-/math-intrinsics-1.1.0.tgz",
"integrity": "sha512-/IXtbwEk5HTPyEwyKX6hGkYXxM9nbj64B+ilVJnC/R6B0pH5G4V3b0pVbL7DBj4tkhBAppbQUlf6F6Xl9LHu1g==",
"license": "MIT",
"engines": {
"node": ">= 0.4"
}
},
"node_modules/mime-db": {
"version": "1.52.0",
"resolved": "https://registry.npmjs.org/mime-db/-/mime-db-1.52.0.tgz",
"integrity": "sha512-sPU4uV7dYlvtWJxwwxHD0PuihVNiE7TyAbQ5SWxDCB9mUYvOgroQOwYQQOKPJ8CIbE+1ETVlOoK1UC2nU3gYvg==",
"license": "MIT",
"engines": {
"node": ">= 0.6"
}
},
"node_modules/mime-types": {
"version": "2.1.35",
"resolved": "https://registry.npmjs.org/mime-types/-/mime-types-2.1.35.tgz",
"integrity": "sha512-ZDY+bPm5zTTF+YpCrAU9nK0UgICYPT0QtT1NZWFv4s++TNkcgVaT0g6+4R2uI4MjQjzysHB1zxuWL50hzaeXiw==",
"license": "MIT",
"dependencies": {
"mime-db": "1.52.0"
},
"engines": {
"node": ">= 0.6"
}
},
"node_modules/proxy-from-env": {
"version": "1.1.0",
"resolved": "https://registry.npmjs.org/proxy-from-env/-/proxy-from-env-1.1.0.tgz",
"integrity": "sha512-D+zkORCbA9f1tdWRK0RaCR3GPv50cMxcrz4X8k5LTSUD1Dkw47mKJEZQNunItRTkWwgtaUSo1RVFRIG9ZXiFYg==",
"license": "MIT"
}
}
}
FILE:package.json
{
"name": "skill-dropshipping-product-launcher",
"version": "1.0.0",
"description": "CJ Dropshipping → WooCommerce one-command product launcher",
"main": "scripts/launch.js",
"scripts": {
"launch": "node scripts/launch.js",
"fetch": "node scripts/cj-fetch.js"
},
"dependencies": {
"axios": "^1.6.0",
"form-data": "^4.0.0"
}
}
FILE:scripts/cj-fetch.js
#!/usr/bin/env node
/**
* cj-fetch.js — Fetch product data from CJ Dropshipping API
*/
const axios = require('axios');
const fs = require('fs');
const path = require('path');
const CJ_CONFIG_PATH = process.env.CJ_API_PATH || require('os').homedir() + '/cj-api.json';
function loadConfig() {
return JSON.parse(fs.readFileSync(CJ_CONFIG_PATH, 'utf8'));
}
async function getAccessToken(config) {
// If token is still valid (with 5min buffer), reuse it
if (config.accessToken && config.tokenExpiry) {
const expiry = new Date(config.tokenExpiry).getTime();
if (Date.now() < expiry - 5 * 60 * 1000) {
return config.accessToken;
}
}
// Refresh token using apiKey
const res = await axios.post('https://developers.cjdropshipping.com/api2.0/v1/authentication/getAccessToken', {
apiKey: config.apiKey,
});
if (!res.data || res.data.result !== true) {
throw new Error(`CJ auth failed: JSON.stringify(res.data)`);
}
const { accessToken, accessTokenExpiryDate } = res.data.data;
// Update config file with new token
const updated = { ...config, accessToken, tokenExpiry: accessTokenExpiryDate };
fs.writeFileSync(CJ_CONFIG_PATH, JSON.stringify(updated, null, 2));
return accessToken;
}
async function fetchProduct(productId) {
const config = loadConfig();
const token = await getAccessToken(config);
const baseUrl = config.baseUrl || 'https://developers.cjdropshipping.com/api2.0/v1';
// Fetch product details
const res = await axios.get(`baseUrl/product/query`, {
params: { pid: productId },
headers: { 'CJ-Access-Token': token },
});
if (!res.data || res.data.result !== true) {
throw new Error(`CJ product fetch failed: JSON.stringify(res.data)`);
}
const product = res.data.data;
// Fetch variants if not included
let variants = product.variants || product.productVariants || [];
if (variants.length === 0) {
try {
const varRes = await axios.get(`baseUrl/product/variant/query`, {
params: { pid: productId },
headers: { 'CJ-Access-Token': token },
});
if (varRes.data && varRes.data.result === true) {
variants = varRes.data.data || [];
}
} catch (e) {
// variants optional
}
}
return { product, variants };
}
module.exports = { fetchProduct, getAccessToken };
// CLI usage: node cj-fetch.js <product_id>
if (require.main === module) {
const productId = process.argv[2];
if (!productId) {
console.error('Usage: node cj-fetch.js <cj_product_id>');
process.exit(1);
}
fetchProduct(productId)
.then(data => console.log(JSON.stringify(data, null, 2)))
.catch(err => { console.error(err.message); process.exit(1); });
}
FILE:scripts/launch.js
#!/usr/bin/env node
/**
* launch.js — CJ Dropshipping → WooCommerce Product Launcher
*
* Usage:
* node launch.js --product <cj_product_id> --price <sell_price_aed> [--category <name_or_id>]
*
* Example:
* node launch.js --product CJA123456789 --price 149.99 --category Electronics
*/
const axios = require('axios');
const fs = require('fs');
const path = require('path');
const https = require('https');
const http = require('http');
const { fetchProduct } = require('./cj-fetch');
const { uploadImages, createProduct } = require('./woo-create');
const WOO_CONFIG_PATH = process.env.WOO_API_PATH || require('os').homedir() + '/woo-api.json';
const IMAGE_DIR = '/tmp/product-images';
// ─── CLI Arg Parser ───────────────────────────────────────────────────────────
function parseArgs() {
const args = process.argv.slice(2);
const opts = {};
for (let i = 0; i < args.length; i++) {
if (args[i] === '--product' || args[i] === '-p' || args[i] === '--cj-pid') opts.productId = args[++i];
else if (args[i] === '--price') opts.sellPrice = parseFloat(args[++i]);
else if (args[i] === '--category' || args[i] === '-c') opts.category = args[++i];
else if (args[i] === '--status') opts.status = args[++i]; // draft|publish
else if (args[i] === '--images') opts.imageCount = parseInt(args[++i], 10);
else if (args[i] === '--dry-run') opts.dryRun = true;
}
return opts;
}
// ─── Image Downloader ─────────────────────────────────────────────────────────
function downloadImage(url, dest) {
return new Promise((resolve, reject) => {
const proto = url.startsWith('https') ? https : http;
const file = fs.createWriteStream(dest);
proto.get(url, res => {
if (res.statusCode === 301 || res.statusCode === 302) {
file.close();
fs.unlink(dest, () => {});
return downloadImage(res.headers.location, dest).then(resolve).catch(reject);
}
if (res.statusCode !== 200) {
file.close();
fs.unlink(dest, () => {});
return reject(new Error(`HTTP res.statusCode for url`));
}
res.pipe(file);
file.on('finish', () => file.close(resolve));
file.on('error', reject);
}).on('error', reject);
});
}
async function downloadImages(imageUrls, productId) {
fs.mkdirSync(IMAGE_DIR, { recursive: true });
const dir = path.join(IMAGE_DIR, productId.replace(/[^a-z0-9]/gi, '_'));
fs.mkdirSync(dir, { recursive: true });
const localPaths = [];
for (let i = 0; i < imageUrls.length; i++) {
const url = imageUrls[i];
const ext = (url.split('?')[0].match(/\.(jpg|jpeg|png|webp|gif)$/i) || ['', '.jpg'])[1];
const dest = path.join(dir, `image_i + 1ext`);
try {
console.log(` ⬇ Downloading image i + 1/imageUrls.length...`);
await downloadImage(url, dest);
localPaths.push(dest);
} catch (e) {
console.warn(` ⚠ Failed to download image i + 1: e.message`);
}
}
return localPaths;
}
// ─── Margin Calculator ────────────────────────────────────────────────────────
function calcMargin(sellPriceAED, cjPriceUSD) {
const cjPriceAED = cjPriceUSD * 3.67;
const margin = ((sellPriceAED - cjPriceAED) / sellPriceAED * 100);
return margin;
}
// ─── Main ─────────────────────────────────────────────────────────────────────
async function main() {
const opts = parseArgs();
if (!opts.productId || !opts.sellPrice) {
console.error('❌ Missing required args: --product <cj_product_id> --price <sell_price_aed>');
console.error(' Example: node launch.js --product CJA123456789 --price 149.99');
process.exit(1);
}
const { productId, sellPrice, category, dryRun } = opts;
const status = opts.status || 'draft';
const imageCount = opts.imageCount || 4;
const wooConfig = JSON.parse(fs.readFileSync(WOO_CONFIG_PATH, 'utf8'));
console.log(`\n🚀 Product Launcher`);
console.log(` CJ Product ID : productId`);
console.log(` Sell Price : sellPrice AED`);
if (category) console.log(` Category : category`);
if (dryRun) console.log(` Mode : DRY RUN (no WooCommerce writes)`);
// ── Step 1: Fetch from CJ ──────────────────────────────────────────────────
console.log(`\n📦 Fetching product from CJ Dropshipping...`);
const { product, variants } = await fetchProduct(productId);
const title = product.productNameEn || product.productName || 'Untitled Product';
const description = product.description || product.productNameEn || '';
const cjPriceUSD = parseFloat(product.sellPrice || product.productPrice || 0);
// Collect image URLs
const imageUrls = [];
if (product.productImage) imageUrls.push(product.productImage);
if (Array.isArray(product.productImageSet)) {
product.productImageSet.forEach(img => {
const u = typeof img === 'string' ? img : img.imageUrl || img.url || img;
if (u && !imageUrls.includes(u)) imageUrls.push(u);
});
}
console.log(` ✅ "title"`);
console.log(` CJ Price : $cjPriceUSD USD`);
console.log(` Images found : imageUrls.length`);
console.log(` Variants : variants.length`);
// ── Step 2: Margin Check ───────────────────────────────────────────────────
const margin = calcMargin(sellPrice, cjPriceUSD);
const marginStr = margin.toFixed(1);
console.log(`\n💰 Margin: marginStr%`);
if (margin < 30) {
console.warn(` ⚠️ WARNING: Margin below 30%! Consider raising price or sourcing cheaper.`);
} else {
console.log(` ✅ Margin looks good.`);
}
if (dryRun) {
console.log('\n✅ Dry run complete. No products created.');
console.log({ title, cjPriceUSD, sellPrice, margin: marginStr, variants: variants.length, images: imageUrls.length });
return;
}
// ── Step 3: Download Images ────────────────────────────────────────────────
console.log(`\n🖼 Downloading images...`);
const localPaths = imageUrls.length > 0 ? await downloadImages(imageUrls.slice(0, imageCount), productId) : [];
console.log(` Downloaded localPaths.length images.`);
// ── Step 4: Upload to WooCommerce ──────────────────────────────────────────
console.log(`\n☁️ Uploading images to WooCommerce...`);
const uploadedImages = localPaths.length > 0 ? await uploadImages(localPaths, wooConfig) : [];
console.log(` Uploaded uploadedImages.length images.`);
// ── Step 5: Create Product ─────────────────────────────────────────────────
console.log(`\n🛒 Creating WooCommerce draft product...`);
const wooProduct = await createProduct({
name: title,
description,
sellPrice,
images: uploadedImages,
variants,
cjProductId: productId,
category,
}, wooConfig);
const productUrl = `wooConfig.url/?p=wooProduct.id`;
const adminUrl = `wooConfig.url/wp-admin/post.php?post=wooProduct.id&action=edit`;
// ── Output ─────────────────────────────────────────────────────────────────
console.log(`\n✅ Product created!`);
console.log(` WooCommerce ID : wooProduct.id`);
console.log(` Margin : marginStr%`);
console.log(` Admin URL : adminUrl`);
console.log(` Product URL : wooProduct.permalink || productUrl`);
console.log();
// Machine-readable output for pipeline use
const result = {
wooProductId: wooProduct.id,
margin: marginStr,
productUrl: wooProduct.permalink || productUrl,
adminUrl,
title,
cjProductId: productId,
sellPriceAED: sellPrice,
cjPriceUSD,
};
process.stdout.write('\n__RESULT__\n' + JSON.stringify(result, null, 2) + '\n');
}
main().catch(err => {
console.error(`\n❌ Error: err.message`);
if (process.env.DEBUG) console.error(err.stack);
process.exit(1);
});
FILE:scripts/woo-create.js
#!/usr/bin/env node
/**
* woo-create.js — Upload images & create WooCommerce draft product
*/
const axios = require('axios');
const fs = require('fs');
const path = require('path');
const FormData = require('form-data');
const WOO_CONFIG_PATH = process.env.WOO_API_PATH || require('os').homedir() + '/woo-api.json';
function loadConfig() {
return JSON.parse(fs.readFileSync(WOO_CONFIG_PATH, 'utf8'));
}
function wooAuth(config) {
return {
username: config.consumerKey,
password: config.consumerSecret,
};
}
/**
* Upload a local image file to WooCommerce media library.
* Returns the WP media object with { id, source_url }
*/
async function uploadImage(localPath, config) {
const form = new FormData();
form.append('file', fs.createReadStream(localPath), path.basename(localPath));
const res = await axios.post(`config.url/wp-json/wp/v2/media`, form, {
auth: wooAuth(config),
headers: form.getHeaders(),
maxContentLength: Infinity,
maxBodyLength: Infinity,
});
return { id: res.data.id, src: res.data.source_url };
}
/**
* Upload multiple images, return array of { id, src }
*/
async function uploadImages(localPaths, config) {
const results = [];
for (const p of localPaths) {
try {
console.log(` Uploading path.basename(p)...`);
const img = await uploadImage(p, config);
results.push(img);
} catch (e) {
console.warn(` ⚠ Failed to upload p: e.message`);
}
}
return results;
}
/**
* Create a WooCommerce draft product.
* @param {Object} opts
* @param {string} opts.name
* @param {string} opts.description
* @param {string} opts.sellPrice — AED price string
* @param {Array} opts.images — [{ id, src }]
* @param {Array} opts.variants — CJ variant objects (optional)
* @param {string} opts.cjProductId
* @param {string} opts.category — optional category name/id
* @param {Object} config — woo config
*/
async function createProduct(opts, config) {
const { name, description, sellPrice, images, variants, cjProductId, category } = opts;
const hasVariants = variants && variants.length > 1;
const body = {
name,
status: 'draft',
description,
regular_price: String(sellPrice),
images: images.map(i => ({ id: i.id, src: i.src })),
meta_data: [{ key: '_cj_product_id', value: cjProductId }],
type: hasVariants ? 'variable' : 'simple',
};
if (category) {
// Accept numeric id or name
if (!isNaN(category)) {
body.categories = [{ id: parseInt(category) }];
} else {
// Try to find or create category
const catId = await findOrCreateCategory(category, config);
if (catId) body.categories = [{ id: catId }];
}
}
// Create the product
const res = await axios.post(`config.url/wp-json/wc/v3/products`, body, {
auth: wooAuth(config),
});
const product = res.data;
// Add variations if variable product
if (hasVariants) {
await createVariations(product.id, variants, sellPrice, config);
}
return product;
}
async function findOrCreateCategory(name, config) {
try {
const res = await axios.get(`config.url/wp-json/wc/v3/products/categories`, {
auth: wooAuth(config),
params: { search: name, per_page: 5 },
});
if (res.data && res.data.length > 0) return res.data[0].id;
// Create it
const created = await axios.post(`config.url/wp-json/wc/v3/products/categories`, { name }, {
auth: wooAuth(config),
});
return created.data.id;
} catch (e) {
console.warn(` ⚠ Could not resolve category "name": e.message`);
return null;
}
}
async function createVariations(productId, variants, sellPrice, config) {
for (const v of variants.slice(0, 100)) {
try {
const attrs = [];
if (v.variantName) attrs.push({ name: 'Variant', option: v.variantName });
await axios.post(`config.url/wp-json/wc/v3/products/productId/variations`, {
regular_price: String(sellPrice),
sku: v.variantSku || v.vid || '',
attributes: attrs,
meta_data: [
{ key: '_cj_variant_id', value: v.vid || v.variantId || '' },
{ key: '_cj_variant_sku', value: v.variantSku || '' },
],
}, { auth: wooAuth(config) });
} catch (e) {
console.warn(` ⚠ Failed to create variation: e.message`);
}
}
}
module.exports = { uploadImages, createProduct };
Generate and stitch short videos via Google Veo 3.x using the Gemini API (google-genai). Use when you need to create video clips from prompts (ads, UGC-style...
---
name: skill-veo3-video-gen
description: Generate and stitch short videos via Google Veo 3.x using the Gemini API (google-genai). Use when you need to create video clips from prompts (ads, UGC-style clips, product demos) and want a reproducible CLI workflow (generate, poll, download MP4, optionally stitch multiple segments).
---
# Veo 3 Video Generation (Gemini API)
Use the bundled script to generate an MP4 from a text prompt.
## Generate (text → video)
```bash
uv run {baseDir}/scripts/generate_video.py \
--prompt "A close up of ..." \
--filename "out.mp4" \
--model "veo-3.1-generate-preview" \
--aspect-ratio "9:16" \
--poll-seconds 10
```
## Generate a longer video by stitching segments
Veo commonly outputs ~8s clips per request. Use `--segments` to generate multiple clips and concatenate them with ffmpeg.
**Important:** This skill sends **one prompt per segment** (one Veo request per segment). Use `--base-style` to keep style consistent across segments.
```bash
uv run {baseDir}/scripts/generate_video.py \
--prompt "Same scene, consistent style..." \
--filename "out-24s.mp4" \
--model "veo-3.1-generate-preview" \
--aspect-ratio "9:16" \
--segments 3 \
--segment-style continuation
```
Options:
- `--base-style "..."`: prepended to every segment prompt (recommended).
- `--segment-prompt "..."` (repeatable): provide one prompt per segment (overrides `--prompt`).
- `--segment-style continuation` (default): appends continuity instructions per segment (only when using `--prompt`).
- `--segment-style same`: uses the exact same prompt for each segment (only when using `--prompt`).
- `--use-last-frame`: for segment >=2, extract previous segment last frame and pass it as `lastFrame` for continuity.
- `--emit-segment-media`: print `MEDIA:` for each segment as it finishes (useful for progress).
- `--keep-segments`: keep intermediate `*.segXX.mp4` files.
- `--reference-image path.jpg` (repeatable): guide generation with product/style references.
## Requirements
- `GEMINI_API_KEY` env var (or `--api-key`).
- `ffmpeg` on PATH when using `--segments > 1`.
## Troubleshooting
- 429/RESOURCE_EXHAUSTED: API key has no quota/billing for video.
- 503/UNAVAILABLE: model overloaded; retry later.
## Rate Limits & Fallback
- Veo 3.1 has per-minute quota — on 429, wait 60s and retry
- Fallback to Runway Gen4 Turbo when rate-limited (use skill-runway-video-gen)
- Key location: GEMINI_API_KEY env var (second key in openclaw.json)
## Rate Limits & Fallback
- Veo 3.1 has per-minute quota — on 429, wait 60s and retry
- Fallback to Runway Gen4 Turbo when rate-limited (use skill-runway-video-gen)
- Key location: `GEMINI_API_KEY` env var
FILE:_meta.json
{
"ownerId": "kn73qsnm7x84wphe31p1kr3tm98023y2",
"slug": "skill-veo3-video-gen",
"version": "1.0.0",
"publishedAt": 1769609324187,
"name": "skill-veo3-video-gen"
}
FILE:scripts/generate_video.py
#!/usr/bin/env python3
# /// script
# requires-python = ">=3.10"
# dependencies = [
# "google-genai>=1.0.0",
# "pillow>=10.0.0",
# ]
# ///
"""Generate videos using Google Veo 3.x via Gemini API (google-genai).
Supports multi-segment generation + automatic concatenation, with per-segment prompting and optional continuity via the previous segment's last frame.
Usage:
uv run generate_video.py \
--prompt "..." \
--filename "out.mp4" \
[--model veo-3.1-generate-preview] \
[--aspect-ratio 16:9|9:16|1:1] \
[--segments 1] \
[--poll-seconds 10] \
[--timeout-seconds 900] \
[--api-key KEY]
Notes:
- Requires GEMINI_API_KEY if --api-key not provided.
- Veo 3.x commonly generates ~8s clips per request; use --segments to build longer videos.
- If --segments > 1, requires ffmpeg on PATH to concatenate the clips.
- Prints a MEDIA: line for Clawdbot to auto-attach on supported providers.
"""
from __future__ import annotations
import argparse
import os
import shlex
import subprocess
import sys
import time
from pathlib import Path
def get_api_key(provided_key: str | None) -> str | None:
if provided_key:
return provided_key
return os.environ.get("GEMINI_API_KEY")
def require_bin(name: str) -> None:
if subprocess.run(["bash", "-lc", f"command -v {shlex.quote(name)}"], capture_output=True).returncode != 0:
raise RuntimeError(f"Required binary not found on PATH: {name}")
def extract_last_frame_png(video_path: Path, out_png: Path) -> Path:
"""Extract the last frame of a video to a PNG using ffmpeg."""
require_bin("ffmpeg")
out_png.parent.mkdir(parents=True, exist_ok=True)
# Use -sseof to seek from the end; grab a single frame.
cmd = [
"ffmpeg",
"-y",
"-sseof",
"-0.05",
"-i",
str(video_path),
"-frames:v",
"1",
"-q:v",
"2",
str(out_png),
]
p = subprocess.run(cmd, capture_output=True, text=True)
if p.returncode != 0:
raise RuntimeError(f"ffmpeg last-frame extract failed: {p.stderr[-2000:]}")
return out_png
def poll_until_done(client, operation, poll_seconds: int, timeout_seconds: int):
started = time.time()
while not getattr(operation, "done", False):
elapsed = int(time.time() - started)
if elapsed > timeout_seconds:
raise TimeoutError(f"Timed out after {timeout_seconds}s waiting for video generation")
print("Waiting for video generation to complete…")
time.sleep(poll_seconds)
operation = client.operations.get(operation)
return operation
def extract_first_video_handle(client, operation, wait_seconds: int = 60, poll_seconds: int = 5):
"""Extract the first video handle from a finished operation.
In practice, some operations may flip to done=True before the response payload is
populated (eventual consistency). We retry `operations.get()` briefly. Also surface
operation.error when present.
"""
deadline = time.time() + wait_seconds
while True:
# Surface explicit operation error if present.
op_err = getattr(operation, "error", None) or getattr(operation, "Error", None)
if op_err:
raise RuntimeError(f"Operation error: {op_err}")
resp = getattr(operation, "response", None) or getattr(operation, "Response", None)
if resp is not None:
break
if time.time() >= deadline:
raise RuntimeError("Operation finished but no response found")
# Retry fetch
time.sleep(poll_seconds)
try:
operation = client.operations.get(operation)
except Exception:
# If polling fails intermittently, keep trying until deadline.
pass
vids = (
getattr(resp, "generated_videos", None)
or getattr(resp, "generatedVideos", None)
or getattr(resp, "GeneratedVideos", None)
)
if not vids:
raise RuntimeError(f"No generated_videos in response (resp={type(resp).__name__})")
first = vids[0]
video_file = getattr(first, "video", None)
if video_file is None:
raise RuntimeError("generated video file handle missing")
return video_file
def save_video_file(client, video_file, out_path: Path) -> None:
out_path.parent.mkdir(parents=True, exist_ok=True)
# Some SDK variants need an explicit download call before save().
try:
client.files.download(file=video_file)
except Exception:
pass
video_file.save(str(out_path))
def ffmpeg_concat(inputs: list[Path], out_path: Path) -> None:
require_bin("ffmpeg")
out_path.parent.mkdir(parents=True, exist_ok=True)
# Create concat list file.
lst = out_path.with_suffix(out_path.suffix + ".concat.txt")
lines = [f"file '{p.as_posix()}'" for p in inputs]
lst.write_text("\n".join(lines) + "\n", encoding="utf-8")
cmd = [
"ffmpeg",
"-y",
"-f",
"concat",
"-safe",
"0",
"-i",
str(lst),
"-c",
"copy",
str(out_path),
]
p = subprocess.run(cmd, capture_output=True, text=True)
if p.returncode != 0:
# Fallback: re-encode (more compatible, slower)
print("ffmpeg stream copy concat failed; falling back to re-encode…", file=sys.stderr)
cmd2 = [
"ffmpeg",
"-y",
"-f",
"concat",
"-safe",
"0",
"-i",
str(lst),
"-c:v",
"libx264",
"-preset",
"veryfast",
"-crf",
"18",
"-c:a",
"aac",
"-b:a",
"192k",
str(out_path),
]
p2 = subprocess.run(cmd2, capture_output=True, text=True)
if p2.returncode != 0:
raise RuntimeError(
"ffmpeg concat failed.\n"
f"copy stderr:\n{p.stderr[-2000:]}\n\n"
f"reencode stderr:\n{p2.stderr[-2000:]}"
)
def main() -> None:
parser = argparse.ArgumentParser(description="Generate videos via Veo 3.x (Gemini API)")
parser.add_argument("--prompt", "-p", required=False, help="Text prompt (used for all segments unless --segment-prompt is provided)")
parser.add_argument(
"--segment-prompt",
action="append",
default=[],
help="Per-segment prompt. Repeat this flag for each segment (e.g. 3 times). Overrides --prompt.",
)
parser.add_argument("--filename", "-f", required=True, help="Output .mp4 path")
parser.add_argument(
"--model",
"-m",
default="veo-3.1-generate-preview",
help="Model id (e.g., veo-3.1-generate-preview)",
)
parser.add_argument(
"--aspect-ratio",
default=None,
choices=["16:9", "9:16", "1:1"],
help="Optional aspect ratio (if supported by the model/config)",
)
parser.add_argument(
"--segments",
type=int,
default=1,
help="Number of segments to generate and concatenate (use for longer videos)",
)
parser.add_argument(
"--reference-image",
action="append",
default=[],
help="Path to a reference image to guide generation (repeatable).",
)
parser.add_argument(
"--reference-type",
default="asset",
choices=["asset", "style", "subject"],
help="Reference type hint passed to the API (default: asset).",
)
parser.add_argument(
"--generate-audio",
default=True,
action=argparse.BooleanOptionalAction,
help="Whether to generate audio (default: true).",
)
parser.add_argument(
"--base-style",
default=None,
help="Style/system prompt prepended to every segment prompt (recommended for consistency).",
)
parser.add_argument(
"--segment-style",
choices=["same", "continuation"],
default="continuation",
help="How to prompt segments: same prompt each time, or continuation prompts.",
)
parser.add_argument(
"--use-last-frame",
action="store_true",
help="For segment >=2 (single run), extract the previous segment's last frame and pass it as lastFrame to encourage continuity.",
)
parser.add_argument(
"--last-frame-image",
default=None,
help="Explicit last frame image path to pass as lastFrame (useful when running segments as separate commands).",
)
parser.add_argument(
"--emit-segment-media",
action="store_true",
help="Print MEDIA: lines for each segment as they finish (useful for progress updates).",
)
parser.add_argument(
"--poll-seconds",
type=int,
default=10,
help="Polling interval seconds",
)
parser.add_argument(
"--timeout-seconds",
type=int,
default=900,
help="Max time to wait for each segment generation to finish",
)
parser.add_argument(
"--api-key",
"-k",
help="API key (overrides GEMINI_API_KEY)",
)
parser.add_argument(
"--keep-segments",
action="store_true",
help="Keep intermediate segment mp4 files",
)
args = parser.parse_args()
api_key = get_api_key(args.api_key)
if not api_key:
print("Error: No API key provided.", file=sys.stderr)
print("Set GEMINI_API_KEY or pass --api-key", file=sys.stderr)
sys.exit(1)
if args.segments < 1:
print("Error: --segments must be >= 1", file=sys.stderr)
sys.exit(1)
# Determine per-segment prompts
seg_prompts: list[str] = []
if args.segment_prompt:
seg_prompts = [p for p in args.segment_prompt if p and p.strip()]
if not seg_prompts:
print("Error: --segment-prompt provided but empty", file=sys.stderr)
sys.exit(1)
if args.segments != 1 and args.segments != len(seg_prompts):
print(
f"Error: --segments ({args.segments}) does not match number of --segment-prompt ({len(seg_prompts)})",
file=sys.stderr,
)
sys.exit(1)
args.segments = len(seg_prompts)
else:
if not args.prompt or not args.prompt.strip():
print("Error: provide --prompt or one/more --segment-prompt", file=sys.stderr)
sys.exit(1)
seg_prompts = [args.prompt]
# Lazy import after validation
from google import genai
from google.genai import types
import mimetypes
client = genai.Client(api_key=api_key)
out = Path(args.filename)
out.parent.mkdir(parents=True, exist_ok=True)
reference_images = []
for p in args.reference_image or []:
try:
img_path = Path(p)
b = img_path.read_bytes()
mt, _ = mimetypes.guess_type(str(img_path))
mt = mt or "image/jpeg"
img = types.Image(imageBytes=b, mimeType=mt)
except Exception as e:
print(f"Error loading reference image {p}: {e}", file=sys.stderr)
sys.exit(1)
reference_images.append(
types.VideoGenerationReferenceImage(
image=img,
referenceType=args.reference_type,
)
)
# Build base config (use canonical field names)
base_cfg_kwargs = {}
if args.aspect_ratio:
base_cfg_kwargs["aspectRatio"] = args.aspect_ratio
if reference_images:
base_cfg_kwargs["referenceImages"] = reference_images
# Optional explicit lastFrame image
if args.last_frame_image:
import mimetypes
p = Path(args.last_frame_image)
b = p.read_bytes()
mt, _ = mimetypes.guess_type(str(p))
mt = mt or "image/png"
base_cfg_kwargs["lastFrame"] = types.Image(imageBytes=b, mimeType=mt)
# Empirically, some Veo requests reject combining lastFrame + referenceImages.
# To maximize compatibility, drop referenceImages when lastFrame is provided.
if "referenceImages" in base_cfg_kwargs:
print(
"Note: lastFrame provided; dropping referenceImages for compatibility.",
file=sys.stderr,
)
base_cfg_kwargs.pop("referenceImages", None)
# Note: As of google-genai SDK 1.60.0, generate_audio is NOT supported on the Gemini API
# for Veo video generation (SDK raises ValueError). We keep the CLI flag for forward-compat,
# but do not send it today.
if args.generate_audio is not None and args.generate_audio is not True:
print("Note: --no-generate-audio requested, but Gemini API does not currently support the generateAudio flag; proceeding.")
seg_paths: list[Path] = []
base = out.with_suffix("")
for i in range(1, args.segments + 1):
# Segment prompt strategy:
# - Always send exactly ONE segment prompt per Veo request.
# - If --segment-prompt is provided, use that prompt for this segment.
# - Otherwise, use --prompt for all segments and optionally append continuation guidance.
raw_seg_prompt = seg_prompts[min(i - 1, len(seg_prompts) - 1)]
if not args.segment_prompt and args.segments > 1 and args.segment_style != "same":
raw_seg_prompt = raw_seg_prompt + (
f"\n\nThis is segment {i} of {args.segments}. Continue seamlessly from the previous segment with consistent characters, lighting, camera style, and setting."
)
if args.base_style:
seg_prompt = args.base_style.strip() + "\n\n" + raw_seg_prompt.strip()
else:
seg_prompt = raw_seg_prompt.strip()
seg_out = base.parent / f"{base.name}.seg{i:02d}.mp4"
# Build per-segment config, optionally with lastFrame.
seg_cfg_kwargs = dict(base_cfg_kwargs)
if args.use_last_frame and i >= 2:
try:
last_png = base.parent / f"{base.name}.seg{i-1:02d}.last.png"
extract_last_frame_png(seg_paths[-1], last_png)
b = last_png.read_bytes()
import mimetypes
mt, _ = mimetypes.guess_type(str(last_png))
mt = mt or "image/png"
seg_cfg_kwargs["lastFrame"] = types.Image(imageBytes=b, mimeType=mt)
# Empirically, some Veo requests reject combining lastFrame + referenceImages.
# Drop referenceImages for segment>=2 to improve success rate.
seg_cfg_kwargs.pop("referenceImages", None)
except Exception as e:
print(f"Warning: failed to attach lastFrame for segment {i}: {e}", file=sys.stderr)
seg_config = types.GenerateVideosConfig(**seg_cfg_kwargs) if seg_cfg_kwargs else None
print(f"Starting video generation segment {i}/{args.segments} (model={args.model})…")
operation = client.models.generate_videos(
model=args.model,
prompt=seg_prompt,
config=seg_config,
)
try:
operation = poll_until_done(client, operation, args.poll_seconds, args.timeout_seconds)
except Exception as e:
print(f"Error while waiting for segment {i}: {e}", file=sys.stderr)
sys.exit(2)
try:
video_file = extract_first_video_handle(client, operation)
save_video_file(client, video_file, seg_out)
except Exception as e:
print(f"Error downloading/saving segment {i}: {e}", file=sys.stderr)
sys.exit(3)
seg_full = seg_out.resolve()
print(f"Saved segment {i}: {seg_full}")
if args.emit_segment_media:
print(f"MEDIA: {seg_full}")
seg_paths.append(seg_out)
if args.segments == 1:
# Rename seg file to requested filename
seg_paths[0].replace(out)
full = out.resolve()
print(f"Generated video saved: {full}")
print(f"MEDIA: {full}")
return
# Concatenate segments
try:
ffmpeg_concat(seg_paths, out)
except Exception as e:
print(f"Error concatenating segments: {e}", file=sys.stderr)
sys.exit(4)
full = out.resolve()
print(f"Generated stitched video saved: {full}")
print(f"MEDIA: {full}")
if not args.keep_segments:
for p in seg_paths:
try:
p.unlink(missing_ok=True)
except Exception:
pass
if __name__ == "__main__":
main()
End-to-end TikTok ad video pipeline. Product script → Veo base video → animated caption overlay → audio mix → final MP4. One command, full automation.
---
name: skill-tiktok-video-pipeline
version: 2.0.0
# Updated: pipeline.py wired to overlay v3 engine with --audio and --slowmo support
description: End-to-end TikTok ad video pipeline. Product script → Veo base video → animated caption overlay → audio mix → final MP4. One command, full automation.
metadata:
openclaw:
requires: { bins: ["uv", "ffmpeg", "node"] }
---
# skill-tiktok-video-pipeline v2
Full end-to-end pipeline for TikTok product ads. Takes a `product_id` + `script_text` and outputs a publish-ready vertical short-form video with captions, optional logo watermark, and background music.
## Architecture
```
script_text + product_id
│
▼
Step 1: Veo 3 base video generation (9:16, ~8s)
│
▼
Step 2: Caption overlay + logo watermark
└── tiktok_overlay_engine_v3.py (ffmpeg drawtext)
│
▼
Step 3: Background audio mix (20% volume, ffmpeg amix)
│
▼
output/tiktok/<product_id>_<lang>_final.mp4
```
## Requirements
- `GEMINI_API_KEY` env var (for Veo generation)
- `ffmpeg` on PATH
- `uv` on PATH (for Python scripts)
- `veo3-video-gen` skill installed at `skills/veo3-video-gen/`
## Usage
```bash
node scripts/generate.js \
--product-id rain_cloud \
--script-text "Stop dry air!|Ultrasonic mist|Whisper-quiet|Get yours today" \
--lang EN
```
### With logo and custom audio
```bash
node scripts/generate.js \
--product-id hydro_bottle \
--script-text "Hydrogen water|Boosts energy|Pure & clean|Shop now" \
--lang EN \
--logo /path/to/brand_logo.png \
--audio /path/to/bgm.mp3
```
### Arabic (AR) captions
```bash
node scripts/generate.js \
--product-id mini_cam \
--script-text "صوّر كل لحظة|دقة عالية|خفيف وصغير|اطلب الآن" \
--lang AR
```
### Dry-run (no API calls, generates dummy video for testing overlay)
```bash
node scripts/generate.js \
--product-id test \
--script-text "Line 1|Line 2|Line 3" \
--dry-run
```
## Inputs
| Argument | Required | Default | Description |
|---|---|---|---|
| `--product-id` | ✅ | — | Product identifier (used in output filename) |
| `--script-text` | ✅ | — | Caption lines separated by `\|` |
| `--lang` | ❌ | `EN` | Language: `EN` or `AR` |
| `--logo` | ❌ | none | Path to logo PNG for watermark (top-right) |
| `--audio` | ❌ | `assets/bgm_default.mp3` | Background music path |
| `--veo-model` | ❌ | `veo-3.1-generate-preview` | Veo model to use |
| `--prompt` | ❌ | auto | Custom Veo generation prompt |
| `--segments` | ❌ | `1` | Number of Veo segments to generate & stitch |
| `--dry-run` | ❌ | false | Skip Veo API call; use dummy black video |
## Outputs
| File | Description |
|---|---|
| `output/tiktok/<product_id>_<lang>_final.mp4` | Final publish-ready TikTok video |
## Scripts
| Script | Description |
|---|---|
| `scripts/generate.js` | Main Node.js orchestrator |
| `scripts/tiktok_overlay_engine_v3.py` | Python/ffmpeg caption overlay engine |
## Caption Format
Captions are split by `|` and timed evenly across the video duration.
**Example:** `"Hook line!|Feature 1|Feature 2|CTA here"` → 4 pills, each shown for ~2s on an 8s video.
Pill style: dark semi-transparent box, white text, centered at 75% height.
## Default Audio
Place a royalty-free BGM file at `assets/bgm_default.mp3` in this skill folder to auto-mix audio in all runs. If no audio is found, the video is output without BGM.
## Pipeline Steps Detail
```
Step 1 Veo 3 generates a 9:16 base MP4 ~60–120s
Step 2 Python overlays timed caption pills ~5s
Step 3 ffmpeg mixes BGM at 20% volume ~5s
─────────────────────────────────────────────────────────
Output Final branded MP4 ready to post
```
## pipeline.py (v2.0.0 — Python orchestrator)
Direct Python pipeline wired to overlay engine via subprocess.
```bash
uv run scripts/pipeline.py \
--product rain_cloud \
--image product.jpg \
--output final.mp4 \
--audio /path/to/music.mp3 \
--slowmo
```
### New flags (v2.0.0)
| Flag | Default | Description |
|---|---|---|
| `--audio` | `$DEFAULT_AUDIO` env or bundled Hyperfun.mp3 | Audio file passed to overlay step |
| `--slowmo` | false | Apply 0.83x speed → fills ~12s. Overrides `--extend-to` auto-stretch |
### Environment Variables
| Var | Default | Description |
|---|---|---|
| `DEFAULT_AUDIO` | workspace root `audio_Hyperfun.mp3` | Default audio if `--audio` not set |
FILE:_meta.json
{
"name": "skill-tiktok-video-pipeline",
"version": "2.0.0",
"description": "End-to-end TikTok ad video pipeline. Product image → Runway/Veo video → slowmo → caption overlay → final MP4.",
"author": "Zero2Ai-hub",
"license": "MIT"
}
FILE:assets/README.md
# Assets\nPlace bgm_default.mp3 here for automatic background music mixing.
FILE:config/products.json
{
"rain_cloud": {"name": "Rain Cloud Humidifier", "accent": [140, 180, 255], "cta_price": "129 AED"},
"hydro_bottle": {"name": "Hydrogen Water Bottle", "accent": [0, 200, 255], "cta_price": "149 AED"},
"mini_cam": {"name": "Mini Clip Camera", "accent": [255, 100, 60], "cta_price": "89 AED"}
}
FILE:scripts/generate.js
#!/usr/bin/env node
/**
* TikTok Video Pipeline v2 — Main Orchestrator
* =============================================
* End-to-end: script_text → Veo base video → caption overlay → audio mix → final MP4
*
* Usage:
* node generate.js \
* --product-id rain_cloud \
* --script-text "Hook|Feature 1|Feature 2|Buy now" \
* [--lang EN|AR] \
* [--logo /path/to/logo.png] \
* [--audio /path/to/bgm.mp3] \
* [--veo-model veo-3.1-generate-preview] \
* [--prompt "Cinematic mist floating..."] \
* [--segments 1] \
* [--dry-run]
*
* Env vars:
* GEMINI_API_KEY — required for Veo generation
*/
const { execSync, spawnSync } = require('child_process');
const path = require('path');
const fs = require('fs');
// ── Config ──────────────────────────────────────────────────────────────────
const WORKSPACE = path.resolve(__dirname, '../../..');
const SKILL_DIR = path.resolve(__dirname, '..');
const VEO_SCRIPT = path.resolve(WORKSPACE, 'skills/veo3-video-gen/scripts/generate_video.py');
const OVERLAY_SCRIPT = path.resolve(__dirname, 'tiktok_overlay_engine_v3.py');
const OUTPUT_DIR = path.resolve(WORKSPACE, 'output/tiktok');
const DEFAULT_AUDIO = path.resolve(SKILL_DIR, 'assets/bgm_default.mp3');
const DEFAULT_VEO_MODEL = 'veo-3.1-generate-preview';
// ── Args ─────────────────────────────────────────────────────────────────────
function parseArgs() {
const args = process.argv.slice(2);
const opts = {
productId: null,
scriptText: null,
lang: 'EN',
logo: null,
audio: null,
veoModel: DEFAULT_VEO_MODEL,
prompt: null,
segments: 1,
dryRun: false,
};
for (let i = 0; i < args.length; i++) {
switch (args[i]) {
case '--product-id': opts.productId = args[++i]; break;
case '--script-text': opts.scriptText = args[++i]; break;
case '--lang': opts.lang = args[++i].toUpperCase(); break;
case '--logo': opts.logo = args[++i]; break;
case '--audio': opts.audio = args[++i]; break;
case '--veo-model': opts.veoModel = args[++i]; break;
case '--prompt': opts.prompt = args[++i]; break;
case '--segments': opts.segments = parseInt(args[++i], 10); break;
case '--dry-run': opts.dryRun = true; break;
}
}
if (!opts.productId) { console.error('ERROR: --product-id required'); process.exit(1); }
if (!opts.scriptText) { console.error('ERROR: --script-text required'); process.exit(1); }
return opts;
}
// ── Helpers ──────────────────────────────────────────────────────────────────
function run(cmd, opts = {}) {
console.log(`\n▶ cmd\n`);
return spawnSync('bash', ['-lc', cmd], {
stdio: 'inherit',
env: { ...process.env },
...opts,
});
}
function ensureDir(p) {
fs.mkdirSync(p, { recursive: true });
}
// ── Steps ────────────────────────────────────────────────────────────────────
function step1_generateBaseVideo(opts, baseVideoPath) {
console.log('\n━━ Step 1: Generate base video via Veo ━━');
const prompt = opts.prompt ||
`Cinematic product showcase, 9:16 vertical, smooth motion, professional lighting, no text`;
if (!fs.existsSync(VEO_SCRIPT)) {
throw new Error(`Veo script not found at: VEO_SCRIPT`);
}
const cmd = [
'uv run', JSON.stringify(VEO_SCRIPT),
'--prompt', JSON.stringify(prompt),
'--filename', JSON.stringify(baseVideoPath),
'--model', opts.veoModel,
'--aspect-ratio', '9:16',
'--segments', opts.segments,
'--poll-seconds', '10',
].join(' ');
if (opts.dryRun) {
console.log('[dry-run] Would run:', cmd);
// Create dummy mp4 for dry-run testing
execSync(`ffmpeg -y -f lavfi -i color=c=black:s=1080x1920:d=8 -c:v libx264 JSON.stringify(baseVideoPath) 2>/dev/null`);
return;
}
const r = run(cmd);
if (r.status !== 0) throw new Error('Veo generation failed');
}
function step2_overlayCaption(opts, baseVideoPath, overlayedPath) {
console.log('\n━━ Step 2: Caption + logo overlay ━━');
const logoFlag = opts.logo && fs.existsSync(opts.logo)
? `--logo JSON.stringify(opts.logo)`
: '';
const cmd = [
'uv run', JSON.stringify(OVERLAY_SCRIPT),
'--input', JSON.stringify(baseVideoPath),
'--output', JSON.stringify(overlayedPath),
'--captions', JSON.stringify(opts.scriptText),
'--lang', opts.lang,
logoFlag,
].filter(Boolean).join(' ');
const r = run(cmd);
if (r.status !== 0) throw new Error('Overlay engine failed');
}
function step3_mixAudio(opts, overlayedPath, finalPath) {
console.log('\n━━ Step 3: Mix background audio at 20% ━━');
// Use provided audio, default asset, or skip
const audioSrc = opts.audio || (fs.existsSync(DEFAULT_AUDIO) ? DEFAULT_AUDIO : null);
if (!audioSrc) {
console.log('No background audio found — copying overlayed video as final (no bgm).');
fs.copyFileSync(overlayedPath, finalPath);
return;
}
// amix: original audio at 100% + background at 20%
const cmd = [
'ffmpeg -y',
`-i JSON.stringify(overlayedPath)`,
`-i JSON.stringify(audioSrc)`,
'-filter_complex',
'"[0:a]volume=1.0[va];[1:a]volume=0.20[bgm];[va][bgm]amix=inputs=2:duration=first:dropout_transition=2[aout]"',
'-map 0:v',
'-map "[aout]"',
'-c:v copy',
'-c:a aac -b:a 192k',
'-shortest',
JSON.stringify(finalPath),
].join(' ');
const r = run(cmd);
if (r.status !== 0) {
console.warn('Audio mix failed — falling back to overlayed video without bgm');
fs.copyFileSync(overlayedPath, finalPath);
}
}
// ── Main ─────────────────────────────────────────────────────────────────────
async function main() {
const opts = parseArgs();
ensureDir(OUTPUT_DIR);
const tmpDir = path.join(OUTPUT_DIR, `.tmp_opts.productId_Date.now()`);
ensureDir(tmpDir);
const baseVideoPath = path.join(tmpDir, 'base.mp4');
const overlayedPath = path.join(tmpDir, 'overlayed.mp4');
const finalPath = path.join(OUTPUT_DIR, `opts.productId_opts.lang_final.mp4`);
console.log(`\n🎬 TikTok Pipeline v2`);
console.log(` Product : opts.productId`);
console.log(` Lang : opts.lang`);
console.log(` Output : finalPath`);
console.log(` Dry-run : opts.dryRun`);
try {
step1_generateBaseVideo(opts, baseVideoPath);
step2_overlayCaption(opts, baseVideoPath, overlayedPath);
step3_mixAudio(opts, overlayedPath, finalPath);
// Cleanup tmp
fs.rmSync(tmpDir, { recursive: true, force: true });
console.log(`\n✅ Pipeline complete!`);
console.log(`MEDIA:finalPath`);
} catch (err) {
console.error(`\n❌ Pipeline failed: err.message`);
process.exit(1);
}
}
main();
FILE:scripts/pipeline.py
#!/usr/bin/env python3
# /// script
# requires-python = ">=3.10"
# dependencies = ["moviepy>=2.0"]
# ///
"""
TikTok Video Pipeline v2.0.0 — full orchestrator
Product image → base video (Runway/Veo) → [slowmo] → caption overlay → final MP4
"""
import argparse
import json
import os
import subprocess
import sys
import tempfile
import time
SKILLS_BASE = os.path.join(os.path.dirname(__file__), "..", "..", "..")
RUNWAY_SCRIPT = os.path.join(SKILLS_BASE, "skill-runway-video-gen", "scripts", "generate_video.py")
OVERLAY_SCRIPT = os.path.join(SKILLS_BASE, "skill-tiktok-ads-video", "scripts", "overlay.py")
PRODUCTS_JSON = os.path.join(os.path.dirname(__file__), "..", "config", "products.json")
DEFAULT_AUDIO = os.environ.get(
"DEFAULT_AUDIO",
os.path.join(os.path.dirname(__file__), "..", "..", "..", "audio_Hyperfun.mp3"),
)
def load_products():
with open(PRODUCTS_JSON) as f:
return json.load(f)
def get_video_duration(path):
from moviepy import VideoFileClip
clip = VideoFileClip(path)
dur = clip.duration
clip.close()
return dur
def apply_slowmo(input_path, output_path, factor=0.83):
"""Apply slowmo at fixed factor (e.g. 0.83x = ~12s from 10s base)."""
from moviepy import VideoFileClip
from moviepy import vfx
t0 = time.time()
print(f"[pipeline] ⏳ Applying slowmo at {factor}x speed factor ...")
clip = VideoFileClip(input_path)
slow = clip.with_effects([vfx.MultiplySpeed(factor)])
slow.write_videofile(output_path, fps=30, logger=None)
clip.close()
slow.close()
print(f"[pipeline] ✅ Slowmo done → {output_path} ({time.time()-t0:.1f}s)")
def apply_stretch(input_path, output_path, target_duration):
"""Stretch video to fill target_duration seconds."""
from moviepy import VideoFileClip
from moviepy import vfx
t0 = time.time()
print(f"[pipeline] ⏳ Stretching to {target_duration}s ...")
clip = VideoFileClip(input_path)
speed = clip.duration / target_duration
slow = clip.with_effects([vfx.MultiplySpeed(speed)])
slow.write_videofile(output_path, fps=30, logger=None)
clip.close()
slow.close()
print(f"[pipeline] ✅ Stretch done → {output_path} ({time.time()-t0:.1f}s)")
def run_runway(image, prompt, output, duration=10, ratio="720:1280"):
t0 = time.time()
print(f"[pipeline] ⏳ Step 1: Generating video via Runway Gen4 Turbo ...")
cmd = [
"uv", "run", RUNWAY_SCRIPT,
"--image", image,
"--prompt", prompt,
"--output", output,
"--duration", str(duration),
"--ratio", ratio,
]
result = subprocess.run(cmd, check=True)
print(f"[pipeline] ✅ Runway done ({time.time()-t0:.1f}s)")
return result.returncode == 0
def run_veo(image, prompt, output):
"""Try Veo via veo3-video-gen skill. Falls back to Runway on 429."""
veo_script = os.path.join(SKILLS_BASE, "veo3-video-gen", "scripts", "generate.py")
if not os.path.exists(veo_script):
print("[pipeline] Veo script not found, falling back to Runway ...")
return False
t0 = time.time()
print(f"[pipeline] ⏳ Step 1: Generating video via Veo ...")
cmd = ["uv", "run", veo_script, "--image", image, "--prompt", prompt, "--output", output]
result = subprocess.run(cmd)
if result.returncode == 0:
print(f"[pipeline] ✅ Veo done ({time.time()-t0:.1f}s)")
return True
print("[pipeline] Veo failed (429 or error) — falling back to Runway ...")
return False
def run_overlay(video, product, style, output, audio=None):
t0 = time.time()
print(f"[pipeline] ⏳ Step 3: Applying caption overlay ({style}) ...")
cmd = [
"uv", "run", "--with", "moviepy", "--with", "pillow",
OVERLAY_SCRIPT,
"--video", video,
"--product", product,
"--style", style,
"--output", output,
]
if audio:
cmd += ["--audio", audio]
subprocess.run(cmd, check=True)
print(f"[pipeline] ✅ Overlay done → {output} ({time.time()-t0:.1f}s)")
def main():
parser = argparse.ArgumentParser(description="TikTok Video Pipeline v2.0.0 — full end-to-end")
parser.add_argument("--product", required=True, choices=["rain_cloud", "hydro_bottle", "mini_cam"])
parser.add_argument("--image", required=True, help="Source product image path")
parser.add_argument("--output", required=True, help="Final output MP4 path")
parser.add_argument("--style", default="subtitle_talk", choices=["subtitle_talk", "phrase_slam", "random"])
parser.add_argument("--engine", default="auto", choices=["runway", "veo", "auto"])
parser.add_argument("--extend-to", type=float, default=12.0, dest="extend_to",
help="Target duration in seconds for auto-stretch (default: 12)")
parser.add_argument("--prompt", default="", help="Motion description for video generation")
parser.add_argument("--audio", default=None,
help=f"Audio file to mix into overlay (default: $DEFAULT_AUDIO env or bundled Hyperfun)")
parser.add_argument("--slowmo", action="store_true",
help="Apply 0.83x slowmo stretch to fill ~12s before overlay (uses moviepy)")
args = parser.parse_args()
# Resolve audio path
audio_path = args.audio or DEFAULT_AUDIO
if not os.path.exists(audio_path):
print(f"[pipeline] ⚠️ Audio file not found: {audio_path} — overlay will run without audio")
audio_path = None
products = load_products()
product = products[args.product]
pipeline_start = time.time()
print(f"\n[pipeline] === TikTok Video Pipeline v2.0.0 ===")
print(f"[pipeline] Product: {product['name']} | Style: {args.style} | Engine: {args.engine}")
print(f"[pipeline] Slowmo: {'yes (0.83x)' if args.slowmo else 'auto-stretch to ' + str(args.extend_to) + 's'}")
print(f"[pipeline] Audio: {audio_path or 'none'}\n")
with tempfile.TemporaryDirectory() as tmp:
base_video = os.path.join(tmp, "base.mp4")
processed_video = os.path.join(tmp, "processed.mp4")
# Step 1 — Generate base video
prompt = args.prompt or f"product in action, cinematic, smooth motion, {product['name']}"
if args.engine == "runway":
run_runway(args.image, prompt, base_video)
elif args.engine == "veo":
if not run_veo(args.image, prompt, base_video):
run_runway(args.image, prompt, base_video)
else: # auto
if not run_veo(args.image, prompt, base_video):
run_runway(args.image, prompt, base_video)
if not os.path.exists(base_video):
print("[pipeline] ERROR: Base video generation failed", file=sys.stderr)
sys.exit(1)
# Step 2 — Slowmo / Stretch
dur = get_video_duration(base_video)
print(f"[pipeline] Step 2: Base video duration = {dur:.1f}s")
if args.slowmo:
apply_slowmo(base_video, processed_video, factor=0.83)
caption_input = processed_video
elif args.extend_to > dur:
apply_stretch(base_video, processed_video, args.extend_to)
caption_input = processed_video
else:
print(f"[pipeline] No stretch needed ({dur:.1f}s >= {args.extend_to}s)")
caption_input = base_video
# Step 3 — Caption overlay
run_overlay(caption_input, args.product, args.style, args.output, audio=audio_path)
elapsed = time.time() - pipeline_start
print(f"\n[pipeline] ✅ Final video saved: {args.output} (total: {elapsed:.1f}s)")
if __name__ == "__main__":
main()
FILE:scripts/tiktok_overlay_engine_v3.py
#!/usr/bin/env python3
# /// script
# requires-python = ">=3.10"
# dependencies = [
# "pillow>=10.0.0",
# ]
# ///
"""
TikTok Overlay Engine v3
========================
Adds animated pill-style captions + optional logo watermark to an MP4 using ffmpeg.
Usage:
uv run tiktok_overlay_engine_v3.py \
--input base.mp4 \
--output overlayed.mp4 \
--captions "Hook line!|Feature 1|Feature 2|CTA here" \
[--logo /path/to/logo.png] \
[--lang EN|AR] \
[--font-size 52] \
[--pill-color "#000000AA"] \
[--text-color white]
Captions are split by | and timed evenly across the video duration.
"""
from __future__ import annotations
import argparse
import subprocess
import sys
import tempfile
from pathlib import Path
def get_video_duration(path: Path) -> float:
result = subprocess.run(
["ffprobe", "-v", "quiet", "-print_format", "json",
"-show_format", path],
capture_output=True, text=True
)
import json
data = json.loads(result.stdout)
return float(data["format"]["duration"])
def build_drawtext_filter(captions: list[str], duration: float, font_size: int,
text_color: str, pill_color: str, lang: str) -> str:
"""Build ffmpeg drawtext filter chain with timed captions."""
segment = duration / len(captions)
filters = []
# For AR we flip alignment; default LTR for EN
align_x = "w/2" if lang == "AR" else "w/2"
for i, caption in enumerate(captions):
start = i * segment
end = start + segment
# Escape special chars for ffmpeg
safe = (caption
.replace("'", "\\'")
.replace(":", "\\:")
.replace(",", "\\,"))
f = (
f"drawtext=text='{safe}'"
f":fontsize={font_size}"
f":fontcolor={text_color}"
f":x=(w-text_w)/2"
f":y=h*0.75"
f":box=1:boxcolor={pill_color}:boxborderw=20"
f":enable='between(t,{start:.2f},{end:.2f})'"
f":alpha='if(lt(t-{start:.2f},0.3),(t-{start:.2f})/0.3,if(gt(t,{end:.2f}-0.3),(({end:.2f}-t)/0.3),1))'"
)
filters.append(f)
return ",".join(filters)
def main() -> None:
parser = argparse.ArgumentParser(description="TikTok Overlay Engine v3")
parser.add_argument("--input", required=True, help="Input MP4 path")
parser.add_argument("--output", required=True, help="Output MP4 path")
parser.add_argument("--captions", required=True,
help="Caption lines separated by |")
parser.add_argument("--logo", default=None, help="Optional logo PNG path")
parser.add_argument("--lang", default="EN", choices=["EN", "AR"],
help="Language (affects text direction)")
parser.add_argument("--font-size", type=int, default=52)
parser.add_argument("--pill-color", default="[email protected]",
help="Box/pill background color (ffmpeg color format)")
parser.add_argument("--text-color", default="white")
args = parser.parse_args()
inp = Path(args.input)
out = Path(args.output)
out.parent.mkdir(parents=True, exist_ok=True)
if not inp.exists():
print(f"ERROR: input file not found: {inp}", file=sys.stderr)
sys.exit(1)
captions = [c.strip() for c in args.captions.split("|") if c.strip()]
if not captions:
print("ERROR: no captions provided", file=sys.stderr)
sys.exit(1)
duration = get_video_duration(inp)
print(f"Video duration: {duration:.2f}s, {len(captions)} captions")
drawtext = build_drawtext_filter(
captions, duration, args.font_size,
args.text_color, args.pill_color, args.lang
)
# Build filter_complex
if args.logo and Path(args.logo).exists():
# Overlay logo top-right at 10% width
filter_complex = (
f"[0:v]{drawtext}[captioned];"
f"[1:v]scale=w*0.10:-1[logo];"
f"[captioned][logo]overlay=W-w-20:20[out]"
)
cmd = [
"ffmpeg", "-y",
"-i", str(inp),
"-i", args.logo,
"-filter_complex", filter_complex,
"-map", "[out]",
"-map", "0:a?",
"-c:v", "libx264", "-preset", "fast", "-crf", "20",
"-c:a", "aac", "-b:a", "192k",
str(out)
]
else:
cmd = [
"ffmpeg", "-y",
"-i", str(inp),
"-vf", drawtext,
"-c:v", "libx264", "-preset", "fast", "-crf", "20",
"-c:a", "aac", "-b:a", "192k",
str(out)
]
print(f"Running ffmpeg overlay...")
result = subprocess.run(cmd, capture_output=True, text=True)
if result.returncode != 0:
print(f"ERROR:\n{result.stderr}", file=sys.stderr)
sys.exit(result.returncode)
print(f"✅ Overlay done → {out}")
print(f"MEDIA:{out}")
if __name__ == "__main__":
main()