--- name: jasper-recall version: 0.3.1 description: Local RAG system for agent memory using ChromaDB and SkillBoss API Hub embeddings. v0.3.0 adds multi-agent mesh (N agents sharing memory), OpenClaw plugin with autoRecall, and agent-specific collections. Commands: recall, index-digests, digest-sessions, privacy-check, sync-shared, serve, recall-mesh. requires.env: [SKILLBOSS_API_KEY] --- # Jasper Recall v0.2.3 Local RAG (Retrieval-Augmented Generation) system for AI agent memory. Gives your agent the ability to remember and search past conversations. **New in v0.2.2:** Shared ChromaDB Collections — separate collections for private, shared, and learnings content. Better isolation for multi-agent setups. **New in v0.2.1:** Recall Server — HTTP API for Docker-isolated agents that can't run CLI directly. **New in v0.2.0:** Shared Agent Memory — bidirectional learning between main and sandboxed agents with privacy controls. ## When to Use - **Memory recall**: Search past sessions for context before answering - **Continuous learning**: Index daily notes and decisions for future reference - **Session continuity**: Remember what happened across restarts - **Knowledge base**: Build searchable documentation from your agent's experience ## Quick Start ### Setup One command installs everything: ```bash npx jasper-recall setup ``` This creates: - Python venv at `~/.openclaw/rag-env` - ChromaDB database at `~/.openclaw/chroma-db` - CLI scripts in `~/.local/bin/` - OpenClaw plugin config in `openclaw.json` ### Why Python? The core search and embedding functionality uses Python libraries: - **ChromaDB** — Vector database for semantic search - **sentence-transformers** — Local embedding models (no API needed) These are the gold standard for local RAG. There are no good Node.js equivalents that work fully offline. ### Why a Separate Venv? The venv at `~/.openclaw/rag-env` provides: | Benefit | Why It Matters | |---------|----------------| | **Isolation** | Won't conflict with your other Python projects | | **No sudo** | Installs to your home directory, no root needed | | **Clean uninstall** | Delete the folder and it's gone | | **Reproducibility** | Same versions everywhere | The dependencies are heavy (~200MB total with the embedding model), but this is a one-time download that runs entirely locally. ### Basic Usage **Search your memory:** ```bash recall "what did we decide about the API design" recall "hopeIDS patterns" --limit 10 recall "meeting notes" --json ``` **Index your files:** ```bash index-digests # Index memory files into ChromaDB ``` **Create session digests:** ```bash digest-sessions # Process new sessions digest-sessions --dry-run # Preview what would be processed ``` ## How It Works ### Three Components 1. **digest-sessions** — Extracts key info from session logs (topics, tools used) 2. **index-digests** — Chunks and embeds markdown files into ChromaDB 3. **recall** — Semantic search across your indexed memory ### What Gets Indexed By default, indexes files from `~/.openclaw/workspace/memory/`: - `*.md` — Daily notes, MEMORY.md - `session-digests/*.md` — Session summaries - `repos/*.md` — Project documentation - `founder-logs/*.md` — Development logs (if present) ### Embedding Model Uses SkillBoss API Hub (`/v1/pilot` embedding type) for vector embeddings: - Automatically routes to optimal embedding model via SkillBoss API Hub - No local model download required - Requires `SKILLBOSS_API_KEY` environment variable ## Agent Integration ### Memory-Augmented Responses ```python # Before answering questions about past work results = exec("recall 'project setup decisions' --json") # Include relevant context in your response ``` ### Automated Indexing (Heartbeat) Add to HEARTBEAT.md: ```markdown ## Memory Maintenance - [ ] New session logs? → `digest-sessions` - [ ] Memory files updated? → `index-digests` ``` ### Cron Job Schedule regular indexing: ```json { "schedule": { "kind": "cron", "expr": "0 */6 * * *" }, "payload": { "kind": "agentTurn", "message": "Run index-digests to update the memory index" }, "sessionTarget": "isolated" } ``` ## Shared Agent Memory (v0.2.0+) For multi-agent setups where sandboxed agents need access to some memories: ### Memory Tagging Tag entries in daily notes: ```markdown ## 2026-02-05 [public] - Feature shipped This is visible to all agents. ## 2026-02-05 [private] - Personal note This is main agent only (default if untagged). ## 2026-02-05 [learning] - Pattern discovered Learnings shared bidirectionally between agents. ``` ### ChromaDB Collections (v0.2.2+) Memory is stored in separate collections for isolation: | Collection | Purpose | Who accesses | |------------|---------|--------------| | `private_memories` | Main agent's private content | Main agent only | | `shared_memories` | [public] tagged content | Sandboxed agents | | `agent_learnings` | Learnings from any agent | All agents | | `jasper_memory` | Legacy unified (backward compat) | Fallback | **Collection selection:** ```bash # Main agent (default) - searches private_memories recall "api design" # Sandboxed agents - searches shared_memories only recall "product info" --public-only # Search learnings only recall "patterns" --learnings # Search all collections (merged results) recall "everything" --all # Specific collection recall "something" --collection private_memories # Legacy mode (single collection) recall "old way" --legacy ``` ### Sandboxed Agent Access ```bash # Sandboxed agents use --public-only recall "product info" --public-only # Main agent can see everything recall "product info" ``` ### Moltbook Agent Setup (v0.4.0+) For the moltbook-scanner (or any sandboxed agent), use the built-in setup: ```bash # Configure sandboxed agent with --public-only restriction npx jasper-recall moltbook-setup # Verify the setup is correct npx jasper-recall moltbook-verify ``` This creates: - `~/bin/recall` — Wrapper that forces `--public-only` flag - `shared/` — Symlink to main workspace's shared memory The sandboxed agent can then use: ```bash ~/bin/recall "query" # Automatically restricted to public memories ``` **Privacy model:** 1. Main agent tags memories as `[public]` or `[private]` in daily notes 2. `sync-shared` extracts `[public]` content to `memory/shared/` 3. Sandboxed agents can ONLY search the `shared` collection ### Privacy Workflow ```bash # Check for sensitive data before sharing privacy-check "text to scan" privacy-check --file notes.md # Extract [public] entries to shared directory sync-shared sync-shared --dry-run # Preview first ``` ## CLI Reference ### recall ``` recall "query" [OPTIONS] Options: -n, --limit N Number of results (default: 5) --json Output as JSON -v, --verbose Show similarity scores and collection source --public-only Search shared_memories only (sandboxed agents) --learnings Search agent_learnings only --all Search all collections (merged results) --collection X Search specific collection by name --legacy Use legacy jasper_memory collection ``` ### serve (v0.2.1+) ``` npx jasper-recall serve [OPTIONS] Options: --port, -p N Port to listen on (default: 3458) --host, -h H Host to bind (default: 127.0.0.1) Starts HTTP API server for Docker-isolated agents. Endpoints: GET /recall?q=query&limit=5 Search memories GET /health Health check Security: public_only=true enforced by default. Set RECALL_ALLOW_PRIVATE=true to allow private queries. ``` **Example (from Docker container):** ```bash curl "http://host.docker.internal:3458/recall?q=product+info" ``` ### privacy-check (v0.2.0+) ``` privacy-check "text" # Scan inline text privacy-check --file X # Scan a file Detects: emails, API keys, internal IPs, home paths, credentials. Returns: CLEAN or list of violations. ``` ### sync-shared (v0.2.0+) ``` sync-shared [OPTIONS] Options: --dry-run Preview without writing --all Process all daily notes Extracts [public] tagged entries to memory/shared/. ``` ### index-digests ``` index-digests Indexes markdown files from: ~/.openclaw/workspace/memory/*.md ~/.openclaw/workspace/memory/session-digests/*.md ~/.openclaw/workspace/memory/repos/*.md ~/.openclaw/workspace/memory/founder-logs/*.md Skips files that haven't changed (content hash check). ``` ### digest-sessions ``` digest-sessions [OPTIONS] Options: --dry-run Preview without writing --all Process all sessions (not just new) --recent N Process only N most recent sessions ``` ## Configuration ### Custom Paths Set environment variables: ```bash export RECALL_WORKSPACE=~/.openclaw/workspace export RECALL_CHROMA_DB=~/.openclaw/chroma-db export RECALL_SESSIONS_DIR=~/.openclaw/agents/main/sessions ``` ### Chunking Default settings in index-digests: - Chunk size: 500 characters - Overlap: 100 characters ## Security Considerations ⚠️ **Review these settings before enabling in production:** ### Server Binding The `serve` command defaults to `127.0.0.1` (localhost only). **Do not use `--host 0.0.0.0`** unless you explicitly intend to expose the API externally and have secured it appropriately. ### Private Memory Access The server enforces `public_only=true` by default. The env var `RECALL_ALLOW_PRIVATE=true` bypasses this restriction. **Never set this on public/shared hosts** — it exposes your private memories to any client. ### autoRecall Plugin When `autoRecall: true` in the OpenClaw plugin config, memories are automatically injected before every agent message. Consider: - Set `publicOnly: true` in plugin config for sandboxed agents - Review which collections will be searched - Use `minScore` to filter low-relevance injections **What's automatically skipped (no recall triggered):** - Heartbeat polls (`HEARTBEAT`, `Read HEARTBEAT.md`, `HEARTBEAT_OK`) - Messages containing `NO_REPLY` - Messages < 10 characters - Agent-to-agent messages (cron jobs, workers, spawned agents) - Automated reports (`📋 PR Review`, `🤖 Codex Watch`, `ANNOUNCE_*`) - Messages from senders starting with `agent:` or `worker-` **Safer config for untrusted contexts:** ```json "jasper-recall": { "enabled": true, "config": { "autoRecall": true, "publicOnly": true, "minScore": 0.5 } } ``` ### Environment Variables The following env vars affect behavior — set them explicitly rather than relying on defaults: | Variable | Default | Purpose | |----------|---------|---------| | `RECALL_WORKSPACE` | `~/.openclaw/workspace` | Memory files location | | `RECALL_CHROMA_DB` | `~/.openclaw/chroma-db` | Vector database path | | `RECALL_SESSIONS_DIR` | `~/.openclaw/agents/main/sessions` | Session logs | | `RECALL_ALLOW_PRIVATE` | `false` | Server private access | | `RECALL_PORT` | `3458` | Server port | | `RECALL_HOST` | `127.0.0.1` | Server bind address | ### Dry-Run First Before sharing or syncing, use dry-run options to preview what will be exposed: ```bash privacy-check --file notes.md # Scan for sensitive data sync-shared --dry-run # Preview public extraction digest-sessions --dry-run # Preview session processing ``` ### Sandboxed Environments For maximum isolation, run jasper-recall in a container or dedicated account: - Limits risk of accidental data exposure - Separates private memory from shared contexts - Recommended for multi-agent setups with untrusted agents ## Troubleshooting **"No index found"** ```bash index-digests # Create the index first ``` **"Collection not found"** ```bash rm -rf ~/.openclaw/chroma-db # Clear and rebuild index-digests ``` **Model download slow** First run downloads ~80MB model. Subsequent runs are instant. ## Links - **GitHub**: https://github.com/E-x-O-Entertainment-Studios-Inc/jasper-recall - **npm**: https://www.npmjs.com/package/jasper-recall - **ClawHub**: https://clawhub.ai/skills/jasper-recall FILE:CHANGELOG.md # Changelog All notable changes to Jasper Recall will be documented in this file. ## [0.3.0] - 2026-02-05 ### Added (JR-19: Multi-Agent Mesh) - **Multi-agent mesh** — N agents can share memory, not just 2 - **Agent-specific collections** — Each agent gets its own collection (`agent_sonnet`, `agent_qwen`, etc.) - **`recall-mesh` script** — Enhanced recall with `--agent` and `--mesh` flags - **`index-digests-mesh` script** — Index into agent-specific collections - **Mesh queries** — Query multiple agents' collections: `--mesh sonnet,qwen,opus` - **Backward compatibility** — Legacy collections still work (`private_memories`) - **Documentation** — Comprehensive guide in `docs/MULTI-AGENT-MESH.md` ### Features - `recall-mesh "query" --agent sonnet` — Query as specific agent - `recall-mesh "query" --mesh sonnet,qwen` — Query multiple agents - `index-digests-mesh --agent sonnet` — Index for specific agent - Agent memory remains private by default - Shared and learnings collections accessible to all agents ### Technical - Each agent collection is isolated in ChromaDB - Collections queried in parallel and results merged - Relevance-based sorting across all collections - Automatic collection creation on first index ## [0.2.1] - 2026-02-05 ### Added - **`serve` command** — HTTP API server for sandboxed/Docker agents - `npx jasper-recall serve --port 3458` - `GET /recall?q=query` endpoint - Public-only enforced by default for security - CORS enabled for browser/agent access - Sandboxed agents can now query memories without CLI access - Server exports for programmatic use ### Security - API server enforces `public_only=true` by default - Private content access requires `RECALL_ALLOW_PRIVATE=true` env var ## [0.2.0] - 2026-02-05 ### Added - **Memory tagging** — Mark entries `[public]` or `[private]` in daily notes - **`--public-only` flag** — Sandboxed agents query only shared content - **`privacy-check` command** — Scan text/files for sensitive data before sharing - **`sync-shared` command** — Extract `[public]` entries to shared memory directory - **Bidirectional learning** — Main and sandboxed agents share knowledge safely ### Changed - `recall` now supports post-filtering for privacy-tagged content - README updated with shared memory documentation ## [0.1.0] - 2026-02-04 ### Added - Initial release - `recall` — Semantic search over indexed memories - `index-digests` — Index markdown files into ChromaDB - `digest-sessions` — Extract summaries from session logs - `npx jasper-recall setup` — One-command installation - Local embeddings via sentence-transformers (all-MiniLM-L6-v2) - ChromaDB persistent vector storage - Incremental indexing with content hashing ## [0.2.2] - 2026-02-05 ### Fixed - `serve` command now properly passes CLI arguments (--help, --port, etc.) - Server runCLI function exported for programmatic use ## [0.2.3] - 2026-02-05 ### Added - **Automatic update check** — Notifies you when new versions are available - `update` command — Manually check for updates: `npx jasper-recall update` - Update checks cached for 24 hours (non-intrusive) ## [0.2.4] - 2026-02-05 ### Added - **Configuration management** — `npx jasper-recall config` shows settings - Config file: `~/.jasper-recall/config.json` - `config init` creates config file with defaults - Environment variables override config file - Documented all configuration options in help FILE:cli/config.js /** * Configuration management for jasper-recall * * Priority: ENV vars > config file > defaults * Config file: ~/.jasper-recall/config.json */ const fs = require('fs'); const path = require('path'); const os = require('os'); const CONFIG_DIR = path.join(os.homedir(), '.jasper-recall'); const CONFIG_FILE = path.join(CONFIG_DIR, 'config.json'); const DEFAULTS = { workspace: path.join(os.homedir(), '.openclaw', 'workspace'), chromaDb: path.join(os.homedir(), '.openclaw', 'chroma-db'), venv: path.join(os.homedir(), '.openclaw', 'rag-env'), serverPort: 3458, serverHost: '127.0.0.1', publicOnly: true, // Default for API access memoryPaths: ['memory/'], sharedMemoryPath: 'memory/shared/' }; /** * Load config from file */ function loadConfigFile() { try { if (fs.existsSync(CONFIG_FILE)) { const raw = fs.readFileSync(CONFIG_FILE, 'utf8'); return JSON.parse(raw); } } catch (err) { console.error(`Warning: Could not load config from ${CONFIG_FILE}:`, err.message); } return {}; } /** * Get config value with priority: ENV > file > default */ function get(key) { const envMap = { workspace: 'RECALL_WORKSPACE', chromaDb: 'RECALL_CHROMA_DB', venv: 'RECALL_VENV', serverPort: 'RECALL_PORT', serverHost: 'RECALL_HOST', publicOnly: 'RECALL_PUBLIC_ONLY' }; // Check env var first const envKey = envMap[key]; if (envKey && process.env[envKey]) { const val = process.env[envKey]; // Handle booleans if (val === 'true') return true; if (val === 'false') return false; // Handle numbers if (!isNaN(val)) return parseInt(val, 10); return val; } // Check config file const fileConfig = loadConfigFile(); if (key in fileConfig) { return fileConfig[key]; } // Return default return DEFAULTS[key]; } /** * Get all config */ function getAll() { const fileConfig = loadConfigFile(); const config = { ...DEFAULTS, ...fileConfig }; // Override with env vars for (const key of Object.keys(DEFAULTS)) { config[key] = get(key); } return config; } /** * Save config to file */ function save(config) { if (!fs.existsSync(CONFIG_DIR)) { fs.mkdirSync(CONFIG_DIR, { recursive: true }); } fs.writeFileSync(CONFIG_FILE, JSON.stringify(config, null, 2)); console.log(`Config saved to ${CONFIG_FILE}`); } /** * Initialize config interactively */ function init(options = {}) { const config = { workspace: options.workspace || DEFAULTS.workspace, chromaDb: options.chromaDb || DEFAULTS.chromaDb, venv: options.venv || DEFAULTS.venv, serverPort: options.serverPort || DEFAULTS.serverPort }; save(config); return config; } /** * Show current config */ function show() { console.log('\nJasper Recall Configuration'); console.log('===========================\n'); console.log(`Config file: ${CONFIG_FILE}`); console.log(`Exists: ${fs.existsSync(CONFIG_FILE) ? 'yes' : 'no'}\n`); const config = getAll(); for (const [key, value] of Object.entries(config)) { const source = process.env[`RECALL_${key.toUpperCase()}`] ? '(env)' : loadConfigFile()[key] !== undefined ? '(file)' : '(default)'; console.log(` ${key}: ${value} ${source}`); } console.log(''); } module.exports = { CONFIG_DIR, CONFIG_FILE, DEFAULTS, get, getAll, save, init, show, loadConfigFile }; FILE:cli/doctor.js /** * Jasper Recall Doctor * System health check for RAG dependencies */ const { execSync } = require('child_process'); const fs = require('fs'); const path = require('path'); const os = require('os'); const VENV_PATH = path.join(os.homedir(), '.openclaw', 'rag-env'); const CHROMA_PATH = path.join(os.homedir(), '.openclaw', 'chroma-db'); const MEMORY_PATH = path.join(os.homedir(), '.openclaw', 'workspace', 'memory'); function exec(cmd, opts = {}) { try { const result = execSync(cmd, { encoding: 'utf8', stdio: opts.silent !== false ? 'pipe' : 'inherit', ...opts }); return { success: true, output: result.trim() }; } catch (e) { return { success: false, output: e.message, stderr: e.stderr?.toString() }; } } function checkVersion(requirement, actual) { const reqParts = requirement.replace('>=', '').split('.').map(Number); const actParts = actual.split('.').map(Number); for (let i = 0; i < reqParts.length; i++) { if (actParts[i] > reqParts[i]) return true; if (actParts[i] < reqParts[i]) return false; } return true; } function formatTime(ms) { const seconds = Math.floor(ms / 1000); const minutes = Math.floor(seconds / 60); const hours = Math.floor(minutes / 60); const days = Math.floor(hours / 24); if (days > 0) return `${days}d ago`; if (hours > 0) return `${hours}h ago`; if (minutes > 0) return `${minutes}m ago`; return `${seconds}s ago`; } function getLastIndexTime() { try { if (!fs.existsSync(CHROMA_PATH)) return null; const files = fs.readdirSync(CHROMA_PATH, { recursive: true }); let latestMtime = 0; for (const file of files) { const fullPath = path.join(CHROMA_PATH, file); const stats = fs.statSync(fullPath); if (stats.isFile() && stats.mtimeMs > latestMtime) { latestMtime = stats.mtimeMs; } } if (latestMtime === 0) return null; return Date.now() - latestMtime; } catch (e) { return null; } } function countCollections() { try { if (!fs.existsSync(CHROMA_PATH)) return 0; const sqliteFile = path.join(CHROMA_PATH, 'chroma.sqlite3'); if (!fs.existsSync(sqliteFile)) return 0; // Try to count collections from the database const result = exec(`sqlite3 "${sqliteFile}" "SELECT COUNT(*) FROM collections;"`, { silent: true }); if (result.success) { return parseInt(result.output.trim()) || 0; } // Fallback: count directories const entries = fs.readdirSync(CHROMA_PATH, { withFileTypes: true }); return entries.filter(e => e.isDirectory() && !e.name.startsWith('.')).length; } catch (e) { return 0; } } function countMemoryFiles() { try { if (!fs.existsSync(MEMORY_PATH)) return 0; const files = fs.readdirSync(MEMORY_PATH); return files.filter(f => f.endsWith('.md') && !f.startsWith('.')).length; } catch (e) { return 0; } } function runDoctor(options = {}) { const { fix = false, dryRun = false } = options; const verbose = dryRun; console.log('🏥 Jasper Recall Doctor\n'); if (fix) { console.log('🔧 Fix mode enabled - will attempt to repair issues\n'); } else if (dryRun) { console.log('👁️ Dry-run mode - showing what --fix would do\n'); } const checks = []; const fixes = []; // Node.js version check const nodeResult = exec('node --version'); const nodeVersion = nodeResult.output.replace('v', ''); const nodeOk = nodeResult.success && checkVersion('18.0.0', nodeVersion); checks.push({ label: 'Node.js', status: nodeOk ? '✅' : '❌', value: nodeResult.success ? `v${nodeVersion}` : 'not found', ok: nodeOk, fixable: false, fixMessage: 'Please upgrade Node.js manually: https://nodejs.org/' }); // Python version check const pythonResult = exec('python3 --version'); const pythonMatch = pythonResult.output.match(/Python (\d+\.\d+\.\d+)/); const pythonVersion = pythonMatch ? pythonMatch[1] : null; const pythonOk = pythonResult.success && pythonVersion; checks.push({ label: 'Python', status: pythonOk ? '✅' : '❌', value: pythonVersion || 'not found', ok: pythonOk, fixable: false, fixMessage: 'Please install Python 3: https://www.python.org/downloads/' }); // Virtual environment check const venvExists = fs.existsSync(VENV_PATH); checks.push({ label: 'Venv', status: venvExists ? '✅' : '❌', value: venvExists ? VENV_PATH : 'not found', ok: venvExists, fixable: !venvExists && pythonOk, fixMessage: !venvExists ? `create virtual environment at ${VENV_PATH}` : null, fixCommand: `python3 -m venv ${VENV_PATH}`, fixAction: () => { console.log(` 🔧 Creating virtual environment...`); const result = exec(`python3 -m venv ${VENV_PATH}`, { silent: false }); if (result.success) { console.log(` ✅ Virtual environment created at ${VENV_PATH}`); return true; } else { console.log(` ❌ Failed to create virtual environment`); return false; } } }); // ChromaDB check const pipPath = path.join(VENV_PATH, 'bin', 'pip'); const chromaResult = exec(`${pipPath} show chromadb 2>/dev/null || pip3 show chromadb 2>/dev/null`); const chromaMatch = chromaResult.output.match(/Version: ([\d.]+)/); const chromaVersion = chromaMatch ? chromaMatch[1] : null; const chromaOk = chromaResult.success && chromaVersion; checks.push({ label: 'ChromaDB', status: chromaOk ? '✅' : '❌', value: chromaVersion ? `installed (${chromaVersion})` : 'not installed', ok: chromaOk, fixable: !chromaOk && venvExists, fixMessage: !chromaOk ? 'install chromadb via pip' : null, fixCommand: `${pipPath} install chromadb`, fixAction: () => { console.log(` 🔧 Installing ChromaDB...`); const result = exec(`${pipPath} install chromadb`, { silent: false }); if (result.success) { console.log(` ✅ ChromaDB installed successfully`); return true; } else { console.log(` ❌ Failed to install ChromaDB`); return false; } } }); // Sentence-transformers check const transformersResult = exec(`${pipPath} show sentence-transformers 2>/dev/null || pip3 show sentence-transformers 2>/dev/null`); const transformersMatch = transformersResult.output.match(/Version: ([\d.]+)/); const transformersVersion = transformersMatch ? transformersMatch[1] : null; const transformersOk = transformersResult.success && transformersVersion; checks.push({ label: 'Transformers', status: transformersOk ? '✅' : '❌', value: transformersVersion ? 'sentence-transformers installed' : 'not installed', ok: transformersOk, fixable: !transformersOk && venvExists, fixMessage: !transformersOk ? 'install sentence-transformers via pip' : null, fixCommand: `${pipPath} install sentence-transformers`, fixAction: () => { console.log(` 🔧 Installing sentence-transformers...`); const result = exec(`${pipPath} install sentence-transformers`, { silent: false }); if (result.success) { console.log(` ✅ sentence-transformers installed successfully`); return true; } else { console.log(` ❌ Failed to install sentence-transformers`); return false; } } }); // ChromaDB directory check const chromaExists = fs.existsSync(CHROMA_PATH); const collections = countCollections(); checks.push({ label: 'Database', status: chromaExists ? '✅' : '❌', value: chromaExists ? `${CHROMA_PATH} (${collections} collections)` : 'not found', ok: chromaExists, fixable: !chromaExists, fixMessage: !chromaExists ? `create database directory at ${CHROMA_PATH}` : null, fixCommand: `mkdir -p ${CHROMA_PATH}`, fixAction: () => { console.log(` 🔧 Creating ChromaDB directory...`); try { fs.mkdirSync(CHROMA_PATH, { recursive: true }); console.log(` ✅ Created directory: ${CHROMA_PATH}`); return true; } catch (e) { console.log(` ❌ Failed to create directory: ${e.message}`); return false; } } }); // Memory files check const memoryExists = fs.existsSync(MEMORY_PATH); const memoryCount = countMemoryFiles(); checks.push({ label: 'Memory files', status: memoryExists ? '✅' : '⚠️', value: memoryExists ? `${memoryCount} files in memory/` : 'directory not found', ok: memoryExists, fixable: !memoryExists, fixMessage: !memoryExists ? `create memory directory at ${MEMORY_PATH}` : null, fixCommand: `mkdir -p ${MEMORY_PATH}`, fixAction: () => { console.log(` 🔧 Creating memory directory...`); try { fs.mkdirSync(MEMORY_PATH, { recursive: true }); console.log(` ✅ Created directory: ${MEMORY_PATH}`); return true; } catch (e) { console.log(` ❌ Failed to create directory: ${e.message}`); return false; } } }); // Last index time / collections check const lastIndexMs = getLastIndexTime(); const needsIndex = collections === 0 && chromaExists; const lastIndexOk = !needsIndex && (lastIndexMs !== null && lastIndexMs < 7 * 24 * 60 * 60 * 1000); // < 7 days checks.push({ label: 'Last indexed', status: lastIndexMs === null ? '⚠️' : (lastIndexOk ? '✅' : '⚠️'), value: needsIndex ? 'no collections - needs initial index' : (lastIndexMs === null ? 'never' : formatTime(lastIndexMs)), ok: lastIndexMs !== null && !needsIndex, fixable: needsIndex, fixMessage: needsIndex ? 'run initial indexing with index-digests' : null, fixCommand: 'index-digests', fixAction: () => { console.log(` 🔧 Running initial index...`); const indexScript = path.join(__dirname, 'index-digests.js'); const result = exec(`node ${indexScript}`, { silent: false }); if (result.success) { console.log(` ✅ Initial indexing complete`); return true; } else { console.log(` ⚠️ Indexing may have completed with warnings`); return true; // Don't treat warnings as failure } } }); // Print results const maxLabelLength = Math.max(...checks.map(c => c.label.length)); for (const check of checks) { const padding = ' '.repeat(maxLabelLength - check.label.length); console.log(` ${check.label}:${padding} ${check.status} ${check.value}`); // Show fix suggestions in default/dry-run mode if (!check.ok && !fix) { if (check.fixable && check.fixMessage) { if (verbose && check.fixCommand) { console.log(` ${dryRun ? '📋' : '→'} Would run: ${check.fixCommand}`); } else { console.log(` → run with --fix to ${check.fixMessage}`); } } else if (!check.fixable && check.fixMessage) { console.log(` ❌ ${check.fixMessage}`); } } } console.log(''); // Apply fixes if requested if (fix) { const fixableIssues = checks.filter(c => !c.ok && c.fixable && c.fixAction); if (fixableIssues.length === 0) { const unfixableIssues = checks.filter(c => !c.ok && !c.fixable); if (unfixableIssues.length > 0) { console.log('⚠️ Some issues require manual intervention:\n'); for (const issue of unfixableIssues) { console.log(` ❌ ${issue.label}: ${issue.fixMessage}`); } console.log(''); } } else { console.log('🔧 Applying fixes...\n'); for (const issue of fixableIssues) { const success = issue.fixAction(); fixes.push({ issue: issue.label, success }); console.log(''); } const successCount = fixes.filter(f => f.success).length; const failCount = fixes.filter(f => !f.success).length; if (failCount === 0) { console.log(`✅ All ${successCount} issue${successCount > 1 ? 's' : ''} fixed!\n`); } else { console.log(`⚠️ Fixed ${successCount}/${fixes.length} issues (${failCount} failed)\n`); } // Check for remaining unfixable issues const unfixableIssues = checks.filter(c => !c.ok && !c.fixable); if (unfixableIssues.length > 0) { console.log('⚠️ Remaining issues require manual intervention:\n'); for (const issue of unfixableIssues) { console.log(` ❌ ${issue.label}: ${issue.fixMessage}`); } console.log(''); } } } // Summary const allOk = checks.every(c => c.ok); if (allOk) { console.log('✅ All systems operational!\n'); return 0; } else { const failed = checks.filter(c => !c.ok); if (!fix) { console.log(`⚠️ ${failed.length} issue${failed.length > 1 ? 's' : ''} detected.\n`); const hasFixableIssues = failed.some(c => c.fixable); if (hasFixableIssues) { console.log('→ Run with --fix to automatically repair issues\n'); } } return fixes.length > 0 && fixes.every(f => f.success) ? 0 : 1; } } module.exports = { runDoctor }; // Allow direct execution if (require.main === module) { const args = process.argv.slice(2); const options = { fix: args.includes('--fix'), dryRun: args.includes('--dry-run') }; process.exit(runDoctor(options)); } FILE:cli/jasper-recall.js #!/usr/bin/env node /** * Jasper Recall CLI * Local RAG system for AI agent memory * * Usage: * npx jasper-recall setup # Install dependencies and create scripts * npx jasper-recall recall # Run a query (alias) * npx jasper-recall index # Index files (alias) * npx jasper-recall digest # Digest sessions (alias) */ const { execSync, spawn } = require('child_process'); const fs = require('fs'); const path = require('path'); const os = require('os'); // Read version from package.json const packageJson = require('../package.json'); const VERSION = packageJson.version; // Check for updates in background (non-blocking) const { checkInBackground } = require('./update-check'); checkInBackground(); const VENV_PATH = path.join(os.homedir(), '.openclaw', 'rag-env'); const CHROMA_PATH = path.join(os.homedir(), '.openclaw', 'chroma-db'); const BIN_PATH = path.join(os.homedir(), '.local', 'bin'); const SCRIPTS_DIR = path.join(__dirname, '..', 'scripts'); const EXTENSIONS_DIR = path.join(__dirname, '..', 'extensions'); const OPENCLAW_CONFIG = path.join(os.homedir(), '.openclaw', 'openclaw.json'); const OPENCLAW_SKILLS = path.join(os.homedir(), '.openclaw', 'workspace', 'skills'); function log(msg) { console.log(`🦊 ${msg}`); } function error(msg) { console.error(`❌ ${msg}`); } function run(cmd, opts = {}) { try { return execSync(cmd, { stdio: opts.silent ? 'pipe' : 'inherit', ...opts }); } catch (e) { if (!opts.ignoreError) { error(`Command failed: ${cmd}`); process.exit(1); } return null; } } function setupOpenClawIntegration() { log('Setting up OpenClaw integration...'); // Check if OpenClaw is installed const openclawDir = path.join(os.homedir(), '.openclaw'); if (!fs.existsSync(openclawDir)) { console.log(' ⚠ OpenClaw not detected (~/.openclaw not found)'); console.log(' → Skipping OpenClaw integration'); return false; } // Install SKILL.md to skills directory const skillSrc = path.join(EXTENSIONS_DIR, 'openclaw-plugin', 'SKILL.md'); const skillDest = path.join(OPENCLAW_SKILLS, 'jasper-recall', 'SKILL.md'); if (fs.existsSync(skillSrc)) { fs.mkdirSync(path.dirname(skillDest), { recursive: true }); fs.copyFileSync(skillSrc, skillDest); console.log(` ✓ Installed SKILL.md: ${skillDest}`); } else { console.log(' ⚠ SKILL.md not found in package (try reinstalling)'); } // Update openclaw.json with plugin config if (fs.existsSync(OPENCLAW_CONFIG)) { try { const configRaw = fs.readFileSync(OPENCLAW_CONFIG, 'utf8'); const config = JSON.parse(configRaw); // Initialize plugins structure if needed if (!config.plugins) config.plugins = {}; if (!config.plugins.entries) config.plugins.entries = {}; // Check if already configured if (config.plugins.entries['jasper-recall']) { console.log(' ✓ Plugin already configured in openclaw.json'); } else { // Add plugin config config.plugins.entries['jasper-recall'] = { enabled: true, config: { autoRecall: true, minScore: 0.3, defaultLimit: 5 } }; // Write back with nice formatting fs.writeFileSync(OPENCLAW_CONFIG, JSON.stringify(config, null, 2) + '\n'); console.log(' ✓ Added jasper-recall plugin to openclaw.json'); console.log(' → Restart OpenClaw gateway to activate: openclaw gateway restart'); } } catch (e) { console.log(` ⚠ Could not update openclaw.json: ${e.message}`); console.log(' → Manually add plugin config (see docs)'); } } else { console.log(' ⚠ openclaw.json not found'); console.log(' → Create config or manually add jasper-recall plugin'); } return true; } function setup() { log('Jasper Recall — Setup'); console.log('=' .repeat(40)); // Check Python log('Checking Python...'); let python = 'python3'; try { const version = execSync(`${python} --version`, { encoding: 'utf8' }); console.log(` ✓ ${version.trim()}`); } catch { error('Python 3 is required. Install it first.'); process.exit(1); } // Create venv log('Creating Python virtual environment...'); fs.mkdirSync(path.dirname(VENV_PATH), { recursive: true }); if (!fs.existsSync(VENV_PATH)) { run(`${python} -m venv ${VENV_PATH}`); console.log(` ✓ Created: ${VENV_PATH}`); } else { console.log(` ✓ Already exists: ${VENV_PATH}`); } // Install Python dependencies log('Installing Python dependencies (this may take a minute)...'); const pip = path.join(VENV_PATH, 'bin', 'pip'); run(`${pip} install --quiet chromadb sentence-transformers`); console.log(' ✓ Installed: chromadb, sentence-transformers'); // Create bin directory fs.mkdirSync(BIN_PATH, { recursive: true }); // Copy scripts log('Installing CLI scripts...'); const scripts = [ { src: 'recall.py', dest: 'recall', shebang: `#!${path.join(VENV_PATH, 'bin', 'python3')}` }, { src: 'index-digests.py', dest: 'index-digests', shebang: `#!${path.join(VENV_PATH, 'bin', 'python3')}` }, { src: 'digest-sessions.sh', dest: 'digest-sessions', shebang: '#!/bin/bash' }, { src: 'summarize-old.py', dest: 'summarize-old', shebang: `#!${path.join(VENV_PATH, 'bin', 'python3')}` } ]; for (const script of scripts) { const srcPath = path.join(SCRIPTS_DIR, script.src); const destPath = path.join(BIN_PATH, script.dest); let content = fs.readFileSync(srcPath, 'utf8'); // Replace generic shebang with specific one for Python scripts if (script.src.endsWith('.py')) { content = content.replace(/^#!.*python3?\n/, script.shebang + '\n'); } fs.writeFileSync(destPath, content); fs.chmodSync(destPath, 0o755); console.log(` ✓ Installed: ${destPath}`); } // Create chroma directory fs.mkdirSync(CHROMA_PATH, { recursive: true }); // Verify PATH const pathEnv = process.env.PATH || ''; if (!pathEnv.includes(BIN_PATH)) { console.log(''); log('Add to your PATH (add to ~/.bashrc or ~/.zshrc):'); console.log(` export PATH="$HOME/.local/bin:$PATH"`); } console.log(''); // OpenClaw integration setupOpenClawIntegration(); console.log(''); console.log('=' .repeat(40)); log('Setup complete!'); console.log(''); console.log('Next steps:'); console.log(' 1. index-digests # Index your memory files'); console.log(' 2. recall "query" # Search your memory'); console.log(' 3. digest-sessions # Process session logs'); } function showHelp() { console.log(` Jasper Recall v${VERSION} Local RAG system for AI agent memory USAGE: npx jasper-recall COMMANDS: setup Install dependencies and CLI scripts doctor Run system health check Flags: --fix (auto-repair issues), --dry-run (verbose output) recall Search your memory (alias for the recall command) index Index memory files (alias for index-digests) digest Process session logs (alias for digest-sessions) summarize Compress old entries to save tokens (alias for summarize-old) serve Start HTTP API server (for sandboxed agents) config Show or set configuration update Check for updates moltbook-setup Configure moltbook agent with --public-only restriction moltbook-verify Verify moltbook agent setup help Show this help message CONFIGURATION: Config file: ~/.jasper-recall/config.json Environment variables (override config file): RECALL_WORKSPACE Memory workspace path RECALL_CHROMA_DB ChromaDB storage path RECALL_VENV Python venv path RECALL_PORT Server port (default: 3458) RECALL_HOST Server host (default: 127.0.0.1) EXAMPLES: npx jasper-recall setup recall "what did we discuss yesterday" index-digests digest-sessions --dry-run npx jasper-recall serve --port 3458 `); } // Main const command = process.argv[2]; switch (command) { case 'setup': setup(); break; case 'recall': // Pass through to recall script const recallScript = path.join(BIN_PATH, 'recall'); if (fs.existsSync(recallScript)) { const args = process.argv.slice(3); spawn(recallScript, args, { stdio: 'inherit' }); } else { error('Run "npx jasper-recall setup" first'); } break; case 'index': const indexScript = path.join(BIN_PATH, 'index-digests'); if (fs.existsSync(indexScript)) { spawn(indexScript, [], { stdio: 'inherit' }); } else { error('Run "npx jasper-recall setup" first'); } break; case 'digest': const digestScript = path.join(BIN_PATH, 'digest-sessions'); if (fs.existsSync(digestScript)) { const args = process.argv.slice(3); spawn(digestScript, args, { stdio: 'inherit' }); } else { error('Run "npx jasper-recall setup" first'); } break; case 'summarize': const summarizeScript = path.join(BIN_PATH, 'summarize-old'); if (fs.existsSync(summarizeScript)) { const args = process.argv.slice(3); spawn(summarizeScript, args, { stdio: 'inherit' }); } else { error('Run "npx jasper-recall setup" first'); } break; case 'serve': case 'server': // Start the HTTP server for sandboxed agents const { runCLI } = require('./server'); runCLI(process.argv.slice(3)); break; case 'update': case 'check-update': // Check for updates explicitly const { checkForUpdates } = require('./update-check'); checkForUpdates().then(result => { if (result && !result.updateAvailable) { console.log(`✓ You're on the latest version (${result.current})`); } else if (!result) { console.log('Could not check for updates'); } }); break; case 'doctor': // Run system health check const { runDoctor } = require('./doctor'); const args = process.argv.slice(3); const options = { fix: args.includes('--fix'), dryRun: args.includes('--dry-run') }; process.exit(runDoctor(options)); break; case 'moltbook-setup': case 'moltbook': // Set up moltbook agent integration process.argv = [process.argv[0], process.argv[1], 'setup']; require('../extensions/moltbook-setup/setup.js'); break; case 'moltbook-verify': // Verify moltbook agent setup process.argv = [process.argv[0], process.argv[1], 'verify']; require('../extensions/moltbook-setup/setup.js'); break; case 'config': // Configuration management const config = require('./config'); const configArg = process.argv[3]; if (configArg === 'init') { config.init(); } else if (configArg === 'path') { console.log(config.CONFIG_FILE); } else { config.show(); } break; case '--version': case '-v': console.log(VERSION); break; case 'help': case '--help': case '-h': case undefined: showHelp(); break; default: error(`Unknown command: ${command}`); showHelp(); process.exit(1); } FILE:cli/server.js /** * Jasper Recall Server * HTTP API for memory search - designed for sandboxed agents * * Security: public_only is enforced by default */ const http = require('http'); const { execSync } = require('child_process'); const path = require('path'); const os = require('os'); const url = require('url'); const BIN_PATH = path.join(os.homedir(), '.local', 'bin'); const RECALL_SCRIPT = path.join(BIN_PATH, 'recall'); /** * Execute recall query */ function executeRecall(query, options = {}) { const { publicOnly = true, limit = 5 } = options; let cmd = `${RECALL_SCRIPT} "${query.replace(/"/g, '\\"')}"`; // Security: always add --public-only unless explicitly disabled if (publicOnly) { cmd += ' --public-only'; } cmd += ` --limit ${parseInt(limit) || 5}`; try { const output = execSync(cmd, { encoding: 'utf8', timeout: 30000, env: { ...process.env, HOME: os.homedir() } }); return { ok: true, output }; } catch (err) { // Check if it's just "no results" if (err.stdout?.includes('No results') || err.status === 0) { return { ok: true, output: err.stdout || 'No results found' }; } return { ok: false, error: err.message, stderr: err.stderr }; } } /** * Parse recall output into structured results */ function parseResults(output) { const results = []; // Try to parse structured output const blocks = output.split(/={3,}\s*(?:Result\s+\d+|---)/i); for (const block of blocks) { if (!block.trim()) continue; const result = {}; const scoreMatch = block.match(/score:\s*([\d.]+)/i); if (scoreMatch) result.score = parseFloat(scoreMatch[1]); const fileMatch = block.match(/File:\s*(.+)/i); if (fileMatch) result.file = fileMatch[1].trim(); const linesMatch = block.match(/Lines?:\s*(\d+(?:-\d+)?)/i); if (linesMatch) result.lines = linesMatch[1]; // Content is everything else let content = block .replace(/score:\s*[\d.]+/gi, '') .replace(/File:\s*.+/gi, '') .replace(/Lines?:\s*\d+(?:-\d+)?/gi, '') .trim(); if (content) { result.content = content.substring(0, 1000); results.push(result); } } // Fallback for unparseable output if (results.length === 0 && output.trim()) { results.push({ content: output.trim().substring(0, 2000), raw: true }); } return results; } /** * Handle HTTP request */ function handleRequest(req, res) { // CORS headers for browser/agent access res.setHeader('Access-Control-Allow-Origin', '*'); res.setHeader('Access-Control-Allow-Methods', 'GET, OPTIONS'); res.setHeader('Content-Type', 'application/json'); if (req.method === 'OPTIONS') { res.writeHead(200); res.end(); return; } const parsedUrl = url.parse(req.url, true); const pathname = parsedUrl.pathname; const query = parsedUrl.query; // Health check if (pathname === '/health' || pathname === '/') { res.writeHead(200); res.end(JSON.stringify({ ok: true, service: 'jasper-recall', version: '0.2.1' })); return; } // Recall endpoint if (pathname === '/recall' || pathname === '/api/recall') { const searchQuery = query.q || query.query; if (!searchQuery) { res.writeHead(400); res.end(JSON.stringify({ ok: false, error: 'q or query parameter required' })); return; } // Security: public_only defaults to true // Only allow disabling if explicitly set AND RECALL_ALLOW_PRIVATE=true let publicOnly = true; if (query.public_only === 'false' && process.env.RECALL_ALLOW_PRIVATE === 'true') { publicOnly = false; } const result = executeRecall(searchQuery, { publicOnly, limit: query.limit || 5 }); if (result.ok) { const parsed = parseResults(result.output); res.writeHead(200); res.end(JSON.stringify({ ok: true, query: searchQuery, public_only: publicOnly, count: parsed.length, results: parsed, raw: result.output })); } else { res.writeHead(500); res.end(JSON.stringify({ ok: false, error: result.error, stderr: result.stderr?.substring(0, 500) })); } return; } // 404 res.writeHead(404); res.end(JSON.stringify({ ok: false, error: 'Not found' })); } /** * Start the server */ function startServer(port = 3458, host = '127.0.0.1') { const server = http.createServer(handleRequest); server.listen(port, host, () => { console.log(`🦊 Jasper Recall Server running on http://${host}:${port}`); console.log(''); console.log('Endpoints:'); console.log(` GET /recall?q=query Search memories (public-only by default)`); console.log(` GET /health Health check`); console.log(''); console.log('Security: public_only=true is enforced by default'); console.log('Press Ctrl+C to stop'); }); return server; } /** * Parse CLI args and start server */ function runCLI(args) { let port = 3458; let host = '127.0.0.1'; for (let i = 0; i < args.length; i++) { if (args[i] === '--port' || args[i] === '-p') { port = parseInt(args[++i]) || 3458; } if (args[i] === '--host' || args[i] === '-h') { host = args[++i] || '127.0.0.1'; } if (args[i] === '--help') { console.log(` Jasper Recall Server HTTP API for memory search Usage: npx jasper-recall serve [options] Options: --port, -p Port to listen on (default: 3458) --host, -h Host to bind to (default: 127.0.0.1) --help Show this help Environment: RECALL_ALLOW_PRIVATE=true Allow public_only=false queries (dangerous!) Examples: npx jasper-recall serve npx jasper-recall serve --port 8080 npx jasper-recall serve --host 0.0.0.0 `); process.exit(0); } } startServer(port, host); } // Export for programmatic use module.exports = { startServer, executeRecall, parseResults, runCLI }; // CLI entry point if (require.main === module) { runCLI(process.argv.slice(2)); } FILE:cli/update-check.js /** * Check for updates and notify user * Non-blocking, caches check for 24 hours */ const https = require('https'); const fs = require('fs'); const path = require('path'); const os = require('os'); const PACKAGE_NAME = 'jasper-recall'; const CACHE_FILE = path.join(os.homedir(), '.openclaw', '.jasper-recall-update-check'); const CHECK_INTERVAL_MS = 24 * 60 * 60 * 1000; // 24 hours /** * Get current package version */ function getCurrentVersion() { try { const pkg = require('../package.json'); return pkg.version; } catch { return null; } } /** * Check if we should run update check */ function shouldCheck() { try { if (fs.existsSync(CACHE_FILE)) { const stat = fs.statSync(CACHE_FILE); const age = Date.now() - stat.mtimeMs; if (age < CHECK_INTERVAL_MS) { return false; // Checked recently } } } catch { // Ignore errors, just check } return true; } /** * Save check timestamp */ function saveCheckTime(latestVersion) { try { const dir = path.dirname(CACHE_FILE); if (!fs.existsSync(dir)) { fs.mkdirSync(dir, { recursive: true }); } fs.writeFileSync(CACHE_FILE, JSON.stringify({ checked: new Date().toISOString(), latest: latestVersion })); } catch { // Ignore errors } } /** * Fetch latest version from npm */ function fetchLatestVersion() { return new Promise((resolve, reject) => { const req = https.get(`https://registry.npmjs.org/${PACKAGE_NAME}/latest`, { timeout: 3000, headers: { 'Accept': 'application/json' } }, (res) => { let data = ''; res.on('data', chunk => data += chunk); res.on('end', () => { try { const pkg = JSON.parse(data); resolve(pkg.version); } catch (e) { reject(e); } }); }); req.on('error', reject); req.on('timeout', () => { req.destroy(); reject(new Error('timeout')); }); }); } /** * Compare semver versions */ function isNewer(latest, current) { const l = latest.split('.').map(Number); const c = current.split('.').map(Number); for (let i = 0; i < 3; i++) { if ((l[i] || 0) > (c[i] || 0)) return true; if ((l[i] || 0) < (c[i] || 0)) return false; } return false; } /** * Check for updates (non-blocking) */ async function checkForUpdates(silent = false) { if (!shouldCheck()) { return null; } const current = getCurrentVersion(); if (!current) return null; try { const latest = await fetchLatestVersion(); saveCheckTime(latest); if (isNewer(latest, current)) { if (!silent) { console.log(''); console.log(`📦 Update available: ${current} → ${latest}`); console.log(` Run: npm update -g jasper-recall`); console.log(''); } return { current, latest, updateAvailable: true }; } return { current, latest, updateAvailable: false }; } catch { // Silent fail - don't block user return null; } } /** * Run check in background (fire and forget) */ function checkInBackground() { // Don't await - let it run async checkForUpdates().catch(() => {}); } module.exports = { checkForUpdates, checkInBackground, getCurrentVersion }; FILE:docs/MULTI-AGENT-MESH.md # Multi-Agent Mesh (JR-19) ## Overview The multi-agent mesh feature allows N agents to share memory, not just 2. Each agent can have its own private collection while selectively sharing with other agents. ## Architecture ### Collection Types 1. **Agent-specific collections**: `agent_` (e.g., `agent_sonnet`, `agent_qwen`) - Private memory for each agent - Created when indexing with `--agent ` 2. **Shared collections** (accessible to all agents): - `shared_memories`: Public/shared content - `agent_learnings`: Meta-learnings about agent operation 3. **Legacy collection** (backward compatibility): - `private_memories`: Original main agent collection ## Usage ### Indexing for Specific Agents ```bash # Index memory for SONNET agent index-digests-mesh --agent sonnet # Index memory for QWEN agent index-digests-mesh --agent qwen # Index memory for legacy/main agent (no agent flag) index-digests-mesh ``` ### Querying as a Specific Agent ```bash # Query as SONNET (sees: agent_sonnet + shared + learnings) recall-mesh "query" --agent sonnet # Query as QWEN (sees: agent_qwen + shared + learnings) recall-mesh "query" --agent qwen # Query legacy mode (sees: private_memories + shared + learnings) recall-mesh "query" ``` ### Multi-Agent Mesh Queries ```bash # Query across multiple agents (mesh mode) recall-mesh "query" --mesh sonnet,qwen,opus # This queries: # - agent_sonnet # - agent_qwen # - agent_opus # - shared_memories # - agent_learnings ``` ### Public-Only Mode (for sandboxed agents) ```bash # Only query shared content (backward compat with JR-17) recall-mesh "query" --public-only # This queries: # - shared_memories # - agent_learnings ``` ## Content Classification Files are automatically classified based on path and tags: | Type | Collection | Criteria | |------|------------|----------| | **Learning** | `agent_learnings` | Path contains `learnings/` OR filename is `AGENTS.md` or `TOOLS.md` | | **Public** | `shared_memories` | Path contains `shared/` OR content includes `[public]` tag | | **Private** | `agent_` or `private_memories` | Default for all other content | ### Tagging Content Use inline tags to control visibility: ```markdown # Example Memory Entry [public] This content is visible to all agents. [private] This content is only visible to the indexing agent. ``` ## Installation The mesh scripts are in `scripts/` and need to be installed to `~/.local/bin/`: ```bash # Install mesh scripts cp scripts/recall-mesh ~/.local/bin/recall-mesh cp scripts/index-digests-mesh ~/.local/bin/index-digests-mesh chmod +x ~/.local/bin/recall-mesh ~/.local/bin/index-digests-mesh ``` Or create symlinks for development: ```bash ln -sf ~/projects/jasper-recall/scripts/recall-mesh ~/.local/bin/recall-mesh ln -sf ~/projects/jasper-recall/scripts/index-digests-mesh ~/.local/bin/index-digests-mesh ``` ## Backward Compatibility All existing functionality is preserved: - Scripts without flags work exactly as before - Legacy `private_memories` collection still works - `--public-only` flag (JR-17) still works - Existing indexes are not affected ## Examples ### Scenario 1: Two Worker Agents Sharing Knowledge ```bash # SONNET indexes its work index-digests-mesh --agent sonnet # QWEN indexes its work index-digests-mesh --agent qwen # SONNET queries both agents' memory recall-mesh "how did QWEN implement this?" --mesh sonnet,qwen # QWEN queries both agents' memory recall-mesh "what did SONNET decide?" --mesh qwen,sonnet ``` ### Scenario 2: Main Agent Coordinating Workers ```bash # Workers index their own memory index-digests-mesh --agent worker1 index-digests-mesh --agent worker2 index-digests-mesh --agent worker3 # Main agent queries all workers recall-mesh "what have the workers accomplished?" --mesh worker1,worker2,worker3 # Individual worker queries only its own + shared recall-mesh "query" --agent worker1 ``` ### Scenario 3: Gradual Migration ```bash # Keep using legacy collection index-digests # Uses private_memories recall "query" # Queries private_memories + shared + learnings # Start using agent-specific collections index-digests-mesh --agent main recall-mesh "query" --agent main # Both work simultaneously (different collections) ``` ## API Integration The mesh feature can be integrated with the recall server: ```bash # Start server with agent support # (Future enhancement - server needs update) npx jasper-recall serve --agent sonnet # Query via HTTP curl "http://localhost:9876/recall?q=query&agent=sonnet&mesh=qwen,opus" ``` ## Performance Considerations - **Mesh queries** search multiple collections, so they're slightly slower - Each collection is queried in parallel internally - Results are merged and sorted by relevance - Larger meshes (more agents) = more collections to query ### Optimization Tips 1. **Use specific agent queries** when you know which agent's memory you need 2. **Use mesh queries** only when you need cross-agent knowledge 3. **Limit mesh size** to agents that are actually relevant 4. **Keep shared content minimal** to avoid duplication ## Directory Structure ``` ~/.openclaw/ ├── chroma-db/ # ChromaDB persistent storage │ ├── agent_sonnet/ # SONNET's collection │ ├── agent_qwen/ # QWEN's collection │ ├── agent_opus/ # OPUS's collection │ ├── private_memories/# Legacy main agent │ ├── shared_memories/ # Shared across all agents │ └── agent_learnings/ # Meta-learnings └── workspace/ └── memory/ # Source markdown files ``` ## Testing ```bash # 1. Index some content for different agents echo "SONNET learned this" > ~/.openclaw/workspace/memory/sonnet-test.md echo "QWEN learned this" > ~/.openclaw/workspace/memory/qwen-test.md echo "[public] Everyone knows this" > ~/.openclaw/workspace/memory/shared-test.md # 2. Index for each agent index-digests-mesh --agent sonnet index-digests-mesh --agent qwen # 3. Test queries recall-mesh "learned" --agent sonnet # Should find SONNET + shared recall-mesh "learned" --agent qwen # Should find QWEN + shared recall-mesh "learned" --mesh sonnet,qwen # Should find both + shared ``` ## Troubleshooting ### Collections not found ```bash # List all collections python3 -c "import chromadb; client = chromadb.PersistentClient('~/.openclaw/chroma-db'); print([c.name for c in client.list_collections()])" ``` ### Empty results ```bash # Check collection contents recall-mesh "test" --agent sonnet -v # Verbose shows collections queried ``` ### Performance issues ```bash # Check collection sizes python3 -c " import chromadb client = chromadb.PersistentClient('~/.openclaw/chroma-db') for col in client.list_collections(): print(f'{col.name}: {col.count()} chunks') " ``` ## Future Enhancements - [ ] Agent-to-agent memory sharing permissions - [ ] Automatic mesh discovery (query all available agents) - [ ] Memory replication across agents - [ ] Cross-agent memory deduplication - [ ] Agent memory quotas - [ ] Memory access audit logs ## See Also - [JR-17: Shared ChromaDB Collections](../CHANGELOG.md#v020) - [Main README](../README.md) - [REQUIREMENTS.md](../../task-dashboard/docs/jasper-recall/REQUIREMENTS.md) FILE:docs/SHARED-MEMORY-SPEC.md # Jasper Recall v0.2.0 Spec: Shared Agent Memory > Bidirectional learning between main and sandboxed agents with privacy controls ## Overview **Problem:** Sandboxed agents (like moltbook-scanner) operate in isolation. They can't: - Learn from main agent's daily work and decisions - Share their learnings back to main - Access relevant product context for authentic engagement **Solution:** Tagged memory system with access control: - `[public]` memories visible to all agents - `[private]` memories restricted to main - Bidirectional sync with privacy filtering ## Architecture ``` ┌─────────────────────────────────────────────────────────────────────┐ │ MEMORY LAYER │ │ │ │ ┌──────────────────────┐ ┌──────────────────────┐ │ │ │ PRIVATE ZONE │ │ SHARED ZONE │ │ │ │ (main only) │ │ (all agents) │ │ │ │ │ │ │ │ │ │ • memory/*.md │ ───► │ • memory/shared/ │ │ │ │ [private] tagged │filter│ auto-extracted │ │ │ │ • MEMORY.md │ │ • product-updates.md │ │ │ │ • USER.md │ │ • learnings.md │ │ │ └──────────────────────┘ └──────────────────────┘ │ │ │ │ │ │ ▼ ▼ │ │ ┌──────────────────────────────────────────────────────┐ │ │ │ ChromaDB │ │ │ │ │ │ │ │ collection: private_memories ◄── main only │ │ │ │ collection: shared_memories ◄── all agents │ │ │ │ collection: agent_learnings ◄── sandboxed writes │ │ │ └──────────────────────────────────────────────────────┘ │ └─────────────────────────────────────────────────────────────────────┘ ▲ ▲ │ │ ┌─────┴─────┐ ┌──────┴──────┐ │ JASPER │ │ MOLTBOOK │ │ (main) │ │ SCANNER │ │ │ │ (sandboxed)│ │ • rw all │ │ • r shared │ │ • tag mem │ │ • w learnings│ └───────────┘ └─────────────┘ ``` ## Memory Tagging Convention ### Syntax Tags appear at the start of a section header: ```markdown ## 2026-02-05 [public] - Shipped jasper-recall v0.1.0 Released the npm package, got good community reception. ## 2026-02-05 [private] - User mentioned upcoming travel Will be unavailable Feb 10-15. ``` ### Classification Rules | Category | Tag | Examples | |----------|-----|----------| | Product work | `[public]` | Feature releases, bug fixes, decisions | | Technical learnings | `[public]` | Patterns, best practices, gotchas | | Community engagement | `[public]` | Moltbook posts, feedback, reactions | | Public decisions | `[public]` | Architecture choices, roadmap | | Personal info | `[private]` | Names, locations, schedule | | Secrets | `[private]` | Keys, tokens, credentials | | Internal ops | `[private]` | Server IPs, internal paths | | User preferences | `[private]` | Habits, communication style | ### Default Behavior - Untagged content defaults to `[private]` (safe default) - Explicit `[public]` required for sharing ## File Structure ``` ~/.openclaw/workspace/ ├── memory/ │ ├── 2026-02-05.md # Daily notes (tagged) │ ├── YYYY-MM-DD.md # More daily notes │ └── shared/ # PUBLIC ZONE │ ├── product-updates.md # Auto-extracted from daily notes │ ├── learnings.md # Aggregated insights │ └── moltbook/ # Engagement data │ └── posts.md # What was posted, reactions │ ~/.openclaw/workspace-moltbook/ ├── shared -> ~/.openclaw/workspace/memory/shared/ # SYMLINK ├── AGENTS.md └── PRODUCT-CONTEXT.md # Deprecated, use shared/ ``` ## CLI Changes ### recall (updated) ```bash # Existing behavior (searches all) recall "query" # New: public-only mode for sandboxed agents recall "query" --public-only # New: specify collection recall "query" --collection shared_memories recall "query" --collection agent_learnings ``` ### index-digests (updated) ```bash # Index with tag extraction index-digests # Parses [public]/[private] tags # Routes to appropriate collection ``` ### New: sync-shared ```bash # Extract [public] content from daily notes sync-shared # Options sync-shared --dry-run # Preview only sync-shared --force # Re-extract all sync-shared --since 7d # Last 7 days only ``` ### New: privacy-check ```bash # Scan content for private data before writing privacy-check "text to check" privacy-check --file /path/to/file.md # Returns: CLEAN or list of detected patterns ``` ## Privacy Filter Patterns Reuses patterns from hopeIDS where applicable: ```javascript const PRIVATE_PATTERNS = [ // Personal identifiers { name: 'email', pattern: /[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}/g }, { name: 'phone', pattern: /\b\d{3}[-.]?\d{3}[-.]?\d{4}\b/g }, // Paths and infrastructure { name: 'home_path', pattern: /\/home\/\w+\//g }, { name: 'internal_ip', pattern: /\b(?:10|172\.(?:1[6-9]|2\d|3[01])|192\.168)\.\d{1,3}\.\d{1,3}\b/g }, // Secrets { name: 'api_key', pattern: /sk-[a-zA-Z0-9-_]{20,}/g }, { name: 'token', pattern: /\b[a-zA-Z0-9]{32,}\b/g }, // Generic long tokens // Keywords { name: 'secret_keyword', pattern: /\b(password|secret|private|internal|confidential)\b/gi }, // Names (configurable allowlist) { name: 'product_names', allowlist: ['jasper-recall', 'hopeIDS', 'Jasper', 'OpenClaw'] }, ]; ``` ## Implementation Plan ### Phase 1: Foundation (Day 1) 1. **JR-10**: Memory tagging convention - Update AGENTS.md with tagging rules - Add examples to daily note template 2. **JR-11**: Shared memory directory - Create `memory/shared/` structure - Symlink to moltbook-scanner workspace - Create initial files ### Phase 2: Privacy (Day 1-2) 3. **JR-13**: Privacy filter - Create `scripts/privacy-check.py` - Integrate hopeIDS patterns - Add CLI command 4. **JR-16**: Reflection workflow - Update moltbook-scanner AGENTS.md - Add pre-post checklist ### Phase 3: Indexing (Day 2) 5. **JR-12**: Public-only recall - Update `scripts/recall.py` with --public-only - Add collection routing in index-digests - Create shared_memories collection ### Phase 4: Sync (Day 2-3) 6. **JR-14**: Bidirectional sync cron - Create `scripts/sync-shared.py` - Extract [public] entries - Schedule via OpenClaw cron 7. **JR-15**: Moltbook learnings capture - Update post-comment.js to log engagement - Write to shared/moltbook/posts.md ### Phase 5: Polish (Day 3) 8. **JR-17**: ChromaDB collections - Migrate to multi-collection setup - Update all scripts ## Success Criteria 1. ✅ Moltbook-scanner can query recall for product info 2. ✅ Private data never appears in shared memory 3. ✅ Main agent sees moltbook engagement data 4. ✅ New product updates auto-sync to sandboxed agents 5. ✅ Privacy filter catches 95%+ of sensitive patterns ## Timeline | Day | Tasks | Deliverable | |-----|-------|-------------| | 1 | JR-10, JR-11, JR-13 | Tagging + shared dir + privacy filter | | 2 | JR-12, JR-14, JR-16 | Public recall + sync + reflection | | 3 | JR-15, JR-17 | Learnings capture + collections | **Target:** v0.2.0 release by Feb 7, 2026 ## Future Considerations - **v0.3.0**: Multi-agent memory mesh (N agents, not just 2) - **v0.3.0**: Encrypted shared memories for sensitive-but-shareable - **v0.3.0**: Memory summarization (compress old entries) FILE:extensions/jasper-recall/index.ts /** * Jasper Recall OpenClaw Plugin * * Semantic search over indexed memory using ChromaDB. * "Remember everything. Recall what matters." * * Features: * - `recall` tool for manual searches * - `/recall` command for quick lookups * - Auto-recall: inject relevant memories before agent processing */ import { execFileSync, execSync } from 'child_process'; import * as path from 'path'; import * as os from 'os'; interface PluginConfig { enabled?: boolean; autoRecall?: boolean; defaultLimit?: number; publicOnly?: boolean; minScore?: number; logLevel?: 'debug' | 'info' | 'warn' | 'error'; } interface PluginApi { config: { plugins?: { entries?: { 'jasper-recall'?: { config?: PluginConfig; }; }; }; }; logger: { info: (msg: string) => void; warn: (msg: string) => void; error: (msg: string) => void; debug: (msg: string) => void; }; registerTool: (tool: any) => void; registerCommand: (cmd: any) => void; registerGatewayMethod: (name: string, handler: any) => void; on: (event: string, handler: (event: any) => Promise) => void; } const BIN_PATH = path.join(os.homedir(), '.local', 'bin'); function runRecall(query: string, options: { limit?: number; json?: boolean; publicOnly?: boolean } = {}): string { const args = [JSON.stringify(query)]; if (options.limit) args.push('-n', String(options.limit)); if (options.json) args.push('--json'); if (options.publicOnly) args.push('--public-only'); const recallPath = path.join(BIN_PATH, 'recall'); try { return execFileSync(recallPath, args, { encoding: 'utf8', timeout: 30000 }); } catch (err: any) { throw new Error(`Recall failed: ${err.message}`); } } function getSimilarity(result: any): number { return typeof result?.similarity === 'number' ? result.similarity : result?.score ?? 0; } export default function register(api: PluginApi) { const cfg = api.config.plugins?.entries?.['jasper-recall']?.config ?? {}; if (cfg.enabled === false) { api.logger.info('[jasper-recall] Plugin disabled'); return; } const defaultLimit = cfg.defaultLimit ?? 5; const publicOnly = cfg.publicOnly ?? false; const autoRecall = cfg.autoRecall ?? false; const minScore = cfg.minScore ?? 0.3; api.logger.info(`[jasper-recall] Initialized (limit=${defaultLimit}, publicOnly=${publicOnly}, autoRecall=${autoRecall})`); // ============================================================================ // Auto-Recall: inject relevant memories before agent processes the message // ============================================================================ if (autoRecall) { api.on('before_agent_start', async (event: { prompt?: string }) => { // Skip if no prompt or too short if (!event.prompt || event.prompt.length < 10) { return; } // Skip system/internal prompts if (event.prompt.startsWith('HEARTBEAT') || event.prompt.includes('NO_REPLY')) { return; } try { const results = runRecall(event.prompt, { limit: 3, json: true, publicOnly, }); const parsed = JSON.parse(results); // Filter by minimum score const relevant = parsed.filter((r: any) => getSimilarity(r) >= minScore); if (relevant.length === 0) { api.logger.debug?.('[jasper-recall] No relevant memories found for auto-recall'); return; } // Format memories for context injection const memoryContext = relevant .map((r: any) => `- [${r.source || 'memory'}] ${r.content.slice(0, 500)}${r.content.length > 500 ? '...' : ''}`) .join('\n'); api.logger.info(`[jasper-recall] Auto-injecting ${relevant.length} memories into context`); return { prependContext: `\nThe following memories may be relevant to this conversation:\n${memoryContext}\n`, }; } catch (err: any) { api.logger.warn(`[jasper-recall] Auto-recall failed: ${err.message}`); } }); } // ============================================================================ // Tool: recall // ============================================================================ api.registerTool({ name: 'recall', description: 'Semantic search over indexed memory (daily notes, session digests, documentation). Use to find context from past conversations, decisions, and learnings.', parameters: { type: 'object', properties: { query: { type: 'string', description: 'Search query - natural language question or keywords', }, limit: { type: 'number', description: 'Maximum number of results to return (default: 5)', }, }, required: ['query'], }, execute: async (_id: string, { query, limit }: { query: string; limit?: number }) => { try { const results = runRecall(query, { limit: limit ?? defaultLimit, json: true, publicOnly, }); const parsed = JSON.parse(results); // Format results for agent consumption let formatted = `## Recall Results for: "${query}"\n\n`; if (parsed.length === 0) { formatted += '_No relevant memories found._\n'; } else { for (const result of parsed) { formatted += `### ${result.source || 'Memory'}\n`; formatted += `**Similarity:** ${(getSimilarity(result) * 100).toFixed(1)}%\n\n`; formatted += `${result.content}\n\n---\n\n`; } } api.logger.info(`[jasper-recall] Query "${query}" returned ${parsed.length} results`); return { content: [{ type: 'text', text: formatted }] }; } catch (err: any) { api.logger.error(`[jasper-recall] Error: ${err.message}`); return { content: [{ type: 'text', text: `Recall error: ${err.message}` }] }; } }, }); // ============================================================================ // Command: /recall // ============================================================================ api.registerCommand({ name: 'recall', description: 'Search memory for relevant context', acceptsArgs: true, requireAuth: true, handler: async (ctx: { args?: string }) => { const query = ctx.args?.trim(); if (!query) { return { text: '⚠️ Usage: /recall ' }; } try { const results = runRecall(query, { limit: defaultLimit, publicOnly }); return { text: `🧠 **Recall Results**\n\n${results}` }; } catch (err: any) { return { text: `❌ Recall failed: ${err.message}` }; } }, }); // ============================================================================ // Command: /index // ============================================================================ api.registerCommand({ name: 'index', description: 'Re-index memory files into ChromaDB', acceptsArgs: false, requireAuth: true, handler: async () => { try { const indexPath = path.join(BIN_PATH, 'index-digests'); const output = execSync(indexPath, { encoding: 'utf8', timeout: 120000 }); return { text: `🔄 **Memory Indexed**\n\n${output}` }; } catch (err: any) { return { text: `❌ Index failed: ${err.message}` }; } }, }); // ============================================================================ // RPC Methods // ============================================================================ api.registerGatewayMethod('recall.search', async ({ params, respond }: any) => { try { const { query, limit } = params; const results = runRecall(query, { limit: limit ?? defaultLimit, json: true, publicOnly }); respond(true, JSON.parse(results)); } catch (err: any) { respond(false, { error: err.message }); } }); api.registerGatewayMethod('recall.index', async ({ respond }: any) => { try { const indexPath = path.join(BIN_PATH, 'index-digests'); execSync(indexPath, { encoding: 'utf8', timeout: 120000 }); respond(true, { status: 'indexed' }); } catch (err: any) { respond(false, { error: err.message }); } }); } export const id = 'jasper-recall'; export const name = 'Jasper Recall - Local RAG Memory'; FILE:extensions/jasper-recall/openclaw.plugin.json { "id": "jasper-recall", "name": "Jasper Recall - Local RAG Memory", "version": "0.2.0", "description": "Semantic search over indexed memory using ChromaDB with auto-recall", "homepage": "https://github.com/E-x-O-Entertainment-Studios-Inc/jasper-recall", "configSchema": { "type": "object", "additionalProperties": false, "properties": { "enabled": { "type": "boolean", "default": true }, "autoRecall": { "type": "boolean", "default": false, "description": "Automatically inject relevant memories before agent processing" }, "defaultLimit": { "type": "number", "default": 5, "description": "Default number of results to return" }, "minScore": { "type": "number", "default": 0.3, "description": "Minimum similarity score for auto-recall (0-1)" }, "publicOnly": { "type": "boolean", "default": false, "description": "Only search public memory (for sandboxed agents)" }, "logLevel": { "type": "string", "enum": ["debug", "info", "warn", "error"], "default": "info" } } }, "uiHints": { "enabled": { "label": "Enable Jasper Recall" }, "autoRecall": { "label": "Auto-Recall", "help": "Inject relevant memories into context before processing" }, "defaultLimit": { "label": "Default Result Limit" }, "minScore": { "label": "Minimum Score", "help": "Threshold for auto-recall relevance (0.3 = 30%)" }, "publicOnly": { "label": "Public Memory Only" }, "logLevel": { "label": "Log Level" } } } FILE:extensions/jasper-recall/package.json { "name": "@jasper-recall/openclaw-plugin", "version": "0.1.0", "description": "OpenClaw plugin for Jasper Recall semantic memory search", "main": "index.ts", "type": "module", "dependencies": {} } FILE:extensions/jasper-recall/SKILL.md # Jasper Recall - OpenClaw Plugin Semantic search over indexed memory using ChromaDB. Automatically injects relevant context before agent processing. ## Features - **`recall` tool** — Manual semantic search over memory - **`/recall` command** — Quick lookups from chat - **`/index` command** — Re-index memory files - **Auto-recall** — Automatically inject relevant memories before processing --- ## Auto-Recall (The Magic ✨) When `autoRecall` is enabled, jasper-recall hooks into the agent lifecycle and automatically searches your memory before every message is processed. ### How It Works ``` ┌─────────────────────────────────────────────────────────────┐ │ 1. Message arrives from user │ │ 2. before_agent_start hook fires │ │ 3. jasper-recall searches ChromaDB with message as query │ │ 4. Results filtered by minScore (default: 30%) │ │ 5. Relevant memories injected via prependContext │ │ 6. Agent sees memories + original message │ │ 7. Agent responds with full context │ └─────────────────────────────────────────────────────────────┘ ``` ### What Gets Injected ```xml The following memories may be relevant to this conversation: - [memory/2026-02-05.md] Worker orchestration decisions... - [MEMORY.md] Git workflow: feature → develop → main... - [memory/sops/codex-integration-sop.md] Codex Cloud sync... ``` ### What's Skipped Auto-recall won't run for: - Heartbeat polls (`HEARTBEAT...`) - System prompts containing `NO_REPLY` - Messages shorter than 10 characters --- ## Configuration In `openclaw.json`: ```json { "plugins": { "entries": { "jasper-recall": { "enabled": true, "config": { "autoRecall": true, "minScore": 0.3, "defaultLimit": 5, "publicOnly": false } } } } } ``` ### Options | Option | Type | Default | Description | |--------|------|---------|-------------| | `enabled` | boolean | `true` | Enable/disable plugin | | `autoRecall` | boolean | `false` | Auto-inject memories before processing | | `minScore` | number | `0.3` | Minimum similarity score (0-1) for auto-recall | | `defaultLimit` | number | `5` | Default number of results | | `publicOnly` | boolean | `false` | Only search public memory (sandboxed agents) | ### Score Tuning - `minScore: 0.3` — Include loosely related memories (more context, may include noise) - `minScore: 0.5` — Only moderately relevant (balanced) - `minScore: 0.7` — Only highly relevant (precise, may miss useful context) --- ## Tools ### `recall` Manual semantic search over memory. **Parameters:** - `query` (string, required): Natural language search query - `limit` (number, optional): Max results (default: 5) **Example:** ``` recall query="what did we decide about the API design" limit=3 ``` **Returns:** Formatted markdown with matching memories, scores, and sources. --- ## Commands ### `/recall ` Quick memory search from chat. ``` /recall worker orchestration decisions ``` ### `/index` Re-index memory files into ChromaDB. Run after updating notes. ``` /index ``` --- ## RPC Methods For external integrations: ### `recall.search` ```json { "query": "search terms", "limit": 5 } ``` ### `recall.index` Re-index memory files (no params). --- ## Requirements - `recall` command in `~/.local/bin/` - ChromaDB index at `~/.openclaw/chroma-db` - Python venv at `~/.openclaw/rag-env` ## Installation ```bash npx jasper-recall setup ``` This sets up: 1. Python venv with ChromaDB + sentence-transformers 2. `recall`, `index-digests`, `digest-sessions` scripts 3. Initial index of memory files --- ## When Auto-Recall Helps ✅ **Great for:** - Questions about past decisions ("what did we decide about X?") - Following up on previous work ("where were we with the worker setup?") - Context about people, preferences, projects - Finding SOPs and procedures ⚠️ **Less useful for:** - Brand new topics with no memory - Simple commands ("list files") - Real-time data (weather, time) --- ## Sandboxed Agents For agents processing untrusted input, use `publicOnly`: ```json { "jasper-recall": { "config": { "publicOnly": true, "autoRecall": true } } } ``` This restricts searches to `memory/shared/` and public-tagged content, preventing leakage of private memories. --- ## Links - **GitHub**: https://github.com/E-x-O-Entertainment-Studios-Inc/jasper-recall - **npm**: `npx jasper-recall setup` - **ClawHub**: `clawhub install jasper-recall` FILE:extensions/moltbook-setup/setup.js #!/usr/bin/env node /** * Moltbook Agent Setup for jasper-recall * * Configures a sandboxed agent to use jasper-recall with --public-only restriction. * This ensures the agent can only access shared/public memories, not private ones. */ const fs = require('fs'); const path = require('path'); const os = require('os'); const readline = require('readline'); const MOLTBOOK_WORKSPACE = path.join(os.homedir(), '.openclaw', 'workspace-moltbook'); const MAIN_WORKSPACE = path.join(os.homedir(), '.openclaw', 'workspace'); const RECALL_BIN = path.join(os.homedir(), '.local', 'bin', 'recall'); function log(msg) { console.log(`🦞 ${msg}`); } function warn(msg) { console.log(`⚠️ ${msg}`); } function error(msg) { console.error(`❌ ${msg}`); } function success(msg) { console.log(`✅ ${msg}`); } async function prompt(question) { const rl = readline.createInterface({ input: process.stdin, output: process.stdout }); return new Promise(resolve => { rl.question(question, answer => { rl.close(); resolve(answer.trim()); }); }); } async function setup() { console.log(''); log('Moltbook Agent — jasper-recall Integration Setup'); console.log('='.repeat(55)); console.log(''); console.log(' This configures the moltbook-scanner agent to use jasper-recall'); console.log(' with the --public-only restriction for privacy.'); console.log(''); console.log(' What it does:'); console.log(' 1. Creates ~/bin/recall wrapper (forces --public-only)'); console.log(' 2. Symlinks shared/ folder from main workspace'); console.log(' 3. Verifies jasper-recall is installed'); console.log(''); // Check prerequisites if (!fs.existsSync(MOLTBOOK_WORKSPACE)) { error(`Moltbook workspace not found: ${MOLTBOOK_WORKSPACE}`); console.log(' Create it first or check your OpenClaw agent config.'); process.exit(1); } if (!fs.existsSync(RECALL_BIN)) { error(`jasper-recall not installed: ${RECALL_BIN}`); console.log(' Install it first: npx jasper-recall setup'); process.exit(1); } const proceed = await prompt(' Continue? (y/n): '); if (proceed.toLowerCase() !== 'y' && proceed.toLowerCase() !== 'yes') { console.log('\n Setup cancelled.\n'); process.exit(0); } console.log(''); // Step 1: Create bin directory and wrapper const binDir = path.join(MOLTBOOK_WORKSPACE, 'bin'); const wrapperPath = path.join(binDir, 'recall'); fs.mkdirSync(binDir, { recursive: true }); const wrapperScript = `#!/bin/bash # Sandboxed recall wrapper - forces --public-only for privacy # This agent can ONLY access shared/public memory exec ${RECALL_BIN} "$@" --public-only `; fs.writeFileSync(wrapperPath, wrapperScript); fs.chmodSync(wrapperPath, '755'); success(`Created recall wrapper: ${wrapperPath}`); // Step 2: Create shared folder symlink const sharedSource = path.join(MAIN_WORKSPACE, 'memory', 'shared'); const sharedTarget = path.join(MOLTBOOK_WORKSPACE, 'shared'); // Ensure source exists fs.mkdirSync(sharedSource, { recursive: true }); // Remove existing symlink/dir if needed try { const stat = fs.lstatSync(sharedTarget); if (stat.isSymbolicLink()) { fs.unlinkSync(sharedTarget); } else if (stat.isDirectory()) { warn(`${sharedTarget} is a directory, not a symlink. Skipping.`); } } catch (e) { // Doesn't exist, that's fine } if (!fs.existsSync(sharedTarget)) { fs.symlinkSync(sharedSource, sharedTarget); success(`Created symlink: shared/ → ${sharedSource}`); } // Step 3: Verify setup console.log(''); log('Verifying setup...'); const issues = verify({ quiet: true }); if (issues.length === 0) { console.log(''); console.log('='.repeat(55)); success('Setup complete!'); console.log(''); console.log(' The moltbook-scanner agent can now use:'); console.log(' ~/bin/recall "query" — searches public memories only'); console.log(' shared/ — symlink to main agent\'s shared memory'); console.log(''); console.log(' Test it:'); console.log(` ${wrapperPath} "test query"`); console.log(''); } else { console.log(''); warn('Setup completed with issues:'); issues.forEach(issue => console.log(` - ${issue}`)); } } function verify(options = {}) { const { quiet = false } = options; const issues = []; if (!quiet) { console.log(''); log('Moltbook Agent — jasper-recall Verification'); console.log('='.repeat(55)); console.log(''); } // Check 1: Workspace exists if (!fs.existsSync(MOLTBOOK_WORKSPACE)) { issues.push(`Workspace missing: ${MOLTBOOK_WORKSPACE}`); } else if (!quiet) { success(`Workspace exists: ${MOLTBOOK_WORKSPACE}`); } // Check 2: Recall wrapper exists and is executable const wrapperPath = path.join(MOLTBOOK_WORKSPACE, 'bin', 'recall'); if (!fs.existsSync(wrapperPath)) { issues.push(`Recall wrapper missing: ${wrapperPath}`); } else { // Check it has --public-only const content = fs.readFileSync(wrapperPath, 'utf8'); if (!content.includes('--public-only')) { issues.push('Recall wrapper missing --public-only flag!'); } else if (!quiet) { success('Recall wrapper has --public-only restriction'); } } // Check 3: Shared folder is a symlink const sharedPath = path.join(MOLTBOOK_WORKSPACE, 'shared'); try { const stat = fs.lstatSync(sharedPath); if (!stat.isSymbolicLink()) { issues.push(`shared/ is not a symlink (should link to main workspace)`); } else { const target = fs.readlinkSync(sharedPath); if (!quiet) { success(`shared/ symlink → ${target}`); } } } catch (e) { issues.push(`shared/ folder missing`); } // Check 4: jasper-recall is installed if (!fs.existsSync(RECALL_BIN)) { issues.push(`jasper-recall not installed: ${RECALL_BIN}`); } else if (!quiet) { success(`jasper-recall installed: ${RECALL_BIN}`); } // Check 5: AGENTS.md mentions recall restrictions const agentsMd = path.join(MOLTBOOK_WORKSPACE, 'AGENTS.md'); if (fs.existsSync(agentsMd)) { const content = fs.readFileSync(agentsMd, 'utf8'); if (!content.includes('public-only') && !content.includes('public_only')) { issues.push('AGENTS.md should document --public-only restriction'); } else if (!quiet) { success('AGENTS.md documents recall restrictions'); } } if (!quiet) { console.log(''); if (issues.length === 0) { console.log('='.repeat(55)); success('All checks passed! Moltbook agent is properly configured.'); } else { console.log('='.repeat(55)); warn(`Found ${issues.length} issue(s):`); issues.forEach(issue => console.log(` ❌ ${issue}`)); console.log(''); console.log(' Run setup to fix: npx jasper-recall moltbook-setup'); } console.log(''); } return issues; } function showHelp() { console.log(` Moltbook Agent — jasper-recall Integration USAGE: npx jasper-recall moltbook-setup Configure moltbook agent npx jasper-recall moltbook-verify Verify configuration WHAT IT DOES: Sets up the moltbook-scanner agent to use jasper-recall with privacy restrictions. The agent can only access shared/public memories, not private ones from the main workspace. COMPONENTS: ~/bin/recall Wrapper script that forces --public-only flag shared/ Symlink to main workspace's shared memory folder PRIVACY MODEL: Main agent tags memories as [public] or [private] in daily notes. sync-shared.py extracts [public] content to memory/shared/. Sandboxed agents can ONLY search the shared collection. `); } // Main const command = process.argv[2]; switch (command) { case 'setup': case 'install': setup().catch(err => { error(err.message); process.exit(1); }); break; case 'verify': case 'check': verify(); break; case 'help': case '--help': case '-h': case undefined: showHelp(); break; default: error(`Unknown command: ${command}`); showHelp(); process.exit(1); } FILE:extensions/openclaw-plugin/index.ts /** * Jasper Recall OpenClaw Plugin * * Semantic search over indexed memory using ChromaDB. * "Remember everything. Recall what matters." * * Features: * - `recall` tool for manual searches * - `/recall` command for quick lookups * - Auto-recall: inject relevant memories before agent processing */ import { execSync } from 'child_process'; import * as path from 'path'; import * as os from 'os'; interface PluginConfig { enabled?: boolean; autoRecall?: boolean; defaultLimit?: number; publicOnly?: boolean; minScore?: number; logLevel?: 'debug' | 'info' | 'warn' | 'error'; } interface PluginApi { config: { plugins?: { entries?: { 'jasper-recall'?: { config?: PluginConfig; }; }; }; }; logger: { info: (msg: string) => void; warn: (msg: string) => void; error: (msg: string) => void; debug: (msg: string) => void; }; registerTool: (tool: any) => void; registerCommand: (cmd: any) => void; registerGatewayMethod: (name: string, handler: any) => void; on: (event: string, handler: (event: any) => Promise) => void; } const BIN_PATH = path.join(os.homedir(), '.local', 'bin'); function runRecall(query: string, options: { limit?: number; json?: boolean; publicOnly?: boolean } = {}): string { const args = [JSON.stringify(query)]; if (options.limit) args.push('-n', String(options.limit)); if (options.json) args.push('--json'); if (options.publicOnly) args.push('--public-only'); const recallPath = path.join(BIN_PATH, 'recall'); try { return execSync(`${recallPath} ${args.join(' ')}`, { encoding: 'utf8', timeout: 30000 }); } catch (err: any) { throw new Error(`Recall failed: ${err.message}`); } } export default function register(api: PluginApi) { const cfg = api.config.plugins?.entries?.['jasper-recall']?.config ?? {}; if (cfg.enabled === false) { api.logger.info('[jasper-recall] Plugin disabled'); return; } const defaultLimit = cfg.defaultLimit ?? 5; const publicOnly = cfg.publicOnly ?? false; const autoRecall = cfg.autoRecall ?? false; const minScore = cfg.minScore ?? 0.3; api.logger.info(`[jasper-recall] Initialized (limit=${defaultLimit}, publicOnly=${publicOnly}, autoRecall=${autoRecall})`); // ============================================================================ // Auto-Recall: inject relevant memories before agent processes the message // ============================================================================ if (autoRecall) { api.on('before_agent_start', async (event: { prompt?: string; senderId?: string; source?: string }) => { // Skip if no prompt or too short if (!event.prompt || event.prompt.length < 10) { return; } const prompt = event.prompt; // Skip heartbeats and system prompts if (prompt.startsWith('HEARTBEAT') || prompt.startsWith('Read HEARTBEAT.md') || prompt.includes('NO_REPLY') || prompt.includes('HEARTBEAT_OK')) { return; } // Skip agent-to-agent messages (cron jobs, workers, spawned agents) if (event.source?.startsWith('cron:') || event.source?.startsWith('agent:') || event.source?.startsWith('spawn:') || event.source === 'sessions_send' || event.senderId?.startsWith('agent:') || event.senderId?.startsWith('worker-')) { return; } // Skip common automated patterns if (prompt.startsWith('Agent-to-agent') || prompt.startsWith('📋 PR Review') || prompt.startsWith('🤖 Codex Watch') || prompt.startsWith('ANNOUNCE_')) { return; } try { const results = runRecall(event.prompt, { limit: 3, json: true, publicOnly, }); const parsed = JSON.parse(results); // Filter by minimum score const relevant = parsed.filter((r: any) => r.score >= minScore); if (relevant.length === 0) { api.logger.debug?.('[jasper-recall] No relevant memories found for auto-recall'); return; } // Format memories for context injection const memoryContext = relevant .map((r: any) => `- [${r.source || 'memory'}] ${r.content.slice(0, 500)}${r.content.length > 500 ? '...' : ''}`) .join('\n'); api.logger.info(`[jasper-recall] Auto-injecting ${relevant.length} memories into context`); return { prependContext: `\nThe following memories may be relevant to this conversation:\n${memoryContext}\n`, }; } catch (err: any) { api.logger.warn(`[jasper-recall] Auto-recall failed: ${err.message}`); } }); } // ============================================================================ // Tool: recall // ============================================================================ api.registerTool({ name: 'recall', description: 'Semantic search over indexed memory (daily notes, session digests, documentation). Use to find context from past conversations, decisions, and learnings.', parameters: { type: 'object', properties: { query: { type: 'string', description: 'Search query - natural language question or keywords', }, limit: { type: 'number', description: 'Maximum number of results to return (default: 5)', }, }, required: ['query'], }, execute: async (_id: string, { query, limit }: { query: string; limit?: number }) => { try { const results = runRecall(query, { limit: limit ?? defaultLimit, json: true, publicOnly, }); const parsed = JSON.parse(results); // Format results for agent consumption let formatted = `## Recall Results for: "${query}"\n\n`; if (parsed.length === 0) { formatted += '_No relevant memories found._\n'; } else { for (const result of parsed) { formatted += `### ${result.source || 'Memory'}\n`; formatted += `**Score:** ${(result.score * 100).toFixed(1)}%\n\n`; formatted += `${result.content}\n\n---\n\n`; } } api.logger.info(`[jasper-recall] Query "${query}" returned ${parsed.length} results`); return { content: [{ type: 'text', text: formatted }] }; } catch (err: any) { api.logger.error(`[jasper-recall] Error: ${err.message}`); return { content: [{ type: 'text', text: `Recall error: ${err.message}` }] }; } }, }); // ============================================================================ // Command: /recall // ============================================================================ api.registerCommand({ name: 'recall', description: 'Search memory for relevant context', acceptsArgs: true, requireAuth: true, handler: async (ctx: { args?: string }) => { const query = ctx.args?.trim(); if (!query) { return { text: '⚠️ Usage: /recall ' }; } try { const results = runRecall(query, { limit: defaultLimit, publicOnly }); return { text: `🧠 **Recall Results**\n\n${results}` }; } catch (err: any) { return { text: `❌ Recall failed: ${err.message}` }; } }, }); // ============================================================================ // Command: /index // ============================================================================ api.registerCommand({ name: 'index', description: 'Re-index memory files into ChromaDB', acceptsArgs: false, requireAuth: true, handler: async () => { try { const indexPath = path.join(BIN_PATH, 'index-digests'); const output = execSync(indexPath, { encoding: 'utf8', timeout: 120000 }); return { text: `🔄 **Memory Indexed**\n\n${output}` }; } catch (err: any) { return { text: `❌ Index failed: ${err.message}` }; } }, }); // ============================================================================ // RPC Methods // ============================================================================ api.registerGatewayMethod('recall.search', async ({ params, respond }: any) => { try { const { query, limit } = params; const results = runRecall(query, { limit: limit ?? defaultLimit, json: true, publicOnly }); respond(true, JSON.parse(results)); } catch (err: any) { respond(false, { error: err.message }); } }); api.registerGatewayMethod('recall.index', async ({ respond }: any) => { try { const indexPath = path.join(BIN_PATH, 'index-digests'); execSync(indexPath, { encoding: 'utf8', timeout: 120000 }); respond(true, { status: 'indexed' }); } catch (err: any) { respond(false, { error: err.message }); } }); } export const id = 'jasper-recall'; export const name = 'Jasper Recall - Local RAG Memory'; FILE:extensions/openclaw-plugin/openclaw.plugin.json { "id": "jasper-recall", "name": "Jasper Recall - Local RAG Memory", "version": "0.2.0", "description": "Semantic search over indexed memory using ChromaDB with auto-recall", "homepage": "https://github.com/E-x-O-Entertainment-Studios-Inc/jasper-recall", "configSchema": { "type": "object", "additionalProperties": false, "properties": { "enabled": { "type": "boolean", "default": true }, "autoRecall": { "type": "boolean", "default": false, "description": "Automatically inject relevant memories before agent processing" }, "defaultLimit": { "type": "number", "default": 5, "description": "Default number of results to return" }, "minScore": { "type": "number", "default": 0.3, "description": "Minimum similarity score for auto-recall (0-1)" }, "publicOnly": { "type": "boolean", "default": false, "description": "Only search public memory (for sandboxed agents)" }, "logLevel": { "type": "string", "enum": ["debug", "info", "warn", "error"], "default": "info" } } }, "uiHints": { "enabled": { "label": "Enable Jasper Recall" }, "autoRecall": { "label": "Auto-Recall", "help": "Inject relevant memories into context before processing" }, "defaultLimit": { "label": "Default Result Limit" }, "minScore": { "label": "Minimum Score", "help": "Threshold for auto-recall relevance (0.3 = 30%)" }, "publicOnly": { "label": "Public Memory Only" }, "logLevel": { "label": "Log Level" } } } FILE:extensions/openclaw-plugin/package.json { "name": "@jasper-recall/openclaw-plugin", "version": "0.1.0", "description": "OpenClaw plugin for Jasper Recall semantic memory search", "main": "index.ts", "type": "module", "dependencies": {} } FILE:extensions/openclaw-plugin/SKILL.md # Jasper Recall - OpenClaw Plugin Semantic search over indexed memory using ChromaDB. Automatically injects relevant context before agent processing. ## Features - **`recall` tool** — Manual semantic search over memory - **`/recall` command** — Quick lookups from chat - **`/index` command** — Re-index memory files - **Auto-recall** — Automatically inject relevant memories before processing --- ## Auto-Recall (The Magic ✨) When `autoRecall` is enabled, jasper-recall hooks into the agent lifecycle and automatically searches your memory before every message is processed. ### How It Works ``` ┌─────────────────────────────────────────────────────────────┐ │ 1. Message arrives from user │ │ 2. before_agent_start hook fires │ │ 3. jasper-recall searches ChromaDB with message as query │ │ 4. Results filtered by minScore (default: 30%) │ │ 5. Relevant memories injected via prependContext │ │ 6. Agent sees memories + original message │ │ 7. Agent responds with full context │ └─────────────────────────────────────────────────────────────┘ ``` ### What Gets Injected ```xml The following memories may be relevant to this conversation: - [memory/2026-02-05.md] Worker orchestration decisions... - [MEMORY.md] Git workflow: feature → develop → main... - [memory/sops/codex-integration-sop.md] Codex Cloud sync... ``` ### What's Skipped Auto-recall won't run for: - Heartbeat polls (`HEARTBEAT...`) - System prompts containing `NO_REPLY` - Messages shorter than 10 characters --- ## Configuration In `openclaw.json`: ```json { "plugins": { "entries": { "jasper-recall": { "enabled": true, "config": { "autoRecall": true, "minScore": 0.3, "defaultLimit": 5, "publicOnly": false } } } } } ``` ### Options | Option | Type | Default | Description | |--------|------|---------|-------------| | `enabled` | boolean | `true` | Enable/disable plugin | | `autoRecall` | boolean | `false` | Auto-inject memories before processing | | `minScore` | number | `0.3` | Minimum similarity score (0-1) for auto-recall | | `defaultLimit` | number | `5` | Default number of results | | `publicOnly` | boolean | `false` | Only search public memory (sandboxed agents) | ### Score Tuning - `minScore: 0.3` — Include loosely related memories (more context, may include noise) - `minScore: 0.5` — Only moderately relevant (balanced) - `minScore: 0.7` — Only highly relevant (precise, may miss useful context) --- ## Tools ### `recall` Manual semantic search over memory. **Parameters:** - `query` (string, required): Natural language search query - `limit` (number, optional): Max results (default: 5) **Example:** ``` recall query="what did we decide about the API design" limit=3 ``` **Returns:** Formatted markdown with matching memories, scores, and sources. --- ## Commands ### `/recall ` Quick memory search from chat. ``` /recall worker orchestration decisions ``` ### `/index` Re-index memory files into ChromaDB. Run after updating notes. ``` /index ``` --- ## RPC Methods For external integrations: ### `recall.search` ```json { "query": "search terms", "limit": 5 } ``` ### `recall.index` Re-index memory files (no params). --- ## Requirements - `recall` command in `~/.local/bin/` - ChromaDB index at `~/.openclaw/chroma-db` - Python venv at `~/.openclaw/rag-env` ## Installation ```bash npx jasper-recall setup ``` This sets up: 1. Python venv with ChromaDB + sentence-transformers 2. `recall`, `index-digests`, `digest-sessions` scripts 3. Initial index of memory files --- ## When Auto-Recall Helps ✅ **Great for:** - Questions about past decisions ("what did we decide about X?") - Following up on previous work ("where were we with the worker setup?") - Context about people, preferences, projects - Finding SOPs and procedures ⚠️ **Less useful for:** - Brand new topics with no memory - Simple commands ("list files") - Real-time data (weather, time) --- ## Sandboxed Agents For agents processing untrusted input, use `publicOnly`: ```json { "jasper-recall": { "config": { "publicOnly": true, "autoRecall": true } } } ``` This restricts searches to `memory/shared/` and public-tagged content, preventing leakage of private memories. --- ## Links - **GitHub**: https://github.com/E-x-O-Entertainment-Studios-Inc/jasper-recall - **npm**: `npx jasper-recall setup` - **ClawHub**: `clawhub install jasper-recall` FILE:package.json { "name": "jasper-recall", "version": "0.4.0", "description": "Local RAG system for AI agent memory using ChromaDB and sentence-transformers", "main": "src/index.js", "bin": { "jasper-recall": "./cli/jasper-recall.js" }, "scripts": { "test": "node cli/jasper-recall.js --version" }, "keywords": [ "rag", "chromadb", "embeddings", "memory", "ai-agent", "openclaw", "semantic-search", "vector-database" ], "author": "E.x.O. Entertainment Studios Inc.", "license": "MIT", "repository": { "type": "git", "url": "https://github.com/E-x-O-Entertainment-Studios-Inc/jasper-recall.git" }, "homepage": "https://exohaven.online/products/jasper-recall", "bugs": { "url": "https://github.com/E-x-O-Entertainment-Studios-Inc/jasper-recall/issues" }, "engines": { "node": ">=18.0.0" }, "files": [ "cli/", "scripts/", "src/", "extensions/", "SKILL.md", "README.md" ] } FILE:README.md # Jasper Recall Published via SkillPublisher. ## Installation ```bash clawhub install qui-jasper-recall ``` > More info: https://skillboss.co/skills/jasper-recall ## Usage See SKILL.md for details. ## License MIT FILE:scripts/digest-sessions.sh #!/bin/bash # digest-sessions — Extract learnings from session logs # Usage: digest-sessions [--all | --recent N | --dry-run] set -e # Support custom paths via environment WORKSPACE="${RECALL_WORKSPACE:-$HOME/.openclaw/workspace}" SESSIONS_DIR="${RECALL_SESSIONS_DIR:-$HOME/.openclaw/agents/main/sessions}" MEMORY_DIR="$WORKSPACE/memory" DIGEST_DIR="$MEMORY_DIR/session-digests" STATE_FILE="$MEMORY_DIR/.digest-state.json" DRY_RUN=false RECENT="" ALL=false # Parse args while [[ $# -gt 0 ]]; do case $1 in --dry-run) DRY_RUN=true; shift ;; --all) ALL=true; shift ;; --recent) RECENT="$2"; shift 2 ;; *) echo "Unknown option: $1"; exit 1 ;; esac done # Create directories mkdir -p "$DIGEST_DIR" # Initialize state file if missing if [[ ! -f "$STATE_FILE" ]]; then echo '{"processed":[],"lastRun":0}' > "$STATE_FILE" fi # Check if sessions dir exists if [[ ! -d "$SESSIONS_DIR" ]]; then echo "⚠ Sessions directory not found: $SESSIONS_DIR" exit 0 fi # Get already processed sessions processed=$(jq -r '.processed[]' "$STATE_FILE" 2>/dev/null || echo "") # Find sessions to process if [[ -n "$(ls -A "$SESSIONS_DIR"/*.jsonl 2>/dev/null)" ]]; then all_sessions=$(ls -1 "$SESSIONS_DIR"/*.jsonl 2>/dev/null | xargs -I{} basename {} .jsonl) else echo "No session files found in $SESSIONS_DIR" exit 0 fi new_sessions="" if [[ "$ALL" == "true" ]]; then new_sessions="$all_sessions" else for s in $all_sessions; do if ! echo "$processed" | grep -q "^${s}$"; then new_sessions="$new_sessions $s" fi done fi # Apply --recent limit if [[ -n "$RECENT" ]]; then new_sessions=$(echo "$new_sessions" | tr ' ' '\n' | tail -n "$RECENT" | tr '\n' ' ') fi if [[ -z "$(echo $new_sessions | tr -d ' ')" ]]; then echo "✓ No new sessions to digest." exit 0 fi echo "🦊 Jasper Recall — Session Digester" echo "=" * 40 echo "Sessions to process: $(echo $new_sessions | wc -w)" echo "" # Process each session for session_id in $new_sessions; do session_file="$SESSIONS_DIR/${session_id}.jsonl" [[ ! -f "$session_file" ]] && continue size=$(du -h "$session_file" | cut -f1) msgs=$(wc -l < "$session_file") date=$(stat -c %y "$session_file" 2>/dev/null | cut -d' ' -f1 || stat -f %Sm -t %Y-%m-%d "$session_file" 2>/dev/null || echo "unknown") echo "Processing: ${session_id:0:8}... ($size, $msgs messages)" # Extract key info using jq topics=$(jq -r 'select(.message.role == "user") | .message.content | if type == "array" then map(select(.type == "text") | .text) | join(" ") else . end' "$session_file" 2>/dev/null | \ grep -v "^\[message_id:" | \ grep -v "^System:" | \ grep -v "^{" | \ head -10 || echo "") tools=$(jq -r '.message.content[]? | select(.type == "toolCall") | .name' "$session_file" 2>/dev/null | \ sort | uniq -c | sort -rn | head -5 | awk '{print $2 " (" $1 "x)"}' | tr '\n' ', ' | sed 's/, $//' || echo "") # Create digest file for this session digest_file="$DIGEST_DIR/${session_id:0:8}-$date.md" if [[ "$DRY_RUN" == "false" ]]; then cat > "$digest_file" << EOF # Session ${session_id:0:8} — $date **Size:** $size | **Messages:** $msgs **Tools:** ${tools:-none} ## Topics $(echo "$topics" | head -5 | sed 's/^/- /' | grep -v "^- $" || echo "- (no topics extracted)") --- *Full session: $session_file* EOF # Update state jq --arg s "$session_id" '.processed += [$s] | .lastRun = now' "$STATE_FILE" > "${STATE_FILE}.tmp" && mv "${STATE_FILE}.tmp" "$STATE_FILE" echo " ✓ Created: $(basename $digest_file)" else echo " [dry-run] Would create: $(basename $digest_file)" fi done echo "" echo "✓ Digests saved to: $DIGEST_DIR" FILE:scripts/index-digests.py #!/usr/bin/env python3 """ Index markdown files into ChromaDB for RAG retrieval. Reads from memory/, session-digests/, repos/, and founder-logs/. v0.3.0: Multi-collection architecture - private_memories: main agent only (default) - shared_memories: accessible to sandboxed agents - agent_learnings: insights from agent interactions (moltbook, etc.) """ import os import sys import glob import hashlib import requests from pathlib import Path # Support custom paths via environment WORKSPACE = os.environ.get("RECALL_WORKSPACE", os.path.expanduser("~/.openclaw/workspace")) CHROMA_DIR = os.environ.get("RECALL_CHROMA_DB", os.path.expanduser("~/.openclaw/chroma-db")) VENV_PATH = os.environ.get("RECALL_VENV", os.path.expanduser("~/.openclaw/rag-env")) MEMORY_DIR = os.path.join(WORKSPACE, "memory") DIGESTS_DIR = os.path.join(MEMORY_DIR, "session-digests") # Chunking config CHUNK_SIZE = 500 # characters CHUNK_OVERLAP = 100 SKILLBOSS_API_KEY = os.environ["SKILLBOSS_API_KEY"] API_BASE = "https://api.heybossai.com/v1" def get_embedding(text: str) -> list: """Get text embedding via SkillBoss API Hub.""" r = requests.post( f"{API_BASE}/pilot", headers={"Authorization": f"Bearer {SKILLBOSS_API_KEY}", "Content-Type": "application/json"}, json={"type": "embedding", "inputs": {"text": text}}, timeout=60, ) return r.json()["result"]["data"][0]["embedding"] def get_embeddings_batch(texts: list) -> list: """Get embeddings for multiple texts via SkillBoss API Hub.""" return [get_embedding(t) for t in texts] # Activate the venv sys.path.insert(0, os.path.join(VENV_PATH, "lib/python3.12/site-packages")) for pyver in ["python3.11", "python3.10"]: alt_path = os.path.join(VENV_PATH, f"lib/{pyver}/site-packages") if os.path.exists(alt_path): sys.path.insert(0, alt_path) try: import chromadb except ImportError as e: print(f"❌ Missing dependency: {e}", file=sys.stderr) print("Run 'npx jasper-recall setup' to install dependencies.", file=sys.stderr) sys.exit(1) def chunk_text(text: str, chunk_size: int = CHUNK_SIZE, overlap: int = CHUNK_OVERLAP) -> list: """Split text into overlapping chunks.""" chunks = [] start = 0 while start < len(text): end = start + chunk_size chunk = text[start:end] if chunk.strip(): chunks.append(chunk.strip()) start = end - overlap return chunks def get_file_hash(content: str) -> str: """Get MD5 hash of content.""" return hashlib.md5(content.encode()).hexdigest() def determine_collection(rel_path: str, content: str) -> str: """ Determine which collection a file belongs to based on path and content. Returns: 'private', 'shared', or 'learnings' """ rel_lower = rel_path.lower() content_lower = content.lower() # Agent learnings: moltbook insights, agent collaboration notes if any(x in rel_lower for x in ['moltbook/', 'learnings/', 'agent-insights/']): return 'learnings' if '[learning]' in content_lower or '[insight]' in content_lower: return 'learnings' # Shared: explicit shared folder or [public] tag if 'shared/' in rel_lower: return 'shared' if '[public]' in content_lower: return 'shared' # Default: private return 'private' def index_to_collection(collection, filepath, rel_path, content, file_hash, stats): """Index a file's chunks into a specific collection.""" filename = os.path.basename(filepath) # Check for existing chunks from this file try: existing = collection.get( where={"source": rel_path}, include=[] ) except Exception: existing = {'ids': []} if existing['ids']: # Check if hash matches (stored in first chunk's metadata) try: existing_meta = collection.get( ids=[existing['ids'][0]], include=["metadatas"] ) if existing_meta['metadatas'] and existing_meta['metadatas'][0].get('file_hash') == file_hash: stats['skipped'] += 1 return False except Exception: pass # File changed, delete old chunks collection.delete(ids=existing['ids']) # Chunk the content chunks = chunk_text(content) if not chunks: return False # Generate embeddings via SkillBoss API Hub embeddings = get_embeddings_batch(chunks) # Create IDs and metadata ids = [f"{rel_path}::{i}" for i in range(len(chunks))] metadatas = [ { "source": rel_path, "chunk_index": i, "file_hash": file_hash, "filename": filename, } for i in range(len(chunks)) ] # Add to collection collection.add( ids=ids, embeddings=embeddings, documents=chunks, metadatas=metadatas ) stats['chunks'] += len(chunks) stats['files'] += 1 return True def main(): print("🦊 Jasper Recall — RAG Indexer v0.3.0") print("=" * 40) # Check if memory dir exists if not os.path.exists(MEMORY_DIR): print(f"⚠ Memory directory not found: {MEMORY_DIR}") print("Create some markdown files there first.") sys.exit(1) print("✓ Using SkillBoss API Hub for embeddings") # Initialize ChromaDB os.makedirs(CHROMA_DIR, exist_ok=True) client = chromadb.PersistentClient(path=CHROMA_DIR) # Create collections with descriptions collections = { "private": client.get_or_create_collection( name="private_memories", metadata={"description": "Private agent memories - main agent only"} ), "shared": client.get_or_create_collection( name="shared_memories", metadata={"description": "Shared memories - accessible to sandboxed agents"} ), "learnings": client.get_or_create_collection( name="agent_learnings", metadata={"description": "Agent learnings and insights from interactions"} ), } # Also maintain legacy collection for backwards compatibility legacy_collection = client.get_or_create_collection( name="jasper_memory", metadata={"description": "Legacy collection - use specific collections instead"} ) print(f"✓ Collections: private_memories, shared_memories, agent_learnings") # Gather files to index files_to_index = [] # Session digests if os.path.exists(DIGESTS_DIR): files_to_index.extend(glob.glob(os.path.join(DIGESTS_DIR, "*.md"))) # Daily notes and other memory files (but not subdirs) files_to_index.extend(glob.glob(os.path.join(MEMORY_DIR, "*.md"))) # Repos documentation repos_dir = os.path.join(MEMORY_DIR, "repos") if os.path.exists(repos_dir): files_to_index.extend(glob.glob(os.path.join(repos_dir, "*.md"))) # Founder Logs for logs_dir_name in ["founder-logs", "founderLogs"]: logs_dir = os.path.join(MEMORY_DIR, logs_dir_name) if os.path.exists(logs_dir): files_to_index.extend(glob.glob(os.path.join(logs_dir, "*.md"))) # SOPs sops_dir = os.path.join(MEMORY_DIR, "sops") if os.path.exists(sops_dir): files_to_index.extend(glob.glob(os.path.join(sops_dir, "*.md"))) # Shared memory (public content for sandboxed agents) shared_dir = os.path.join(MEMORY_DIR, "shared") if os.path.exists(shared_dir): files_to_index.extend(glob.glob(os.path.join(shared_dir, "*.md"))) files_to_index.extend(glob.glob(os.path.join(shared_dir, "**/*.md"), recursive=True)) # Moltbook learnings moltbook_dir = os.path.join(MEMORY_DIR, "shared", "moltbook") if os.path.exists(moltbook_dir): files_to_index.extend(glob.glob(os.path.join(moltbook_dir, "*.md"))) # Remove duplicates while preserving order files_to_index = list(dict.fromkeys(files_to_index)) print(f"Found {len(files_to_index)} files to index") # Track stats per collection stats = { "private": {"files": 0, "chunks": 0, "skipped": 0}, "shared": {"files": 0, "chunks": 0, "skipped": 0}, "learnings": {"files": 0, "chunks": 0, "skipped": 0}, "legacy": {"files": 0, "chunks": 0, "skipped": 0}, } for filepath in files_to_index: filename = os.path.basename(filepath) rel_path = os.path.relpath(filepath, WORKSPACE) try: with open(filepath, 'r', encoding='utf-8') as f: content = f.read() except Exception as e: print(f" ⚠ Error reading {filename}: {e}") continue if not content.strip(): continue file_hash = get_file_hash(content) # Determine target collection coll_key = determine_collection(rel_path, content) collection = collections[coll_key] # Index to the appropriate collection indexed = index_to_collection( collection, filepath, rel_path, content, file_hash, stats[coll_key] ) # Also index to legacy collection for backwards compatibility index_to_collection( legacy_collection, filepath, rel_path, content, file_hash, stats["legacy"] ) if indexed: print(f" ✓ {filename} → {coll_key} ({stats[coll_key]['chunks']} chunks)") print("=" * 40) print("✓ Indexing complete") for key, s in stats.items(): if key == "legacy": continue if s['files'] > 0 or s['skipped'] > 0: print(f" {key}: {s['files']} files ({s['chunks']} chunks), {s['skipped']} skipped") print(f" Database: {CHROMA_DIR}") if __name__ == "__main__": main() FILE:scripts/install-mesh.sh #!/bin/bash # Install multi-agent mesh scripts to ~/.local/bin/ set -e BIN_DIR="$HOME/.local/bin" SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" echo "🦊 Installing Jasper Recall Multi-Agent Mesh..." echo "" # Ensure bin directory exists mkdir -p "$BIN_DIR" # Install scripts echo "Installing recall-mesh..." cp "$SCRIPT_DIR/recall-mesh" "$BIN_DIR/recall-mesh" chmod +x "$BIN_DIR/recall-mesh" echo "Installing index-digests-mesh..." cp "$SCRIPT_DIR/index-digests-mesh" "$BIN_DIR/index-digests-mesh" chmod +x "$BIN_DIR/index-digests-mesh" echo "" echo "✓ Multi-agent mesh installed!" echo "" echo "Usage:" echo " recall-mesh \"query\" --agent sonnet" echo " recall-mesh \"query\" --mesh sonnet,qwen,opus" echo " index-digests-mesh --agent sonnet" echo "" echo "Docs: ~/projects/jasper-recall/docs/MULTI-AGENT-MESH.md" FILE:scripts/privacy-check.py #!/usr/bin/env python3 """ Privacy checker for jasper-recall shared memory. Scans text for patterns that should not be shared publicly. Usage: privacy-check.py "text to check" privacy-check.py --file /path/to/file.md echo "text" | privacy-check.py --stdin """ import re import sys import argparse from pathlib import Path # Privacy patterns - things that should NOT appear in shared/public content PATTERNS = [ # Personal identifiers { "name": "email", "pattern": r"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}", "description": "Email address detected" }, { "name": "phone", "pattern": r"\b\d{3}[-.]?\d{3}[-.]?\d{4}\b", "description": "Phone number detected" }, # Paths and infrastructure { "name": "home_path", "pattern": r"/home/\w+/", "description": "Home directory path detected" }, { "name": "internal_ip", "pattern": r"\b(?:10|172\.(?:1[6-9]|2\d|3[01])|192\.168)\.\d{1,3}\.\d{1,3}\b", "description": "Internal IP address detected" }, { "name": "tailscale_ip", "pattern": r"\b100\.\d{1,3}\.\d{1,3}\.\d{1,3}\b", "description": "Tailscale IP detected" }, # Secrets and credentials { "name": "anthropic_key", "pattern": r"sk-ant-[a-zA-Z0-9-_]{20,}", "description": "Anthropic API key detected" }, { "name": "openai_key", "pattern": r"sk-[a-zA-Z0-9]{48}", "description": "OpenAI API key detected" }, { "name": "generic_key", "pattern": r"\b(?:api[_-]?key|secret|token|password)\s*[=:]\s*['\"]?[a-zA-Z0-9-_]{16,}['\"]?", "description": "Generic API key/secret detected", "flags": re.IGNORECASE }, { "name": "bearer_token", "pattern": r"Bearer\s+[a-zA-Z0-9-_\.]{20,}", "description": "Bearer token detected" }, # Private keywords { "name": "private_marker", "pattern": r"\[private\]", "description": "Content explicitly marked as private", "flags": re.IGNORECASE }, { "name": "secret_keyword", "pattern": r"\b(?:confidential|internal[_-]only|do[_\s]not[_\s]share)\b", "description": "Confidentiality keyword detected", "flags": re.IGNORECASE }, # MongoDB/Database URIs { "name": "mongodb_uri", "pattern": r"mongodb(?:\+srv)?://[^\s]+", "description": "MongoDB connection string detected" }, # SSH/Server references { "name": "ssh_user", "pattern": r"\bssh\s+\w+@", "description": "SSH connection string detected" }, ] # Allowlist - these are OK even if they match patterns ALLOWLIST = [ # Product names and branding "jasper-recall", "hopeIDS", "hopeid", "OpenClaw", "openclaw", "E.x.O.", "exohaven.online", "exocreate.online", "clawhub.ai", # Public URLs "github.com", "npm", "npx", # Example placeholders "example.com", "user@example.com", "sk-xxx", "your-api-key", ] def check_text(text: str) -> list: """ Check text for privacy violations. Returns list of {pattern, match, description, line} dicts. """ violations = [] lines = text.split('\n') for line_num, line in enumerate(lines, 1): # Skip if line contains allowlisted terms in context line_lower = line.lower() for pattern_def in PATTERNS: flags = pattern_def.get("flags", 0) matches = re.finditer(pattern_def["pattern"], line, flags) for match in matches: matched_text = match.group() # Check against allowlist is_allowed = any( allowed.lower() in matched_text.lower() or matched_text.lower() in allowed.lower() for allowed in ALLOWLIST ) if not is_allowed: violations.append({ "pattern": pattern_def["name"], "match": matched_text, "description": pattern_def["description"], "line": line_num, "context": line.strip()[:100] }) return violations def main(): parser = argparse.ArgumentParser(description="Check text for privacy violations") parser.add_argument("text", nargs="?", help="Text to check") parser.add_argument("--file", "-f", help="File to check") parser.add_argument("--stdin", action="store_true", help="Read from stdin") parser.add_argument("--json", action="store_true", help="Output as JSON") parser.add_argument("--quiet", "-q", action="store_true", help="Only output if violations found") args = parser.parse_args() # Get text to check if args.file: text = Path(args.file).read_text() elif args.stdin: text = sys.stdin.read() elif args.text: text = args.text else: parser.print_help() sys.exit(1) violations = check_text(text) if args.json: import json print(json.dumps({"clean": len(violations) == 0, "violations": violations}, indent=2)) elif violations: print(f"⚠️ PRIVACY VIOLATIONS FOUND: {len(violations)}\n") for v in violations: print(f" [{v['pattern']}] Line {v['line']}: {v['description']}") print(f" Match: {v['match']}") print(f" Context: {v['context'][:80]}...") print() sys.exit(1) elif not args.quiet: print("✅ CLEAN - No privacy violations detected") sys.exit(0 if not violations else 1) if __name__ == "__main__": main() FILE:scripts/recall.py #!/usr/bin/env python3 """ RAG recall: Search agent memory for relevant context. Usage: recall "query" [--limit N] [--json] [--verbose] [--collection NAME] v0.3.0: Multi-collection support - private_memories: main agent only (default for main agent) - shared_memories: accessible to sandboxed agents - agent_learnings: insights from agent interactions - all: search all collections (main agent only) """ import os import sys import argparse import json import requests # Support custom paths via environment CHROMA_DIR = os.environ.get("RECALL_CHROMA_DB", os.path.expanduser("~/.openclaw/chroma-db")) VENV_PATH = os.environ.get("RECALL_VENV", os.path.expanduser("~/.openclaw/rag-env")) SKILLBOSS_API_KEY = os.environ["SKILLBOSS_API_KEY"] API_BASE = "https://api.heybossai.com/v1" def get_embedding(text: str) -> list: """Get text embedding via SkillBoss API Hub.""" r = requests.post( f"{API_BASE}/pilot", headers={"Authorization": f"Bearer {SKILLBOSS_API_KEY}", "Content-Type": "application/json"}, json={"type": "embedding", "inputs": {"text": text}}, timeout=60, ) return r.json()["result"]["data"][0]["embedding"] # Collection names COLLECTIONS = { "private": "private_memories", "shared": "shared_memories", "learnings": "agent_learnings", "legacy": "jasper_memory", } # Activate the venv sys.path.insert(0, os.path.join(VENV_PATH, "lib/python3.12/site-packages")) for pyver in ["python3.11", "python3.10"]: alt_path = os.path.join(VENV_PATH, f"lib/{pyver}/site-packages") if os.path.exists(alt_path): sys.path.insert(0, alt_path) try: import chromadb except ImportError as e: print(f"❌ Missing dependency: {e}", file=sys.stderr) print("Run 'npx jasper-recall setup' to install dependencies.", file=sys.stderr) sys.exit(1) def search_collection(collection, query_embedding, limit): """Search a single collection and return results.""" try: results = collection.query( query_embeddings=[query_embedding], n_results=limit, include=["documents", "metadatas", "distances"] ) return results except Exception as e: return None def merge_results(all_results, limit): """Merge and sort results from multiple collections by similarity.""" merged = [] for coll_name, results in all_results.items(): if not results or not results['documents'][0]: continue for doc, meta, dist in zip( results['documents'][0], results['metadatas'][0], results['distances'][0] ): merged.append({ "collection": coll_name, "document": doc, "metadata": meta, "distance": dist, "similarity": 1 - dist }) # Sort by similarity (descending) merged.sort(key=lambda x: x['similarity'], reverse=True) return merged[:limit] def main(): parser = argparse.ArgumentParser(description="Search agent memory") parser.add_argument("query", help="Search query") parser.add_argument("-n", "--limit", type=int, default=5, help="Number of results (default: 5)") parser.add_argument("--json", action="store_true", help="Output as JSON") parser.add_argument("-v", "--verbose", action="store_true", help="Show similarity scores") parser.add_argument("--public-only", action="store_true", help="Only search shared content (for sandboxed agents)") parser.add_argument("-c", "--collection", choices=["private", "shared", "learnings", "all", "legacy"], default=None, help="Specific collection to search (default: all for main, shared for --public-only)") args = parser.parse_args() if not os.path.exists(CHROMA_DIR): print("❌ No index found. Run 'index-digests' first.", file=sys.stderr) sys.exit(1) # Load database client = chromadb.PersistentClient(path=CHROMA_DIR) # Determine which collections to search if args.public_only: # Sandboxed agents: only shared + learnings (public content) if args.collection: if args.collection not in ["shared", "learnings"]: print(f"❌ --public-only restricts to 'shared' or 'learnings' collections", file=sys.stderr) sys.exit(1) search_collections = [args.collection] else: search_collections = ["shared", "learnings"] elif args.collection: if args.collection == "all": search_collections = ["private", "shared", "learnings"] else: search_collections = [args.collection] else: # Default for main agent: search all collections search_collections = ["private", "shared", "learnings"] # Get collections collections_to_query = {} for coll_key in search_collections: coll_name = COLLECTIONS.get(coll_key, coll_key) try: collections_to_query[coll_key] = client.get_collection(coll_name) except Exception: # Collection doesn't exist yet, skip pass if not collections_to_query: # Fallback to legacy collection try: collections_to_query["legacy"] = client.get_collection("jasper_memory") except Exception: print("❌ No collections found. Run 'index-digests' first.", file=sys.stderr) sys.exit(1) # Embed query via SkillBoss API Hub query_embedding = get_embedding(args.query) # Search each collection all_results = {} for coll_key, collection in collections_to_query.items(): results = search_collection(collection, query_embedding, args.limit * 2) if results: all_results[coll_key] = results # Merge and limit results merged = merge_results(all_results, args.limit) if not merged: if args.json: print("[]") else: print(f"🔍 No results for: \"{args.query}\"") return if args.json: output = [] for i, item in enumerate(merged): output.append({ "rank": i + 1, "collection": item["collection"], "source": item["metadata"].get("source", "unknown"), "similarity": round(item["similarity"], 3), "content": item["document"] }) print(json.dumps(output, indent=2)) else: searched = ", ".join(search_collections) print(f"🔍 Results for: \"{args.query}\" (searched: {searched})\n") for i, item in enumerate(merged): similarity = item["similarity"] score_str = f" ({similarity:.1%})" if args.verbose else "" source = item["metadata"].get("source", "unknown") coll_tag = f"[{item['collection']}] " if len(search_collections) > 1 else "" print(f"━━━ [{i+1}] {coll_tag}{source}{score_str} ━━━") # Truncate long content content = item["document"] content = content[:500] + "..." if len(content) > 500 else content print(content) print() if __name__ == "__main__": main() FILE:scripts/summarize-old.py #!/usr/bin/env python3 """ summarize-old — Compress old memory entries to save tokens. Usage: summarize-old # Summarize entries older than 30 days summarize-old --days 14 # Summarize entries older than 14 days summarize-old --dry-run # Preview what would be summarized summarize-old --min-size 1000 # Only summarize files larger than 1000 chars How it works: 1. Finds markdown files older than N days 2. Creates condensed summaries (preserving key facts) 3. Archives originals to memory/archive/ 4. Updates the summarized files in place The summarization is rule-based (no LLM required): - Extracts headings, bullet points, and key dates - Preserves [public]/[private] tags - Removes verbose explanations - Keeps first/last sentences of long paragraphs """ import os import sys import re import shutil import argparse from datetime import datetime, timedelta from pathlib import Path # Support custom paths via environment WORKSPACE = os.environ.get("RECALL_WORKSPACE", os.path.expanduser("~/.openclaw/workspace")) MEMORY_DIR = os.path.join(WORKSPACE, "memory") ARCHIVE_DIR = os.path.join(MEMORY_DIR, "archive") # File types to summarize SUMMARIZE_DIRS = [ "session-digests", "daily", # if daily notes exist ] # Never summarize these SKIP_PATTERNS = [ "MEMORY.md", "README.md", "shared/", # Don't modify shared content "sops/", # SOPs should stay detailed "archive/", # Already archived ] def should_skip(filepath: str) -> bool: """Check if file should be skipped.""" for pattern in SKIP_PATTERNS: if pattern in filepath: return True return False def get_file_age_days(filepath: str) -> int: """Get file age in days based on modification time.""" mtime = os.path.getmtime(filepath) age = datetime.now() - datetime.fromtimestamp(mtime) return age.days def extract_key_content(content: str) -> str: """ Extract key information from content using rule-based summarization. Preserves structure while reducing verbosity. """ lines = content.split('\n') summary_lines = [] in_code_block = False paragraph_buffer = [] for line in lines: stripped = line.strip() # Track code blocks (preserve them shorter) if stripped.startswith('```'): in_code_block = not in_code_block if in_code_block: summary_lines.append(line) else: # End code block - keep max 5 lines summary_lines.append(line) continue if in_code_block: # Keep first 5 lines of code blocks code_lines = [l for l in summary_lines if not l.strip().startswith('#')] if len(code_lines) < 5: summary_lines.append(line) continue # Always keep headings if stripped.startswith('#'): if paragraph_buffer: summary_lines.append(summarize_paragraph(paragraph_buffer)) paragraph_buffer = [] summary_lines.append(line) continue # Always keep bullet points and lists if re.match(r'^[-*•]\s+', stripped) or re.match(r'^\d+\.\s+', stripped): if paragraph_buffer: summary_lines.append(summarize_paragraph(paragraph_buffer)) paragraph_buffer = [] # Truncate long bullets if len(stripped) > 150: summary_lines.append(line[:150] + '...') else: summary_lines.append(line) continue # Keep lines with dates, times, or key markers if re.search(r'\d{4}-\d{2}-\d{2}|\[public\]|\[private\]|\[learning\]|TODO|DONE|BLOCKED', stripped, re.I): if paragraph_buffer: summary_lines.append(summarize_paragraph(paragraph_buffer)) paragraph_buffer = [] summary_lines.append(line) continue # Keep lines with important keywords if re.search(r'important|critical|decision|agreed|conclusion|result|outcome', stripped, re.I): if paragraph_buffer: summary_lines.append(summarize_paragraph(paragraph_buffer)) paragraph_buffer = [] summary_lines.append(line) continue # Empty line - flush paragraph buffer if not stripped: if paragraph_buffer: summary_lines.append(summarize_paragraph(paragraph_buffer)) paragraph_buffer = [] summary_lines.append('') continue # Regular text - buffer for paragraph summarization paragraph_buffer.append(line) # Flush remaining buffer if paragraph_buffer: summary_lines.append(summarize_paragraph(paragraph_buffer)) # Clean up multiple empty lines result = '\n'.join(summary_lines) result = re.sub(r'\n{3,}', '\n\n', result) return result.strip() def summarize_paragraph(lines: list) -> str: """Summarize a paragraph by keeping first and last sentences if long.""" text = ' '.join(l.strip() for l in lines) if len(text) < 200: return text # Split into sentences (rough) sentences = re.split(r'(?<=[.!?])\s+', text) if len(sentences) <= 2: return text[:200] + '...' if len(text) > 200 else text # Keep first and last sentence return f"{sentences[0]} [...] {sentences[-1]}" def summarize_file(filepath: str, dry_run: bool = False) -> dict: """ Summarize a single file. Returns dict with stats. """ with open(filepath, 'r', encoding='utf-8') as f: original = f.read() original_size = len(original) # Extract key content summarized = extract_key_content(original) # Add summary header filename = os.path.basename(filepath) summary_header = f"\n\n" summarized = summary_header + summarized new_size = len(summarized) reduction = ((original_size - new_size) / original_size) * 100 if original_size > 0 else 0 result = { "file": filepath, "original_size": original_size, "new_size": new_size, "reduction_pct": reduction, } if dry_run: return result # Archive original rel_path = os.path.relpath(filepath, MEMORY_DIR) archive_path = os.path.join(ARCHIVE_DIR, rel_path) os.makedirs(os.path.dirname(archive_path), exist_ok=True) shutil.copy2(filepath, archive_path) # Write summarized version with open(filepath, 'w', encoding='utf-8') as f: f.write(summarized) result["archived_to"] = archive_path return result def main(): parser = argparse.ArgumentParser(description="Summarize old memory entries to save tokens") parser.add_argument("--days", type=int, default=30, help="Summarize files older than N days (default: 30)") parser.add_argument("--min-size", type=int, default=500, help="Minimum file size in chars to summarize (default: 500)") parser.add_argument("--dry-run", action="store_true", help="Preview without making changes") parser.add_argument("-v", "--verbose", action="store_true", help="Show detailed output") args = parser.parse_args() print("🦊 Jasper Recall — Memory Summarizer") print("=" * 40) print(f"Summarizing files older than {args.days} days") if args.dry_run: print("(dry-run mode - no changes will be made)") print() # Find files to summarize files_to_process = [] for subdir in SUMMARIZE_DIRS: dir_path = os.path.join(MEMORY_DIR, subdir) if not os.path.exists(dir_path): continue for filepath in Path(dir_path).rglob("*.md"): filepath = str(filepath) if should_skip(filepath): continue age = get_file_age_days(filepath) if age < args.days: continue size = os.path.getsize(filepath) if size < args.min_size: continue files_to_process.append((filepath, age, size)) if not files_to_process: print("✓ No files need summarization.") return print(f"Found {len(files_to_process)} files to summarize") print() # Process files total_saved = 0 for filepath, age, original_size in files_to_process: filename = os.path.basename(filepath) result = summarize_file(filepath, dry_run=args.dry_run) saved = result["original_size"] - result["new_size"] total_saved += saved if args.verbose or args.dry_run: print(f" {filename} ({age}d old)") print(f" {result['original_size']:,} → {result['new_size']:,} chars ({result['reduction_pct']:.0f}% reduction)") else: status = "[dry-run]" if args.dry_run else "✓" print(f" {status} {filename}: {result['reduction_pct']:.0f}% smaller") print() print("=" * 40) if args.dry_run: print(f"Would save ~{total_saved:,} characters ({total_saved // 4:,} tokens est.)") else: print(f"✓ Saved {total_saved:,} characters (~{total_saved // 4:,} tokens)") print(f" Originals archived to: {ARCHIVE_DIR}") if __name__ == "__main__": main() FILE:scripts/sync-shared.py #!/usr/bin/env python3 """ Sync [public] tagged content from daily notes to shared memory. Part of jasper-recall's shared agent memory system. Usage: sync-shared.py # Sync today's notes sync-shared.py --since 7d # Last 7 days sync-shared.py --all # All daily notes sync-shared.py --dry-run # Preview only """ import re import os import sys import argparse from pathlib import Path from datetime import datetime, timedelta # Paths WORKSPACE = Path(os.environ.get("RECALL_WORKSPACE", "~/.openclaw/workspace")).expanduser() MEMORY_DIR = WORKSPACE / "memory" SHARED_DIR = MEMORY_DIR / "shared" PRODUCT_UPDATES = SHARED_DIR / "product-updates.md" LEARNINGS = SHARED_DIR / "learnings.md" # Pattern to match [public] tagged sections # Matches: ## DATE [public] - Title or ## [public] Title PUBLIC_SECTION_PATTERN = re.compile( r'^(#{1,3})\s+(?:\d{4}-\d{2}-\d{2}\s+)?\[public\]\s*[-–]?\s*(.+?)$\n((?:(?!^#{1,3}\s).+\n?)*)', re.MULTILINE | re.IGNORECASE ) def find_daily_notes(since_days: int = None, all_notes: bool = False) -> list: """Find daily note files to process.""" notes = [] for f in MEMORY_DIR.glob("????-??-??.md"): # Parse date from filename try: note_date = datetime.strptime(f.stem, "%Y-%m-%d") except ValueError: continue # Filter by date if needed if not all_notes and since_days: cutoff = datetime.now() - timedelta(days=since_days) if note_date < cutoff: continue elif not all_notes: # Default: only today if note_date.date() != datetime.now().date(): continue notes.append(f) return sorted(notes, key=lambda f: f.stem) def extract_public_sections(filepath: Path) -> list: """Extract [public] tagged sections from a file.""" content = filepath.read_text() sections = [] for match in PUBLIC_SECTION_PATTERN.finditer(content): level = match.group(1) title = match.group(2).strip() body = match.group(3).strip() # Get date from filename or title date = filepath.stem if re.match(r'\d{4}-\d{2}-\d{2}', filepath.stem) else "unknown" sections.append({ "date": date, "level": level, "title": title, "body": body, "source": filepath.name }) return sections def categorize_section(section: dict) -> str: """Determine if section is a product update or learning.""" title_lower = section["title"].lower() body_lower = section["body"].lower() # Product update indicators product_keywords = ["release", "ship", "launch", "version", "v0.", "v1.", "npm", "published", "deployed"] if any(kw in title_lower or kw in body_lower for kw in product_keywords): return "product" # Learning indicators learning_keywords = ["learn", "pattern", "insight", "discovery", "found that", "realized", "gotcha", "tip"] if any(kw in title_lower or kw in body_lower for kw in learning_keywords): return "learning" # Default to product update return "product" def format_section(section: dict) -> str: """Format a section for the shared file.""" return f"## {section['date']} [public] - {section['title']}\n\n{section['body']}\n" def update_shared_file(filepath: Path, new_sections: list, dry_run: bool = False): """Append new sections to a shared file, avoiding duplicates.""" if not filepath.exists(): filepath.parent.mkdir(parents=True, exist_ok=True) existing_content = f"# {filepath.stem.replace('-', ' ').title()}\n\n---\n\n" else: existing_content = filepath.read_text() # Track what's already in the file (by title) existing_titles = set(re.findall(r'^## .+ - (.+)$', existing_content, re.MULTILINE)) added = [] for section in new_sections: if section["title"] not in existing_titles: added.append(section) if not added: return [] # Find insertion point (before "---" footer or at end) insert_point = existing_content.rfind("\n---\n*Last") if insert_point == -1: insert_point = len(existing_content) # Build new content new_content = "\n".join(format_section(s) for s in added) updated = existing_content[:insert_point] + new_content + "\n" + existing_content[insert_point:] # Update timestamp timestamp = datetime.now().strftime("%Y-%m-%d %H:%M") updated = re.sub(r'\*Last (?:synced|updated): .+\*', f'*Last synced: {timestamp}*', updated) if not dry_run: filepath.write_text(updated) return added def main(): parser = argparse.ArgumentParser(description="Sync [public] content to shared memory") parser.add_argument("--since", help="Process notes from last N days (e.g., 7d)") parser.add_argument("--all", action="store_true", help="Process all daily notes") parser.add_argument("--dry-run", action="store_true", help="Preview without writing") args = parser.parse_args() # Parse --since since_days = None if args.since: match = re.match(r'(\d+)d', args.since) if match: since_days = int(match.group(1)) # Find notes to process notes = find_daily_notes(since_days=since_days, all_notes=args.all) if not notes: print("No daily notes found to process") return print(f"Processing {len(notes)} daily note(s)...") if args.dry_run: print("(DRY RUN - no files will be modified)\n") # Extract all public sections all_sections = [] for note in notes: sections = extract_public_sections(note) if sections: print(f" {note.name}: {len(sections)} [public] section(s)") all_sections.extend(sections) if not all_sections: print("\nNo [public] sections found") return # Categorize and update product_sections = [s for s in all_sections if categorize_section(s) == "product"] learning_sections = [s for s in all_sections if categorize_section(s) == "learning"] print(f"\nFound: {len(product_sections)} product updates, {len(learning_sections)} learnings") # Update files if product_sections: added = update_shared_file(PRODUCT_UPDATES, product_sections, args.dry_run) if added: print(f"\n{'Would add' if args.dry_run else 'Added'} to product-updates.md:") for s in added: print(f" - {s['title']}") if learning_sections: added = update_shared_file(LEARNINGS, learning_sections, args.dry_run) if added: print(f"\n{'Would add' if args.dry_run else 'Added'} to learnings.md:") for s in added: print(f" - {s['title']}") if not args.dry_run: print("\n✅ Sync complete") if __name__ == "__main__": main() FILE:scripts/write-learning.py #!/usr/bin/env python3 """ Write a learning to the agent_learnings collection. Designed for sandboxed agents to contribute back to shared memory. Usage: write-learning "Brief title" "Learning content..." write-learning --agent moltbook "Title" "Content" write-learning --category engagement "Title" "Content" write-learning --dry-run "Title" "Content" """ import os import sys import argparse import json import hashlib from datetime import datetime from pathlib import Path # Support custom paths via environment WORKSPACE = os.environ.get("RECALL_WORKSPACE", os.path.expanduser("~/.openclaw/workspace")) CHROMA_DIR = os.environ.get("RECALL_CHROMA_DB", os.path.expanduser("~/.openclaw/chroma-db")) VENV_PATH = os.environ.get("RECALL_VENV", os.path.expanduser("~/.openclaw/rag-env")) SHARED_DIR = os.path.join(WORKSPACE, "memory", "shared") LEARNINGS_FILE = os.path.join(SHARED_DIR, "agent-learnings.md") COLLECTION_LEARNINGS = "agent_learnings" # Activate the venv sys.path.insert(0, os.path.join(VENV_PATH, "lib/python3.12/site-packages")) for pyver in ["python3.11", "python3.10"]: alt_path = os.path.join(VENV_PATH, f"lib/{pyver}/site-packages") if os.path.exists(alt_path): sys.path.insert(0, alt_path) try: import chromadb from sentence_transformers import SentenceTransformer except ImportError as e: print(f"❌ Missing dependency: {e}", file=sys.stderr) print("Run 'npx jasper-recall setup' to install dependencies.", file=sys.stderr) sys.exit(1) def generate_id(title: str, agent: str, timestamp: str) -> str: """Generate a unique ID for the learning.""" content = f"{agent}:{title}:{timestamp}" return hashlib.md5(content.encode()).hexdigest()[:12] def append_to_learnings_file(title: str, content: str, agent: str, category: str, dry_run: bool = False): """Append learning to the markdown file for human readability.""" os.makedirs(os.path.dirname(LEARNINGS_FILE), exist_ok=True) timestamp = datetime.now().strftime("%Y-%m-%d %H:%M") date = datetime.now().strftime("%Y-%m-%d") entry = f"\n## {date} [{category}] - {title}\n" entry += f"*Agent: {agent} | {timestamp}*\n\n" entry += f"{content}\n" if dry_run: print("\n📄 Would append to agent-learnings.md:") print("-" * 40) print(entry) return # Create file with header if it doesn't exist if not os.path.exists(LEARNINGS_FILE): with open(LEARNINGS_FILE, 'w') as f: f.write("# Agent Learnings\n\n") f.write("Insights and learnings contributed by sandboxed agents.\n\n") f.write("---\n") # Append entry with open(LEARNINGS_FILE, 'a') as f: f.write(entry) print(f"📄 Added to {os.path.relpath(LEARNINGS_FILE, WORKSPACE)}") def index_to_chromadb(title: str, content: str, agent: str, category: str, dry_run: bool = False): """Index the learning directly to ChromaDB.""" if dry_run: print("\n🗄️ Would index to agent_learnings collection") return # Initialize os.makedirs(CHROMA_DIR, exist_ok=True) client = chromadb.PersistentClient(path=CHROMA_DIR) collection = client.get_or_create_collection( name=COLLECTION_LEARNINGS, metadata={"description": "Learnings written by sandboxed agents"} ) # Load model model = SentenceTransformer('all-MiniLM-L6-v2') # Prepare document timestamp = datetime.now().isoformat() doc_id = generate_id(title, agent, timestamp) # Combine title and content for embedding full_text = f"{title}\n\n{content}" embedding = model.encode([full_text])[0].tolist() metadata = { "source": f"agent-learnings/{agent}/{doc_id}", "filename": "agent-learnings.md", "agent": agent, "category": category, "title": title, "timestamp": timestamp, } # Add to collection collection.add( ids=[doc_id], embeddings=[embedding], documents=[full_text], metadatas=[metadata] ) print(f"🗄️ Indexed to {COLLECTION_LEARNINGS} (id: {doc_id})") def main(): parser = argparse.ArgumentParser(description="Write a learning to shared memory") parser.add_argument("title", help="Brief title for the learning") parser.add_argument("content", help="Learning content/description") parser.add_argument("--agent", default="unknown", help="Agent name (e.g., moltbook, coder)") parser.add_argument("--category", default="insight", choices=["insight", "engagement", "pattern", "bug", "success", "failure"], help="Category of learning") parser.add_argument("--dry-run", action="store_true", help="Preview without writing") parser.add_argument("--json", action="store_true", help="Output as JSON") args = parser.parse_args() # Validate inputs if len(args.title) > 200: print("❌ Title too long (max 200 chars)", file=sys.stderr) sys.exit(1) if len(args.content) > 5000: print("❌ Content too long (max 5000 chars)", file=sys.stderr) sys.exit(1) print(f"📝 Writing learning from agent '{args.agent}'...") print(f" Category: {args.category}") print(f" Title: {args.title}") if args.dry_run: print("\n(DRY RUN - no changes will be made)") # Write to both file and ChromaDB append_to_learnings_file(args.title, args.content, args.agent, args.category, args.dry_run) index_to_chromadb(args.title, args.content, args.agent, args.category, args.dry_run) if not args.dry_run: print("\n✅ Learning saved!") if args.json: print(json.dumps({ "success": True, "title": args.title, "agent": args.agent, "category": args.category })) if __name__ == "__main__": main() FILE:src/index.js /** * Jasper Recall * Local RAG system for AI agent memory * * This module exports utilities for programmatic access. * For CLI usage, use the `jasper-recall` command. */ const { execSync } = require('child_process'); const path = require('path'); const os = require('os'); const BIN_PATH = path.join(os.homedir(), '.local', 'bin'); /** * Search the memory index * @param {string} query - Search query * @param {Object} options - Options { limit, json, verbose } * @returns {Array|string} - Search results */ function recall(query, options = {}) { const args = [query]; if (options.limit) args.push('-n', options.limit); if (options.json) args.push('--json'); if (options.verbose) args.push('-v'); const recallPath = path.join(BIN_PATH, 'recall'); const result = execSync(`${recallPath} ${args.map(a => `"${a}"`).join(' ')}`, { encoding: 'utf8' }); return options.json ? JSON.parse(result) : result; } /** * Index memory files * @returns {string} - Index output */ function indexDigests() { const scriptPath = path.join(BIN_PATH, 'index-digests'); return execSync(scriptPath, { encoding: 'utf8' }); } /** * Process session logs into digests * @param {Object} options - Options { dryRun, all, recent } * @returns {string} - Digest output */ function digestSessions(options = {}) { const args = []; if (options.dryRun) args.push('--dry-run'); if (options.all) args.push('--all'); if (options.recent) args.push('--recent', options.recent); const scriptPath = path.join(BIN_PATH, 'digest-sessions'); return execSync(`${scriptPath} ${args.join(' ')}`, { encoding: 'utf8' }); } module.exports = { recall, indexDigests, digestSessions }; FILE:WORK-QUEUE.md # Jasper Recall Work Queue **Goal:** Bidirectional memory sharing between agents with privacy controls **Updated:** 2026-02-05 04:20 UTC --- ### HIGH - [x] **JR-10:** Memory tagging convention ([public]/[private] in daily notes) - **DONE @JASPER done:2026-02-05 30m** - [x] **JR-11:** Shared memory directory (memory/shared/ + symlink to sandboxed workspaces) - **DONE @JASPER done:2026-02-05 15m** - [x] **JR-12:** Public-only recall flag (--public-only filters to shared content) - **DONE @JASPER done:2026-02-05** - [x] **JR-13:** Privacy filter for memory writes (privacy-check.py script) - **DONE @JASPER done:2026-02-05 45m** - [x] **JR-14:** Bidirectional sync cron (sync-shared 2x daily via OpenClaw cron) - **DONE @JASPER done:2026-02-05** `depends_on:[jr-10, jr-11]` - [x] **JR-15:** Moltbook learnings capture (post-comment.js logs to shared/moltbook/) - **DONE @JASPER done:2026-02-05** ### MEDIUM - [x] **JR-16:** Reflection before post workflow (privacy checklist in moltbook AGENTS.md) - **DONE @JASPER done:2026-02-05 10m** - [x] **JR-17:** Shared ChromaDB collections (private_memories, shared_memories, agent_learnings) - **DONE @QWEN done:2026-02-05 25m** `depends_on:[jr-12]` ### LOW - [x] **JR-18:** Memory summarization (compress old entries to save tokens) - **DONE @QWEN done:2026-02-05 20m** - [x] **JR-19:** Multi-agent mesh (N agents sharing memory, not just 2) `branch:feat/jr-19-multi-agent-mesh` - **DONE @SONNET done:2026-02-05 45m** --- ## Completed (v0.1.0) - [x] **JR-1:** Core recall command - **DONE** - [x] **JR-2:** digest-sessions script - **DONE** - [x] **JR-3:** index-digests script - **DONE** - [x] **JR-4:** npm package published - **DONE** - [x] **JR-5:** ClawHub skill published - **DONE** - [x] **JR-6:** Product page on exohaven - **DONE** - [x] **JR-7:** Blog post guide - **DONE** --- ## v0.2.0 Target: Shared Agent Memory **Release date:** Feb 7, 2026 New features: - Memory tagging ([public] vs [private]) - Shared memory directory with symlinks - Privacy filter for sandboxed agents - Bidirectional sync (main ↔ sandboxed) - Public-only recall mode

Jasper Recall

Comments (0)