EncryptShawn

---
name: openclaw_projects
description: >
  Builds a Project — a coordinated multi-agent team setup — inside OpenClaw, for
  any kind of team: software development, marketing, real estate, content, sales,
  operations, research, customer success, or anything else where multiple agents
  need to work together toward shared goals. Use this skill whenever someone says
  "set up a project", "create a project", "add a project to my team", "build a
  team", "make my agents work together", "configure agent coordination",
  "set up agent collaboration", "I want a team of agents", "how do I run multiple
  agents on one project", "wire up Asana for my agents", "wire up ClickUp for my
  agents", "add a new project", or anything similar — even if they don't explicitly
  say "project" or "team". Also trigger when someone asks how multiple agents
  should coordinate, share work, escalate, or hand off tasks. This skill creates
  the entire project folder structure (PROJECT.md rulebook, project.json config,
  queue files for inter-agent messaging, project-lock.json phase tracking,
  decision/issue/runbook documents, shared workspace) and updates OpenClaw config
  to wire agent-to-agent communication. It works through a structured interview:
  team identity, work structure, then a comprehensive AI-rewritten team plan for
  user review and fine-tuning before building. Supports any task manager backend
  (Asana, ClickUp) via the user's separately installed dependency skill.
  Multiple projects can coexist; agents can participate in multiple projects.
  Requires the openclaw-administrator skill (EncryptShawn) to be loaded.
  Recommends openclaw-recovery-manager (EncryptShawn) for safety.
  This skill does not make task-manager API calls itself — those are delegated
  to the user's installed Asana or ClickUp dependency skill.
  This skill does not read or store any credentials or secret values.
---

# OpenClaw Projects

This skill adds the concept of a **Project** to OpenClaw — a coordinated multi-agent team setup that lets one or more agents work together toward shared goals. Projects can be any team type: software development, marketing, real estate, content, sales, customer success, research, or anything else.

A Project is:
- A folder at `~/.openclaw/projects/[project-id]/` containing the team rulebook, configuration, shared workspace, and inter-agent message queues
- A defined workflow that takes work from intake through delivery
- A wiring layer that connects multiple agents through OpenClaw's agent-to-agent communication
- A coordination unit that uses one task manager (Asana or ClickUp) as the source of truth for task ownership and status

Multiple projects can coexist. Agents can participate in multiple projects.

---

## Contents

- [What This Skill Does](#what-this-skill-does)
- [Prerequisites](#prerequisites)
- [Credential and Security Model](#credential-and-security-model)
- [Step 0 — Safety First](#step-0--safety-first)
- [Step 1 — Discover Existing Setup](#step-1--discover-existing-setup)
- [Step 2 — Pass 1: Team Identity Interview](#step-2--pass-1-team-identity-interview)
- [Step 3 — Pass 2: Work Structure Interview](#step-3--pass-2-work-structure-interview)
- [Step 4 — Pass 3: AI-Drafted Team Plan for Review](#step-4--pass-3-ai-drafted-team-plan-for-review)
- [Step 5 — Capability Check](#step-5--capability-check)
- [Step 6 — Task Manager Setup](#step-6--task-manager-setup)
- [Step 7 — Create Project Folder Structure](#step-7--create-project-folder-structure)
- [Step 8 — Update Agent Workspaces](#step-8--update-agent-workspaces)
- [Step 9 — Update OpenClaw Config](#step-9--update-openclaw-config)
- [Step 10 — Smoke Test](#step-10--smoke-test)
- [Step 11 — Post-Setup Snapshot and Handoff](#step-11--post-setup-snapshot-and-handoff)
- [If Anything Goes Wrong](#if-anything-goes-wrong)

Reference files (read when needed):
- `references/project-files.md` — Full specification of every project folder file
- `references/workflow.md` — Universal workflow phases, escalation rules, queue formats
- `references/interview-questions.md` — Full interview question banks for Pass 1 and Pass 2
- `references/team-archetypes.md` — Common team patterns to draw on for examples
- `references/templates.md` — Parameterized templates for every file this skill generates (PROJECT.md, project.json, queue files, etc.) plus a placeholder reference table

---

## What This Skill Does

1. **Interviews the user** through a structured three-pass discovery process to understand the team
2. **Drafts a comprehensive team plan** — AI-rewrites the user's answers into a complete operational plan, filling gaps and tightening loose answers
3. **Reviews the plan with the user** for fine-tuning and approval before building
4. **Checks agent capabilities** against the planned work and surfaces any concerns (e.g., a vision-required task assigned to an agent without a vision-capable model)
5. **Creates the project folder** at `~/.openclaw/projects/[project-id]/` with all coordination files
6. **Updates each participating agent's workspace** with project references in their AGENTS.md
7. **Updates OpenClaw config** to enable agent-to-agent communication between project participants
8. **Walks through a smoke test** to verify the project is operational

## What This Skill Does NOT Do

- Create agents — agents must already exist (use OpenClaw's agent creation flow first)
- Choose models for agents — that's the user's decision, made when creating the agent
- Hold credentials — the task manager dependency skill handles that
- Make task manager API calls directly — delegated to the user's Asana or ClickUp skill

---

## Prerequisites

**Must be installed before running this skill:**
- **openclaw-administrator** (EncryptShawn) — used to update OpenClaw config and write workspace files

**Must already exist in OpenClaw:**
- Each agent that will participate in the project
- Each agent must have a model and fallback configured (this skill verifies but does not set models)

**Must be installed on the agents that will use it:**
- A task manager dependency skill — either Asana or ClickUp — installed on every agent that needs to read/write tasks. The user is responsible for installing this and configuring its credential (PAT, API key, or token) in their secret management system.

**Strongly recommended:**
- **openclaw-recovery-manager** (EncryptShawn) — provides config snapshots and rollback

**If the user has not yet created their agents**, stop and tell them:
> "Before we build a project, you need to create the agents that will participate in it. Each agent should have its model and fallback configured, and you should install the task manager skill (Asana or ClickUp) on each agent that will read or write tasks. Once your agents exist, come back and we'll set up the project."

---

## Credential and Security Model

**This skill never reads, stores, requests, or transmits credential values.**

This skill collects only the *names* of env vars (e.g., `PROJ_ASANA_PAT`) — never their values. The dependency skills the user installed (Asana skill, ClickUp skill, any other domain-specific skill) hold and use credentials. This skill passes credential env var *names* to those skills so they know which value to pull from the agent runtime environment.

Credentials must be stored in the user's secret management system (Kubernetes ConfigMap/Secret, .env file, or equivalent) before this skill runs.

---

## Step 0 — Safety First

1. Check if **openclaw-recovery-manager** is installed.
   - **Yes:** Take a snapshot. Label: `pre-project-setup-[project-id]-[date]`
   - **No:** Ask:
     > "I recommend installing openclaw-recovery-manager (EncryptShawn on ClawHub) before we proceed — it lets us roll back if anything goes wrong. Want to install it first, or proceed without it?"

---

## Step 1 — Discover Existing Setup

Before interviewing, gather context using openclaw-administrator:

1. List existing agents and their configured models
2. List existing projects in `~/.openclaw/projects/` (if any)
3. Check which task manager skills are installed on each agent (Asana, ClickUp, or both)

This context informs the interview — for example, if the user has 5 agents already, you can show them the list to pick from rather than asking them to type names.

If `~/.openclaw/projects/` doesn't exist yet, this is the first project. Note this — the agents won't have any "Active Projects" section in their AGENTS.md yet.

---

## Step 2 — Pass 1: Team Identity Interview

Read `references/interview-questions.md` for the full question bank. The goal of Pass 1 is to understand **who is on this team and what they each do.**

Ask the user the questions in order. After each block, briefly summarize what they said back to confirm before moving on.

Core Pass 1 questions:

```
Pass 1 — Team Identity

1. What is this project for? (one-sentence purpose, e.g., "Build and maintain
   the EZBI analytics platform" or "Generate marketing content for client
   campaigns")

2. What kind of team is this?
   - Software development
   - Marketing / creative
   - Real estate
   - Content / editorial
   - Sales / outreach
   - Customer success / support
   - Operations
   - Research
   - Other (describe it briefly)

3. What should this project be called?
   - Display name (e.g., "EZBI Platform")
   - Project ID (lowercase, hyphens or underscores, used in folder names — e.g., "ezbi")

4. Which agents will be on this team? (You can pick from your existing agents.)
   For each agent, what is their role on this project?
   (One agent might be "Project Manager" on this project and "Researcher" on
   another — the role is per-project.)

5. Which agent is client-facing? (Equivalent of a PM — receives requirements,
   talks to clients, owns the intake. There should be exactly one.)

6. Which agent validates feasibility? (Domain expert who reviews whether the
   work is doable before committing — e.g., engineer for dev work, strategist
   for marketing, broker for real estate.)

7. Which agent does quality review? (Reviews completed work before delivery
   to the client — e.g., QA for dev, creative director for marketing.)

8. Is there a human operator? (Final authority for merges, unresolvable
   escalations, client engagement when agents can't reach the client.)
   Yes / No — if yes, what's their alias? (e.g., "operator")

9. Are there any other roles? (Specialized contributors — designers,
   researchers, copywriters, etc.)
```

Record all answers. Confirm back before moving on.

---

## Step 3 — Pass 2: Work Structure Interview

The goal of Pass 2 is to understand **how the team works together day-to-day.**

```
Pass 2 — Work Structure

1. Task manager — which one?
   - Asana
   - ClickUp
   (You should already have the corresponding dependency skill installed on
   your agents. Confirm which one.)

2. What are the stages a piece of work goes through? (These become the
   task manager columns. Default suggestion based on team type — confirm
   or customize.)

3. What is the shared working medium?
   - Where does this team actually produce deliverables?
   - Examples: a git repo (devs), a Google Drive folder (marketing),
     a Notion workspace (content), a CRM system (sales/real estate),
     a shared file folder, or none / not applicable
   - If a git repo: SSH URL(s) for cloning
   - If a folder/workspace: path or link
   - If a CRM/external system: how do agents access it?

4. What does "done" look like before work goes to client review?
   - Devs: PR opened, all tests pass, QA reviewed
   - Marketing: copy approved by creative director, brand guidelines met
   - Real estate: listing complete, photos verified, pricing confirmed
   - Whatever fits this team

5. How does the team handle requirements?
   - One sprint at a time (recommended): one agreed scope completed before
     next is accepted
   - Continuous flow: new requirements can come in any time
   (One-sprint-at-a-time is strongly recommended for teams that need
   focused execution. Continuous flow is appropriate for teams handling
   high-volume small tasks.)

6. Escalation thresholds:
   - How long should an agent be stuck on the same problem before stopping
     and surfacing to a human? (Default: 24 hours of active work)
   - How many times can an agent re-escalate the same issue before stopping?
     (Default: 2 escalations to the feasibility-reviewer)
   - How long with no client response before involving the operator?
     (Default: 48 hours)

7. Does this team produce or consume any visual / media assets?
   - Yes: describe (mockups, photos, video, audio, diagrams)
   - No
   (If yes: assets will be stored as task-manager attachments primarily,
   with a fallback location in the project workspace.)

8. Anything else specific to this team that other agents would need to know?
   (Free-form — house style, client communication preferences, specific
   tools/platforms, compliance requirements, etc.)
```

Record all answers. Confirm back before moving on.

---

## Step 4 — Pass 3: AI-Drafted Team Plan for Review

This is the most important step. Take everything from Pass 1 and Pass 2 and produce a **comprehensive, operational team plan** — not a transcription of the user's answers, but an AI-rewritten, gap-filled version that any agent could read and immediately know how to operate.

The plan must include:

- **Team identity:** project name, ID, purpose, kind
- **Roster:** every agent, their role on this project, what they own, what they do not own
- **Task manager configuration:** which one, the column structure with each column's meaning
- **Shared working medium:** what it is, how agents access it, conventions for using it
- **Workflow phases:** every phase from intake through close, with who owns each, what triggers transitions
- **Escalation rules:** thresholds, who escalates to whom, when work stops
- **Communication protocol:** queue files between agents, who reads which queue
- **Visual/media handling:** if applicable, how assets are stored and referenced
- **What "done" means** at each phase
- **Anything specific** the user mentioned

Write this plan in clear, complete prose. Fill gaps the user didn't address explicitly — for example, if the user said "we have a designer" but didn't say what triggers the designer's involvement, infer reasonable defaults based on team type and write them in. Mark inferred items clearly so the user can correct them.

Present the plan to the user:

```
I've turned your answers into a full team plan. Read through it carefully —
this is what will go into PROJECT.md, which is what every agent on the team
reads to understand how to operate.

Items I inferred (not directly asked) are marked with [INFERRED].
Anything that doesn't match what you want, tell me and I'll revise.

[FULL PLAN HERE]

Does this match what you want? Anything to change before we build?
```

**Do not move to Step 5 until the user explicitly approves the plan.** This is the gate that prevents vague PROJECT.md files. Iterate as many times as needed.

---

## Step 5 — Capability Check

Before building, look at the approved plan and check whether the assigned agents can actually do what the plan asks of them.

Using openclaw-administrator, fetch each agent's configured model. Cross-reference against the plan:

- **Vision required?** If the plan involves the agent reviewing mockups, photos, screenshots, or any visual asset, check that the agent's model has vision capability. If not, flag it.
- **Long context required?** If the agent needs to read large documents (long specs, full codebases, large research corpora), check the model's context window. If under 200k and the work seems heavy, flag it.
- **Code-heavy work?** If the agent is doing software development, check the model has reasonable coding benchmarks. (If the user picked something obviously weak, mention it.)
- **Hallucination-sensitive work?** If the agent does requirements translation, client communication, or QA-style validation, a high-hallucination model is risky. Flag it.

**Output format:**

```
Capability check on the assigned agents:

✅ [agent-id] (role: PM) — model [model-name]
   No concerns.

⚠️  [agent-id] (role: FE Designer) — model [model-name]
   Concern: This role will review visual mockups, but [model-name] does not
   support vision. Consider using a vision-capable model for tasks that
   involve images, or assigning that work to a different agent.

⚠️  [agent-id] (role: QA) — model [model-name]
   Concern: This model has a [X]% hallucination rate per public benchmarks,
   which is high for QA work that needs precise pass/fail judgment.
   This is advice — not a blocker.

These are advisory only. You can proceed as-is, change agent models in
your OpenClaw config, or reassign work to different agents.
Proceed? (yes / make changes first)
```

Wait for the user. If they want to change agent models, that's their job — point them at openclaw-administrator. This skill does not set models.

---

## Step 6 — Task Manager Setup

Based on the user's choice in Pass 2:

### If task manager board does NOT exist yet

```
Create the task manager board manually:

1. Log into [Asana / ClickUp]
2. Create a new project / space named: [project_display_name]
3. Set up the columns in this exact order:
   [column list from the approved plan]
4. Invite all agent accounts as members
5. Copy the project ID / GID from the board URL
6. Share the ID here when ready
```

### Once the project ID is confirmed

Using the task manager dependency skill (via the client-facing agent, since they own task creation), add a project description / pinned note with:

```
Project: [project_display_name]
Project ID: [project-id]
Client-facing agent: [agent-id]
Feasibility reviewer: [agent-id]
QA: [agent-id]
Operator: [operator-alias or N/A]
Shared workspace: [path or description]
Task manager column meanings: [brief column legend]
```

Record the task manager project ID — it goes in `project.json`.

---

## Step 7 — Create Project Folder Structure

This creates the entire project at `~/.openclaw/projects/[project-id]/`. Read `references/project-files.md` for the full specification of each file.

### Folder layout

```
~/.openclaw/projects/[project-id]/
├── PROJECT.md                    ← Team rulebook from the approved plan
├── project.json                  ← Machine-readable config
├── project-lock.json             ← Phase tracker (initialized to "idle")
├── STATE.md                      ← Human-readable status
├── SHARED_MEMORY.md              ← Cross-agent knowledge store
├── DECISIONS.md                  ← Append-only decision log
├── KNOWN_ISSUES.md               ← Accepted limitations / debt
├── RUNBOOK.md                    ← Project operating guide (stub initially)
├── workspace/
│   ├── [shared medium]           ← Repo, folder, files, depending on team
│   ├── [media-folder/]           ← Only if team uses visual/media assets
│   ├── SPEC-CURRENT.md           ← Current accepted spec / brief
│   └── DELIVERABLES_GUIDE.md     ← Feasibility-reviewer's task plan (was IMPLEMENTATION_GUIDE.md)
└── queues/
    ├── to-[client-facing-role].md
    ├── to-[feasibility-reviewer-role].md
    ├── to-[feasibility-reviewer-role]-feasibility.md
    ├── to-[qa-role].md
    ├── to-[operator].md          ← Only if operator was specified
    └── to-[other-role].md         ← One per other role on the team
```

### Building each file

**PROJECT.md** — generate from the PROJECT.md template in `references/templates.md`, filling every placeholder with the approved plan content from Step 4. This is the single most important file — it must be complete and operational. Use the placeholder reference table at the bottom of `references/templates.md` to map each placeholder to its source.

**project.json** — generate from the project.json template in `references/templates.md`. Validate the result is valid JSON before writing. Fill in:
- `id` and `name` from Pass 1
- `task_manager` block — type (asana / clickup), project ID, columns
- `participants` — every agent with their project role and OpenClaw workspace path
- `client_facing_role`, `feasibility_reviewer_role`, `qa_role`, `operator` — pointers to the right roles
- `shared_workspace` and `shared_medium` — type and path/URL
- `visual_assets` block — only if the team uses media (Pass 2 #7)
- `queues` — file paths for each role's queue
- `escalation_rules` — values from Pass 2 #6

**project-lock.json** — initialize:
```json
{
  "phase": "idle",
  "sprint_id": null,
  "sprint_opened": null,
  "waiting_on": null,
  "last_updated": "[today]",
  "last_updated_by": "operator",
  "context": "Project initialized. Ready to receive first work.",
  "blocked_tasks": []
}
```

**STATE.md** — initialize:
```markdown
# [project_display_name] — Current State
**Phase:** Idle — Ready for first work
**Last updated:** [today] by operator
```

**Queue files** — initialize each one with header:
```
# Queue: to-[role]
# Format: [YYYY-MM-DD HH:MM] [FROM: agent-id] [TO: agent-id] [TASK: task-id or N/A]
# Append-only. Never delete entries. Mark processed with [READ].
```

**SHARED_MEMORY.md, DECISIONS.md, KNOWN_ISSUES.md** — each gets a header and an empty body.

**RUNBOOK.md** — generate a stub with section headers appropriate for the team type, plus a note:
```
This is a starting stub. The feasibility-reviewer should expand each section
as they learn the project. Devs / contributors read this before starting work.
```

**Shared medium initialization:**
- Git repo: `cd ~/.openclaw/projects/[project-id]/workspace && git clone [ssh-url] [repo-name]`
- Folder/Drive: create `workspace/[folder-name]/` and add a `LINKS.md` file with the external URL if not local
- CRM/external: skip — write a `workspace/EXTERNAL_SYSTEM.md` describing where work happens
- None: skip

**Media folder** — only if Pass 2 #7 was yes:
```bash
mkdir -p ~/.openclaw/projects/[project-id]/workspace/[media-folder-name]
```

---

## Step 8 — Update Agent Workspaces

For each participating agent, append (or update) an Active Projects section in their `AGENTS.md`:

```markdown
## Active Projects
- **[project_display_name]** — I am the [role] on this project.
  - Full rules: ~/.openclaw/projects/[project-id]/PROJECT.md
  - My queue: ~/.openclaw/projects/[project-id]/queues/to-[my-role].md
  - Shared workspace: ~/.openclaw/projects/[project-id]/workspace/
  - Check my queue at the start of every session before doing anything else.
  - Check ~/.openclaw/projects/[project-id]/project-lock.json to know what
    phase we are in before acting.
```

If the agent is already on other projects, **append** — do not overwrite. The agent should see all their active projects.

---

## Step 9 — Update OpenClaw Config

Use openclaw-administrator to update each participating agent's `agent_to_agent` allow list so agents on this project can communicate. The allow list should include every other agent on the project.

Be careful: if the agent is already on other projects, they may already have entries in their allow list for those project members. **Merge, don't replace.**

Example: if agent `engineer` is on projects A and B:
- Project A members: `pm-agent-a, dev-fe, dev-be, qa`
- Project B members: `pm-agent-b, designer, copywriter, qa`
- Final allow list for `engineer`: `pm-agent-a, dev-fe, dev-be, qa, pm-agent-b, designer, copywriter`

After updating, verify:
```
openclaw agents list --verbose
Confirm each agent's allow list includes all project members.
```

---

## Step 10 — Smoke Test

```
SMOKE TEST

Step 1: Manually create a test task in the [task-manager] [first-stage column]:
  Title: [TEST] Smoke test — verify [project-id]
  Description: Test task. The [client-facing role] agent should pick this up,
  acknowledge it, and either move it forward or post to a queue.

Step 2: Wait for the [client-facing role] agent's next heartbeat (up to 30 min).
  Watch for: task gaining a comment, or moving to another column.

Step 3: Confirm the agent is reading from the project folder.
  - Check [client-facing role] agent's session log
  - Should see references to ~/.openclaw/projects/[project-id]/PROJECT.md
    and the agent's queue file

Step 4: Confirm queue files are writable.
  - Either: trigger a small interaction that produces a queue entry
  - Or: manually write a test entry to one queue file and verify the
    receiving agent picks it up next session

Heartbeat confirmed working? (yes / no — describe what happened)
```

If something fails here, do not move to Step 11. Diagnose:
- Agent didn't pick up task → check their AGENTS.md has the project reference, check their task manager skill is installed and authenticated
- Agent picked up task but didn't write to queue → check `project-lock.json` is readable and `queues/` files exist with correct permissions
- Agent-to-agent message didn't arrive → check OpenClaw config allow list from Step 9

---

## Step 11 — Post-Setup Snapshot and Handoff

### Snapshot
If openclaw-recovery-manager is installed:
```
Take post-setup snapshot.
Label: post-project-setup-[project-id]-[date]-confirmed
```

### Handoff Summary

```
PROJECT SETUP COMPLETE

Project: [project_display_name] ([project-id])
Type: [team_type]
Folder: ~/.openclaw/projects/[project-id]/

ROSTER:
[role]               | [agent-id]              | [model]
[client-facing role] | [agent-id]              | [model]
[feasibility role]   | [agent-id]              | [model]
[qa role]            | [agent-id]              | [model]
[operator]           | human

TASK MANAGER:
Type: [Asana / ClickUp]
Project ID: [id]
Stages: [column list]

SHARED WORKSPACE:
[path or description]
[Repo SSH URL if applicable]

HOW TO START WORK:
Send your first requirements / brief / intake to [client-facing-agent-id].
The team will:
  1. Validate feasibility through [feasibility-reviewer]
  2. Get your sign-off on the plan
  3. Execute through [executing-roles]
  4. Quality-review through [qa-role]
  5. Notify [operator or you] when ready for sign-off

ESCALATION:
Stuck > [X]h or [Y] re-escalations → work stops, [operator] notified
Client no response > [Z]h → [operator] gets a message

ADD ANOTHER PROJECT:
Run this skill again. Same agents can join multiple projects without conflict.

RECOVERY:
Pre-setup snapshot:  pre-project-setup-[project-id]-[date]
Post-setup snapshot: post-project-setup-[project-id]-[date]-confirmed
```

---

## If Anything Goes Wrong

```
Option 1 — Recover using openclaw-recovery-manager:
  Restore: pre-project-setup-[project-id]-[date]
  Returns config to the state before setup began.

Option 2 — Diagnose with openclaw-administrator:
  Run diagnostics, identify what failed, retry just that step.

Option 3 — Describe what step failed and what error appeared.
  I can walk through the failed step again.
```

---

## Adding Agents to an Existing Project Later

If the user runs this skill against an existing project with the same project ID:
1. Detect the existing project folder
2. Ask: "Project [id] already exists. Are you adding agents, changing the structure, or something else?"
3. If adding agents: run only Pass 1 question 4 (agent assignment), check capabilities, update participants in `project.json`, append to AGENTS.md for new agents only, update allow lists in OpenClaw config
4. If changing structure: walk through the relevant interview sections, regenerate PROJECT.md, leave history files (DECISIONS.md, SHARED_MEMORY.md) intact

Do not overwrite history files (DECISIONS.md, SHARED_MEMORY.md, queue archives) under any circumstance.

FILE:references/templates.md
# Templates

Parameterized templates for files this skill generates. Read this when filling out
the project folder in Step 7 of SKILL.md. Substitute every `{{placeholder}}` with the
appropriate value from the user's interview answers and approved plan.

---

## Table of Contents

- [PROJECT.md Template](#projectmd-template)
- [project.json Template](#projectjson-template)
- [project-lock.json (Initial State)](#project-lockjson-initial-state)
- [STATE.md (Initial)](#statemd-initial)
- [Empty File Headers](#empty-file-headers)
- [AGENTS.md Active Projects Block](#agentsmd-active-projects-block)
- [Placeholder Reference](#placeholder-reference)

---

## PROJECT.md Template

The team rulebook. This is the single most important generated file — every agent
reads it. Fill in every placeholder, expand every conditional. If a section
doesn't apply (e.g. `{{#if visual_assets_enabled}}` is false), omit the entire
section rather than leaving an empty heading.

```markdown
# Project: {{project_display_name}}

**Project ID:** {{project_id}}
**Type:** {{team_type}}
**Purpose:** {{project_purpose}}

---

## The Team

{{#each participants}}
- **{{role}} ({{agentId}})** — {{role_description}}
{{/each}}
{{#if operator}}
- **Operator (Human, alias: {{operator}})** — Final authority. Sign-off, unresolvable escalations, client engagement when agents can't reach the client.
{{/if}}

---

## Source of Truth

| What | Where |
|---|---|
| Task ownership and status | {{task_manager_type}} |
| Accepted scope | `workspace/SPEC-CURRENT.md` |
| How to produce deliverables | `workspace/DELIVERABLES_GUIDE.md` |
| Cross-agent knowledge | `SHARED_MEMORY.md` |
| Decision history | `DECISIONS.md` |
| Accepted limitations | `KNOWN_ISSUES.md` |
| Project conventions | `RUNBOOK.md` |
| Current phase and ownership | `project-lock.json` |
| Human-readable status | `STATE.md` |

---

## Stages ({{task_manager_type}} Columns)

| Stage | Meaning | Owner |
|---|---|---|
{{#each stages}}
| {{name}} | {{purpose}} | {{owner}} |
{{/each}}

---

## Shared Working Medium

**Type:** {{shared_medium_type}}
**Location:** {{shared_medium_location}}

{{shared_medium_conventions}}

---

{{#if visual_assets_enabled}}
## Visual / Media Assets

When a task involves visual or media reference material:

- **Primary storage:** {{task_manager_type}} task attachments (retrieved via the installed {{task_manager_type}} skill)
- **Fallback storage:** `./workspace/{{media_folder_name}}/`
- **Naming convention:** `{{visual_naming_convention}}`
- **Task description must reference the asset filename** so executors and QA can locate it

**Vision-required roles:** {{vision_required_roles_list}}

These roles should use a vision-capable model when working with tasks that reference visual assets. If the asset cannot be retrieved from either source, treat it as a blocker and escalate per normal escalation rules.

---
{{/if}}

## Workflow

### Phase 1: Intake

**Owner:** {{client_facing_role}}
**Lock phase:** `intake`

1. {{client_facing_role}} receives or drafts {{intake_term}} from the client.
2. {{client_facing_role}} writes a draft to `workspace/SPEC-v[N]-[YYYY-MM-DD].md` (new version, never overwrite). Updates `SPEC-CURRENT.md`.
3. {{client_facing_role}} posts to `queues/to-{{feasibility_reviewer_role}}-feasibility.md`: "New scope draft ready for feasibility review."
4. {{feasibility_reviewer_role}} reviews for {{feasibility_concerns}}. Posts numbered issues back.
5. {{client_facing_role}} translates issues to client-friendly language. Sends to client via email skill or `to-{{operator}}.md` for relay.
6. Client responds to each numbered issue: Accept / Provide solution / Descope.
7. {{client_facing_role}} logs response in `DECISIONS.md` verbatim with date.
8. Loop until all issues resolved. {{feasibility_reviewer_role}} marks `SPEC-CURRENT.md` ACCEPTED.
9. `project-lock.json` → phase: `planning`.

**Client no-response rule:** No response in {{client_no_response_hours}}h → client-facing follows up. Still no response → posts to `to-{{operator}}.md`. Task moves to Blocked.

### Phase 2: Planning

**Owner:** {{feasibility_reviewer_role}}
**Lock phase:** `planning`

1. {{feasibility_reviewer_role}} writes `workspace/DELIVERABLES_GUIDE.md`. Each numbered section = one task.
2. Updates `KNOWN_ISSUES.md` with limitations accepted during intake.
3. Posts to `to-{{client_facing_role}}.md`: "Deliverables guide ready."
4. {{client_facing_role}} creates tasks in {{task_manager_type}} from guide, assigns to roles, places in {{first_stage_name}}.
5. `project-lock.json` → phase: `execution`, sprint_id set.

### Phase 3: Execution

**Owner:** Executors (per assigned task)
**Escalation owner:** {{feasibility_reviewer_role}}

1. Executor picks up task → moves to "In Progress" stage.
2. Reads DELIVERABLES_GUIDE.md, RUNBOOK.md, their queue.
{{#if visual_assets_enabled}}
3. If task references a visual asset and executor's role requires vision: switch to vision-capable model, retrieve asset.
{{/if}}
4. Produces deliverable in shared medium ({{shared_medium_type}}).
5. When complete: {{completion_action}}, move task to "In Review" stage, post to `to-{{qa_role}}.md`.

**Hard stop rule (universal):** If executor escalates same issue {{stuck_re_escalations_threshold}}x to reviewer OR is stuck {{stuck_hours_threshold}}h, executor stops. Posts full summary to `to-{{client_facing_role}}.md`. {{client_facing_role}} posts to `to-{{operator}}.md`. Task moves to Blocked. **No further AI cycles spent until operator resolves.**

### Phase 4: Review

**Owner:** {{qa_role}}

1. {{qa_role}} picks up task from "In Review" stage.
2. Reads KNOWN_ISSUES.md, SPEC-CURRENT.md, DELIVERABLES_GUIDE.md.
{{#if visual_assets_enabled}}
3. If deliverable includes visual output and task references a mockup: use vision to compare output to reference.
{{/if}}
4. Reviews against all references.

**Pass:** Move task to "Completed". Post to `to-{{operator}}.md` with deliverable pointer.
**Fail:** Post specific failures to `to-{{feasibility_reviewer_role}}.md`. Move task back to "In Progress".

{{#if operator}}
### Phase 5: {{operator}} Sign-off

1. {{operator}} reviews `to-{{operator}}.md`.
2. {{operator}} validates the deliverable (pulls/reviews/tests as appropriate for medium).
3. If satisfied: {{operator}} approves delivery via {{delivery_action}}.
4. `project-lock.json` → `phase: close`.
{{/if}}

### Phase 6: Close

**Owner:** {{client_facing_role}}

1. Verify all sprint tasks are "Completed" in {{task_manager_type}}.
2. Archive completed tasks.
3. Verify DECISIONS.md and KNOWN_ISSUES.md are current.
4. Write sprint summary to SHARED_MEMORY.md.
5. Update STATE.md: "Sprint [N] closed. Ready for next intake."
6. Archive queue entries (mark READ, do not delete).
7. `project-lock.json` → `phase: idle`.
8. Post to `to-{{operator}}.md`: "Sprint closed."

{{#if sprint_mode_one_at_a_time}}
**One sprint at a time:** {{client_facing_role}} does not accept new intake until project-lock.json is `idle`.
{{else}}
**Continuous flow:** {{client_facing_role}} can accept new intake at any time.
{{/if}}

---

## Escalation Rules

| Situation | Action | Threshold |
|---|---|---|
| Client not responding | {{client_facing_role}} follows up; then to operator | {{client_no_response_hours}}h |
| Executor stuck on task | Escalate to {{feasibility_reviewer_role}} | Immediately |
| Same issue re-escalated {{stuck_re_escalations_threshold}}x | Hard stop; client-facing → operator | {{stuck_re_escalations_threshold}} escalations |
| Executor stuck {{stuck_hours_threshold}}h | Hard stop; client-facing → operator | {{stuck_hours_threshold}}h |
| Task in Blocked with no movement | Client-facing → operator | {{blocked_task_operator_escalation_hours}}h |

**No agent continues spending AI cycles on a blocked path.**

---

## Communication Protocol

All inter-agent communication uses queue files in `queues/`.

**Format:**

    [YYYY-MM-DD HH:MM] [FROM: agent-id] [TO: agent-id] [TASK: task-id or N/A]
    Message body. Be specific.
    ---

- Queues are append-only. Never delete entries.
- Mark processed entries with `[READ]` — do not remove the line.
- Archive at sprint close.
- Each agent checks their queue at session start, before any other action.
- `to-{{feasibility_reviewer_role}}-feasibility.md` is used **only during intake phase**.

---

## Reference Block for Each Agent's AGENTS.md

Every participating agent's workspace AGENTS.md must include the block defined
in [AGENTS.md Active Projects Block](#agentsmd-active-projects-block) below.

{{#each user_specific_notes}}
---

## {{section_title}}

{{section_body}}
{{/each}}
```

---

## project.json Template

Machine-readable project config. Substitute every placeholder, then validate the
result is valid JSON before writing to disk.

```json
{
  "id": "{{project_id}}",
  "name": "{{project_display_name}}",
  "purpose": "{{project_purpose}}",
  "team_type": "{{team_type}}",
  "task_manager": {
    "type": "{{task_manager_type}}",
    "project_id": "{{task_manager_project_id}}",
    "columns": {
      "{{stage_key_1}}": { "id": "{{stage_id_1}}", "purpose": "{{stage_purpose_1}}" },
      "{{stage_key_2}}": { "id": "{{stage_id_2}}", "purpose": "{{stage_purpose_2}}" },
      "{{stage_key_3}}": { "id": "{{stage_id_3}}", "purpose": "{{stage_purpose_3}}" },
      "{{stage_key_4}}": { "id": "{{stage_id_4}}", "purpose": "{{stage_purpose_4}}" },
      "{{stage_key_5}}": { "id": "{{stage_id_5}}", "purpose": "{{stage_purpose_5}}" },
      "blocked": { "id": "{{blocked_stage_id}}", "purpose": "Work waiting on external resolution. Client-facing agent owns escalation." }
    }
  },
  "participants": [
    {
      "agentId": "{{participant_agent_id}}",
      "workspace": "{{participant_workspace_path}}",
      "role": "{{participant_role_display}}",
      "role_key": "{{participant_role_key}}"
    }
  ],
  "client_facing_role": "{{client_facing_role_key}}",
  "feasibility_reviewer_role": "{{feasibility_reviewer_role_key}}",
  "qa_role": "{{qa_role_key}}",
  "operator": "{{operator_alias_or_null}}",
  "shared_workspace": "./workspace",
  "shared_medium": {
    "type": "{{shared_medium_type}}",
    "path_or_url": "{{shared_medium_location}}",
    "convention_notes": "{{shared_medium_conventions}}"
  },
  "spec_path": "./workspace/SPEC-CURRENT.md",
  "deliverables_guide_path": "./workspace/DELIVERABLES_GUIDE.md",
  "shared_memory": "./SHARED_MEMORY.md",
  "decisions_log": "./DECISIONS.md",
  "known_issues": "./KNOWN_ISSUES.md",
  "runbook": "./RUNBOOK.md",
  "queues": {
    "{{client_facing_role_key}}": "./queues/to-{{client_facing_role_key}}.md",
    "{{feasibility_reviewer_role_key}}": "./queues/to-{{feasibility_reviewer_role_key}}.md",
    "{{feasibility_reviewer_role_key}}_feasibility": "./queues/to-{{feasibility_reviewer_role_key}}-feasibility.md",
    "{{qa_role_key}}": "./queues/to-{{qa_role_key}}.md",
    "operator": "./queues/to-{{operator_alias_or_null}}.md"
  },
  "visual_assets": {
    "enabled": false,
    "primary_storage": "task_manager_attachments",
    "fallback_storage": "./workspace/{{media_folder_name}}",
    "naming_convention": "[task-id]-[short-description].[ext]",
    "vision_required_roles": []
  },
  "escalation_rules": {
    "client_no_response_hours": 48,
    "stuck_re_escalations_threshold": 2,
    "stuck_hours_threshold": 24,
    "blocked_task_operator_escalation_hours": 48
  },
  "sprint_mode": "one_at_a_time"
}
```

If `visual_assets.enabled` is set to `true`, populate `vision_required_roles` with
the role keys that need vision capability. Set `fallback_storage` to the actual
media folder path the skill creates in `workspace/`.

If there is no operator, set `"operator": null` (without quotes) and **omit the
`"operator"` key from the `queues` block entirely** rather than leaving it
pointing at a queue file that won't exist.

---

## project-lock.json (Initial State)

Initialize fresh on every project creation:

```json
{
  "phase": "idle",
  "sprint_id": null,
  "sprint_opened": null,
  "waiting_on": null,
  "last_updated": "{{today_iso_date}}",
  "last_updated_by": "operator",
  "context": "Project initialized. Ready to receive first work.",
  "blocked_tasks": []
}
```

---

## STATE.md (Initial)

Initialize fresh on every project creation:

```markdown
# {{project_display_name}} — Current State
**Phase:** Idle — Ready for first work
**Last updated:** {{today_iso_date}} by operator
```

---

## Empty File Headers

Use these for the files that start empty but need a header so agents know what
they are.

### SHARED_MEMORY.md

```markdown
# {{project_display_name}} — Shared Memory

Cross-agent knowledge that needs to persist across sessions but doesn't belong
in the task manager. Append entries with date and author.

Format for new entries:

    ## [YYYY-MM-DD] [agent-id] — [topic]
    Content here.
    ---
```

### DECISIONS.md

```markdown
# {{project_display_name}} — Decision Log

Append-only record of every significant decision made during intake or scope
negotiation. Never edit existing entries. Written by {{client_facing_role}}.

Format for new entries:

    ## [YYYY-MM-DD] — [Sprint ID]: [Decision Topic]

    **Issue surfaced by [feasibility reviewer]:** ...

    **Client response (received [date]):** ...

    **Resolution:** Accept as known outcome / Client-proposed alternative / Descoped

    **Accepted by:** [Client name], [feasibility reviewer agent], [client-facing agent]
    **Logged by:** [client-facing agent]
    ---
```

### KNOWN_ISSUES.md

```markdown
# {{project_display_name}} — Known Issues

Accepted limitations and trade-offs. QA reads this before reviewing — do not
file failures against items here. Written by {{feasibility_reviewer_role}}.

Format for new entries:

    ## [Sprint ID] — [Issue Title]
    - **Accepted:** [date]
    - **Context:** [why this limitation exists]
    - **Impact:** [what users/clients/operators will experience]
    ---
```

### RUNBOOK.md (stub)

```markdown
# {{project_display_name}} — Runbook

This is a starting stub. The {{feasibility_reviewer_role}} should expand each
section as they learn the project. All agents read this before starting work.

## Local Setup / Access

How to access the shared working medium ({{shared_medium_type}}).

## Conventions

How this team formats work, names things, and structures deliverables.

## Definition of Done

What counts as complete on this project.

## Known Gotchas

Pitfalls specific to this project — things that have tripped up agents before.

{{#each team_type_specific_sections}}
## {{section_title}}

{{section_body_or_placeholder}}
{{/each}}
```

### Queue Files

Initialize each queue file with:

```markdown
# Queue: to-{{role_key}}

Format for entries:

    [YYYY-MM-DD HH:MM] [FROM: agent-id] [TO: agent-id] [TASK: task-id or N/A]
    Message body. Be specific.
    ---

Append-only. Never delete entries. Mark processed entries with [READ] prepended.
Archive at sprint close (mark READ, leave in place).
```

---

## AGENTS.md Active Projects Block

Insert (or append) this block in each participating agent's workspace AGENTS.md.
If the agent is already on other projects, append — never overwrite.

```markdown
## Active Projects

- **{{project_display_name}}** — I am the {{my_role}} on this project.
  - Full rules: ~/.openclaw/projects/{{project_id}}/PROJECT.md
  - My queue: ~/.openclaw/projects/{{project_id}}/queues/to-{{my_role_key}}.md
  - Shared workspace: ~/.openclaw/projects/{{project_id}}/workspace/
  - Check my queue at the start of every session before doing anything else.
  - Check ~/.openclaw/projects/{{project_id}}/project-lock.json to know what
    phase we are in before acting.
```

---

## Placeholder Reference

Every placeholder used across the templates above. Source column tells you
where the value comes from.

| Placeholder | Source |
|---|---|
| `{{project_id}}` | Pass 1 #3 |
| `{{project_display_name}}` | Pass 1 #3 |
| `{{project_purpose}}` | Pass 1 #1 |
| `{{team_type}}` | Pass 1 #2 |
| `{{participants}}` (list) | Pass 1 #4 — each entry has `agentId`, `role`, `role_key`, `role_description`, `workspace` |
| `{{operator}}` | Pass 1 #8 (alias) or null |
| `{{client_facing_role}}` | Pass 1 #5 (display name) |
| `{{client_facing_role_key}}` | Pass 1 #5 (slug form) |
| `{{feasibility_reviewer_role}}` | Pass 1 #6 (display name) |
| `{{feasibility_reviewer_role_key}}` | Pass 1 #6 (slug) |
| `{{qa_role}}` | Pass 1 #7 (display name) |
| `{{qa_role_key}}` | Pass 1 #7 (slug) |
| `{{task_manager_type}}` | Pass 2 #1 — `asana` or `clickup` |
| `{{task_manager_project_id}}` | From Step 6 (after board creation) |
| `{{stages}}` (list) | Pass 2 #2 — each has `name`, `key`, `id`, `purpose`, `owner` |
| `{{first_stage_name}}` | Pass 2 #2 (first column) |
| `{{shared_medium_type}}` | Pass 2 #3 — `git`, `folder`, `external_system`, or `none` |
| `{{shared_medium_location}}` | Pass 2 #3 |
| `{{shared_medium_conventions}}` | Inferred in Step 4, user-confirmed |
| `{{visual_assets_enabled}}` | Pass 2 #7 (boolean) |
| `{{media_folder_name}}` | Inferred from team type — e.g. `mockups`, `photos`, `media` |
| `{{visual_naming_convention}}` | Default: `[task-id]-[short-description].[ext]` |
| `{{vision_required_roles_list}}` | Inferred in Step 4 from team type |
| `{{intake_term}}` | Per team type — e.g. "requirements", "brief", "lead" |
| `{{feasibility_concerns}}` | Per team type — see team-archetypes.md |
| `{{completion_action}}` | Per team type — e.g. "push to sprint branch and update PR" |
| `{{delivery_action}}` | Per team type — e.g. "merge to main", "send to client", "publish to MLS" |
| `{{sprint_mode_one_at_a_time}}` | Pass 2 #5 (boolean) |
| `{{client_no_response_hours}}` | Pass 2 #6 (default 48) |
| `{{stuck_hours_threshold}}` | Pass 2 #6 (default 24) |
| `{{stuck_re_escalations_threshold}}` | Pass 2 #6 (default 2) |
| `{{blocked_task_operator_escalation_hours}}` | Default 48 |
| `{{user_specific_notes}}` | Pass 2 #8 (free-form additions) |
| `{{today_iso_date}}` | Today's date in YYYY-MM-DD |
| `{{my_role}}` / `{{my_role_key}}` | Per-agent when generating their AGENTS.md block |

FILE:references/project-files.md
# Project Files Reference

Full specification of every file in `~/.openclaw/projects/[project-id]/`.
Universal — applies to any team type.

---

## Table of Contents

- [PROJECT.md — Team Rulebook](#projectmd--team-rulebook)
- [project.json — Machine Config](#projectjson--machine-config)
- [project-lock.json — Phase Tracker](#project-lockjson--phase-tracker)
- [STATE.md — Human Status](#statemd--human-status)
- [SHARED_MEMORY.md — Cross-Agent Knowledge](#shared_memorymd--cross-agent-knowledge)
- [DECISIONS.md — Decision Log](#decisionsmd--decision-log)
- [KNOWN_ISSUES.md — Accepted Limitations](#known_issuesmd--accepted-limitations)
- [RUNBOOK.md — Project Operating Guide](#runbookmd--project-operating-guide)
- [workspace/SPEC-CURRENT.md](#workspacespec-currentmd)
- [workspace/DELIVERABLES_GUIDE.md](#workspacedeliverables_guidemd)
- [Queue Files](#queue-files)
- [File Responsibility Matrix](#file-responsibility-matrix)

---

## PROJECT.md — Team Rulebook

**The most important file in the project folder.** Every participating agent has this referenced from their AGENTS.md and reads it before taking action. It is the single source of truth for how the team operates on this project.

### Who reads it
All agents on the project, at every session start.

### Who writes it
Generated by this skill from the user's interview answers (after Step 4 approval). Updated by operator only when workflow rules change.

### Required sections

A complete PROJECT.md must contain:

1. **Project header** — display name, ID, purpose, team type
2. **The Team** — every agent on the project, their role, what they own and do not own
3. **Source of Truth** — table mapping "what" to "where"
4. **Stages** (task manager columns) — name, meaning, who owns it
5. **Shared Working Medium** — what it is, how to access, conventions for use
6. **Visual / Media Assets** — only if applicable: storage convention, vision-required roles
7. **Workflow** — every phase from intake to close, with steps, owners, and queue references
8. **Escalation Rules** — thresholds, who escalates to whom, when work stops
9. **Communication Protocol** — queue file format, when to check, append-only rules
10. **AGENTS.md Reference Block** — the snippet that goes in each agent's AGENTS.md

See the PROJECT.md template in `references/templates.md` for the parameterized template.

---

## project.json — Machine Config

Machine-readable project configuration. Agents read this to resolve file paths, task manager IDs, and participant details without relying on hardcoded values.

### Who reads it
All agents.

### Who writes it
Generated by this skill. Updated by operator when project structure changes.

### Schema

```json
{
  "id": "string — lowercase project identifier",
  "name": "string — display name",
  "purpose": "string — one-line description of project goal",
  "team_type": "string — software_dev | marketing | real_estate | content | sales | customer_success | operations | research | custom",
  "task_manager": {
    "type": "asana | clickup",
    "project_id": "string — task manager's project/workspace ID",
    "columns": {
      "stage_key": { "id": "string", "purpose": "string" }
    }
  },
  "participants": [
    {
      "agentId": "string",
      "workspace": "string — relative path to agent's OpenClaw workspace",
      "role": "string — display name (e.g. 'Project Manager')",
      "role_key": "string — machine slug (e.g. 'pm')"
    }
  ],
  "client_facing_role": "string — role_key of client-facing agent",
  "feasibility_reviewer_role": "string — role_key",
  "qa_role": "string — role_key",
  "operator": "string — operator alias, or null if no operator",
  "shared_workspace": "./workspace",
  "shared_medium": {
    "type": "git | folder | external_system | none",
    "path_or_url": "string — local path or external URL",
    "convention_notes": "string — brief notes on usage"
  },
  "spec_path": "./workspace/SPEC-CURRENT.md",
  "deliverables_guide_path": "./workspace/DELIVERABLES_GUIDE.md",
  "shared_memory": "./SHARED_MEMORY.md",
  "decisions_log": "./DECISIONS.md",
  "known_issues": "./KNOWN_ISSUES.md",
  "runbook": "./RUNBOOK.md",
  "queues": {
    "role_key": "string — relative path to queue file"
  },
  "visual_assets": {
    "enabled": "boolean",
    "primary_storage": "task_manager_attachments | workspace_folder | none",
    "fallback_storage": "string — relative path to media folder, or null",
    "naming_convention": "string — e.g. '[task-id]-[short-description].[ext]'",
    "vision_required_roles": ["array of role_keys"]
  },
  "escalation_rules": {
    "client_no_response_hours": "number",
    "stuck_re_escalations_threshold": "number",
    "stuck_hours_threshold": "number",
    "blocked_task_operator_escalation_hours": "number"
  },
  "sprint_mode": "one_at_a_time | continuous"
}
```

### Notes
- All file paths use relative paths from project root
- Task manager column IDs must be filled in from the actual board after creation
- `visual_assets.enabled: false` if the team doesn't use media — the entire block can still be present, just disabled
- `operator: null` for fully autonomous teams (rare)

---

## project-lock.json — Phase Tracker

Prevents agents from acting out of phase or moving forward when waiting on another agent or the operator.

### Who reads it
All agents check this at session start before any action.

### Who writes it
Client-facing agent (most phase transitions), feasibility-reviewer (planning → execution), QA (after sign-off), operator (close → idle).

### Phase progression
`idle` → `intake` → `planning` → `execution` → `review` → `close` → `idle`

These phase names replace the dev-specific names. Translation:
- `intake` = formerly "requirements" (universal: receiving and validating new work)
- `planning` = same (defining how the work gets done)
- `execution` = formerly "implementation" (universal: doing the work)
- `review` = formerly "qa" (universal: quality review before delivery)
- `close` = formerly "sprint-close" (universal: wrap up and reset)

### Format

```json
{
  "phase": "string — one of the phase names above",
  "sprint_id": "string or null",
  "sprint_opened": "string ISO date or null",
  "waiting_on": "string agent-id, 'client', 'operator', or null",
  "last_updated": "string ISO date",
  "last_updated_by": "string — agent-id or 'operator'",
  "context": "string — human-readable description of current state",
  "blocked_tasks": ["array of task identifiers"]
}
```

### Agent behavior rules
- Phase doesn't match my expected action → stop and post to relevant queue
- `waiting_on` is me → act immediately
- `blocked_tasks` contains my task → treat as stopped, do not work on it

---

## STATE.md — Human Status

Operator's quick-glance status without digging through queues or task manager.

### Who reads it
Operator primarily. Agents may read for context.

### Who writes it
All agents update when they complete a significant action.

### Format

```markdown
# [Project Name] — Current State
**Phase:** [phase] ([sprint_id if applicable])
**Last updated:** [YYYY-MM-DD HH:MM] by [agent-id]

## Sprint / Work in Flight
- ✅ [Task ID] — [description] (delivered)
- 🔄 [Task ID] — [description] (in progress, [agent])
- ⏳ [Task ID] — [description] (queued)
- 🚫 [Task ID] — [description] (blocked — [reason])

## Operator Queue Summary
[Summary of items in to-operator.md awaiting action]
```

---

## SHARED_MEMORY.md — Cross-Agent Knowledge

Living document for project knowledge that persists across sessions but doesn't belong in the task manager.

### What goes here (universal)
- Domain knowledge agents have learned about this project
- Client preferences and communication style
- Conventions specific to this client or this work
- Cross-sprint summaries (added by client-facing agent at close)
- Anything one agent needs to tell another that doesn't fit a task

### What does NOT go here
- Task status → task manager
- Accepted scope → SPEC files
- How-to-do-the-work → DELIVERABLES_GUIDE.md
- Decisions and acceptances → DECISIONS.md
- Limitations → KNOWN_ISSUES.md

### Format
```markdown
## [YYYY-MM-DD] [agent-id] — [topic]
Content here.
---
```

---

## DECISIONS.md — Decision Log

**Append-only.** Every significant decision made during intake or scope negotiation. Never edit existing entries.

### Who reads it
Client-facing agent, feasibility reviewer, operator. QA references when filing failures.

### Who writes it
Client-facing agent, during intake phase as decisions are made.

### Purpose
When a client later disputes what was agreed, this is the record. It captures what was proposed, what issue was surfaced, what the client said, what was accepted.

### Format

```markdown
## [YYYY-MM-DD] — [Sprint ID]: [Decision Topic]

**Issue surfaced by [feasibility reviewer]:** [description of the issue, options if any]

**Client response (received [date]):** [exact words or close paraphrase, attributed]

**Resolution:** [Accept as known outcome / Client-proposed alternative / Descoped]

**Accepted by:** [Client name], [feasibility reviewer agent], [client-facing agent]
**Logged by:** [client-facing agent]
---
```

---

## KNOWN_ISSUES.md — Accepted Limitations

Accepted limitations and trade-offs. QA reads before reviewing — does not file failures against items here.

### Who reads it
QA (before every review), all agents for context, operator.

### Who writes it
Feasibility reviewer, during planning and as new limitations are accepted.

### Format

```markdown
## [Sprint ID] — [Issue Title]
- **Accepted:** [date]
- **Context:** [why this limitation exists, what decision led to it]
- **Impact:** [what users/clients/operators will experience as a result]
---
```

---

## RUNBOOK.md — Project Operating Guide

Maintained by feasibility reviewer. All contributors read before starting work to avoid unnecessary escalations.

### Who reads it
All agents before starting work, operator for reference.

### Who writes it
Initial stub generated by this skill (with section headers appropriate for team type). Feasibility reviewer fills in details after first session with the project.

### Universal sections (always present)
- Local Setup / Access — how to access the shared medium
- Conventions — how the team formats work, names things, etc.
- Definition of Done — what counts as complete
- Known Gotchas — pitfalls specific to this project

### Team-type-specific sections
See `references/team-archetypes.md` for sections appropriate to each team type.

---

## workspace/SPEC-CURRENT.md

Reference to the currently active accepted specification / brief / scope document.

### Versioning rules (universal)
- Every new draft gets its own versioned file: `SPEC-v[N]-[YYYY-MM-DD].md`
- Specs are **never overwritten** — increment version
- `SPEC-CURRENT.md` updates to point to or contain the latest accepted version
- Version history is the audit trail

### Status markers
Feasibility reviewer adds one when reviewing:
```
STATUS: DRAFT — Under feasibility review
STATUS: ACCEPTED — [date] — [feasibility-reviewer-agent] + [client-facing-agent]
```

---

## workspace/DELIVERABLES_GUIDE.md

Written by feasibility reviewer after spec is accepted. Task-oriented blueprint for what gets produced and how.

This file replaces the dev-specific "IMPLEMENTATION_GUIDE.md" with a universal name. It serves the same function across all team types: turn the accepted scope into discrete tasks the executors can work on.

### Format

```markdown
# Deliverables Guide — [Sprint ID]

## Task 1: [Task Title]
**Assigned to:** [role / agent]
**Task manager ID:** [created by client-facing agent after this guide is written]

### What to produce
[Description of the deliverable]

### Inputs / dependencies
[What this task depends on or requires]

### Approach
[How to do it — high-level. Not full execution.]

### Acceptance criteria
[How QA will verify this is complete]

### Notes / edge cases
[Anything the executor should know]
---

## Task 2: ...
```

---

## Queue Files

Located in `queues/`. One file per role-recipient.

Standard set:
- `to-[client-facing-role].md`
- `to-[feasibility-reviewer-role].md`
- `to-[feasibility-reviewer-role]-feasibility.md` (intake phase only — separate from execution escalations)
- `to-[qa-role].md`
- `to-[operator].md` (only if operator exists)
- `to-[other-role].md` for each additional role

### Format (universal — every entry must use this)

```
[YYYY-MM-DD HH:MM] [FROM: agent-id] [TO: agent-id] [TASK: task-id or N/A]
Message body. Be specific. Include task IDs, file references, error messages.
If multiple items, number them clearly.
---
```

### Rules (universal)
- Append-only — never delete entries
- Mark processed entries by prepending `[READ]` — do not remove the line
- Archive at sprint close (mark READ, do not remove)
- Each agent checks their queue at session start, before any other action
- Feasibility queue is **only** used during intake phase — keeps it separate from execution escalations

---

## File Responsibility Matrix

| File | Created by | Updated by | Read by | Mutability |
|---|---|---|---|---|
| `PROJECT.md` | This skill | Operator | All agents | Rare (workflow changes only) |
| `project.json` | This skill | Operator | All agents | Structure changes only |
| `project-lock.json` | This skill | All agents (per phase) | All agents | Every phase change |
| `STATE.md` | This skill | All agents | Operator, all agents | Frequently |
| `SHARED_MEMORY.md` | This skill | All agents (append) | All agents | Frequently |
| `DECISIONS.md` | This skill | Client-facing agent (append) | Client-facing, feasibility, operator | Append-only |
| `KNOWN_ISSUES.md` | This skill | Feasibility reviewer (append) | QA, all agents | Append-only |
| `RUNBOOK.md` | This skill (stub) | Feasibility reviewer | All agents | As patterns evolve |
| `SPEC-vN-*.md` | Client-facing agent | Never | Client-facing, feasibility | Immutable |
| `SPEC-CURRENT.md` | Client-facing agent | Client-facing (per sprint) | All agents | Per sprint |
| `DELIVERABLES_GUIDE.md` | Feasibility reviewer | Feasibility reviewer | All executors, QA | Per sprint |
| `queues/to-*.md` | This skill (init) | Named sender (append) | Named recipient | Append-only |

FILE:references/workflow.md
# Workflow Reference

Universal workflow for any project type managed under OpenClaw Projects. Read this when:
- Drafting the workflow section of PROJECT.md in Step 4
- Troubleshooting agent behavior during a sprint
- Explaining how the team should operate

---

## Table of Contents

- [Phase Overview](#phase-overview)
- [Phase 1: Intake](#phase-1-intake)
- [Phase 2: Planning](#phase-2-planning)
- [Phase 3: Execution](#phase-3-execution)
- [Phase 4: Review](#phase-4-review)
- [Phase 5: Operator Sign-off](#phase-5-operator-sign-off)
- [Phase 6: Close](#phase-6-close)
- [Escalation Rules](#escalation-rules)
- [Queue Message Format](#queue-message-format)
- [Agent Session Start Checklist](#agent-session-start-checklist)

---

## Phase Overview

```
idle → intake → planning → execution → close → idle
                                ↕
                              review
```

Each phase tracked in `project-lock.json`. Agents check this file before acting.
If current phase doesn't match the agent's intended action, the agent stops and posts to their relevant queue.

The phase names are universal across team types. The *content* of each phase is team-specific (see `references/team-archetypes.md`), but the structure is the same.

---

## Phase 1: Intake

**Owner:** Client-facing agent
**Lock phase:** `intake`
**Queue used:** `to-[feasibility-reviewer]-feasibility.md`

### Steps

1. Client-facing agent receives or drafts work intake (requirements, brief, request, lead — whatever the team type).
2. Client-facing agent creates a versioned spec file: `workspace/SPEC-v[N]-[YYYY-MM-DD].md`
   - Never overwrite — always increment version
   - Updates `SPEC-CURRENT.md` to reference this draft
   - Marks file: `STATUS: DRAFT — Under feasibility review`
3. Client-facing agent posts to feasibility queue:
   ```
   [date] [FROM: client-facing] [TO: feasibility-reviewer]
   New scope draft ready for feasibility review.
   File: workspace/SPEC-v[N]-[YYYY-MM-DD].md
   ---
   ```
4. Feasibility reviewer reads spec and reviews for the team-specific concerns. Examples:
   - **Software dev:** technical feasibility, architecture conflicts, ambiguities
   - **Marketing:** brand fit, channel feasibility, budget alignment
   - **Real estate:** pricing realism, market fit, compliance
   - **Whatever fits the team**
5. Reviewer posts numbered issues to feasibility queue:
   ```
   [date] [FROM: feasibility-reviewer] [TO: client-facing]
   Feasibility review complete. [N] issues to resolve before accepting.

   Issue 1: [title]
   [description, concrete impact, options if available]

   Issue 2: ...
   ---
   ```
6. Client-facing agent translates issues into client-friendly language and sends to client (via email skill if available, or via `to-operator.md` if not).
7. Client responds to each numbered issue:
   - **Accept as known outcome**
   - **Provide a solution** for reviewer to evaluate
   - **Descope** — remove the requirement
8. Client-facing agent logs response in `DECISIONS.md` with date.
9. If client proposes solution, reviewer evaluates. Loop repeats until all issues resolved.
10. When resolved:
    - Reviewer updates `SPEC-CURRENT.md`: `STATUS: ACCEPTED — [date] — [reviewer] + [client-facing]`
    - Client-facing agent logs final acceptance in `DECISIONS.md`
    - Client-facing agent updates `project-lock.json` → `phase: planning`

### Client No-Response Rule
- No response in [client_no_response_hours] → client-facing agent sends follow-up
- Still no response → client-facing agent posts to `to-operator.md`:
  ```
  [date] [FROM: client-facing] [TO: operator]
  Client has not responded to feasibility issues for [hours]h. Follow-up sent.
  Please engage client directly. Issues are in queues/to-[feasibility]-feasibility.md.
  ---
  ```
- Task moves to "Blocked" stage in task manager

---

## Phase 2: Planning

**Owner:** Feasibility reviewer
**Lock phase:** `planning`

### Steps

1. Reviewer writes `workspace/DELIVERABLES_GUIDE.md`:
   - Each numbered section = one task in the task manager
   - Detailed enough to execute without ambiguity
   - Approach-level, not full execution
   - References `KNOWN_ISSUES.md` items created from this guide
   - **If scope includes visual/media assets:** reviewer uses vision capability (if model supports it) to review them. Each task referencing a visual asset must include the asset filename so executors and QA can locate it.
2. Reviewer updates `KNOWN_ISSUES.md` with limitations accepted during intake.
3. Reviewer posts to client-facing agent's queue:
   ```
   [date] [FROM: feasibility-reviewer] [TO: client-facing]
   Deliverables guide ready. workspace/DELIVERABLES_GUIDE.md
   [N] tasks defined.
   ---
   ```
4. Client-facing agent reviews guide for completeness.
5. Client-facing agent creates tasks in task manager from guide:
   - One task per numbered section
   - Task description includes the relevant guide section
   - Tasks placed in first stage (Backlog or equivalent), assigned to appropriate roles
6. Client-facing agent posts to feasibility-reviewer's queue:
   ```
   [date] [FROM: client-facing] [TO: feasibility-reviewer]
   Sprint [N] open. [X] tasks created.
   ---
   ```
7. Client-facing agent updates `project-lock.json`:
   ```json
   {
     "phase": "execution",
     "sprint_id": "sprint-[N]",
     "sprint_opened": "[date]"
   }
   ```
8. Client-facing agent updates `STATE.md`.

---

## Phase 3: Execution

**Owner:** Executors (their assigned tasks)
**Escalation owner:** Feasibility reviewer
**Lock phase:** `execution`

### Executor Task Flow

1. Executor picks up assigned task → moves to "In Progress" stage in task manager.
2. Executor reads:
   - `workspace/DELIVERABLES_GUIDE.md` — relevant task section
   - `RUNBOOK.md` — project conventions
   - Their queue — pending messages
3. **If task references a visual/media asset:**
   - Executor switches to vision-capable model (if their config supports it)
   - Retrieves asset from task manager attachment (preferred) or workspace media folder
   - If asset cannot be retrieved → blocker, escalate to feasibility reviewer
4. Executor produces the deliverable in the shared working medium.
5. When complete:
   - **Software dev:** push to sprint branch; first task opens PR, subsequent push updates PR
   - **Marketing/content:** save to shared drive in approved location
   - **Real estate:** update CRM with completion details
   - **Whatever fits the medium**
   - Move task from "In Progress" → "In Review" stage
   - Post to QA's queue:
     ```
     [date] [FROM: executor] [TO: qa] [TASK: task-id]
     Task [id] complete. [Pointer to deliverable — PR URL, file path, etc.]
     What was produced: [brief description]
     ---
     ```
6. Executor moves to next task if available.

### Executor Escalation Rules

**When blocked:**
- Post to feasibility reviewer's queue with task ID, what was tried, specific question
- Reviewer responds in executor's queue
- Executor waits for response

**Hard stop rule — triggers when EITHER condition is met:**
- Same issue escalated [stuck_re_escalations_threshold] times to reviewer without resolution, OR
- Executor stuck on same issue for [stuck_hours_threshold] hours

**When hard stop triggers:**
1. Executor stops work on that task immediately
2. Executor posts full summary to client-facing agent's queue:
   ```
   [date] [FROM: executor] [TO: client-facing] [TASK: task-id]
   HARD STOP — escalation limit reached.
   Issue: [description]
   Escalation history:
     [date] — First escalation: [question]
     [date] — Reviewer response: [response]
     [date] — Second escalation: [question]
     [date] — Reviewer response: [response]
   Still blocked because: [reason]
   Awaiting operator assistance before resuming.
   ---
   ```
3. Task moves to "Blocked" in task manager
4. Client-facing agent posts to `to-operator.md`
5. **No further AI cycles spent on this task until operator resolves**

---

## Phase 4: Review

**Owner:** QA reviewer
**Lock phase:** `execution` (review runs concurrently with ongoing execution)

### QA Flow

1. QA picks up task from "In Review" stage → moves to "QA" or "Review" stage.
2. QA reads:
   - `KNOWN_ISSUES.md` — do not file failures against accepted limitations
   - `workspace/SPEC-CURRENT.md` — accepted scope
   - `workspace/DELIVERABLES_GUIDE.md` — planned approach for this task
3. **If deliverable includes visual output and task references a mockup:**
   - QA uses vision (if model supports) to compare output to reference
   - Visual deviations not in `KNOWN_ISSUES.md` are failures
4. QA reviews deliverable against all references.

### QA Pass
```
[date] [FROM: qa] [TO: operator] [TASK: task-id]
Task [id] — REVIEW PASSED.
Deliverable: [pointer]
Verified against: SPEC-CURRENT.md + DELIVERABLES_GUIDE.md task [N]
No known issues flagged.
---
```
- Move task to "Completed" stage

### QA Fail
```
[date] [FROM: qa] [TO: feasibility-reviewer] [TASK: task-id]
Task [id] — REVIEW FAILED. [N] issues found.

Issue 1: [specific — what was checked, what was expected, what happened]
Issue 2: ...
---
```
- Move task back to "In Progress"
- Executor addresses failures and re-submits
- QA re-reviews

---

## Phase 5: Operator Sign-off

**Owner:** Operator (human, if defined)
**Lock phase:** `close` (after sign-off)

### Flow

1. Operator reviews `to-operator.md` for QA-passed tasks.
2. Operator validates the deliverable:
   - **Software dev:** pull branch, review, test
   - **Marketing/content:** read the deliverable
   - **Real estate:** verify listing data
   - **Whatever fits**
3. If satisfied:
   - **Software dev:** tells QA to merge to main; rebases other repos
   - **Other:** approves delivery to client through whatever channel applies
4. Operator updates `project-lock.json` → `phase: close`.

### If no operator (rare)
QA's pass message is the sign-off. Move directly to close phase.

---

## Phase 6: Close

**Owner:** Client-facing agent
**Lock phase:** `idle` (after close)

### Close Checklist

1. Verify all sprint tasks are in "Completed" stage in task manager.
2. Archive completed tasks (close/archive — do not delete).
3. Verify `DECISIONS.md` has complete record for this sprint.
4. Verify `KNOWN_ISSUES.md` is current.
5. Write sprint summary to `SHARED_MEMORY.md`:
   ```markdown
   ## [date] Sprint [N] Close — [client-facing-agent]
   What was delivered: [summary]
   Issues accepted: [reference to KNOWN_ISSUES entries]
   Client sign-off: [yes/no — how confirmed]
   Carry-over notes: [anything for next sprint]
   ```
6. Update `STATE.md`:
   ```markdown
   # [Project Name] — Current State
   **Phase:** Idle — Sprint [N] closed. Ready for next intake.
   ```
7. Archive queue entries (mark `[READ]`, do not delete).
8. Update `project-lock.json`:
   ```json
   {
     "phase": "idle",
     "sprint_id": null,
     "sprint_opened": null,
     "waiting_on": null,
     "context": "Sprint [N] closed. Ready for next intake."
   }
   ```
9. Post to operator's queue: "Sprint [N] closed. Ready for next intake."

**One sprint at a time mode:** Client-facing agent does not accept new intake until `project-lock.json` is `idle`.

**Continuous flow mode:** Client-facing agent can accept new intake immediately. Each piece of work flows through phases independently. Sprint close still happens for periodic cleanup, but new intake doesn't wait for it.

---

## Escalation Rules

| Situation | Action | Threshold |
|---|---|---|
| Client not responding to scope | Client-facing follows up; then escalates to operator | client_no_response_hours |
| Executor blocked on task | Escalate to feasibility reviewer | Immediately when blocked |
| Same issue re-escalated to reviewer | Hard stop; client-facing escalates to operator | stuck_re_escalations_threshold |
| Executor stuck same issue | Hard stop; client-facing escalates to operator | stuck_hours_threshold |
| Task in Blocked with no movement | Client-facing escalates to operator | blocked_task_operator_escalation_hours |
| QA failing same task repeatedly | QA posts to reviewer; client-facing monitors | — |

**No agent continues spending AI cycles on a blocked path. Stop, surface, wait.**

---

## Queue Message Format

Every queue entry must use this exact format:

```
[YYYY-MM-DD HH:MM] [FROM: agent-id] [TO: agent-id] [TASK: task-id or N/A]
Message body. Be specific. Include task IDs, file references, error messages.
If multiple items, number them clearly.
---
```

### Rules
- Queues are append-only — never delete entries
- Mark processed entries by prepending `[READ]` — do not remove the line
- Archive at sprint close (mark READ, leave in place)
- Each agent checks their queue at session start, before any other action
- Feasibility queue is **only** used during intake phase

---

## Agent Session Start Checklist

Every agent runs this at the start of every session:

1. Read `project-lock.json` for each project they're on — what phase is each in?
2. Read `queues/to-[my-role].md` for each project — any pending messages?
3. If unread queue messages exist, address them before starting new work
4. If phase doesn't match my expected action, post to relevant queue and wait
5. If phase matches and no pending messages, proceed with current task

**Queue and phase check always come before anything else.**

FILE:references/team-archetypes.md
# Team Archetypes

Reference patterns for common team types. Read this when:
- Drafting the team plan in Step 4 and need a reference example
- The user's team is novel and you're inferring conventions
- Writing the RUNBOOK.md stub and need section ideas

These are starting points, not prescriptions. Every team adapts the pattern.

---

## Software Development Team

**Roles:**
- **PM (client-facing)** — receives feature requests, manages scope, talks to client
- **Engineer (feasibility reviewer)** — reviews technical feasibility, writes implementation guides
- **FE Dev / BE Dev** — implements assigned tasks
- **QA** — validates PRs against accepted spec
- **Operator** — final merge authority, handles unresolvable escalations

**Stages:** Backlog → In Progress → In Review → QA → Completed → Blocked

**Shared medium:** Git repository (one cloned copy in `workspace/repo/`, one branch per dev per sprint, PR auto-updates as dev pushes)

**Definition of done:** PR opened, all tests pass, QA reviewed against spec, operator approves merge.

**Visual assets:** Mockups attached to tasks. FE Dev and QA need vision-capable models.

**RUNBOOK sections:**
- Local setup instructions
- Branch naming convention
- PR conventions
- Known codebase gotchas
- Deployment notes

**Workflow nuance:** Devs work one branch per sprint. Multiple tasks ship in one PR that auto-updates. QA re-reviews each push. Operator merges once everything passes.

---

## Marketing / Creative Team

**Roles:**
- **Strategist (client-facing)** — receives briefs, scopes campaigns, manages client relationship
- **Creative Director (feasibility reviewer)** — reviews briefs for fit with brand, channel, and budget
- **Copywriter / Designer / Producer** — produces deliverables
- **Brand Reviewer (QA)** — final quality check against brand guidelines
- **Operator** — handles escalations, approves controversial creative

**Stages:** Brief → Drafting → Internal Review → Client Review → Approved → Blocked

**Shared medium:** Google Drive folder, Notion workspace, or Figma project. Linked from `workspace/LINKS.md` if external.

**Definition of done:** Approved by Brand Reviewer, client has signed off, asset is delivered to client's specified location.

**Visual assets:** Often heavy use — mockups, references, photography. Stored as task attachments primarily.

**RUNBOOK sections:**
- Brand guidelines reference
- Channel-specific requirements (social character limits, email best practices, etc.)
- Asset rights and attribution rules
- Approval chain
- Standard turnaround times

**Workflow nuance:** Client approval is often a back-and-forth. Strategist owns those rounds and must clearly log each round in DECISIONS.md so the team doesn't lose context.

---

## Real Estate Team

**Roles:**
- **Listing Agent (client-facing)** — talks to sellers and buyers, owns the relationship
- **Broker (feasibility reviewer)** — reviews pricing, market fit, compliance
- **Listing Producer** — handles property write-ups, photo coordination, MLS entry
- **Compliance Reviewer (QA)** — checks all disclosures, contract terms before publishing
- **Operator** — handles unusual situations, contested pricing, escalations

**Stages:** New → Listing Prep → Listed → Under Offer → Closed → Blocked

**Shared medium:** CRM platform (BoomTown, Follow Up Boss, etc.). Reference URL in `workspace/EXTERNAL_SYSTEM.md`.

**Definition of done:** Listing live in MLS, photos approved, pricing confirmed, all disclosures complete.

**Visual assets:** Property photos. Stored as task attachments or CRM-hosted.

**RUNBOOK sections:**
- MLS data entry conventions
- Photography requirements (resolution, count, room order)
- Disclosure checklist
- Pricing methodology
- Compliance requirements specific to jurisdiction

**Workflow nuance:** Compliance is non-negotiable. QA reviewer must catch missing disclosures before listing goes live or there are legal consequences.

---

## Content / Editorial Team

**Roles:**
- **Editor in Chief (client-facing)** — owns editorial calendar, talks to publication owner
- **Senior Editor (feasibility reviewer)** — reviews pitches for fit with calendar and audience
- **Writer / Reporter** — produces drafts
- **Copy Editor (QA)** — final pass for grammar, accuracy, style
- **Fact Checker** — separate role if the team is research-heavy
- **Operator** — handles escalations, approves controversial pieces

**Stages:** Idea → Drafting → Editing → Fact Check → Published → Blocked

**Shared medium:** CMS or shared Drive folder. Reference linked.

**Definition of done:** Copy edited, fact-checked, scheduled or published.

**Visual assets:** Hero images, embedded images. Often pulled from stock libraries — sourcing rules matter.

**RUNBOOK sections:**
- House style guide
- SEO requirements (length, keyword conventions)
- Image sourcing and rights
- Citation and attribution conventions
- Publishing checklist

**Workflow nuance:** Fact-checking can introduce significant rework. Plan for it in time estimates.

---

## Sales / Outreach Team

**Roles:**
- **Account Executive (client-facing)** — manages prospects through close
- **Sales Leader (feasibility reviewer)** — reviews lead quality, qualifies opportunities
- **SDR / BDR** — handles initial outreach and qualification
- **Sales Ops (QA)** — reviews pipeline data quality, ensures CRM is clean
- **Operator** — handles escalations, signs off on non-standard deals

**Stages:** Lead → Qualifying → Engaged → Proposal → Closed-Won/Lost → Blocked

**Shared medium:** CRM (Salesforce, HubSpot). External reference in `workspace/EXTERNAL_SYSTEM.md`.

**Definition of done:** Deal closed (won or lost), CRM updated with full context, lessons logged in SHARED_MEMORY.md.

**Visual assets:** Rare. Possibly proposal decks attached to tasks.

**RUNBOOK sections:**
- Qualification criteria (MEDDIC, BANT, whatever the team uses)
- Outreach templates and sequences
- CRM hygiene rules
- Compliance (GDPR, CAN-SPAM, opt-out handling)
- Hand-off triggers between SDR and AE

**Workflow nuance:** Compliance is critical. Bad outreach has legal consequences. QA role focuses on this.

---

## Customer Success / Support Team

**Roles:**
- **CS Manager (client-facing)** — owns customer relationship, manages escalations
- **Tier 2 / Specialist (feasibility reviewer)** — reviews complex tickets, validates solutions
- **Tier 1 Support** — handles standard inquiries
- **QA Reviewer** — audits ticket resolutions for quality
- **Operator** — handles fire escalations, contract-level issues

**Stages:** New Ticket → In Progress → Awaiting Customer → Resolved → Blocked

**Shared medium:** Ticketing system (Zendesk, Intercom). External reference linked.

**Definition of done:** Ticket resolved, customer confirmed, post-resolution survey sent or scheduled.

**Visual assets:** Screenshots from customers. Stored as task attachments.

**RUNBOOK sections:**
- Knowledge base location
- SLA targets per tier
- Escalation triggers
- Tone and communication standards
- Common issue resolution playbooks

**Workflow nuance:** "Awaiting Customer" can sit for days. Have a clear policy for follow-up cadence and when to close as inactive.

---

## Operations Team

**Roles:**
- **Ops Lead (client-facing)** — receives requests from internal stakeholders
- **Senior Ops (feasibility reviewer)** — validates requests are doable / scoped correctly
- **Ops Specialist** — executes operational tasks
- **Audit Reviewer (QA)** — verifies completed work meets compliance requirements
- **Operator** — handles approvals, audit issues

**Stages:** Request → Triage → In Progress → Verification → Complete → Blocked

**Shared medium:** Varies widely. Could be spreadsheets, internal systems, dashboards.

**Definition of done:** Verified by audit reviewer, approval logged, change reflected in source-of-truth system.

**RUNBOOK sections:**
- Systems of record
- Approval thresholds (what requires operator sign-off)
- Audit trail requirements
- Compliance requirements

**Workflow nuance:** Audit trail is mandatory. Every change must be traceable to who requested it, who approved it, who executed it.

---

## Research Team

**Roles:**
- **Lead Researcher (client-facing)** — receives research questions, scopes them
- **Senior Researcher (feasibility reviewer)** — validates questions are answerable, scopes methodology
- **Researcher / Analyst** — does the actual research
- **Peer Reviewer (QA)** — validates findings before publication
- **Operator** — handles ambiguous findings, contested conclusions

**Stages:** Question → Investigating → Drafting → Peer Review → Published → Blocked

**Shared medium:** Notion / shared docs / a research database. Linked from `workspace/`.

**Definition of done:** Findings peer-reviewed, sources cited, report published.

**Visual assets:** Diagrams, charts. Sometimes papers/PDFs as input.

**RUNBOOK sections:**
- Source quality standards
- Citation format
- Peer review criteria
- Publication targets
- Confidence levels and how to express uncertainty

**Workflow nuance:** Research projects can run long. Use SHARED_MEMORY.md aggressively to avoid losing context across sessions.

---

## Custom / Hybrid Teams

If the user describes something that doesn't fit cleanly:

1. Map their description to the closest archetype
2. Adjust roles, stages, and shared medium accordingly
3. Pull RUNBOOK sections from the closest match
4. Mark anything inferred so the user can correct in Step 4 review

Common hybrid patterns:
- **Dev + Marketing for SaaS** — two related projects with shared agents
- **Sales + Customer Success** — different stages, often same team
- **Research + Editorial** — research informs content
- **Real Estate + Marketing** — listing + promotion

For these, suggest creating two separate projects rather than one mega-project. Agents can participate in both.

FILE:references/interview-questions.md
# Interview Question Banks

Extended question banks for the Pass 1 (Team Identity) and Pass 2 (Work Structure) interviews. Use the questions in SKILL.md as the core flow; consult this file when:
- The user answers ambiguously and you need follow-up questions
- The team type is unusual and you need specialized prompts
- You're filling gaps in the AI-drafted plan and need to know what's missing

---

## Table of Contents

- [Pass 1: Team Identity — Core](#pass-1-team-identity--core)
- [Pass 1: Team Identity — Follow-ups](#pass-1-team-identity--follow-ups)
- [Pass 2: Work Structure — Core](#pass-2-work-structure--core)
- [Pass 2: Work Structure — Follow-ups](#pass-2-work-structure--follow-ups)
- [Pass 2: Team-Type-Specific Probes](#pass-2-team-type-specific-probes)
- [Gap-Filling Heuristics](#gap-filling-heuristics)

---

## Pass 1: Team Identity — Core

These are the questions in SKILL.md, restated here for reference:

1. **Project purpose** — one-sentence description
2. **Team type** — software dev, marketing, real estate, content, sales, customer success, operations, research, or other (describe)
3. **Project name and ID** — display name + lowercase ID
4. **Agent roster** — which existing agents will be on this team and their per-project role
5. **Client-facing agent** — who receives intake and talks to the client
6. **Feasibility reviewer** — domain expert who validates work is doable before commitment
7. **Quality reviewer** — who checks completed work before client delivery
8. **Operator** — is there a human in the loop, and what's their alias
9. **Other roles** — specialized contributors

---

## Pass 1: Team Identity — Follow-ups

If a user's answer is vague or missing, dig deeper:

### If they don't know what to call their team type
> "Don't worry about the label — describe what the team produces or does. Examples: 'They write blog posts and social copy for our clients.' Or: 'They handle inbound leads and qualify them.' I can pick a category from there."

### If they say they don't have a clear feasibility reviewer
> "Even if you don't have a formal 'reviewer,' someone needs to look at incoming work and say whether the team can actually do it given current capacity, skills, or constraints. Who would catch a problem like 'this requires expertise nobody has' before the team commits to delivering it?"

### If they say they don't have QA
> "Someone needs to be the last set of eyes before the work goes to the client. It doesn't have to be a separate person — it could be the feasibility reviewer doing a final pass. But there should be someone designated. Who is it?"

### If they say no operator
> "If a problem can't be resolved by any agent — say a client goes silent for a week, or two agents disagree on something the spec doesn't cover — who steps in? Even if it's just you, that's the operator."

### If they want all agents to do everything
Push back gently:
> "Specialized roles produce better results than agents trying to do everything. I'll need at least: a client-facing role, a feasibility reviewer, and a QA reviewer. Pick which agents fill those — they can still do other work too."

---

## Pass 2: Work Structure — Core

Restated from SKILL.md:

1. Task manager (Asana / ClickUp)
2. Stages of work (column structure)
3. Shared working medium
4. Definition of "done"
5. Sprint mode (one-at-a-time vs continuous flow)
6. Escalation thresholds
7. Visual / media assets
8. Anything else specific

---

## Pass 2: Work Structure — Follow-ups

### If they're unsure what columns to use

Suggest a default based on team type, then customize. Sensible defaults below.

### If they say "we don't have a shared working medium"
> "Even if work is mostly conversational, agents need somewhere to read shared context — past decisions, current state, reference material. That can be the project folder itself (`workspace/`). Is that enough, or do you have an external system the team uses?"

### If they're confused about sprint mode
> "Think about how new work arrives. Does the client typically hand you one big chunk to do well, or a steady stream of small things? Big chunks → sprint mode. Steady stream → continuous flow. You can change this later."

### If escalation thresholds feel arbitrary
> "These exist so agents don't spin wheels burning AI tokens. The defaults — 24 hours stuck, 2 re-escalations, 48 hours waiting on client — are reasonable for most teams. If your work is faster-paced (e.g., same-day turnaround), tighten these. If slower (e.g., long research projects), loosen them."

### If they're unclear on visual/media handling
Ask:
- "Will any agent need to look at images, video, or audio to do their work?"
- "Will any agent need to produce images, video, or audio as output?"

If either is yes → visual handling needed.

---

## Pass 2: Team-Type-Specific Probes

Use these to flesh out the plan when the user picks a particular team type.

### Software Development
- Frontend / backend / full-stack split?
- Branch and PR conventions? (default suggested in workflow.md)
- Local dev environment standardized?
- Deployment owned by team or separate?
- Test conventions — required or aspirational?

### Marketing / Creative
- Brand guidelines location?
- Channel mix (social, blog, email, paid)?
- Creative director or copy lead?
- Asset library / DAM in use?
- Approval chain — does client see drafts or only finals?

### Real Estate
- Listing source — MLS or direct?
- Photography handled by team or external?
- Pricing authority — who signs off?
- CRM platform?
- Compliance / disclosure requirements?

### Content / Editorial
- Editorial calendar in place?
- Editor / copy editor / fact-checker chain?
- CMS in use?
- SEO requirements?
- Image sourcing rules (rights, attribution)?

### Sales / Outreach
- CRM platform?
- Lead source(s)?
- Qualification criteria?
- Hand-off to closer / account manager?
- Compliance (GDPR, CAN-SPAM, etc.)?

### Customer Success / Support
- Ticketing platform?
- Tier 1 / Tier 2 split?
- Knowledge base location?
- SLA targets?
- Escalation path for technical issues?

### Operations
- What's the operational scope? (logistics, finance, HR ops, etc.)
- Systems of record?
- Approval workflows?
- Audit / compliance requirements?

### Research
- Research domain?
- Sources (papers, web, internal data)?
- Output format (reports, briefings, structured data)?
- Peer review chain?

### Other / Custom
Ask:
- "Walk me through what the team does end-to-end. Start when work arrives, end when it's delivered."
- "Where does the team produce its work?"
- "Who reviews before delivery?"
- "What can go wrong, and who handles it?"

---

## Default Stage Suggestions by Team Type

Use these as starting points. Confirm with user before locking in.

| Team Type | Suggested Stages |
|---|---|
| Software Dev | Backlog → In Progress → In Review → QA → Completed → Blocked |
| Marketing | Brief → In Drafting → Internal Review → Client Review → Approved → Blocked |
| Real Estate | New → Listing Prep → Listed → Under Offer → Closed → Blocked |
| Content | Idea → Drafting → Editing → Fact Check → Published → Blocked |
| Sales | Lead → Qualifying → Engaged → Proposal → Closed-Won/Lost → Blocked |
| Customer Success | New Ticket → In Progress → Awaiting Customer → Resolved → Blocked |
| Operations | Request → Triage → In Progress → Verification → Complete → Blocked |
| Research | Question → Investigating → Drafting → Peer Review → Published → Blocked |

Every team should have a "Blocked" column for work that's stuck waiting on something external.

---

## Gap-Filling Heuristics

When writing the AI-drafted plan in Step 4, you'll inevitably need to fill gaps the user didn't explicitly answer. Use these heuristics. Mark all inferred items with [INFERRED] in the plan.

### Communication frequency
If unspecified, default: agents check their queue at every session start, before any other action.

### Sprint length
If unspecified and using sprint mode, do not invent a fixed length. Sprints end when all committed work is delivered, not on a calendar.

### Re-work loops
If unspecified, default: failed QA review → back to executor with specific failures listed → re-submit when fixed. Same dev escalation rules apply during rework.

### Idle behavior
If unspecified, default: when nothing is in their queue and no task is assigned to them, agents do nothing (do not invent work).

### Decision authority
If unspecified, default: feasibility-reviewer has technical authority within the project; client-facing agent has client-relationship authority; operator has final authority on anything contested.

### "Done" definition
If unspecified for the team type, default: "QA reviewed and approved against the accepted scope, and the operator (if any) has signed off on delivery." Adjust per team.

### Versioning of accepted scope
Always default: scope documents are versioned (`SPEC-v1-[date].md`) and never overwritten. Latest accepted version is referenced from `SPEC-CURRENT.md`. This is universal across team types.

### Mid-sprint scope changes
If unspecified, default: scope changes mid-sprint require client-facing agent to pause work, log the change in DECISIONS.md, get explicit re-acceptance from client, and update SPEC. Avoid silent scope drift.

---
name: clickhouse
description: >
  Comprehensive ClickHouse skill covering everything you need to work with a ClickHouse analytics database:
  schema design, query optimization, insert strategies, CLI usage, table creation and migrations,
  backend integration (Node.js, Python, Go), Redis caching strategy, cluster vs single-node differences,
  and how to test/debug data in the database. MUST USE whenever the user mentions ClickHouse, asks about
  analytics tables, high-volume insert pipelines, MergeTree schemas, ORDER BY / PRIMARY KEY design,
  materialized views, ClickHouse query performance, connecting to ClickHouse from code, or running
  ClickHouse CLI commands to inspect data.
---

# ClickHouse Skill

A complete reference for designing, operating, querying, and integrating ClickHouse from backend services.
ClickHouse is a **columnar, append-optimized** analytics database — it is NOT a transactional database.
Design everything around that fact.

> Official docs: https://clickhouse.com/docs/best-practices

---

## Quick Reference — Read the Right Section

| Task | Go To |
|------|-------|
| Design a new table | [Schema Design](#schema-design) |
| Write a migration | [Migrations](#migrations) |
| Insert data from code | [Insert Strategy](#insert-strategy) |
| Run queries / inspect the DB | [CLI Reference](#cli-reference) |
| Connect from Node.js / Python / Go | [Backend Integration](#backend-integration) |
| Optimize a slow query | [Query Optimization](#query-optimization) |
| Decide on Redis vs direct query | [Redis Caching Strategy](#redis-caching-strategy) |
| Understand cluster behavior | [Cluster Considerations](#cluster-considerations) |

---

## Core Mental Model

ClickHouse is an **append-only, batch-oriented** analytics database.
The biggest performance wins come from:

1. **Writing large batches** (10K–100K rows), not individual rows
2. **Choosing ORDER BY carefully** — it is immutable and drives all query performance
3. **Using native types** — never store everything as String
4. **Reading many rows across few columns** — not few rows across many columns

Avoid ClickHouse for:
- OLTP workloads (frequent single-row reads/writes)
- Complex multi-table JOINs on huge tables
- Frequent UPDATE/DELETE patterns

---

## Schema Design

### Step 1 — Plan ORDER BY Before Creating Any Table

**ORDER BY is immutable.** Changing it requires creating a new table and migrating all data.
Get it right before writing a single row.

Questions to answer first:
- What columns appear in `WHERE` clauses most often?
- What is the cardinality (number of distinct values) of each filter column?
- Is there a mandatory filter that every query has (e.g. `tenant_id`, `app_id`)?
- Are date ranges a common filter?

```sql
-- BAD: UUID as first ORDER BY column — no index benefit
CREATE TABLE events (
    id UUID,
    timestamp DateTime,
    event_type String,
    user_id UInt64
) ENGINE = MergeTree()
ORDER BY (id);

-- GOOD: Low cardinality first, then date, then higher cardinality
CREATE TABLE events (
    id UUID,
    timestamp DateTime,
    event_type LowCardinality(String),
    user_id UInt64
) ENGINE = MergeTree()
ORDER BY (event_type, toDate(timestamp), user_id);
```

**Cardinality ordering rule:** Put columns with **fewer distinct values first**.

| Position | Cardinality | Examples |
|----------|-------------|----------|
| 1st | Low (2–1,000) | `event_type`, `status`, `country` |
| 2nd | Date (coarse) | `toDate(timestamp)` |
| 3rd+ | Medium-High | `user_id`, `session_id` |
| Last | High (if needed) | `event_id`, UUID |

**Index usage by query pattern** (for `ORDER BY (event_type, event_date, user_id)`):

| Filter | Index Used? |
|--------|-------------|
| `WHERE event_type = 'X'` | ✅ Yes |
| `WHERE event_type = 'X' AND event_date = '...'` | ✅ Yes |
| `WHERE event_date = '...'` | ❌ No — skips first column |
| `WHERE user_id = 123` | ❌ No — skips first two |

For columns that can't be in ORDER BY, add a **data skipping index** (see [Query Optimization](#query-optimization)).

---

### Step 2 — Choose the Right Engine

| Engine | Use When |
|--------|----------|
| `MergeTree` | Standard append-only analytics |
| `ReplacingMergeTree(ver)` | Need logical "upserts" (new version replaces old) |
| `AggregatingMergeTree` | Pre-aggregated data for materialized views |
| `CollapsingMergeTree(sign)` | Logical deletes via insert pattern |
| `SummingMergeTree` | Automatically sum numeric columns on merge |
| `ReplicatedMergeTree` | Any engine on a cluster with replication |

For clusters, prefix engine name with `Replicated`: `ReplicatedMergeTree(...)`.

---

### Step 3 — Pick Native Types (Never Store Everything as String)

| Data | Wrong | Right | Savings |
|------|-------|-------|---------|
| UUID | `String` | `UUID` | 56% |
| Timestamp | `String` | `DateTime` / `DateTime64(3)` | 58–79% |
| Integer ID | `String` | `UInt32` / `UInt64` | varies |
| Boolean | `String` | `Bool` | 75–80% |
| IPv4 | `String` | `IPv4` | 43–73% |
| Decimal amount | `String` | `Decimal(10,2)` | significant |

**Use the smallest numeric type that fits:**

| Type | Range | Use For |
|------|-------|---------|
| `UInt8` | 0–255 | age, rating, status code |
| `UInt16` | 0–65,535 | year, port |
| `UInt32` | 0–4.2B | most IDs, unix timestamps |
| `UInt64` | 0–18E | very large counters |

**Use LowCardinality for repeated strings with < 10,000 unique values:**
```sql
country LowCardinality(String),   -- ~200 unique values
browser LowCardinality(String),   -- ~50 unique values
event_type LowCardinality(String) -- ~100 unique values
```

**Use Enum for fixed, known value sets:**
```sql
-- Provides insert-time validation + 1-byte storage
status Enum8('pending' = 1, 'processing' = 2, 'shipped' = 3, 'delivered' = 4)
```

**Avoid Nullable unless the null is semantically meaningful:**
```sql
-- BAD: Nullable everywhere
name Nullable(String),
login_count Nullable(UInt32)

-- GOOD: Use defaults; Nullable only when null has distinct meaning
name String DEFAULT '',
login_count UInt32 DEFAULT 0,
deleted_at Nullable(DateTime),   -- NULL = "not deleted" is semantically distinct
parent_id Nullable(UInt64)       -- NULL = "no parent" is semantically distinct
```

---

### Step 4 — Partitioning Strategy

**Partition for lifecycle management, NOT for query performance.**
Query performance comes from ORDER BY. Partitions enable fast data expiry.

```sql
-- GOOD: Monthly partitions for TTL and lifecycle
CREATE TABLE events (
    timestamp DateTime,
    event_type LowCardinality(String),
    user_id UInt64
) ENGINE = MergeTree()
PARTITION BY toStartOfMonth(timestamp)
ORDER BY (event_type, toDate(timestamp), user_id)
TTL timestamp + INTERVAL 90 DAY;

-- Instant deletion of a month
ALTER TABLE events DROP PARTITION '2024-01';
```

**Keep partition count between 100–1,000.** Daily partitions grow unbounded; monthly is usually safe.

**Tiered storage:**
```sql
TTL
    timestamp + INTERVAL 7 DAY TO VOLUME 'hot',
    timestamp + INTERVAL 30 DAY TO VOLUME 'warm',
    timestamp + INTERVAL 365 DAY DELETE;
```

**If you're unsure, start without partitioning.** You can add it later by creating a new table, migrating data, and renaming.

---

### Complete Table Example

```sql
CREATE TABLE page_events (
    -- Identifiers
    event_id     UUID DEFAULT generateUUIDv4(),
    tenant_id    UInt32,
    user_id      UInt64,

    -- Low-cardinality dimensions (great for ORDER BY)
    event_type   LowCardinality(String),
    country      LowCardinality(String) DEFAULT '',
    browser      LowCardinality(String) DEFAULT '',
    platform     Enum8('web'=1, 'ios'=2, 'android'=3, 'api'=4),

    -- Timestamps
    occurred_at  DateTime64(3),
    inserted_at  DateTime DEFAULT now(),

    -- Metrics
    duration_ms  UInt32 DEFAULT 0,
    revenue      Decimal(12,4) DEFAULT 0,

    -- Flexible properties
    properties   JSON,

    -- Skipping index for user lookups
    INDEX idx_user_id user_id TYPE bloom_filter GRANULARITY 4
) ENGINE = MergeTree()
PARTITION BY toStartOfMonth(occurred_at)
ORDER BY (tenant_id, event_type, toDate(occurred_at), user_id)
TTL occurred_at + INTERVAL 365 DAY
SETTINGS index_granularity = 8192;
```

---

## Migrations

ClickHouse does NOT support transactional DDL. There is no rollback. Plan carefully.

### ORMs / Migration Tools Compatibility

| Tool | ClickHouse Support |
|------|--------------------|
| Prisma | ❌ No native support — use raw SQL migrations |
| Drizzle | ❌ No native support — use raw SQL migrations |
| TypeORM | ⚠️ Unofficial community driver only |
| Flyway | ✅ Supported via JDBC driver |
| Liquibase | ✅ Supported |
| golang-migrate | ✅ Works well — recommended for Go |
| Custom SQL files | ✅ Always works |

**For Node.js projects:** maintain a `migrations/` folder with numbered `.sql` files and a small runner script.

**For Go projects:** use `golang-migrate` with the ClickHouse driver.

**Do NOT use Prisma/Drizzle to generate ClickHouse DDL.** They have no concept of MergeTree engines, ORDER BY, or PARTITION BY.

---

### Migration Patterns

#### Adding a Column
```sql
-- Safe: adding a column with a DEFAULT
ALTER TABLE events ADD COLUMN IF NOT EXISTS session_id UUID DEFAULT generateUUIDv4();

-- For a cluster: ON CLUSTER must come first
ALTER TABLE events ON CLUSTER '{cluster}' ADD COLUMN IF NOT EXISTS session_id UUID DEFAULT generateUUIDv4();
```

#### Changing an ORDER BY (requires table recreation)
```sql
-- 1. Create new table with correct ORDER BY
CREATE TABLE events_v2 AS events;   -- Copies structure
ALTER TABLE events_v2 MODIFY ORDER BY (tenant_id, event_type, toDate(occurred_at), user_id);

-- Or create from scratch:
CREATE TABLE events_v2 (...) ENGINE = MergeTree() ORDER BY (...);

-- 2. Migrate data
INSERT INTO events_v2 SELECT * FROM events;

-- 3. Swap
RENAME TABLE events TO events_old, events_v2 TO events;

-- 4. Verify, then drop
DROP TABLE events_old;
```

#### Adding a Skipping Index
```sql
ALTER TABLE events ADD INDEX IF NOT EXISTS idx_user_id user_id TYPE bloom_filter GRANULARITY 4;
ALTER TABLE events MATERIALIZE INDEX idx_user_id;  -- Backfill existing data
```

#### Node.js Migration Runner Example
```javascript
// migrations/runner.js
import { createClient } from '@clickhouse/client';
import fs from 'fs';
import path from 'path';

const client = createClient({
  url: process.env.CLICKHOUSE_URL,
  username: process.env.CLICKHOUSE_USER,
  password: process.env.CLICKHOUSE_PASSWORD,
  database: process.env.CLICKHOUSE_DB,
});

// Track applied migrations
await client.command({
  query: `CREATE TABLE IF NOT EXISTS _migrations (
    name String,
    applied_at DateTime DEFAULT now()
  ) ENGINE = MergeTree() ORDER BY (applied_at, name)`
});

const applied = new Set(
  (await client.query({ query: 'SELECT name FROM _migrations', format: 'JSONEachRow' }))
    .json().map(r => r.name)
);

const files = fs.readdirSync('./migrations').filter(f => f.endsWith('.sql')).sort();

for (const file of files) {
  if (applied.has(file)) continue;
  const sql = fs.readFileSync(path.join('./migrations', file), 'utf-8');
  // Execute each statement separately (ClickHouse doesn't support multi-statement by default)
  for (const stmt of sql.split(';').map(s => s.trim()).filter(Boolean)) {
    await client.command({ query: stmt });
  }
  await client.command({ query: `INSERT INTO _migrations (name) VALUES ('file')` });
  console.log(`Applied: file`);
}
```

#### Go Migration Runner (golang-migrate)
```go
import (
    "github.com/golang-migrate/migrate/v4"
    _ "github.com/golang-migrate/migrate/v4/database/clickhouse"
    _ "github.com/golang-migrate/migrate/v4/source/file"
)

m, err := migrate.New(
    "file://migrations",
    "clickhouse://localhost:9000?database=mydb&username=default&password=",
)
if err != nil { log.Fatal(err) }
if err := m.Up(); err != nil && err != migrate.ErrNoChange {
    log.Fatal(err)
}
```

---

## Insert Strategy

### Rule 1 — Batch 10,000–100,000 Rows Per INSERT

Each `INSERT` creates a **data part**. Many small inserts = many small parts = merge pressure = cluster instability.

```python
# BAD: one row at a time
for event in events:
    client.execute("INSERT INTO events VALUES", [event])  # Creates 10,000 parts!

# GOOD: batch appropriately
BATCH_SIZE = 10_000
for i in range(0, len(events), BATCH_SIZE):
    client.execute("INSERT INTO events VALUES", events[i:i+BATCH_SIZE])
```

**Monitor part health:**
```sql
SELECT table, count() as parts, sum(rows) as total_rows
FROM system.parts
WHERE active AND database = currentDatabase()
GROUP BY table
ORDER BY parts DESC;
-- Warning: > 3,000 parts per table is trouble
```

### Rule 2 — Use Async Inserts for Many Small Producers

When batching client-side isn't practical (many microservices, IoT, etc.):

```sql
SET async_insert = 1;
SET async_insert_max_data_size = 10000000;   -- 10MB buffer
SET async_insert_busy_timeout_ms = 1000;      -- Flush every 1s
SET wait_for_async_insert = 1;                -- Wait for durability confirmation
```

### Rule 3 — Avoid Mutations (UPDATE/DELETE)

ClickHouse is not built for mutations. They rewrite entire data parts.

| Need | Use Instead |
|------|-------------|
| UPDATE rows | `ReplacingMergeTree` + insert new version |
| DELETE rows frequently | Lightweight DELETE (23.3+): `DELETE FROM events WHERE ...` |
| Delete old data in bulk | `DROP PARTITION` |
| Track deletions | `CollapsingMergeTree(sign)` with `sign = -1` row |

**ReplacingMergeTree pattern:**
```sql
CREATE TABLE users (
    user_id UInt64,
    name String,
    status LowCardinality(String),
    updated_at DateTime DEFAULT now()
) ENGINE = ReplacingMergeTree(updated_at)
ORDER BY user_id;

-- "Update" by inserting a new version
INSERT INTO users (user_id, name, status) VALUES (123, 'Alice', 'inactive');

-- Query deduplicated (FINAL is slower but consistent)
SELECT * FROM users FINAL WHERE user_id = 123;

-- Or use argMax for better performance at scale
SELECT user_id, argMax(status, updated_at) as status
FROM users GROUP BY user_id;
```

### Rule 4 — Never Run OPTIMIZE TABLE FINAL in Production

Background merges handle part consolidation automatically. Forcing it:
- Blocks other operations
- Causes severe disk I/O spikes
- Provides no lasting benefit

---

## CLI Reference

### Connect

```bash
# Basic
clickhouse-client -h <host> -u <user> --password <pass> -d <database>

# Using env vars (recommended — hides password from process list)
export CLICKHOUSE_PASSWORD=yourpassword
clickhouse-client -h 127.0.0.1 -u app_user -d app_db

# With SSL
clickhouse-client -h <host> -u <user> -d <db> --secure --port 9440
```

Parse a JDBC URL (`jdbc:clickhouse://host:8123/db`):
```bash
JDBC="jdbc:clickhouse://myhost.com:8123/mydb"
HOST=$(echo $JDBC | sed 's|.*://\([^:]*\):.*|\1|')
PORT=$(echo $JDBC | sed 's|.*:\([0-9]*\)/.*|\1|')
DB=$(echo $JDBC   | sed 's|.*/||')
clickhouse-client -h "$HOST" --port "$PORT" -d "$DB"
```

### Inspect the Database

```bash
# List databases
clickhouse-client -h <host> -u <user> -q "SHOW DATABASES;" --format=TSV

# List tables
clickhouse-client -h <host> -u <user> -d <db> -q "SHOW TABLES;" --format=TSV

# Describe a table
clickhouse-client -h <host> -u <user> -d <db> -q "DESCRIBE TABLE my_table;" --format=TSV

# Show CREATE statement (includes engine, ORDER BY, partitioning)
clickhouse-client -h <host> -u <user> -d <db> -q "SHOW CREATE TABLE my_table;" --format=TSV

# Table sizes
clickhouse-client -h <host> -u <user> -d <db> -q "
SELECT table, total_rows as rows,
       formatReadableSize(total_bytes) as size,
       formatReadableSize(data_bytes) as data
FROM system.tables WHERE database = currentDatabase()
ORDER BY total_bytes DESC;" --format=PrettyCompact

# Check primary key / partition key columns
clickhouse-client -h <host> -u <user> -d <db> -q "
SELECT name, type, is_in_primary_key, is_in_partition_key
FROM system.columns
WHERE database = '<db>' AND table = '<table>'
ORDER BY position;" --format=PrettyCompact

# Part health check
clickhouse-client -h <host> -u <user> -d <db> -q "
SELECT table, count() as parts, sum(rows) as rows,
       formatReadableSize(sum(bytes_on_disk)) as size
FROM system.parts WHERE active AND database = currentDatabase()
GROUP BY table ORDER BY parts DESC;" --format=PrettyCompact
```

### Query Data

```bash
# Basic query — JSON output
clickhouse-client -h <host> -u <user> -d <db> \
  -q "SELECT * FROM events LIMIT 10;" --format=JSONEachRow | jq -s '.'

# Aggregation
clickhouse-client -h <host> -u <user> -d <db> \
  -q "SELECT event_type, count() as n FROM events GROUP BY event_type ORDER BY n DESC;" \
  --format=PrettyCompact

# Export to CSV
clickhouse-client -h <host> -u <user> -d <db> \
  -q "SELECT * FROM events FORMAT CSV" > /tmp/events.csv
```

### Analyze Query Performance

```bash
# Execution plan
clickhouse-client -h <host> -u <user> -d <db> \
  -q "EXPLAIN SELECT * FROM events WHERE user_id = 123;" --format=TSV

# With actual timing (ClickHouse 21.1+)
clickhouse-client -h <host> -u <user> -d <db> \
  -q "EXPLAIN ANALYZE SELECT * FROM events WHERE user_id = 123;" --format=TSV

# See which indexes were used
clickhouse-client -h <host> -u <user> -d <db> \
  -q "EXPLAIN indexes = 1 SELECT * FROM events WHERE user_id = 123;" --format=TSV
```

Look for:
- `Rows` in EXPLAIN output — fewer is better
- `Skip` entries showing granules skipped by indexes
- `Full scan` — indicates missing index coverage

### Insert / Modify Data

```bash
# Insert from file (CSV)
clickhouse-client -h <host> -u <user> -d <db> \
  -q "INSERT INTO events FORMAT CSV" < data.csv

# Insert from file (JSONEachRow)
clickhouse-client -h <host> -u <user> -d <db> \
  -q "INSERT INTO events FORMAT JSONEachRow" < data.ndjson

# Run a SQL script
clickhouse-client -h <host> -u <user> -d <db> --multiquery < migration.sql

# Lightweight delete (23.3+)
clickhouse-client -h <host> -u <user> -d <db> \
  -q "DELETE FROM events WHERE occurred_at < '2023-01-01';"

# Drop partition (instant, for lifecycle)
clickhouse-client -h <host> -u <user> -d <db> \
  -q "ALTER TABLE events DROP PARTITION '2023-01';"
```

---

## Backend Integration

### Node.js

**Install:**
```bash
npm install @clickhouse/client
```

**Module setup (`src/clickhouse.js` or `src/clickhouse.ts`):**
```javascript
// src/clickhouse.js
import { createClient } from '@clickhouse/client';

let _client = null;

export function getClickHouseClient() {
  if (_client) return _client;
  _client = createClient({
    url: process.env.CLICKHOUSE_URL ?? 'http://localhost:8123',
    username: process.env.CLICKHOUSE_USER ?? 'default',
    password: process.env.CLICKHOUSE_PASSWORD ?? '',
    database: process.env.CLICKHOUSE_DB ?? 'default',
    clickhouse_settings: {
      async_insert: 1,                       // Buffer small inserts server-side
      wait_for_async_insert: 1,              // Confirm durability
      async_insert_busy_timeout_ms: 1000,
    },
    compression: { request: true },           // Compress inserts
    request_timeout: 30_000,
  });
  return _client;
}
```

**Environment variables (`.env`):**
```env
CLICKHOUSE_URL=http://localhost:8123
CLICKHOUSE_USER=default
CLICKHOUSE_PASSWORD=secret
CLICKHOUSE_DB=analytics
```

**Insert (batch — always batch):**
```javascript
import { getClickHouseClient } from './clickhouse.js';

// Accumulate rows, then flush in a batch
const BATCH_SIZE = 10_000;
const buffer = [];

export async function trackEvent(event) {
  buffer.push(event);
  if (buffer.length >= BATCH_SIZE) {
    await flush();
  }
}

export async function flush() {
  if (buffer.length === 0) return;
  const rows = buffer.splice(0, buffer.length);
  const client = getClickHouseClient();
  await client.insert({
    table: 'events',
    values: rows,
    format: 'JSONEachRow',
  });
}

// Also flush on process exit / interval
setInterval(flush, 5_000);
process.on('beforeExit', flush);
```

**Query (analytics — aggregate, don't fetch rows one by one):**
```javascript
export async function getEventStats({ startDate, endDate, eventType }) {
  const client = getClickHouseClient();
  const result = await client.query({
    query: `
      SELECT
        toStartOfHour(occurred_at) AS hour,
        count() AS events,
        uniq(user_id) AS unique_users
      FROM events
      WHERE
        event_type = {eventType: String}
        AND occurred_at >= {startDate: DateTime}
        AND occurred_at < {endDate: DateTime}
      GROUP BY hour
      ORDER BY hour
    `,
    query_params: { eventType, startDate, endDate },
    format: 'JSONEachRow',
  });
  return result.json();
}
```

**TypeScript types:**
```typescript
interface EventRow {
  event_id: string;
  tenant_id: number;
  event_type: string;
  user_id: number;
  occurred_at: string;  // ClickHouse returns DateTime as string
  properties: Record<string, unknown>;
}

const result = await client.query({
  query: 'SELECT * FROM events LIMIT 100',
  format: 'JSONEachRow',
});
const rows = await result.json<EventRow[]>();
```

---

### Python

**Install:**
```bash
pip install clickhouse-connect   # Official Anthropic-maintained driver
# or
pip install clickhouse-driver    # Older but widely used
```

**Module setup (`clickhouse.py`):**
```python
# clickhouse.py
import os
import clickhouse_connect
from functools import lru_cache

@lru_cache(maxsize=1)
def get_client():
    return clickhouse_connect.get_client(
        host=os.environ.get('CLICKHOUSE_HOST', 'localhost'),
        port=int(os.environ.get('CLICKHOUSE_PORT', 8123)),
        username=os.environ.get('CLICKHOUSE_USER', 'default'),
        password=os.environ.get('CLICKHOUSE_PASSWORD', ''),
        database=os.environ.get('CLICKHOUSE_DB', 'default'),
        settings={
            'async_insert': 1,
            'wait_for_async_insert': 1,
            'async_insert_busy_timeout_ms': 1000,
        },
        compress=True,
    )
```

**Batch insert:**
```python
from clickhouse import get_client
from datetime import datetime

BATCH_SIZE = 10_000

def insert_events(events: list[dict]):
    """Always insert in batches of 10K+ rows."""
    client = get_client()
    # clickhouse_connect expects column-oriented data
    column_names = ['tenant_id', 'event_type', 'user_id', 'occurred_at', 'properties']
    data = [
        [e['tenant_id'] for e in events],
        [e['event_type'] for e in events],
        [e['user_id'] for e in events],
        [e['occurred_at'] for e in events],
        [e.get('properties', {}) for e in events],
    ]
    client.insert('events', data, column_names=column_names)

def batch_insert(events: list[dict]):
    for i in range(0, len(events), BATCH_SIZE):
        insert_events(events[i:i+BATCH_SIZE])
```

**Query:**
```python
def get_event_stats(event_type: str, start_date: str, end_date: str):
    client = get_client()
    result = client.query("""
        SELECT
            toStartOfHour(occurred_at) AS hour,
            count() AS events,
            uniq(user_id) AS unique_users
        FROM events
        WHERE event_type = {event_type:String}
          AND occurred_at >= {start_date:DateTime}
          AND occurred_at < {end_date:DateTime}
        GROUP BY hour
        ORDER BY hour
    """, parameters={'event_type': event_type, 'start_date': start_date, 'end_date': end_date})
    return result.named_results()  # Returns list of dicts
```

**With pandas (for data pipelines):**
```python
def get_dataframe(query: str, params: dict = None):
    client = get_client()
    return client.query_df(query, parameters=params or {})

df = get_dataframe("SELECT event_type, count() as n FROM events GROUP BY event_type")
```

---

### Go

**Install:**
```bash
go get github.com/ClickHouse/clickhouse-go/v2
```

**Module setup (`internal/clickhouse/client.go`):**
```go
package clickhouse

import (
    "context"
    "crypto/tls"
    "fmt"
    "os"
    "sync"
    "time"

    ch "github.com/ClickHouse/clickhouse-go/v2"
    "github.com/ClickHouse/clickhouse-go/v2/lib/driver"
)

var (
    once   sync.Once
    client driver.Conn
)

func GetClient() (driver.Conn, error) {
    var err error
    once.Do(func() {
        options := &ch.Options{
            Addr: []string{fmt.Sprintf("%s:%s",
                getEnv("CLICKHOUSE_HOST", "localhost"),
                getEnv("CLICKHOUSE_PORT", "9000"),
            )},
            Auth: ch.Auth{
                Database: getEnv("CLICKHOUSE_DB", "default"),
                Username: getEnv("CLICKHOUSE_USER", "default"),
                Password: getEnv("CLICKHOUSE_PASSWORD", ""),
            },
            Settings: ch.Settings{
                "async_insert":                1,
                "wait_for_async_insert":       1,
                "async_insert_busy_timeout_ms": 1000,
            },
            DialTimeout:     time.Second * 5,
            MaxOpenConns:    10,
            MaxIdleConns:    5,
            ConnMaxLifetime: time.Hour,
            Compression: &ch.Compression{
                Method: ch.CompressionLZ4,
            },
        }

        // Enable TLS for production
        if os.Getenv("CLICKHOUSE_TLS") == "true" {
            options.TLS = &tls.Config{InsecureSkipVerify: false}
        }

        client, err = ch.Open(options)
    })
    return client, err
}

func getEnv(key, fallback string) string {
    if v := os.Getenv(key); v != "" {
        return v
    }
    return fallback
}
```

**Batch insert:**
```go
package clickhouse

import (
    "context"
    "time"
)

type Event struct {
    TenantID   uint32    `ch:"tenant_id"`
    EventType  string    `ch:"event_type"`
    UserID     uint64    `ch:"user_id"`
    OccurredAt time.Time `ch:"occurred_at"`
}

func InsertEvents(ctx context.Context, events []Event) error {
    conn, err := GetClient()
    if err != nil {
        return err
    }

    batch, err := conn.PrepareBatch(ctx, "INSERT INTO events")
    if err != nil {
        return err
    }

    for _, e := range events {
        if err := batch.AppendStruct(&e); err != nil {
            return err
        }
    }
    return batch.Send()
}
```

**Query:**
```go
type HourlyStats struct {
    Hour        time.Time `ch:"hour"`
    Events      uint64    `ch:"events"`
    UniqueUsers uint64    `ch:"unique_users"`
}

func GetEventStats(ctx context.Context, eventType, start, end string) ([]HourlyStats, error) {
    conn, err := GetClient()
    if err != nil {
        return nil, err
    }

    rows, err := conn.Query(ctx, `
        SELECT toStartOfHour(occurred_at) AS hour,
               count() AS events,
               uniq(user_id) AS unique_users
        FROM events
        WHERE event_type = @eventType
          AND occurred_at >= @start
          AND occurred_at < @end
        GROUP BY hour ORDER BY hour`,
        ch.Named("eventType", eventType),
        ch.Named("start", start),
        ch.Named("end", end),
    )
    if err != nil {
        return nil, err
    }
    defer rows.Close()

    var stats []HourlyStats
    for rows.Next() {
        var s HourlyStats
        if err := rows.ScanStruct(&s); err != nil {
            return nil, err
        }
        stats = append(stats, s)
    }
    return stats, rows.Err()
}
```

---

## Query Optimization

### Use ORDER BY Prefix in Every WHERE Clause

Always filter on the leftmost columns of ORDER BY first.
If you can't, add a data skipping index.

### Data Skipping Indexes

For columns NOT in ORDER BY that you filter on:

```sql
-- Add bloom filter for high-cardinality equality lookups
ALTER TABLE events ADD INDEX idx_user_id user_id TYPE bloom_filter GRANULARITY 4;
ALTER TABLE events MATERIALIZE INDEX idx_user_id;  -- Backfill

-- Index types:
-- bloom_filter: equality on high-cardinality (user IDs, session IDs)
-- set(N):       low-cardinality equality (status IN ('a','b'))  
-- minmax:       range queries (amount > 1000)
-- ngrambf_v1:  text search (LIKE '%term%')
-- tokenbf_v1:  token search (hasToken(text, 'word'))

-- Verify it's being used
EXPLAIN indexes = 1
SELECT * FROM events WHERE user_id = 12345;
-- Look for "Skip" entries in output
```

### JOINs

ClickHouse JOINs load the **right table into memory**. Always put the smaller table on the right.

```sql
-- BAD: large table on right
SELECT * FROM small_table s JOIN large_table l ON l.id = s.id;

-- GOOD: small table on right
SELECT * FROM large_table l JOIN small_table s ON s.id = l.id;
```

**Filter BEFORE joining:**
```sql
-- GOOD: reduce data before the join
SELECT * FROM
    (SELECT * FROM orders WHERE status = 'completed') o
JOIN
    (SELECT * FROM customers WHERE country = 'US') c
ON c.id = o.customer_id;
```

**Choose the right algorithm:**
```sql
SET join_algorithm = 'auto';          -- Default: ClickHouse decides
SET join_algorithm = 'partial_merge'; -- Large-to-large, memory-constrained
SET join_algorithm = 'grace_hash';    -- Large datasets, can spill to disk
```

**Use ANY JOIN when you only need one match:**
```sql
SELECT o.*, c.name
FROM orders o
ANY LEFT JOIN customers c ON c.id = o.customer_id;
-- Faster and less memory when right table may have duplicates
```

**Alternatives to JOINs (often faster):**
```sql
-- Dictionary for dimension lookups
SELECT o.*, dictGet('customers_dict', 'name', o.customer_id) as name
FROM orders o;

-- IN subquery for filtering
SELECT * FROM orders
WHERE customer_id IN (SELECT id FROM customers WHERE country = 'US');
```

### Materialized Views

Use materialized views to pre-aggregate data instead of scanning raw tables.

**Incremental MV (updates in real time):**
```sql
-- Destination table
CREATE TABLE events_hourly (
    hour DateTime,
    event_type LowCardinality(String),
    events AggregateFunction(count, UInt64),
    unique_users AggregateFunction(uniq, UInt64)
) ENGINE = AggregatingMergeTree()
ORDER BY (event_type, hour);

-- MV triggers on every INSERT into events
CREATE MATERIALIZED VIEW events_hourly_mv TO events_hourly AS
SELECT
    toStartOfHour(occurred_at) AS hour,
    event_type,
    countState() AS events,
    uniqState(user_id) AS unique_users
FROM events
GROUP BY hour, event_type;

-- Query (reads thousands instead of billions)
SELECT hour, event_type, countMerge(events), uniqMerge(unique_users)
FROM events_hourly
WHERE hour >= now() - INTERVAL 7 DAY
GROUP BY hour, event_type;
```

**Refreshable MV (periodic rebuild, good for complex JOINs):**
```sql
CREATE MATERIALIZED VIEW customer_summary
REFRESH EVERY 1 HOUR
ENGINE = MergeTree() ORDER BY customer_id
AS SELECT
    c.customer_id, c.name,
    count() as orders, sum(o.amount) as total_spent
FROM customers c
LEFT JOIN orders o ON o.customer_id = c.customer_id
GROUP BY c.customer_id, c.name;

-- Force refresh
SYSTEM REFRESH VIEW customer_summary;
```

### Avoid Small/Single-Row Queries

ClickHouse is built for scanning many rows and returning aggregates.
**Do not use it like a key-value store.**

```javascript
// BAD: Fetching one user's data from ClickHouse on every request
app.get('/user/:id/events', async (req, res) => {
  const events = await ch.query(`SELECT * FROM events WHERE user_id = req.params.id`);
  res.json(events); // This is a key-value access pattern
});

// GOOD: Aggregate query that leverages ClickHouse's strength
app.get('/analytics/summary', async (req, res) => {
  const stats = await ch.query(`
    SELECT event_type, count() as n, uniq(user_id) as users
    FROM events
    WHERE occurred_at >= today() - 7
    GROUP BY event_type
  `);
  res.json(stats);
});
```

---

## Redis Caching Strategy

### When to Use Redis in Front of ClickHouse

| Scenario | Use Redis? | Reason |
|----------|-----------|--------|
| Dashboard with same query run by many users | ✅ Yes | Prevents redundant large scans |
| Single user fetching their own recent events | ✅ Yes | ClickHouse isn't a KV store |
| Aggregation query taking > 500ms | ✅ Yes | Cache computed result |
| Real-time per-user event counts | ✅ Yes | Maintain counter in Redis, bulk-sync to CH |
| Ad-hoc analytics queries (new filters every time) | ❌ No | Cache hit rate will be low |
| Time-series queries where time range keeps moving | ❌ Careful | Invalidation is complex |
| Backfill / batch ETL pipeline | ❌ No | No user-facing latency concern |

### Recommended Cache Pattern

```javascript
// Cache ClickHouse aggregation results in Redis
async function getDashboardStats(tenantId, dateRange) {
  const cacheKey = `stats:tenantId:dateRange`;
  const TTL = 300; // 5 minutes

  // 1. Try cache first
  const cached = await redis.get(cacheKey);
  if (cached) return JSON.parse(cached);

  // 2. Run the (potentially expensive) ClickHouse query
  const result = await clickhouse.query({
    query: `
      SELECT event_type, count() as n, uniq(user_id) as users
      FROM events
      WHERE tenant_id = {tenantId: UInt32}
        AND occurred_at >= {start: DateTime}
      GROUP BY event_type
    `,
    query_params: { tenantId, start: dateRange },
    format: 'JSONEachRow',
  });
  const data = await result.json();

  // 3. Cache for TTL
  await redis.setex(cacheKey, TTL, JSON.stringify(data));
  return data;
}
```

### When to Query ClickHouse Directly (No Redis)

- The query is already fast (< 100ms) due to good schema design and materialized views
- The query parameters are always unique (ad-hoc analytics, no cache benefit)
- You have a materialized view pre-aggregating the data — query the MV directly
- It's an internal/batch process with no latency requirement

**The right answer is usually:** build good materialized views so the ClickHouse query is already fast enough that you don't need Redis.

---

## Cluster Considerations

On a ClickHouse cluster, DDL and certain operations must include `ON CLUSTER`.

### Engine Naming

| Single-node | Cluster |
|-------------|---------|
| `MergeTree` | `ReplicatedMergeTree('/clickhouse/tables/{shard}/{database}/{table}', '{replica}')` |
| `ReplacingMergeTree(ver)` | `ReplicatedReplacingMergeTree(...)` |
| All MergeTree variants | `Replicated` prefix |

In practice, use macros defined in `config.xml`:
```sql
ENGINE = ReplicatedMergeTree('/clickhouse/tables/{shard}/{database}/{table}', '{replica}')
```

### DDL on Cluster

```sql
-- Always include ON CLUSTER for DDL on distributed setups
CREATE TABLE events ON CLUSTER '{cluster}' ( ... )
ENGINE = ReplicatedMergeTree('/clickhouse/tables/{shard}/{database}/{table}', '{replica}')
ORDER BY (...);

ALTER TABLE events ON CLUSTER '{cluster}' ADD COLUMN new_col UInt32 DEFAULT 0;
```

### Distributed Tables

```sql
-- Create local table first (on cluster)
CREATE TABLE events_local ON CLUSTER '{cluster}' ( ... )
ENGINE = ReplicatedMergeTree(...)
ORDER BY (...);

-- Then create a Distributed table as the access layer
CREATE TABLE events ON CLUSTER '{cluster}' ( ... )
ENGINE = Distributed('{cluster}', currentDatabase(), 'events_local', rand());
```

Applications connect to the Distributed table; ClickHouse routes queries to shards transparently.

### INSERT Routing

On a cluster, insert into the Distributed table (not the local table) unless you are doing a shard-local operation intentionally.

### System Queries on Clusters

```sql
-- Check part health across all shards
SELECT hostName(), table, count() as parts
FROM clusterAllReplicas('{cluster}', system.parts)
WHERE active GROUP BY hostName(), table ORDER BY parts DESC;

-- Check replication lag
SELECT database, table, replica_name, queue_size
FROM system.replication_queue
WHERE queue_size > 0;
```

---

## Rules Reference

Detailed per-rule files are in `rules/` (loaded on demand):

- Schema / Primary Key: `rules/schema-pk-*.md`
- Schema / Types: `rules/schema-types-*.md`
- Schema / Partitioning: `rules/schema-partition-*.md`
- Schema / JSON: `rules/schema-json-when-to-use.md`
- Query / JOINs: `rules/query-join-*.md`
- Query / Indexes: `rules/query-index-skipping-indices.md`
- Query / Materialized Views: `rules/query-mv-*.md`
- Insert / Batching: `rules/insert-batch-size.md`
- Insert / Async: `rules/insert-async-small-batches.md`
- Insert / Format: `rules/insert-format-native.md`
- Insert / Mutations: `rules/insert-mutation-*.md`
- Insert / Optimize: `rules/insert-optimize-avoid-final.md`

FILE:rules/insert-mutation-avoid-update.md
---
title: Avoid ALTER TABLE UPDATE
impact: CRITICAL
impactDescription: "Mutations rewrite entire parts; use ReplacingMergeTree instead"
tags: [insert, mutation, UPDATE, ReplacingMergeTree]
---

## Avoid ALTER TABLE UPDATE

**Impact: CRITICAL**

`ALTER TABLE UPDATE` is a mutation - an asynchronous background process that rewrites entire data parts affected by the change. This is extremely expensive for frequent or large-scale operations.

**Why mutations are problematic:**
- **Write amplification:** Rewrite complete parts even for minor changes
- **Disk I/O spike:** Degrades overall cluster performance
- **No rollback:** Cannot be rolled back after submission
- **Inconsistent reads:** SELECT may read mix of mutated and unmutated parts

**Incorrect (mutation for updates):**
```sql
-- Rewrites potentially huge amounts of data
ALTER TABLE users UPDATE status = 'inactive'
WHERE last_login < now() - INTERVAL 90 DAY;

-- Frequent row updates via mutation
ALTER TABLE inventory UPDATE quantity = quantity - 1
WHERE product_id = 123;
-- If product exists across 100 parts, rewrites ALL 100 parts
```

**Correct (ReplacingMergeTree):**
```sql
-- Table design for updates
CREATE TABLE users (
    user_id UInt64,
    name String,
    status LowCardinality(String),
    updated_at DateTime DEFAULT now()
)
ENGINE = ReplacingMergeTree(updated_at)
ORDER BY user_id;

-- "Update" by inserting new version
INSERT INTO users (user_id, name, status)
VALUES (123, 'John', 'inactive');

-- Query with FINAL to get latest version
SELECT * FROM users FINAL WHERE user_id = 123;

-- Or use aggregation
SELECT user_id, argMax(status, updated_at) as status
FROM users GROUP BY user_id;
```

Reference: [Avoid Mutations](https://clickhouse.com/docs/best-practices/avoid-mutations)

FILE:rules/query-mv-incremental.md
---
title: Use Incremental MVs for Real-Time Aggregations
impact: HIGH
impactDescription: "Read thousands of rows instead of billions"
tags: [query, materialized-view, aggregation, real-time]
---

## Use Incremental MVs for Real-Time Aggregations

**Impact: HIGH**

Materialized views can incrementally maintain aggregations, allowing queries to read pre-computed results instead of scanning raw data.

**Problem:** Query scans billions of rows for every request
```sql
-- Slow: scans all raw events
SELECT
    toStartOfHour(timestamp) as hour,
    event_type,
    count() as events,
    uniq(user_id) as unique_users
FROM events
WHERE timestamp >= now() - INTERVAL 7 DAY
GROUP BY hour, event_type;
```

**Solution: Incremental Materialized View**
```sql
-- Create destination table with AggregatingMergeTree
CREATE TABLE events_hourly (
    hour DateTime,
    event_type LowCardinality(String),
    events AggregateFunction(count, UInt64),
    unique_users AggregateFunction(uniq, UInt64)
)
ENGINE = AggregatingMergeTree()
ORDER BY (event_type, hour);

-- Create MV that updates on each insert
CREATE MATERIALIZED VIEW events_hourly_mv
TO events_hourly
AS SELECT
    toStartOfHour(timestamp) as hour,
    event_type,
    countState() as events,
    uniqState(user_id) as unique_users
FROM events
GROUP BY hour, event_type;

-- Query the MV (reads thousands, not billions)
SELECT
    hour,
    event_type,
    countMerge(events) as events,
    uniqMerge(unique_users) as unique_users
FROM events_hourly
WHERE hour >= now() - INTERVAL 7 DAY
GROUP BY hour, event_type;
```

**Key patterns:**
- Use `-State` functions in MV definition
- Use `-Merge` functions in queries
- AggregatingMergeTree for destination table

Reference: [Using Materialized Views](https://clickhouse.com/docs/best-practices/using-materialized-views)

FILE:rules/schema-types-avoid-nullable.md
---
title: Avoid Nullable Unless Semantically Required
impact: HIGH
impactDescription: "Nullable adds storage overhead; use DEFAULT values instead"
tags: [schema, data-types, Nullable, DEFAULT]
---

## Avoid Nullable Unless Semantically Required

**Impact: HIGH**

Nullable columns maintain a separate UInt8 column for tracking null values, increasing storage and degrading performance. Use DEFAULT values instead when feasible.

**Incorrect (Nullable everywhere):**
```sql
CREATE TABLE users (
    id Nullable(UInt64),           -- IDs should never be null
    name Nullable(String),         -- Empty string is fine
    age Nullable(UInt8),           -- 0 is a valid default
    login_count Nullable(UInt32)   -- 0 is a valid default
)
```

**Correct (DEFAULT values, Nullable only when semantic):**
```sql
CREATE TABLE users (
    id UInt64,                          -- Never null
    name String DEFAULT '',             -- Empty = unknown
    age UInt8 DEFAULT 0,                -- 0 = unknown
    login_count UInt32 DEFAULT 0,       -- 0 = never logged in
    deleted_at Nullable(DateTime),      -- NULL = not deleted (semantic!)
    parent_id Nullable(UInt64)          -- NULL = no parent (semantic!)
)
```

**When Nullable IS appropriate:**

| Use Case | Why |
|----------|-----|
| `deleted_at` | NULL = "not deleted", timestamp = "deleted at X" |
| `parent_id` | NULL = "no parent", value = "has parent" |
| `discount_percent` | NULL = "no discount", 0 = "0% discount" |

**Defaults instead of Nullable:**

| Type | Default |
|------|---------|
| String | `''` (empty string) |
| UInt*/Int* | `0` |
| DateTime | `now()` or `toDateTime(0)` |
| UUID | `generateUUIDv4()` |

Reference: [Select Data Types](https://clickhouse.com/docs/best-practices/select-data-types)

FILE:rules/schema-partition-query-tradeoffs.md
---
title: Understand Partition Query Performance Trade-offs
impact: MEDIUM
impactDescription: "Partition pruning helps some queries, hurts others spanning many partitions"
tags: [schema, partitioning, query, performance]
---

## Understand Partition Query Performance Trade-offs

**Impact: MEDIUM**

Partitioning affects query performance in both directions. Queries matching partition keys benefit from pruning, but queries spanning many partitions may suffer.

**Partition pruning benefits:**
```sql
-- Table partitioned by month
CREATE TABLE events (...)
PARTITION BY toStartOfMonth(timestamp)
ORDER BY (event_type, timestamp);

-- Good: Single partition access
SELECT * FROM events
WHERE timestamp >= '2024-01-01' AND timestamp < '2024-02-01';
-- Only reads January partition

-- Good: Few partitions
SELECT * FROM events
WHERE timestamp >= '2024-01-01' AND timestamp < '2024-04-01';
-- Reads 3 partitions
```

**Partition spanning costs:**
```sql
-- Bad: Spanning all partitions
SELECT count() FROM events
WHERE event_type = 'purchase';
-- Must read from every partition

-- Bad: Aggregation across time
SELECT toStartOfWeek(timestamp), count()
FROM events
WHERE event_type = 'click'
GROUP BY 1;
-- Touches many partitions
```

**Query pattern analysis:**

| Query Type | Partitioned Impact |
|------------|-------------------|
| Date-range filtered | ✅ Faster (pruning) |
| Cross-time aggregation | ⚠️ May be slower |
| Non-partition filtered | ➡️ No change |
| Full table scan | ➡️ No change |

**Recommendations:**
1. Analyze your query patterns before partitioning
2. Prefer ORDER BY optimization over partitioning for queries
3. Use partitioning primarily for lifecycle management
4. Test performance with realistic queries

```sql
-- Check which partitions a query touches
EXPLAIN SELECT * FROM events WHERE ...;
-- Look for "Partition" information in output
```

Reference: [Choosing a Partitioning Key](https://clickhouse.com/docs/best-practices/choosing-a-partitioning-key)

FILE:rules/schema-types-minimize-bitwidth.md
---
title: Minimize Bit-Width for Numeric Types
impact: HIGH
impactDescription: "Use smallest type that fits data range for storage efficiency"
tags: [schema, data-types, numeric, optimization]
---

## Minimize Bit-Width for Numeric Types

**Impact: HIGH**

Using UInt64 for all integers wastes storage. Choose the smallest type that accommodates your data range.

**Type selection guide:**

| Type | Range | Use For |
|------|-------|---------|
| UInt8 | 0-255 | age, rating, status codes |
| UInt16 | 0-65,535 | year, port number |
| UInt32 | 0-4.2B | unix timestamp, most IDs |
| UInt64 | 0-18E | very large counters |
| Int8/16/32/64 | signed versions | when negatives needed |

**Incorrect (oversized types):**
```sql
CREATE TABLE users (
    age UInt64,           -- Max age ~120, UInt64 is overkill
    status UInt64,        -- 5 possible values
    year_joined UInt64,   -- Years 2000-2100
    login_count UInt64    -- Rarely exceeds millions
)
```

**Correct (right-sized types):**
```sql
CREATE TABLE users (
    age UInt8,            -- 0-255 is plenty
    status UInt8,         -- 5 values fit in 1 byte
    year_joined UInt16,   -- 0-65535 covers years
    login_count UInt32    -- 0-4.2B is sufficient
)
```

**Storage impact (1 billion rows):**

| Column | UInt64 | Right-sized | Savings |
|--------|--------|-------------|---------|
| age | 8 GB | 1 GB (UInt8) | 87.5% |
| status | 8 GB | 1 GB (UInt8) | 87.5% |
| year | 8 GB | 2 GB (UInt16) | 75% |

**Note:** Consider future growth, but don't over-provision. UInt32 handles most use cases.

Reference: [Select Data Types](https://clickhouse.com/docs/best-practices/select-data-types)

FILE:rules/schema-types-lowcardinality.md
---
title: Use LowCardinality for Repeated Strings
impact: HIGH
impactDescription: "Dictionary encoding for <10K unique values; significant storage reduction"
tags: [schema, data-types, LowCardinality, storage]
---

## Use LowCardinality for Repeated Strings

**Impact: HIGH**

String columns with repeated values store each value repeatedly. LowCardinality uses dictionary encoding for significant storage reduction.

**Incorrect (plain String for repeated values):**
```sql
CREATE TABLE events (
    country String,      -- "United States" stored 500M times
    browser String,      -- "Chrome" stored 300M times
    event_type String    -- "page_view" stored 800M times
)
```

**Correct (LowCardinality for low unique counts):**
```sql
CREATE TABLE events (
    country LowCardinality(String),     -- ~200 unique values
    browser LowCardinality(String),     -- ~50 unique values
    event_type LowCardinality(String)   -- ~100 unique values
)
```

**When to use LowCardinality:**

| Unique Values | Recommendation |
|---------------|----------------|
| < 10,000 | Use LowCardinality |
| > 10,000 | Use regular String |

```sql
-- Check cardinality before deciding
SELECT uniq(column_name) FROM table_name;
```

**LowCardinality vs FixedString:**

Reserve `FixedString` for strictly fixed-length data (e.g., 2-char country codes). For most low-cardinality text, `LowCardinality(String)` outperforms `FixedString`.

```sql
-- FixedString: Only for truly fixed-length data
country_code FixedString(2),  -- "US", "DE", "JP" - always 2 chars

-- LowCardinality: For variable-length low-cardinality strings
country_name LowCardinality(String),  -- "United States", "Germany"
```

Reference: [Select Data Types](https://clickhouse.com/docs/best-practices/select-data-types)

FILE:rules/query-join-null-handling.md
---
title: Optimize NULL Handling in Outer JOINs
impact: MEDIUM
impactDescription: "join_use_nulls=0 uses default values instead of NULL"
tags: [query, JOIN, NULL, optimization]
---

## Optimize NULL Handling in Outer JOINs

**Impact: MEDIUM**

By default, LEFT/RIGHT/FULL JOINs return NULL for non-matching rows. The `join_use_nulls` setting controls this behavior.

**Default behavior (join_use_nulls = 1):**
```sql
SELECT o.order_id, c.name
FROM orders o
LEFT JOIN customers c ON c.id = o.customer_id;

-- Non-matching rows return:
-- order_id | name
-- 123      | NULL  (customer not found)
```

**With join_use_nulls = 0:**
```sql
SET join_use_nulls = 0;

SELECT o.order_id, c.name
FROM orders o
LEFT JOIN customers c ON c.id = o.customer_id;

-- Non-matching rows return default values:
-- order_id | name
-- 123      | ''    (empty string)
```

**When to use join_use_nulls = 0:**

| Scenario | Setting |
|----------|---------|
| Need to distinguish "not found" from "empty" | 1 (default) |
| Default values are acceptable | 0 (faster) |
| Downstream processing handles NULL poorly | 0 |
| Aggregations on results | 0 (avoids NULL handling) |

**Performance consideration:**
```sql
-- join_use_nulls = 0 is slightly faster
-- No need to track NULL status
-- Simpler result handling
SET join_use_nulls = 0;

SELECT
    o.customer_id,
    sum(o.amount) as total,
    c.country  -- Returns '' instead of NULL
FROM orders o
LEFT JOIN customers c ON c.id = o.customer_id
GROUP BY o.customer_id, c.country;
```

Reference: [Minimize and Optimize JOINs](https://clickhouse.com/docs/best-practices/minimize-optimize-joins)

FILE:rules/schema-pk-prioritize-filters.md
---
title: Prioritize Filter Columns in ORDER BY
impact: CRITICAL
impactDescription: "Columns not in ORDER BY cause full table scans"
tags: [schema, primary-key, filters, ORDER BY]
---

## Prioritize Filter Columns in ORDER BY

**Impact: CRITICAL**

Columns frequently used in WHERE clauses should be included in ORDER BY. Filtering on columns not in ORDER BY results in scanning all granules, negating the benefits of ClickHouse's sparse indexing.

**Incorrect (filter columns not in ORDER BY):**
```sql
CREATE TABLE events (
    timestamp DateTime,
    event_type String,
    user_id UInt64,
    country String
)
ENGINE = MergeTree()
ORDER BY (timestamp);

-- Query filters on event_type - full scan required!
SELECT * FROM events WHERE event_type = 'purchase';
```

**Correct (filter columns in ORDER BY):**
```sql
CREATE TABLE events (
    timestamp DateTime,
    event_type LowCardinality(String),
    user_id UInt64,
    country LowCardinality(String)
)
ENGINE = MergeTree()
ORDER BY (event_type, toDate(timestamp), user_id);

-- Query can use index to skip non-matching granules
SELECT * FROM events WHERE event_type = 'purchase';
```

**Analysis steps:**
1. Identify your most common queries
2. List columns in WHERE clauses
3. Determine cardinality of each column
4. Order: low cardinality filters first, then date, then higher cardinality

**When you can't include all filter columns:**
- Use data skipping indices for secondary filter columns
- Consider materialized views with different ORDER BY

Reference: [Choosing a Primary Key](https://clickhouse.com/docs/best-practices/choosing-a-primary-key)

FILE:rules/query-join-consider-alternatives.md
---
title: Consider Alternatives to JOINs
impact: HIGH
impactDescription: "Dictionaries, denormalization, and IN subqueries often outperform JOINs"
tags: [query, JOIN, dictionary, denormalization]
---

## Consider Alternatives to JOINs

**Impact: HIGH**

JOINs in ClickHouse can be expensive. Consider these alternatives for better performance.

**Alternative 1: Dictionaries for Lookups**
```sql
-- Instead of JOIN for dimension lookup
SELECT o.*, dictGet('customers_dict', 'name', o.customer_id) as customer_name
FROM orders o;

-- Create dictionary
CREATE DICTIONARY customers_dict (
    id UInt64,
    name String
)
PRIMARY KEY id
SOURCE(CLICKHOUSE(TABLE 'customers'))
LAYOUT(FLAT())
LIFETIME(3600);
```

**Alternative 2: Denormalization via Materialized Views**
```sql
-- Pre-join data at insert time
CREATE MATERIALIZED VIEW orders_enriched
ENGINE = MergeTree()
ORDER BY (order_date, customer_id)
AS SELECT
    o.*,
    c.name as customer_name,
    c.country as customer_country
FROM orders o
LEFT JOIN customers c ON c.id = o.customer_id;
```

**Alternative 3: IN Subquery for Filtering**
```sql
-- Instead of JOIN just for filtering
SELECT * FROM orders
WHERE customer_id IN (
    SELECT id FROM customers WHERE country = 'US'
);
```

**When to use each:**

| Alternative | Best For |
|-------------|----------|
| Dictionary | Small dimension tables, frequent lookups |
| Denormalization | High-frequency queries, predictable joins |
| IN subquery | Existence checks, filtering |
| Regular JOIN | Ad-hoc analysis, complex conditions |

Reference: [Minimize and Optimize JOINs](https://clickhouse.com/docs/best-practices/minimize-optimize-joins)

FILE:rules/query-index-skipping-indices.md
---
title: Use Data Skipping Indices for Non-ORDER BY Filters
impact: HIGH
impactDescription: "Up to 60x faster queries by skipping irrelevant granules"
tags: [query, index, skipping, bloom_filter]
---

## Use Data Skipping Indices for Non-ORDER BY Filters

**Impact: HIGH**

Queries filtering on columns not in ORDER BY cannot use the primary index and result in full scans. Data skipping indices store metadata about blocks and skip granules that definitely don't match.

**Important:** Skip indices should be considered **after** optimizing data types, primary key selection, and materialized views.

**When to use:**
- High overall cardinality but low cardinality within blocks
- Rare values critical for search (error codes, specific IDs)
- Column correlates with primary key

**When NOT to use:**
- As a first optimization step
- Matching values scattered across many blocks
- Without testing on real data

**Incorrect (filtering on non-ORDER BY column):**
```sql
CREATE TABLE events (
    event_type LowCardinality(String),
    timestamp DateTime,
    user_id UInt64  -- Not in ORDER BY
)
ENGINE = MergeTree()
ORDER BY (event_type, toDate(timestamp));

-- Query filters on user_id - scans all matching event_type
SELECT * FROM events
WHERE event_type = 'click' AND user_id = 12345;
```

**Correct (add skipping index):**
```sql
CREATE TABLE events (
    event_type LowCardinality(String),
    timestamp DateTime,
    user_id UInt64,
    INDEX idx_user_id user_id TYPE bloom_filter GRANULARITY 4
)
ENGINE = MergeTree()
ORDER BY (event_type, toDate(timestamp));

-- Or add to existing table
ALTER TABLE events ADD INDEX idx_user_id user_id TYPE bloom_filter GRANULARITY 4;
ALTER TABLE events MATERIALIZE INDEX idx_user_id;
```

**Index types:**

| Type | Best For | Example Filter |
|------|----------|----------------|
| `bloom_filter` | Equality on high-cardinality | `WHERE user_id = 123` |
| `set(N)` | Low cardinality (N unique values) | `WHERE status IN ('a','b')` |
| `minmax` | Range queries | `WHERE amount > 1000` |
| `ngrambf_v1` | Text search | `WHERE text LIKE '%term%'` |
| `tokenbf_v1` | Token search | `WHERE hasToken(text, 'word')` |

**Validation:**
```sql
EXPLAIN indexes = 1
SELECT * FROM events WHERE user_id = 12345;
-- Look for "Skip" in output showing granules skipped
```

Reference: [Use Data Skipping Indices Where Appropriate](https://clickhouse.com/docs/best-practices/use-data-skipping-indices-where-appropriate)

FILE:rules/query-join-choose-algorithm.md
---
title: Choose the Right JOIN Algorithm
impact: CRITICAL
impactDescription: "Wrong algorithm causes OOM; right algorithm handles large tables efficiently"
tags: [query, JOIN, algorithm, memory]
---

## Choose the Right JOIN Algorithm

**Impact: CRITICAL**

ClickHouse's default hash join loads the RIGHT table entirely into memory. Choose the right algorithm based on table sizes and constraints.

**Algorithm selection:**

| Algorithm | Best For | Trade-off |
|-----------|----------|-----------|
| `parallel_hash` | Small-to-medium in-memory tables | Default since 24.11; fast, concurrent |
| `hash` | General purpose, all join types | Single-threaded hash table build |
| `direct` | Dictionary lookups (INNER/LEFT only) | Fastest; no hash table construction |
| `full_sorting_merge` | Tables already sorted on join key | Skips sort if pre-ordered; low memory |
| `partial_merge` | Large tables, memory-constrained | Minimized memory; slower execution |
| `grace_hash` | Large datasets, tunable memory | Flexible; disk-spilling capability |
| `auto` | Adaptive algorithm selection | Tries hash first, falls back on memory pressure |

**Example usage:**
```sql
-- Let ClickHouse choose automatically
SET join_algorithm = 'auto';

-- For large-to-large joins where memory is constrained
SET join_algorithm = 'partial_merge';
SELECT * FROM large_a JOIN large_b ON large_b.id = large_a.id;

-- When joining by primary key columns, sort-merge skips sorting step
SET join_algorithm = 'full_sorting_merge';
SELECT * FROM table_a a JOIN table_b b ON b.pk_col = a.pk_col;
```

**Note:** ClickHouse 24.12+ automatically positions smaller tables on the right side. For earlier versions, manually ensure the smaller table is on the RIGHT.

Reference: [Minimize and Optimize JOINs](https://clickhouse.com/docs/best-practices/minimize-optimize-joins)

FILE:rules/query-join-filter-before.md
---
title: Filter Tables Before Joining
impact: CRITICAL
impactDescription: "Reduces data processed in JOIN; prevents memory pressure"
tags: [query, JOIN, filter, optimization]
---

## Filter Tables Before Joining

**Impact: CRITICAL**

Apply filters to tables before joining to reduce the data size processed. This prevents memory pressure and speeds up queries significantly.

**Incorrect (filter after join):**
```sql
-- Joins all data, then filters
SELECT *
FROM orders o
JOIN customers c ON c.id = o.customer_id
WHERE o.status = 'completed' AND c.country = 'US';
```

**Correct (filter before join with subqueries):**
```sql
-- Filter each table before joining
SELECT *
FROM (
    SELECT * FROM orders WHERE status = 'completed'
) o
JOIN (
    SELECT * FROM customers WHERE country = 'US'
) c ON c.id = o.customer_id;
```

**Even better (aggregate before join when possible):**
```sql
-- Aggregate first, then join smaller result
SELECT c.country, o.total_orders
FROM (
    SELECT customer_id, count() as total_orders
    FROM orders
    WHERE status = 'completed'
    GROUP BY customer_id
) o
JOIN customers c ON c.id = o.customer_id;
```

**Impact on memory:**

| Scenario | Right Table Size |
|----------|------------------|
| No pre-filter | Full table in memory |
| Pre-filtered | Only matching rows |
| Pre-aggregated | Much smaller result set |

Reference: [Minimize and Optimize JOINs](https://clickhouse.com/docs/best-practices/minimize-optimize-joins)

FILE:rules/schema-pk-filter-on-orderby.md
---
title: Filter on ORDER BY Columns in Queries
impact: CRITICAL
impactDescription: "Skipping prefix columns prevents index usage"
tags: [schema, primary-key, query, filters]
---

## Filter on ORDER BY Columns in Queries

**Impact: CRITICAL**

The primary index only helps when queries filter on the ORDER BY columns in order. Skipping the leading columns prevents the index from being used effectively.

**Example ORDER BY:**
```sql
ORDER BY (event_type, event_date, user_id)
```

**Index usage by query pattern:**

| Query Filter | Index Used? | Why |
|--------------|-------------|-----|
| `WHERE event_type = 'X'` | ✅ Yes | Matches first column |
| `WHERE event_type = 'X' AND event_date = '2024-01-01'` | ✅ Yes | Matches prefix |
| `WHERE event_date = '2024-01-01'` | ❌ No | Skips first column |
| `WHERE user_id = 123` | ❌ No | Skips first two columns |

**Incorrect (skipping ORDER BY prefix):**
```sql
-- ORDER BY (event_type, event_date, user_id)

-- Skips event_type - can't use index efficiently
SELECT * FROM events WHERE event_date = '2024-01-01';

-- Skips event_type and event_date - full scan
SELECT * FROM events WHERE user_id = 123;
```

**Correct (using ORDER BY prefix):**
```sql
-- Always include leading columns when possible
SELECT * FROM events
WHERE event_type = 'click' AND event_date = '2024-01-01';

-- If you must filter on later columns, add skipping index
ALTER TABLE events
ADD INDEX idx_user_id user_id TYPE bloom_filter GRANULARITY 4;
```

Reference: [Choosing a Primary Key](https://clickhouse.com/docs/best-practices/choosing-a-primary-key)

FILE:rules/insert-async-small-batches.md
---
title: Use Async Inserts for High-Frequency Small Batches
impact: HIGH
impactDescription: "Server-side buffering when client batching isn't practical"
tags: [insert, async, buffering, high-frequency]
---

## Use Async Inserts for High-Frequency Small Batches

**Impact: HIGH**

When client-side batching isn't practical (many independent producers, real-time requirements), use async inserts to let ClickHouse buffer and batch server-side.

**Problem scenario:**
- Many microservices sending events
- Real-time data from IoT devices
- Can't coordinate batching across producers

**Incorrect (small sync inserts from many sources):**
```python
# Each producer sends immediately - creates many small parts
client.execute("INSERT INTO events VALUES", [single_event])
```

**Correct (async inserts):**
```python
# Enable async inserts
client.execute("SET async_insert = 1")
client.execute("SET wait_for_async_insert = 0")  # Don't wait for batch

# Inserts are buffered server-side
client.execute("INSERT INTO events VALUES", [single_event])
```

**Server configuration:**
```sql
-- Configure async insert behavior
SET async_insert = 1;
SET async_insert_max_data_size = 10000000;  -- 10MB buffer
SET async_insert_busy_timeout_ms = 1000;     -- Flush every 1s
SET wait_for_async_insert = 1;               -- Wait for durability
```

**Async insert settings:**

| Setting | Description | Recommended |
|---------|-------------|-------------|
| `async_insert` | Enable async mode | 1 |
| `async_insert_max_data_size` | Buffer size before flush | 10-100MB |
| `async_insert_busy_timeout_ms` | Max wait before flush | 1000-5000ms |
| `wait_for_async_insert` | Wait for flush acknowledgment | 1 for durability |

**When to use:**

| Scenario | Async? |
|----------|--------|
| Many small producers | ✅ Yes |
| Single large batch producer | ❌ No, use sync |
| Real-time requirements | ✅ Yes |
| Can batch client-side | ❌ No, batch yourself |

Reference: [Selecting an Insert Strategy](https://clickhouse.com/docs/best-practices/selecting-an-insert-strategy)

FILE:rules/insert-format-native.md
---
title: Use Native Format for Best Insert Performance
impact: MEDIUM
impactDescription: "Native > RowBinary > JSONEachRow for performance"
tags: [insert, format, Native, performance]
---

## Use Native Format for Best Insert Performance

**Impact: MEDIUM**

Different data formats have different parsing overhead. Native format provides the best performance, followed by RowBinary, then text formats like JSONEachRow.

**Format performance ranking:**

| Format | Performance | Use Case |
|--------|-------------|----------|
| Native | Fastest | ClickHouse-to-ClickHouse |
| RowBinary | Very fast | Binary protocols |
| JSONEachRow | Moderate | APIs, human-readable |
| CSV/TSV | Moderate | File imports |
| JSONCompactEachRow | Slower | Compact JSON |

**Using Native format:**
```python
# Python clickhouse-driver uses Native by default
from clickhouse_driver import Client
client = Client('localhost')
client.execute('INSERT INTO events VALUES', data)  # Native format
```

**Using RowBinary:**
```sql
-- For HTTP interface
INSERT INTO events FORMAT RowBinary
-- Binary data follows
```

**JSONEachRow (common but slower):**
```sql
INSERT INTO events FORMAT JSONEachRow
{"event_type": "click", "timestamp": "2024-01-15 10:30:00", "user_id": 123}
{"event_type": "view", "timestamp": "2024-01-15 10:31:00", "user_id": 456}
```

**Performance tips:**
1. Use Native format when possible (native clients)
2. Compress data in transit (LZ4 or ZSTD)
3. Avoid frequent format conversion
4. For HTTP API, consider RowBinary over JSON

```sql
-- Enable compression
SET network_compression_method = 'ZSTD';
SET network_zstd_compression_level = 3;
```

Reference: [Selecting an Insert Strategy](https://clickhouse.com/docs/best-practices/selecting-an-insert-strategy)

FILE:rules/query-mv-refreshable.md
---
title: Use Refreshable MVs for Complex Joins and Batch Workflows
impact: HIGH
impactDescription: "Sub-millisecond queries with periodic refresh"
tags: [query, materialized-view, refresh, joins]
---

## Use Refreshable MVs for Complex Joins and Batch Workflows

**Impact: HIGH**

Refreshable MVs periodically rebuild their contents, making them ideal for complex queries with JOINs or transformations that can't be done incrementally.

**When to use:**
- Complex JOINs across multiple tables
- Data transformations too complex for incremental MVs
- Batch workflows with periodic updates
- Pre-computed dashboards and reports

**Example: Complex JOIN refreshed hourly**
```sql
CREATE MATERIALIZED VIEW customer_orders_summary
REFRESH EVERY 1 HOUR
ENGINE = MergeTree()
ORDER BY customer_id
AS SELECT
    c.customer_id,
    c.name,
    c.country,
    count() as total_orders,
    sum(o.amount) as total_spent,
    max(o.order_date) as last_order
FROM customers c
LEFT JOIN orders o ON o.customer_id = c.customer_id
GROUP BY c.customer_id, c.name, c.country;

-- Queries now sub-millisecond
SELECT * FROM customer_orders_summary
WHERE country = 'US' AND total_spent > 1000;
```

**Refresh modes:**

| Mode | Behavior |
|------|----------|
| REPLACE (default) | Atomically replaces all data |
| APPEND | Adds to existing data |

**Refresh scheduling:**
```sql
-- Every hour
REFRESH EVERY 1 HOUR

-- At specific times
REFRESH EVERY 1 DAY AT '03:00'

-- After another MV refreshes
REFRESH AFTER other_mv
```

**Manual refresh:**
```sql
-- Force immediate refresh
SYSTEM REFRESH VIEW customer_orders_summary;

-- Check refresh status
SELECT * FROM system.view_refreshes WHERE view = 'customer_orders_summary';
```

Reference: [Using Materialized Views](https://clickhouse.com/docs/best-practices/using-materialized-views)

FILE:rules/schema-pk-plan-before-creation.md
---
title: Plan PRIMARY KEY Before Table Creation
impact: CRITICAL
impactDescription: "ORDER BY is immutable after creation; wrong choice requires full data migration"
tags: [schema, primary-key, ORDER BY, planning]
---

## Plan PRIMARY KEY Before Table Creation

**Impact: CRITICAL**

The ORDER BY clause (which defines the primary key in MergeTree tables) cannot be changed after table creation. A wrong choice requires creating a new table and migrating all data.

**Why this matters:**
- ORDER BY defines how data is physically sorted on disk
- The primary index is built from ORDER BY columns
- Query performance depends heavily on ORDER BY alignment with filter patterns
- Changing ORDER BY requires full table recreation

**Questions to answer before creating:**
1. What columns will be in WHERE clauses?
2. What is the cardinality of each filter column?
3. Will queries filter on date ranges?
4. Are there mandatory filter columns (tenant_id, etc.)?

**Incorrect (no planning):**
```sql
-- Created without analyzing query patterns
CREATE TABLE events (
    id UUID,
    timestamp DateTime,
    event_type String,
    user_id UInt64
)
ENGINE = MergeTree()
ORDER BY (id);  -- Wrong! UUID has no query benefit
```

**Correct (planned based on queries):**
```sql
-- Analyzed: most queries filter by event_type, then date range
CREATE TABLE events (
    id UUID,
    timestamp DateTime,
    event_type LowCardinality(String),
    user_id UInt64
)
ENGINE = MergeTree()
ORDER BY (event_type, toDate(timestamp), user_id);
```

**Migration cost when ORDER BY is wrong:**
```sql
-- Must create new table and copy all data
CREATE TABLE events_new (...) ORDER BY (correct_columns);
INSERT INTO events_new SELECT * FROM events;
RENAME TABLE events TO events_old, events_new TO events;
DROP TABLE events_old;
```

Reference: [Choosing a Primary Key](https://clickhouse.com/docs/best-practices/choosing-a-primary-key)

FILE:rules/schema-types-enum.md
---
title: Use Enum for Finite Value Sets
impact: MEDIUM
impactDescription: "Insert-time validation, natural ordering, 1-2 bytes storage"
tags: [schema, data-types, Enum, validation]
---

## Use Enum for Finite Value Sets

**Impact: MEDIUM**

Enum provides insert-time validation, natural ordering, and efficient 1-2 byte storage for columns with a fixed set of values.

**Incorrect (String for finite values):**
```sql
CREATE TABLE orders (
    status String  -- "pending", "processing", "shipped", "delivered"
)
-- No validation: INSERT VALUES ('pendingg') succeeds with typo
```

**Correct (Enum for validation and efficiency):**
```sql
CREATE TABLE orders (
    status Enum8('pending' = 1, 'processing' = 2, 'shipped' = 3, 'delivered' = 4)
)
-- Validation: INSERT VALUES ('pendingg') fails immediately
```

**Enum types:**

| Type | Range | Use For |
|------|-------|---------|
| Enum8 | 1-127 values | Most use cases |
| Enum16 | 1-32767 values | Large value sets |

**Benefits:**
- **Validation**: Invalid values rejected at insert
- **Storage**: 1 byte (Enum8) vs variable String
- **Ordering**: Natural sort by enum position
- **Performance**: Integer comparisons vs string

**When to use Enum vs LowCardinality:**

| Scenario | Recommendation |
|----------|----------------|
| Fixed, known values | Enum |
| Values may change | LowCardinality |
| Need validation | Enum |
| Flexible schema | LowCardinality |

Reference: [Select Data Types](https://clickhouse.com/docs/best-practices/select-data-types)

FILE:rules/insert-mutation-avoid-delete.md
---
title: Avoid ALTER TABLE DELETE
impact: CRITICAL
impactDescription: "Use lightweight DELETE, CollapsingMergeTree, or DROP PARTITION"
tags: [insert, mutation, DELETE, CollapsingMergeTree]
---

## Avoid ALTER TABLE DELETE

**Impact: CRITICAL**

`ALTER TABLE DELETE` is a mutation that rewrites data parts. For frequent deletes, use alternatives that don't require rewriting data.

**Incorrect (mutation for deletes):**
```sql
-- Rewrites parts containing matching rows
ALTER TABLE events DELETE WHERE timestamp < '2023-01-01';

-- Frequent single-row deletes
ALTER TABLE orders DELETE WHERE order_id = 12345;
```

**Alternative 1: Lightweight DELETE (23.3+)**
```sql
-- Marks rows as deleted without immediate rewrite
DELETE FROM events WHERE timestamp < '2023-01-01';

-- Rows cleaned up during normal merges
-- Faster than mutation, less I/O impact
```

**Alternative 2: DROP PARTITION**
```sql
-- Instant deletion of entire partitions
ALTER TABLE events DROP PARTITION '2023-01';

-- Requires proper partitioning strategy
```

**Alternative 3: CollapsingMergeTree**
```sql
CREATE TABLE orders (
    order_id UInt64,
    status String,
    amount Decimal(10,2),
    sign Int8  -- 1 for insert, -1 for delete
)
ENGINE = CollapsingMergeTree(sign)
ORDER BY order_id;

-- "Delete" by inserting with sign = -1
INSERT INTO orders VALUES (12345, 'cancelled', 99.99, -1);
```

**Comparison:**

| Method | Speed | I/O Impact | When to Use |
|--------|-------|------------|-------------|
| ALTER DELETE | Slow | High | Rare, bulk operations |
| Lightweight DELETE | Fast | Low | Frequent deletes |
| DROP PARTITION | Instant | None | Time-based retention |
| CollapsingMergeTree | Fast | None | Frequent record deletes |

Reference: [Avoid Mutations](https://clickhouse.com/docs/best-practices/avoid-mutations)

FILE:rules/insert-batch-size.md
---
title: Batch Inserts Appropriately (10K-100K rows)
impact: CRITICAL
impactDescription: "Each INSERT creates a part; single-row inserts overwhelm merge process"
tags: [insert, batching, parts, performance]
---

## Batch Inserts Appropriately (10K-100K rows)

**Impact: CRITICAL**

Each INSERT creates a new data part. Single-row or small-batch inserts create thousands of tiny parts, overwhelming the merge process and causing cluster instability.

**Incorrect (single-row or tiny batches):**
```python
# Single-row inserts - creates 10,000 parts!
for event in events:
    client.execute("INSERT INTO events VALUES", [event])

# Tiny batches - still too many parts
for batch in chunks(events, 100):  # 100 rows per INSERT
    client.execute("INSERT INTO events VALUES", batch)
```

**Correct (proper batch size):**
```python
# Ideal batch size: 10,000-100,000 rows
BATCH_SIZE = 10_000
for batch in chunks(events, BATCH_SIZE):
    client.execute("INSERT INTO events VALUES", batch)
```

**Recommended batch sizes:**

| Threshold | Value |
|-----------|-------|
| Minimum | 1,000 rows |
| Ideal range | 10,000-100,000 rows |
| Insert rate (sync) | ~1 insert per second |

**Validation:**
```sql
-- Monitor part count (>3000 per partition blocks inserts)
SELECT table, count() as parts, sum(rows) as total_rows
FROM system.parts
WHERE active AND database = 'default'
GROUP BY table
ORDER BY parts DESC;
```

Reference: [Selecting an Insert Strategy](https://clickhouse.com/docs/best-practices/selecting-an-insert-strategy)

FILE:rules/insert-optimize-avoid-final.md
---
title: Avoid OPTIMIZE TABLE FINAL
impact: HIGH
impactDescription: "Let background merges work; FINAL in SELECT for deduplication"
tags: [insert, OPTIMIZE, merge, ReplacingMergeTree]
---

## Avoid OPTIMIZE TABLE FINAL

**Impact: HIGH**

`OPTIMIZE TABLE FINAL` forces immediate merging of all parts into one, which is resource-intensive and blocks other operations. Let background merges work naturally instead.

**Incorrect (forcing merges):**
```sql
-- Expensive! Merges all parts immediately
OPTIMIZE TABLE events FINAL;

-- Running after every batch - unnecessary
INSERT INTO events VALUES (...);
OPTIMIZE TABLE events FINAL;  -- Don't do this!
```

**Correct (let background merges work):**
```sql
-- Insert data normally
INSERT INTO events VALUES (...);

-- Background merge process handles optimization
-- ClickHouse automatically merges parts over time
```

**For ReplacingMergeTree deduplication:**

Instead of OPTIMIZE FINAL, use FINAL modifier in queries:

```sql
-- Table with ReplacingMergeTree
CREATE TABLE users (
    user_id UInt64,
    name String,
    updated_at DateTime
)
ENGINE = ReplacingMergeTree(updated_at)
ORDER BY user_id;

-- Query with FINAL to get deduplicated results
SELECT * FROM users FINAL WHERE user_id = 123;

-- Or use aggregation (often faster)
SELECT user_id, argMax(name, updated_at) as name
FROM users
GROUP BY user_id;
```

**When OPTIMIZE might be acceptable:**

| Scenario | Acceptable? |
|----------|-------------|
| One-time data migration | ⚠️ Maybe, off-peak |
| After bulk historical load | ⚠️ Maybe, off-peak |
| Regular production use | ❌ No |
| After every INSERT | ❌ Never |

Reference: [Avoid Mutations](https://clickhouse.com/docs/best-practices/avoid-mutations)

FILE:rules/schema-partition-start-without.md
---
title: Consider Starting Without Partitioning
impact: MEDIUM
impactDescription: "Add partitioning later when clear lifecycle requirements exist"
tags: [schema, partitioning, design]
---

## Consider Starting Without Partitioning

**Impact: MEDIUM**

Partitioning adds complexity and can hurt query performance in some cases. Start without partitioning and add it later when you have clear lifecycle management requirements.

**Why start without partitioning:**
- Simpler schema management
- No partition key selection mistakes
- Fewer "too many parts" issues
- Query performance depends on ORDER BY, not partitions

**When to add partitioning later:**
1. Data volume grows large enough for lifecycle management
2. Need to DROP old data efficiently
3. Tiered storage requirements emerge
4. Clear retention policy established

**Starting simple:**
```sql
-- Start without partitioning
CREATE TABLE events (
    event_id UUID,
    event_type LowCardinality(String),
    timestamp DateTime,
    user_id UInt64,
    data String
)
ENGINE = MergeTree()
ORDER BY (event_type, toDate(timestamp), user_id);
-- No PARTITION BY clause
```

**Adding partitioning later:**
```sql
-- When lifecycle needs are clear, create new partitioned table
CREATE TABLE events_new (
    event_id UUID,
    event_type LowCardinality(String),
    timestamp DateTime,
    user_id UInt64,
    data String
)
ENGINE = MergeTree()
PARTITION BY toStartOfMonth(timestamp)
ORDER BY (event_type, toDate(timestamp), user_id);

-- Migrate data
INSERT INTO events_new SELECT * FROM events;

-- Swap tables
RENAME TABLE events TO events_old, events_new TO events;
```

**Decision checklist:**
- [ ] Data volume > 100GB?
- [ ] Clear retention policy (e.g., keep 90 days)?
- [ ] Need tiered storage?
- [ ] Frequent DROP PARTITION operations expected?

If all "No", start without partitioning.

Reference: [Choosing a Partitioning Key](https://clickhouse.com/docs/best-practices/choosing-a-partitioning-key)

FILE:rules/schema-partition-lifecycle.md
---
title: Use Partitioning for Data Lifecycle Management
impact: HIGH
impactDescription: "DROP PARTITION is instant; DELETE is expensive"
tags: [schema, partitioning, lifecycle, TTL]
---

## Use Partitioning for Data Lifecycle Management

**Impact: HIGH**

Partitioning's primary benefit is efficient data lifecycle management: instant partition drops, TTL-based retention, and tiered storage. Use it for lifecycle, not query optimization.

**Incorrect (partitioning for queries):**
```sql
-- Partitioning by user_id hoping to speed up user queries
CREATE TABLE events (...)
PARTITION BY user_id  -- Wrong approach!
ORDER BY (timestamp);
```

**Correct (partitioning for lifecycle):**
```sql
CREATE TABLE events (
    timestamp DateTime,
    event_type LowCardinality(String),
    data String
)
ENGINE = MergeTree()
PARTITION BY toStartOfMonth(timestamp)
ORDER BY (event_type, timestamp)
TTL timestamp + INTERVAL 90 DAY;

-- Instant deletion of old data
ALTER TABLE events DROP PARTITION '2023-01';

-- TTL automatically removes expired data
```

**Lifecycle use cases:**

| Use Case | Partition Key | Benefit |
|----------|---------------|---------|
| Retention | toStartOfMonth(ts) | DROP PARTITION for cleanup |
| Archival | toStartOfYear(ts) | Move old partitions to cold storage |
| Compliance | toStartOfMonth(ts) | Delete specific time ranges |

**Tiered storage example:**
```sql
CREATE TABLE events (...)
ENGINE = MergeTree()
PARTITION BY toStartOfMonth(timestamp)
ORDER BY (event_type, timestamp)
TTL
    timestamp + INTERVAL 7 DAY TO VOLUME 'hot',
    timestamp + INTERVAL 30 DAY TO VOLUME 'warm',
    timestamp + INTERVAL 365 DAY DELETE;
```

Reference: [Choosing a Partitioning Key](https://clickhouse.com/docs/best-practices/choosing-a-partitioning-key)

FILE:rules/schema-json-when-to-use.md
---
title: Use JSON Type for Dynamic Schemas
impact: MEDIUM
impactDescription: "JSON for dynamic schemas; typed columns for known fields"
tags: [schema, JSON, dynamic, semi-structured]
---

## Use JSON Type for Dynamic Schemas

**Impact: MEDIUM**

The JSON type enables efficient storage and querying of semi-structured data with dynamic schemas. Use it when field structure varies or is unknown at design time.

**When to use JSON:**
- Event properties that vary by event type
- User attributes from different sources
- API responses with varying structure
- Log data with inconsistent fields

**When NOT to use JSON:**
- Known, stable schema → use typed columns
- High-frequency filtering → use typed columns in ORDER BY
- Aggregations on specific fields → extract to typed columns

**Example: Events with variable properties**
```sql
CREATE TABLE events (
    event_id UUID,
    event_type LowCardinality(String),
    timestamp DateTime,
    user_id UInt64,
    properties JSON  -- Dynamic fields per event type
)
ENGINE = MergeTree()
ORDER BY (event_type, timestamp);

-- Insert with different property structures
INSERT INTO events VALUES
    (generateUUIDv4(), 'page_view', now(), 123,
     '{"url": "/home", "referrer": "google.com", "duration_ms": 5000}'),
    (generateUUIDv4(), 'purchase', now(), 456,
     '{"product_id": 789, "amount": 99.99, "currency": "USD"}');
```

**Querying JSON:**
```sql
-- Access nested fields
SELECT
    event_type,
    properties.url as url,
    properties.amount as amount
FROM events
WHERE properties.currency = 'USD';

-- Type inference works automatically
SELECT
    sum(properties.amount::Float64) as total_revenue
FROM events
WHERE event_type = 'purchase';
```

**Hybrid approach (recommended):**
```sql
-- Known fields as typed columns, unknown as JSON
CREATE TABLE events (
    event_id UUID,
    event_type LowCardinality(String),
    timestamp DateTime,
    user_id UInt64,                    -- Known, frequently filtered
    properties JSON                     -- Variable properties
)
ENGINE = MergeTree()
ORDER BY (event_type, timestamp, user_id);
```

Reference: [Select Data Types](https://clickhouse.com/docs/best-practices/select-data-types)

FILE:rules/schema-pk-cardinality-order.md
---
title: Order Columns by Cardinality (Low to High)
impact: CRITICAL
impactDescription: "Enables granule skipping; high-cardinality first prevents index pruning"
tags: [schema, primary-key, cardinality, ORDER BY]
---

## Order Columns by Cardinality (Low to High)

**Impact: CRITICAL**

Since the sparse primary index operates on data blocks (granules) rather than individual rows, low-cardinality leading columns create more useful index entries that can skip entire blocks. Place lower-cardinality columns before higher-cardinality ones in the ordering key.

**Incorrect (high cardinality first):**
```sql
-- UUID first means no pruning benefit
CREATE TABLE events (...)
ENGINE = MergeTree()
ORDER BY (event_id, event_type, timestamp);
-- Every granule has different event_id values, index can't skip anything
```

**Correct (low cardinality first):**
```sql
-- Low cardinality first enables pruning
CREATE TABLE events (...)
ENGINE = MergeTree()
ORDER BY (event_type, event_date, event_id);
-- Index can skip entire event_type groups
```

**Column Order Guidelines:**

| Position | Cardinality | Examples |
|----------|-------------|----------|
| 1st | Low (few distinct values) | event_type, status, country |
| 2nd | Date (coarse granularity) | toDate(timestamp) |
| 3rd+ | Medium-High | user_id, session_id |
| Last | High (if needed) | event_id, uuid |

**Tip:** Use `toDate(timestamp)` instead of raw `DateTime` columns when day-level filtering suffices - this reduces index size from 32-bit to 16-bit representations.

Reference: [Choosing a Primary Key](https://clickhouse.com/docs/best-practices/choosing-a-primary-key)

FILE:rules/query-join-use-any.md
---
title: Use ANY JOIN When Only One Match Needed
impact: HIGH
impactDescription: "Less memory, faster execution when duplicates don't matter"
tags: [query, JOIN, ANY, optimization]
---

## Use ANY JOIN When Only One Match Needed

**Impact: HIGH**

When you only need one matching row from the right table (not all matches), use `ANY JOIN` instead of regular `JOIN` for better performance.

**Incorrect (regular JOIN when one match suffices):**
```sql
-- Returns all matching rows, even if we only need one
SELECT o.*, c.name
FROM orders o
JOIN customers c ON c.id = o.customer_id;
```

**Correct (ANY JOIN for single match):**
```sql
-- Returns one arbitrary matching row - faster, less memory
SELECT o.*, c.name
FROM orders o
ANY LEFT JOIN customers c ON c.id = o.customer_id;
```

**When to use ANY JOIN:**

| Scenario | Use ANY? |
|----------|----------|
| Lookup from unique key | ✅ Yes |
| Need all matching rows | ❌ No |
| Right table has duplicates but you want one | ✅ Yes |
| Need specific match (latest, first) | ❌ No, use ASOF or subquery |

**ANY vs regular JOIN:**

| Aspect | Regular JOIN | ANY JOIN |
|--------|--------------|----------|
| Multiple matches | Returns all | Returns one |
| Memory usage | Higher | Lower |
| Execution speed | Slower | Faster |
| Result determinism | Deterministic | Non-deterministic |

**Note:** `ANY` returns an arbitrary matching row. If you need a specific one (e.g., latest), use `ASOF JOIN` or a subquery with ordering.

Reference: [Minimize and Optimize JOINs](https://clickhouse.com/docs/best-practices/minimize-optimize-joins)

FILE:rules/schema-partition-low-cardinality.md
---
title: Keep Partition Cardinality Low (100-1,000 Values)
impact: HIGH
impactDescription: "Too many partitions cause part explosion and 'too many parts' errors"
tags: [schema, partitioning, parts]
---

## Keep Partition Cardinality Low (100-1,000 Values)

**Impact: HIGH**

Too many distinct partition values create excessive data parts, eventually triggering "too many parts" errors. ClickHouse enforces limits via `max_parts_in_total` and `parts_to_throw_insert` settings.

**Incorrect (high cardinality partitioning):**
```sql
-- High cardinality = too many partitions
CREATE TABLE events (...)
ENGINE = MergeTree()
PARTITION BY user_id  -- Millions of partitions!
ORDER BY (timestamp);

-- Daily partitions can grow unbounded over years
CREATE TABLE logs (...)
ENGINE = MergeTree()
PARTITION BY toDate(timestamp)  -- 3650 partitions over 10 years
ORDER BY (service, timestamp);
```

**Correct (bounded cardinality):**
```sql
-- Monthly partitions = 12 per year, bounded cardinality
CREATE TABLE events (
    timestamp DateTime,
    event_type LowCardinality(String),
    user_id UInt64
)
ENGINE = MergeTree()
PARTITION BY toStartOfMonth(timestamp)
ORDER BY (event_type, timestamp);
```

**Validation:**
```sql
-- Check partition count and health
SELECT
    partition,
    count() as parts,
    sum(rows) as rows,
    formatReadableSize(sum(bytes_on_disk)) as size
FROM system.parts
WHERE table = 'events' AND active
GROUP BY partition
ORDER BY partition;

-- Warning signs: hundreds or thousands of partitions
```

Reference: [Choosing a Partitioning Key](https://clickhouse.com/docs/best-practices/choosing-a-partitioning-key)

FILE:rules/schema-types-native-types.md
---
title: Use Native Types Instead of String
impact: CRITICAL
impactDescription: "2-10x storage reduction; enables compression and correct semantics"
tags: [schema, data-types, storage, performance]
---

## Use Native Types Instead of String

**Impact: CRITICAL**

Storing everything as String is a common anti-pattern. Native types provide 2-10x storage reduction, better compression, and proper semantics (sorting, arithmetic, etc.).

**Incorrect (String for everything):**
```sql
CREATE TABLE events (
    id String,           -- UUID stored as 36-char string
    timestamp String,    -- DateTime as "2024-01-15 10:30:00"
    user_id String,      -- Integer as "12345678"
    amount String,       -- Decimal as "99.99"
    is_active String     -- Boolean as "true"/"false"
)
```

**Correct (native types):**
```sql
CREATE TABLE events (
    id UUID,                    -- 16 bytes vs 36 bytes
    timestamp DateTime64(3),    -- 8 bytes vs 19 bytes
    user_id UInt64,             -- 8 bytes vs variable
    amount Decimal(10,2),       -- Fixed precision
    is_active Bool              -- 1 byte vs 4-5 bytes
)
```

**Storage comparison:**

| Data | String Size | Native Size | Savings |
|------|-------------|-------------|---------|
| UUID | 36 bytes | 16 bytes | 56% |
| DateTime | 19 bytes | 4-8 bytes | 58-79% |
| Integer | 1-20 bytes | 1-8 bytes | varies |
| Boolean | 4-5 bytes | 1 byte | 75-80% |
| IPv4 | 7-15 bytes | 4 bytes | 43-73% |

**Additional benefits:**
- Proper sorting (numeric vs lexicographic)
- Arithmetic operations work correctly
- Better compression ratios
- Type validation on insert

Reference: [Select Data Types](https://clickhouse.com/docs/best-practices/select-data-types)

---
name: build_development_team
description: >
  Builds and configures a complete AI software development team inside OpenClaw,
  including the full project folder structure, agent coordination workflow, and
  vision-capable model handoffs for mockup-driven UI work. Use this skill whenever
  someone says "build a software development team", "set up my dev team",
  "configure the agent team", "set up a project", "add a project to my team",
  "how do agents work together on a project", "set up project coordination",
  "create a project folder", "wire up Asana for my team", "add mockup support
  to my dev team", or anything similar — even if they don't use the word "skill"
  or explicitly ask for setup. Also trigger when someone is troubleshooting a
  multi-agent dev setup, asking how PM/engineer/QA/devs should coordinate, or
  asking about sprint workflow, branch conventions, queue files, or project-lock
  state. This skill covers the full setup: safety snapshot, agent creation, model
  selection (with hallucination-aware picks and per-agent vision configuration),
  skills installation, Asana project setup, GitHub repo wiring, agent-to-agent
  communication, mockup storage convention (Asana attachments primary, project
  workspace fallback), and the complete project folder structure that coordinates
  agents across sprints — including PROJECT.md, project.json, queue files, shared
  workspace, spec versioning, decision logging, and sprint open/close procedures.
  Requires the openclaw-administrator skill (EncryptShawn) to be loaded.
  Recommends openclaw-recovery-manager (EncryptShawn) for safety.
  This skill does not make Asana or GitHub API calls itself — those are delegated
  to separately installed Asana and Git dependency skills.
  This skill does not read or store any credentials or secret values.
---

# Build a Software Development Team

This skill sets up a complete AI-powered software development team inside your OpenClaw instance. It covers two layers:

1. **Agent setup** — creating and configuring each agent (models, skills, routing, workspace files)
2. **Project setup** — creating the project folder structure that coordinates agents across sprints

Both layers are required for a fully working team. Use the table of contents below to jump to the relevant section.

---

## Contents

- [Credential and Security Model](#credential-and-security-model)
- [Heartbeat Scheduling](#heartbeat-scheduling)
- [Agent Workspace Files](#agent-workspace-files)
- [Before You Start](#before-you-start--required-skills)
- [Step 0 — Safety First](#step-0--safety-first)
- [Step 1 — Gather Project Information](#step-1--gather-project-information)
- [Step 2 — Model Selection](#step-2--model-selection)
- [Step 3 — Skills Selection Per Agent](#step-3--skills-selection-per-agent)
- [Step 4 — Agent Configuration](#step-4--agent-configuration)
- [Step 5 — Asana Project Setup](#step-5--asana-project-setup)
- [Step 6 — Write Agent Workspace Files](#step-6--write-agent-workspace-files)
- [Step 7 — Create Project Folder Structure](#step-7--create-project-folder-structure)
- [Step 8 — Credential Verification](#step-8--credential-verification)
- [Step 9 — Enable Chat Completions](#step-9--enable-chat-completions)
- [Step 10 — Smoke Test](#step-10--smoke-test)
- [Step 11 — Post-Setup Snapshot](#step-11--post-setup-snapshot)
- [Step 12 — Handoff Summary](#step-12--handoff-summary)
- [Project Workflow Reference](#project-workflow-reference)
- [Sprint Management](#sprint-management)
- [If Anything Goes Wrong](#if-anything-goes-wrong)

Reference files (read when needed):
- `references/project-files.md` — Full specification of every project folder file, its format, and its content rules
- `references/workflow.md` — Complete agent workflow: all phases, escalation rules, queue formats, git conventions

---

## Credential and Security Model

**This skill does not read, store, request, or transmit any credentials or secret values.**

All credential handling is performed by:
- The **openclaw-administrator** skill, which uses OpenClaw's own config management to wire agent settings
- The **Asana dependency skill** installed on each agent, which holds and uses the Asana PAT
- The **Git dependency skill** installed on each agent, which holds and uses the GitHub PAT
- The **Email dependency skill** installed on the PM agent, which holds and uses email credentials

This skill collects only the *names* of env vars (e.g., `TA_ASANA_PAT`) — never their values. Those env var names are passed to the dependency skills so they know which credential to pull from the agent runtime environment.

Credentials must be stored in your secret management system (Kubernetes ConfigMap/Secret, .env file, or equivalent) before setup begins.

---

## Heartbeat Scheduling

The 30-minute agent heartbeats configured in this setup are scheduled and triggered by the **OpenClaw platform**, not by this skill. This skill instructs the openclaw-administrator skill to set the heartbeat interval in each agent's OpenClaw config. This skill does not self-invoke, set timers, or persist between sessions.

---

## Agent Workspace Files

This setup process creates and populates the following operator-managed workspace files for each agent:
- **USER.md** — project list, Asana GIDs, repo URLs, team roster. No secret values.
- **TOOLS.md** — available dependency skills and env var name labels each one uses.
- **HEARTBEAT.md** — what to check on each heartbeat run. No secret values.
- **AGENTS.md** — role instructions including active project references and queue check rules.

These are plain text files in each agent's workspace directory. Created once during setup; updated by the operator when projects change.

---

## Before You Start — Required Skills

**Required (install before running this skill):**
- `openclaw-administrator` (EncryptShawn) — performs all agent creation, model assignment, routing config, and workspace file writes.

**Strongly recommended:**
- `openclaw-recovery-manager` (EncryptShawn) — provides config snapshots and emergency rollback.

---

## Step 0 — Safety First

Before touching any configuration:

1. **Check if openclaw-recovery-manager is installed.** If yes, take a snapshot:
   > "Taking a pre-setup configuration snapshot before we begin."
   Label it: `pre-dev-team-setup-[today's date]`

2. If NOT installed, ask:
   > "I recommend installing openclaw-recovery-manager (EncryptShawn on ClawHub) before we proceed. Want to install it first, or proceed without it?"

---

## Step 1 — Gather Project Information

Ask the user for the following. Collect all answers before configuring anything:

```
I need a few details to set up the team correctly.

1. Project name?
   (e.g., talent_avatar — used as the project identifier throughout)

2. Asana workspace GID?
   (Found in your Asana workspace settings or URL)

3. Does an Asana project board already exist for this project?
   - Yes → What is the Asana project GID? (from the board URL)
   - No → I'll guide you through creating one

4. GitHub repository SSH URLs (private repos, SSH access only):
   - Frontend repo: (e.g., [email protected]:org/repo-fe.git)
   - Backend repo: (e.g., [email protected]:org/repo-be.git)
   - Any additional repos?

5. Credential env var names only — do not share token values here.
   What are the names of the env vars in your secret store for:
   - Asana PAT: (e.g., TA_ASANA_PAT)
   - GitHub PAT: (e.g., TA_GITHUB_PAT)
   - Dev Manager email address: (e.g., TA_DEV_MANAGER_EMAIL)

6. PM agent ID for this project?
   (Convention: project_manager_[project_name])
   Press enter to accept the suggested convention.
```

After collecting answers, confirm back with the user before doing anything.

---

## Step 2 — Model Selection

Present model recommendations. Tell the user they can accept or change any agent's model.

**Note: OpenRouter is strongly recommended for access to the full model range.**

```
RECOMMENDED MODEL ASSIGNMENTS
Based on April 2026 SWE-bench Pro and AA-Omniscience hallucination data:

Agent          | Primary                  | Fallback         | Heartbeat       | Vision
---------------|--------------------------|------------------|-----------------|------------------
project_manager| GLM-5.1                  | MiniMax M2.7     | MiniMax M2.7    | —
               | $1.00/$3.20 per 1M       |                  | (cheap loop)    |
               | 34% H% — lowest of all   |                  |                 |
engineer       | Gemini 3.1 Pro           | Claude Opus 4.7  | Gemini 3.1 Pro  | Native (primary)
               | $2/$12 per 1M            | (escalation)     |                 |
dev-fe         | GLM-5.1                  | Qwen 3.6 Plus    | GLM-5.1         | Gemini 3.1 Pro
               | $1/$3.20 per 1M          |                  |                 | (mockup tasks)
dev-be         | GLM-5.1                  | Qwen 3.6 Plus    | GLM-5.1         | —
qa             | Gemini 3.1 Pro           | GPT-5.5          | Gemini 3.1 Pro  | Native (primary)
n8n_engineer   | GLM-5.1 (active)         | Qwen 3.6 Plus    | MiniMax M2.7    | —
               | MiniMax M2.7 (heartbeat) |                  |                 |

Why these picks:
- PM: GLM-5.1 has the lowest hallucination rate (34%) of all 10 frontier models.
  A PM that confidently misrepresents client or engineer statements is dangerous.
  MiniMax M2.7 (39% H%, $0.30/$1.20) is the cheap heartbeat/fallback model.
- Engineer + QA: Gemini 3.1 Pro has native vision — critical for reading mockups,
  rendered UI screenshots, and HTML/CSS visual diffs.
- FE Dev: GLM-5.1 is the default coding model, but switches to Gemini 3.1 Pro
  when a task involves a mockup. See "Vision Tasks" below.
- DeepSeek V4 Pro excluded despite strong benchmarks — 94% hallucination rate
  makes it unsuitable for any agent in this team.

Accept these recommendations? Or tell me which to change.
```

### Vision Tasks (FE Dev and QA)

The FE dev and QA agent both need vision capability for tasks that involve visual
references — mockups, design comps, rendered UI comparisons. Configure them so
Gemini 3.1 Pro is used **only when the task requires vision**, not as the primary
default (which would be more expensive than necessary for non-visual work).

**Configuration approach (via openclaw-administrator):**
- FE dev primary model: GLM-5.1
- FE dev vision model: Gemini 3.1 Pro
- QA primary model: Gemini 3.1 Pro (already vision-capable)
- QA fallback non-vision: GPT-5.5

The FE dev should switch to vision mode when the Asana task description references
a mockup or attached image. The PROJECT.md instructs the FE dev to do this. QA
should use vision mode whenever a PR includes UI changes that need to be compared
against the mockup.

### Mockup Storage

Mockups must be accessible to the engineer (during planning), FE dev (during
implementation), and QA (during review). Two storage paths, in priority order:

1. **Asana attachments (preferred)** — if the Asana skill supports attachment
   upload/retrieval, mockups attach directly to the relevant Asana task. Each
   agent retrieves them via the Asana skill.
2. **Project workspace fallback** — `~/.openclaw/projects/[project_name]/workspace/mockups/`
   - File naming: `[task-id]-[short-description].[ext]` (e.g. `1234-navbar-redesign.png`)
   - The Asana task description must reference the filename so devs and QA can find it
   - This path is pre-created by this skill during Step 7

This mockup convention is documented in PROJECT.md so all agents know the rule.

Wait for confirmation. Record final selections.

---

## Step 3 — Skills Selection Per Agent

Check what's already installed using the openclaw-administrator skill.

### Dependency Skills (Required — Install First)

```
DEPENDENCY SKILLS — install these first on each agent:

All agents:
  - Asana skill (for Asana task management)

Agents that touch code (engineer, dev-fe, dev-be, qa, n8n_engineer):
  - Git skill (for repository operations)

PM agent only:
  - Email skill (for Dev Manager completion alerts)

Search ClawHub for the best-rated version of each if not already installed.
```

### EncryptShawn Workflow Skills (All Agents)

```
- openclaw-administrator
- openclaw-recovery-manager
- approved-self-improver
```

### Role-Specific Workflow Skills

```
project_manager_[project]: dev-project-manager (EncryptShawn)
engineer:                  dev-project-engineer (EncryptShawn)
dev-fe, dev-be, n8n:       project-dev (EncryptShawn)
```

### Tech Stack Skills

```
Tech stack questions:

1. Frontend: Next.js / Nuxt / React / Vue / Astro / Flutter / other?
2. Backend: NestJS / Node.js / Python / Golang / other?
3. Database: MySQL / PostgreSQL / MongoDB / ClickHouse / Redis / other?
4. Using Directus as CMS or API layer? Yes / No
5. Using AWS Cognito for auth? Yes / No
6. Using N8N for automation? Yes / No
7. Using Qdrant for vector search? Yes / No
```

Based on answers, recommend and install appropriate ClawHub skills per agent.

---

## Step 4 — Agent Configuration

Using the openclaw-administrator skill, configure each agent:

### project_manager_[project]
- Model (active): GLM-5.1 (or user choice from Step 2)
- Model (heartbeat): MiniMax M2.7 (cheap heartbeat loop)
- Fallback: MiniMax M2.7
- Heartbeat: every 30 minutes, isolated session, light context
- Repos: read-only access to all repos
- Agent-to-agent allow list: engineer, dev-fe, dev-be, qa, n8n_engineer (if applicable)
- Chat completions: ENABLED

### engineer
- Model (active): Gemini 3.1 Pro (native vision — required for mockup review)
- Fallback: Claude Opus 4.7 (escalation only)
- Heartbeat: Gemini 3.1 Pro
- Heartbeat: every 30 minutes
- Repos: read-only access to all repos (read/write only when explicitly helping a dev)
- Agent-to-agent allow list: [PM agent ID], dev-fe, dev-be, qa, n8n_engineer
- Chat completions: disabled

### dev-fe
- Model (primary): GLM-5.1
- Model (vision): Gemini 3.1 Pro — used when task involves mockup/image references
- Fallback: Qwen 3.6 Plus
- Heartbeat: GLM-5.1
- Heartbeat: every 30 minutes
- Repos: frontend repo (read/write)
- Agent-to-agent allow list: [PM agent ID], engineer, qa
- Chat completions: disabled

### dev-be
- Model (primary): GLM-5.1
- Fallback: Qwen 3.6 Plus
- Heartbeat: GLM-5.1
- Repos: backend repo (read/write)
- Agent-to-agent allow list: [PM agent ID], engineer, qa
- Chat completions: disabled

### qa
- Model (primary): Gemini 3.1 Pro (native vision — required for UI/mockup comparison)
- Fallback (non-vision): GPT-5.5
- Heartbeat: Gemini 3.1 Pro
- Repos: all repos (read-only, plus merge authority on operator instruction)
- Agent-to-agent allow list: [PM agent ID], engineer, dev-fe, dev-be
- Chat completions: disabled

### n8n_engineer (only if N8N confirmed in Step 3)
- Model (active): GLM-5.1
- Model (heartbeat): MiniMax M2.7
- Fallback: Qwen 3.6 Plus
- Repos: backend repo
- Agent-to-agent allow list: [PM agent ID], engineer, qa
- Chat completions: disabled

After all agents configured, verify:
```
openclaw agents list
Confirm all agents show configured status with correct models.
```

---

## Step 5 — Asana Project Setup

### If board does NOT exist yet

```
Create the Asana board manually:

1. Log into Asana → + New Project → name it: [project_name] → Board view
2. Create these columns in this exact order:
   - Backlog
   - PM Queue
   - Engineer Queue
   - Frontend Dev Queue
   - Backend Dev Queue
   - QA Queue
   [- N8N Engineer Queue — only if using N8N]
   - In Progress
   - QA Review
   - Complete
   - Blocked
3. Invite all agent Asana accounts as Members
4. Copy the project GID from the board URL
5. Share the GID here when ready
```

### Once GID is confirmed

Using the Asana dependency skill, add project description to the board:
```
Project: [project_name]
PM Agent: [pm_agent_id]
Repos: FE: [fe_repo] | BE: [be_repo]
Asana Project GID: [GID]
Dev Manager Email env var: [env_var_name — label only]
```

---

## Step 6 — Write Agent Workspace Files

Using the openclaw-administrator skill, write the following files for each agent.

**HEARTBEAT.md** (per agent — under 200 tokens):
Project-specific heartbeat checklists referencing project GID and queue column to check.

**USER.md** (per agent):
- Role and scope
- Active Projects table: project name, PM agent ID, Asana GID, repo URLs
- Repo access scope for this agent
- Env var name labels for credential references (not values)
- Team agent roster with agent IDs

**TOOLS.md** (per agent):
- Available dependency skills
- Env var name label each skill uses for its credential
- sessions_send allowed targets

**AGENTS.md** (per agent) — must include an Active Projects section:
```
## Active Projects
- [project_name] — I am the [role] on this project.
  - Full rules: ~/.openclaw/projects/[project_name]/PROJECT.md
  - My queue: ~/.openclaw/projects/[project_name]/queues/to-[role].md
  - Shared workspace: ~/.openclaw/projects/[project_name]/workspace/
  - Check my queue at the start of every session before doing anything else.
  - Check project-lock.json to understand what phase we are in before acting.
```

---

## Step 7 — Create Project Folder Structure

This is the coordination layer that makes agents work together across sprints. Read `references/project-files.md` for the full specification of each file's content and format.

Create the following structure for each project:

```
~/.openclaw/projects/[project_name]/
├── PROJECT.md                    ← Full team rulebook — every agent reads this
├── project.json                  ← Machine-readable config (agents, Asana, paths)
├── project-lock.json             ← Current phase and ownership
├── STATE.md                      ← Human-readable status at a glance
├── SHARED_MEMORY.md              ← Cross-agent knowledge not suited for Asana
├── DECISIONS.md                  ← Immutable log of all requirement decisions
├── KNOWN_ISSUES.md               ← Accepted technical debt and limitations
├── RUNBOOK.md                    ← How to run the project; branch and PR conventions
├── workspace/
│   ├── repo/                     ← The git repository (cloned here)
│   ├── mockups/                  ← Design mockups (fallback if Asana attachments unavailable)
│   ├── SPEC-CURRENT.md           ← Points to the active accepted spec
│   └── IMPLEMENTATION_GUIDE.md   ← Engineer's task-oriented implementation plan
└── queues/
    ├── to-pm.md
    ├── to-engineer.md
    ├── to-engineer-feasibility.md ← Requirements phase only — kept separate
    ├── to-qa.md
    └── to-operator.md
```

### Creating Each File

**project.json** — fill in from the details collected in Step 1:
```json
{
  "id": "[project_name]",
  "name": "[Project Display Name]",
  "asana": {
    "project_id": "[asana_project_gid]",
    "workspace_id": "[asana_workspace_gid]",
    "columns": {
      "backlog":     { "id": "[col_gid]", "purpose": "Work not yet started. PM creates tasks here." },
      "in_progress": { "id": "[col_gid]", "purpose": "Dev actively working on this." },
      "in_review":   { "id": "[col_gid]", "purpose": "PR submitted. QA should pick this up." },
      "qa":          { "id": "[col_gid]", "purpose": "QA is actively testing." },
      "completed":   { "id": "[col_gid]", "purpose": "QA passed. Awaiting operator merge and sign-off." },
      "blocked":     { "id": "[col_gid]", "purpose": "Waiting on client or external input. PM owns." }
    }
  },
  "participants": [
    { "agentId": "[pm_agent_id]", "workspace": "../../workspace-pm",     "role": "Project Manager" },
    { "agentId": "engineer",      "workspace": "../../workspace-main",   "role": "Lead Engineer" },
    { "agentId": "dev-fe",        "workspace": "../../workspace-dev0fe", "role": "Front-end Developer" },
    { "agentId": "dev-be",        "workspace": "../../workspace-dev0be", "role": "Back-end Developer" },
    { "agentId": "qa",            "workspace": "../../workspace-qa",     "role": "QA Engineer" }
  ],
  "shared_workspace": "./workspace",
  "repo_path": "./workspace/repo",
  "mockups_path": "./workspace/mockups",
  "shared_memory": "./SHARED_MEMORY.md",
  "decisions_log": "./DECISIONS.md",
  "known_issues": "./KNOWN_ISSUES.md",
  "runbook": "./RUNBOOK.md",
  "queues": {
    "pm":                   "./queues/to-pm.md",
    "engineer":             "./queues/to-engineer.md",
    "engineer_feasibility": "./queues/to-engineer-feasibility.md",
    "qa":                   "./queues/to-qa.md",
    "operator":             "./queues/to-operator.md"
  },
  "visual_assets": {
    "primary_storage": "asana_attachments",
    "fallback_storage": "./workspace/mockups",
    "naming_convention": "[task-id]-[short-description].[ext]",
    "vision_required_agents": ["engineer", "dev-fe", "qa"]
  },
  "escalation_rules": {
    "client_no_response_hours": 48,
    "dev_stuck_same_issue_escalations": 2,
    "dev_stuck_same_issue_hours": 24,
    "blocked_task_operator_escalation_hours": 48
  }
}
```

**project-lock.json** — initialize to idle:
```json
{
  "phase": "idle",
  "sprint_id": null,
  "sprint_opened": null,
  "waiting_on": null,
  "last_updated": "[today's date]",
  "last_updated_by": "operator",
  "context": "Project initialized. Ready to receive first requirements.",
  "blocked_tasks": []
}
```

**Initialize all queue files** as empty with a header comment:
```
# Queue: to-[role]
# Format: [YYYY-MM-DD HH:MM] [FROM: agent-id] [TO: agent-id] [TASK: task-id or N/A]
# Queues are append-only. Archive at sprint close. Never delete entries.
```

**Initialize STATE.md**:
```markdown
# [Project Display Name] — Current State
**Phase:** Idle — Ready for first requirements
**Last updated:** [today's date] by operator
```

**Initialize SHARED_MEMORY.md, DECISIONS.md, KNOWN_ISSUES.md** with a header and empty body.

**Clone the repo and create mockups dir** in `./workspace/`:
```bash
cd ~/.openclaw/projects/[project_name]/workspace
git clone [repo_ssh_url] repo
mkdir -p mockups
```

**Write PROJECT.md** — read `references/project-files.md` for the full template and content rules. This is the most important file. Fill it in completely using the project details from Step 1 and the Asana column GIDs from Step 5.

**Write RUNBOOK.md** — stub it out now; the engineer fills in codebase details after their first session with the repo.

---

## Step 8 — Credential Verification

```
Credential check — confirm in your secret management system:

- [ASANA_PAT_ENV_VAR_NAME] — Asana personal access token
- [GITHUB_PAT_ENV_VAR_NAME] — GitHub personal access token
- [DEV_MANAGER_EMAIL_ENV_VAR_NAME] — Dev Manager email address

Confirm they exist before continuing. (yes / one or more missing)
```

---

## Step 9 — Enable Chat Completions

Using the openclaw-administrator skill, enable chat completions for the PM agent.

---

## Step 10 — Smoke Test

```
SMOKE TEST

Step 1: Create a test task in the Asana PM Queue:
  Title: [TEST] Smoke test — verify agent pickup
  Description: Test task. Agent should acknowledge and move task or add a comment.

Step 2: Wait up to 30 minutes for the PM agent's next heartbeat.
  Watch for: task gaining a comment, or task moving to another column.

Step 3: Once PM confirmed working, send via chat completions:
  "Bug report for [project_name]: Login throws a 500 error when
   email field is left blank. Expected: show a validation error."

Step 4: Watch for task creation in the Engineer Queue on the Asana board.

Step 5: Confirm engineer picks up the task and produces an assessment.

Step 6: Verify the project folder exists and queue files are accessible by agents.

Heartbeat confirmed working? (yes / no — describe what happened)
```

---

## Step 11 — Post-Setup Snapshot

If openclaw-recovery-manager is installed:
```
Taking post-setup snapshot.
Label: post-dev-team-setup-[today's date]-confirmed
```

---

## Step 12 — Handoff Summary

```
TEAM SETUP COMPLETE

Project: [project_name]
Asana Project GID: [GID]
PM Agent: [pm_agent_id] — chat completions ENABLED
Project folder: ~/.openclaw/projects/[project_name]/

AGENTS:
Agent            | Primary Model         | Fallback         | Heartbeat       | Vision
[pm_agent_id]    | GLM-5.1               | MiniMax M2.7     | MiniMax M2.7    | —
engineer         | Gemini 3.1 Pro        | Claude Opus 4.7  | Gemini 3.1 Pro  | Native
dev-fe           | GLM-5.1               | Qwen 3.6 Plus    | GLM-5.1         | Gemini 3.1 Pro
dev-be           | GLM-5.1               | Qwen 3.6 Plus    | GLM-5.1         | —
qa               | Gemini 3.1 Pro        | GPT-5.5          | Gemini 3.1 Pro  | Native
[n8n_engineer]   | GLM-5.1               | Qwen 3.6 Plus    | MiniMax M2.7    | —

REPOS (SSH):
Frontend: [fe_repo]
Backend:  [be_repo]
Cloned to: ~/.openclaw/projects/[project_name]/workspace/repo/

HOW PROJECTS WORK:
- All agents read ~/.openclaw/projects/[project_name]/PROJECT.md for workflow rules
- Each agent checks their queue file at every session start
- Asana is the source of truth for task ownership and status
- project-lock.json tracks which phase the sprint is in
- One sprint at a time — PM does not accept new requirements until sprint is closed

TO START THE FIRST SPRINT:
  Send requirements to [pm_agent_id] via chat completions.
  The PM will open a feasibility discussion with the engineer,
  negotiate with the client, and open the sprint once requirements are accepted.

ADDING A NEW PROJECT:
- Run this skill again for the new project
- Add the new project to each shared agent's AGENTS.md Active Projects section
- Add the new PM to agent-to-agent allow lists
- Shared agents pick up new projects automatically on next heartbeat

RECOVERY:
Pre-setup:  pre-dev-team-setup-[date]
Post-setup: post-dev-team-setup-[date]-confirmed
```

---

## Project Workflow Reference

For the complete workflow that agents follow after setup, see:
- `references/workflow.md` — All phases (requirements through sprint close), escalation rules, queue message format, git and branch conventions, QA process, operator merge procedure

Read this file when:
- Troubleshooting agent behavior during a sprint
- Answering operator questions about how the team works
- Explaining why an agent stopped work or escalated

---

## Sprint Management

### Opening a Sprint
The PM opens a sprint automatically when requirements are accepted by both the engineer and client. The PM updates `project-lock.json` phase to `implementation` and creates Asana tasks from the implementation guide.

### During a Sprint
- Agents check `project-lock.json` phase before acting
- Devs work one branch per sprint, pushing updates — PR auto-updates each time
- QA re-reviews after each dev push
- Operator is notified via `to-operator.md` when QA passes all tasks

### Closing a Sprint (Operator-Led)
When `to-operator.md` shows all tasks QA-passed:

1. Operator reviews and pulls the branch locally
2. Operator tells QA agent: "Merge to main"
3. QA merges the PR to main
4. All affected repos rebase to main
5. Operator confirms rebase is clean
6. Operator tells PM: "Sprint closed — ready for next requirements"
7. PM clears Asana, archives queue entries, updates STATE.md to idle
8. `project-lock.json` resets to `idle`

**One sprint at a time.** PM does not accept new requirements until `project-lock.json` is `idle`.

---

## If Anything Goes Wrong

```
Option 1 — Recover using openclaw-recovery-manager:
  Restore: pre-dev-team-setup-[date]

Option 2 — Diagnose with openclaw-administrator:
  Run diagnostics and retry the failed step.

Option 3 — Describe what step failed and what error appeared.
  I can walk through the failed step again.
```

FILE:_meta.json
{
  "ownerId": "kn77nfg6wv2expv6qs7k17dfqs83zp59",
  "slug": "build-dev-team",
  "version": "1.1.1",
  "publishedAt": 1777179327568
}
FILE:references/project-files.md
# Project Files Reference

Full specification of every file in the `~/.openclaw/projects/[project_name]/` folder.

---

## Table of Contents

- [PROJECT.md — The Team Rulebook](#projectmd--the-team-rulebook)
- [project.json — Machine Config](#projectjson--machine-config)
- [project-lock.json — Phase Tracker](#project-lockjson--phase-tracker)
- [STATE.md — Human Status](#statemd--human-status)
- [SHARED_MEMORY.md — Cross-Agent Knowledge](#shared_memorymd--cross-agent-knowledge)
- [DECISIONS.md — Decision Log](#decisionsmd--decision-log)
- [KNOWN_ISSUES.md — Accepted Limitations](#known_issuesmd--accepted-limitations)
- [RUNBOOK.md — Project Setup Guide](#runbookmd--project-setup-guide)
- [workspace/SPEC-CURRENT.md](#workspacespec-currentmd)
- [workspace/IMPLEMENTATION_GUIDE.md](#workspaceimplementation_guidemd)
- [Queue Files](#queue-files)
- [File Responsibility Matrix](#file-responsibility-matrix)

---

## PROJECT.md — The Team Rulebook

**The most important file in the project folder.** Every participating agent has this injected into their context via their AGENTS.md. It is the single source of truth for how the team operates.

### Who reads it
All agents, at every session start (referenced from their AGENTS.md).

### Who writes it
Created by operator during setup. Updated by operator only when workflow rules change.

### Full Template

```markdown
# Project: [Project Display Name]

## The Team
- **PM Agent ([pm_agent_id])** — Client-facing. Owns requirements negotiation,
  Asana task creation, client communication, and sprint open/close.
- **Lead Engineer (engineer)** — Technical authority. Reviews feasibility, writes
  implementation guides, handles all technical escalations.
- **FE Dev (dev-fe)** — Implements front-end tasks from Asana. Escalates to Lead Engineer.
- **BE Dev (dev-be)** — Implements back-end tasks from Asana. Escalates to Lead Engineer.
- **QA Agent (qa)** — Validates PRs against the accepted spec. Merges on operator instruction.
- **Operator (Human)** — Final authority. Merges approval, unresolvable escalations, client engagement.

## Source of Truth

| What | Where |
|---|---|
| Task ownership and status | Asana |
| Accepted requirements | `workspace/SPEC-CURRENT.md` |
| Implementation approach | `workspace/IMPLEMENTATION_GUIDE.md` |
| Cross-agent knowledge | `SHARED_MEMORY.md` |
| Decision history | `DECISIONS.md` |
| Accepted limitations | `KNOWN_ISSUES.md` |
| Project setup / branch conventions | `RUNBOOK.md` |
| Current phase and ownership | `project-lock.json` |
| Human-readable status | `STATE.md` |

## Asana Columns

| Column | Meaning | Owner |
|---|---|---|
| Backlog | Work not yet started | PM creates tasks here |
| In Progress | Dev actively working | Dev owns |
| In Review | PR submitted, awaiting QA | QA picks up |
| QA | QA actively testing | QA owns |
| Completed | QA passed, awaiting operator | Operator reviews and merges |
| Blocked | Waiting on client/external | PM owns, escalates per rules |

## Git and Branch Convention

- **One repo copy** lives at `./workspace/repo/`
- **One active branch per dev per sprint**
  - Format: `[project-id]/[sprint-id]/[task-id]-[short-description]`
  - Example: `ezbi/sprint-3/1234-navbar-redesign`
- **After completing each task:** push to branch, PR opens (first task) or auto-updates (subsequent)
- **QA re-reviews after every push**
- **No new branches mid-sprint** unless operator explicitly approves a hotfix

## Mockups and Visual Assets

When a task involves a mockup, design comp, or visual reference:

- **Primary storage:** Asana task attachments. The Asana skill retrieves these.
- **Fallback storage:** `./workspace/mockups/` — used if Asana attachment retrieval is not available
- **Naming convention:** `[task-id]-[short-description].[ext]` (e.g. `1234-navbar-redesign.png`)
- **Task description must reference the mockup filename** so devs and QA can locate it

**Vision-required agents:**
- **Engineer** — uses vision (Gemini 3.1 Pro, native) when reviewing mockups during planning
  to write accurate implementation guides
- **FE Dev** — switches to its vision model (Gemini 3.1 Pro) when picking up a task whose
  Asana description references a mockup. Returns to GLM-5.1 for non-visual coding work.
- **QA** — uses vision (Gemini 3.1 Pro, native) when a PR includes UI changes that need to
  be compared visually against the mockup. QA is expected to compare rendered output to mockup.

If the FE dev cannot retrieve a referenced mockup from either Asana or `./workspace/mockups/`,
this is treated as a blocker — escalate to engineer per normal escalation rules.

## Workflow

### Phase 1: Requirements & Feasibility
1. PM receives or drafts client requirements
2. PM writes draft to `workspace/SPEC-v[N]-[YYYY-MM-DD].md`, updates `SPEC-CURRENT.md`
3. PM posts to `queues/to-engineer-feasibility.md` — "New spec draft ready"
4. Engineer reviews, posts all issues (numbered) to `queues/to-engineer-feasibility.md`
5. PM translates issues to non-technical language, sends to client via email or posts to
   `queues/to-operator.md` if email not configured
6. Client responds to each numbered issue: Accept / Provide solution / Descope
7. PM logs client response in `DECISIONS.md` verbatim with date
8. Engineer evaluates client solutions. Loop repeats until all issues resolved.
9. Engineer marks spec ACCEPTED in `SPEC-CURRENT.md`. PM logs in `DECISIONS.md`.
10. `project-lock.json` → phase: `planning`

**Client no-response rule:**
- No response in 48h → PM sends follow-up email
- Still no response → PM posts to `queues/to-operator.md`, task moves to Blocked

### Phase 2: Planning
1. Engineer writes `workspace/IMPLEMENTATION_GUIDE.md`
   - Task-oriented: each numbered section = one Asana task
   - Detailed enough to implement without ambiguity — no full code
   - Note items for `KNOWN_ISSUES.md`
2. Engineer updates `KNOWN_ISSUES.md` with accepted limitations
3. Engineer posts to `queues/to-pm.md` — "Implementation guide ready"
4. PM creates Asana tasks from guide, assigns to agents, places in Backlog
5. PM posts to `queues/to-engineer.md` — "Tasks created. Sprint [N] open."
6. `project-lock.json` → phase: `implementation`, sprint_id set

### Phase 3: Implementation
1. Dev picks up task from Backlog → moves to In Progress
2. Dev implements against `IMPLEMENTATION_GUIDE.md` and `RUNBOOK.md`
3. Dev checks `queues/to-[their-role].md` at session start

**Dev escalation rules:**
- If blocked: post to `queues/to-engineer.md` with task ID, what was tried, specific question
- Engineer responds in `queues/to-[dev-agent].md`
- If same issue escalated **2 times** to engineer without resolution, OR dev stuck **24 hours**:
  - Dev **stops work immediately**
  - Dev posts full summary to `queues/to-pm.md`
  - PM posts to `queues/to-operator.md`
  - Task moves to Blocked in Asana
  - **No further AI cycles on this task until operator resolves**

When task complete:
- Push to sprint branch. Open PR (first task) or PR auto-updates (subsequent).
- Move Asana task to In Review
- Post to `queues/to-qa.md` — task ID, PR link, brief description of changes

### Phase 4: QA
1. QA picks up tasks from In Review → moves to QA
2. QA reviews PR against:
   - `workspace/SPEC-CURRENT.md` — meets accepted requirements?
   - `KNOWN_ISSUES.md` — don't file failures against accepted limitations
   - `workspace/IMPLEMENTATION_GUIDE.md` — matches planned approach?
3. **Pass:** Move to Completed. Post to `queues/to-operator.md` — task ID, PR link, "QA passed."
4. **Fail:** Post specific numbered failures to `queues/to-engineer.md`. Move back to In Progress.

### Phase 5: Operator Review and Merge
1. Operator checks `queues/to-operator.md`
2. Operator pulls branch, reviews locally
3. If satisfied: tells QA — "Merge to main"
4. QA merges PR to main
5. QA/devs rebase all affected repos to main
6. Operator confirms rebase is clean
7. `project-lock.json` → phase: `sprint-close`

### Phase 6: Sprint Close
1. PM verifies all sprint tasks are Completed in Asana
2. PM archives completed tasks in Asana
3. PM verifies `DECISIONS.md` has full requirement decision record
4. PM verifies `KNOWN_ISSUES.md` is current
5. PM writes sprint summary to `SHARED_MEMORY.md`
6. PM updates `STATE.md` — "Sprint [N] closed. Ready for next requirements."
7. PM archives queue file entries (marks READ, does not delete)
8. `project-lock.json` → phase: `idle`
9. PM posts to `queues/to-operator.md` — "Sprint closed. Ready for next requirements."

**One sprint at a time. PM does not accept new requirements until project-lock.json is idle.**

## Escalation Rules Summary

| Situation | Action | Threshold |
|---|---|---|
| Client not responding | PM emails, then escalates to operator | 48h |
| Dev stuck on same task | Escalate to Lead Engineer | — |
| Same issue escalated 2x or 24h stuck | Stop work, PM escalates to operator | 24h |
| Task blocked with no movement | PM escalates to operator | 48h |

**No agent continues spending cycles on a blocked path. Stop, surface, wait.**

## Communication Protocol

All inter-agent messages go through the queue files in `queues/`.

**Message format — every entry must use this format:**
```
[YYYY-MM-DD HH:MM] [FROM: agent-id] [TO: agent-id] [TASK: asana-task-id or N/A]
Message body. Be specific. Include task IDs, file names, error messages where relevant.
---
```

- Queues are **append-only**. Never delete entries.
- Archive at sprint close (mark entries READ, do not remove lines).
- Each agent checks their queue at the **start of every session, before anything else**.
- Feasibility discussions go to `to-engineer-feasibility.md` only — separate from escalations.
```

---

## project.json — Machine Config

Machine-readable project configuration. Agents read this to resolve file paths,
Asana GIDs, and participant details without relying on hardcoded values.

### Who reads it
All agents.

### Who writes it
Created by operator during setup. Updated by operator only when structure changes.

### Notes
- All file paths use relative paths from the project root folder
- Asana column GIDs must be filled in from the actual Asana board after creation
- Escalation thresholds here are the canonical values — PROJECT.md references these conceptually but agents read the numbers from this file

---

## project-lock.json — Phase Tracker

Prevents agents from acting out of phase or moving forward when waiting on another agent or the operator.

### Who reads it
All agents check this at session start before taking any action.

### Who writes it
PM (most phase transitions), Engineer (planning → implementation), QA (after merge), Operator (sprint-close → idle).

### Valid phase progression
`idle` → `requirements` → `planning` → `implementation` → `qa` → `sprint-close` → `idle`

### Format
```json
{
  "phase": "idle",
  "sprint_id": null,
  "sprint_opened": null,
  "waiting_on": null,
  "last_updated": "YYYY-MM-DD",
  "last_updated_by": "agent-id or operator",
  "context": "Human-readable description of current state",
  "blocked_tasks": []
}
```

### Agent behavior rules
- If `phase` does not match the agent's expected action → stop and post to relevant queue
- If `waiting_on` is not null and is this agent → act immediately on session start
- If `blocked_tasks` is non-empty and contains this agent's task → treat as stopped

---

## STATE.md — Human Status

The operator's one-file status check. Updated by agents at key moments so the operator
can understand project state without digging through Asana or queue files.

### Who reads it
Operator primarily. Agents may read it for context.

### Who writes it
All agents update it when they complete a significant action.

### Format
```markdown
# [Project Name] — Current State
**Phase:** [phase] ([sprint_id if applicable])
**Last updated:** [YYYY-MM-DD HH:MM] by [agent-id]

## Sprint Progress
- ✅ Task [ID] — [description] (merged)
- 🔄 Task [ID] — [description] (in progress, [agent])
- ⏳ Task [ID] — [description] (backlog)
- 🚫 Task [ID] — [description] (blocked — [reason])

## Operator Queue Summary
[List of items in to-operator.md awaiting action]
```

---

## SHARED_MEMORY.md — Cross-Agent Knowledge

A living document for project knowledge that needs to persist across sessions
but doesn't belong in Asana.

### What goes here
- Codebase quirks and patterns relevant to this project
- Client preferences and communication style notes
- Things learned mid-sprint that other agents should know
- Sprint-close summaries (added by PM at close)

### What does NOT go here
- Task status → Asana
- Accepted requirements → SPEC files
- Implementation approach → IMPLEMENTATION_GUIDE.md
- Decisions and client acceptances → DECISIONS.md

### Format
Free-form markdown. Agents append new information with a date prefix:
```markdown
## [YYYY-MM-DD] [agent-id] — [topic]
Content here.
```

---

## DECISIONS.md — Decision Log

An **immutable, append-only** record of every significant decision made during
requirements negotiation. Never edit or delete existing entries.

### Who reads it
PM, Engineer, Operator. QA references when filing failures.

### Who writes it
PM only. Written during requirements phase as decisions are made.

### Purpose
When a client later says "we never agreed to that," this file is the record.
It captures what was proposed, what issue was surfaced, what the client said, and what was accepted.

### Format
```markdown
## [YYYY-MM-DD] — [Sprint ID]: [Decision Topic]

**Issue surfaced by engineer:** [description of technical issue or conflict]

**Client response (received [date]):** [exact client words or paraphrase, clearly attributed]

**Resolution:** [Accept as known outcome / Client-proposed alternative / Descoped]

**Accepted by:** [Client name], [engineer agent], [pm agent]
**Logged by:** [pm agent]
---
```

---

## KNOWN_ISSUES.md — Accepted Limitations

Documents accepted technical debt and known limitations so QA does not file
failures against intentional decisions, and clients cannot later claim ignorance.

### Who reads it
QA (before every test run), all agents for context, operator.

### Who writes it
Engineer, updated during planning phase and as new limitations are accepted.

### Format
```markdown
## [Sprint ID] — [Issue Title]
- **Accepted:** [date]
- **Context:** [why this limitation exists and what decision led to it]
- **Impact:** [what users or developers will experience as a result]
---
```

---

## RUNBOOK.md — Project Setup Guide

Written and maintained by the Lead Engineer. Devs and QA read this before
starting work on any task to avoid unnecessary escalations.

### Who reads it
All agents before starting work, operator for deployment reference.

### Who writes it
Engineer creates initial stub during setup. Engineer fills in details after
their first session with the repo. Engineer updates as patterns evolve.

### Minimum contents
```markdown
# [Project Name] Runbook

## Local Setup
[How to install dependencies and run the project locally]

## Branch Naming Convention
[project-id]/[sprint-id]/[task-id]-[short-description]
Example: ezbi/sprint-3/1234-navbar-redesign

## PR Conventions
- Title: [[PROJECT-task-id]] Short description
- Body must include: task ID, what changed, how to test, Asana task link

## Known Codebase Patterns
[Common patterns in use — component structure, API conventions, etc.]

## Known Gotchas
[Things that trip up developers — quirky dependencies, env requirements, etc.]

## Deployment Notes
[Steps needed to deploy after merge if applicable]
```

---

## workspace/SPEC-CURRENT.md

A reference that always points to (or contains) the currently active accepted specification.

### Versioning rules
- Every new requirements draft gets its own versioned file: `SPEC-v[N]-[YYYY-MM-DD].md`
- Specs are **never overwritten** — always increment the version number
- `SPEC-CURRENT.md` is updated to point to or contain the latest accepted spec
- The version history is the audit trail

### Status markers
Engineer adds one of these markers at the top when reviewing:
```
STATUS: DRAFT — Under feasibility review
STATUS: ACCEPTED — [date] — [engineer agent] + [pm agent]
```

---

## workspace/IMPLEMENTATION_GUIDE.md

Written by the Lead Engineer after spec is accepted. The task-oriented blueprint
for what needs to be built and how.

### Format rules
- Each numbered section = one Asana task
- Describe approach, files affected, edge cases, dependencies, acceptance criteria
- No full code — approach-level only
- Reference `KNOWN_ISSUES.md` items created from this guide

### Structure
```markdown
# Implementation Guide — [Sprint ID]

## Task 1: [Task Title]
**Assigned to:** [agent role]
**Asana task:** [created by PM after this guide is written]

### What to build
[Description]

### Files affected
[List of files or components]

### Approach
[How to implement it — no full code]

### Acceptance criteria
[How QA will verify this is complete]

### Notes / edge cases
[Anything the dev should know]
---

## Task 2: ...
```

---

## Queue Files

Located in `queues/`. One file per recipient.

| File | Purpose |
|---|---|
| `to-pm.md` | Messages for PM — from engineer (issues ready), from QA (task failures), from devs (stuck escalations) |
| `to-engineer.md` | Messages for Lead Engineer — dev escalations, PM implementation requests |
| `to-engineer-feasibility.md` | Requirements phase only — keeps feasibility back-and-forth separate from escalations |
| `to-qa.md` | Messages for QA — dev task completions with PR links |
| `to-operator.md` | Messages for human operator — QA passes, blocks, client no-response, unresolvable escalations |

### Queue message format (required for every entry)
```
[YYYY-MM-DD HH:MM] [FROM: agent-id] [TO: agent-id] [TASK: asana-task-id or N/A]
Message body. Be specific. Include task IDs, file names, error messages.
---
```

### Queue rules
- Append-only — never delete entries
- Mark entries READ (prepend `[READ]`) when processed — do not remove lines
- Archive at sprint close
- Check your own queue at the start of every session, before any other action

---

## File Responsibility Matrix

| File | Created by | Updated by | Read by | Mutable? |
|---|---|---|---|---|
| `PROJECT.md` | Operator | Operator only | All agents | Rarely |
| `project.json` | Operator | Operator only | All agents | On structure change |
| `project-lock.json` | Operator | PM, Engineer, QA, Operator | All agents | Every phase change |
| `STATE.md` | Operator | All agents | Operator, all agents | Frequently |
| `SHARED_MEMORY.md` | Operator | All agents (append) | All agents | Frequently |
| `DECISIONS.md` | PM | PM only (append) | PM, Engineer, Operator | Append-only |
| `KNOWN_ISSUES.md` | Engineer | Engineer (append) | QA, all agents | Append-only |
| `RUNBOOK.md` | Engineer | Engineer | Devs, QA | As patterns evolve |
| `SPEC-vN-*.md` | PM | Never | PM, Engineer | Immutable |
| `SPEC-CURRENT.md` | PM | PM (points to latest) | All agents | Per sprint |
| `IMPLEMENTATION_GUIDE.md` | Engineer | Engineer | All devs, QA | Per sprint |
| `queues/to-*.md` | Operator (init) | Named sender (append) | Named recipient | Append-only |

FILE:references/workflow.md
# Workflow Reference

Complete agent workflow for projects managed under this team setup.
Read this when troubleshooting agent behavior, explaining the workflow to an operator,
or verifying an agent is acting correctly for the current phase.

---

## Table of Contents

- [Phase Overview](#phase-overview)
- [Phase 1: Requirements and Feasibility](#phase-1-requirements-and-feasibility)
- [Phase 2: Planning](#phase-2-planning)
- [Phase 3: Implementation](#phase-3-implementation)
- [Phase 4: QA](#phase-4-qa)
- [Phase 5: Operator Review and Merge](#phase-5-operator-review-and-merge)
- [Phase 6: Sprint Close](#phase-6-sprint-close)
- [Escalation Rules](#escalation-rules)
- [Git and Branch Conventions](#git-and-branch-conventions)
- [Queue Message Format](#queue-message-format)
- [Agent Session Start Checklist](#agent-session-start-checklist)

---

## Phase Overview

```
idle → requirements → planning → implementation → sprint-close → idle
                                       ↕
                                      qa
```

Each phase is tracked in `project-lock.json`. Agents check this file before acting.
If the current phase does not match the agent's intended action, the agent stops and
posts to their relevant queue rather than proceeding.

---

## Phase 1: Requirements and Feasibility

**Owner:** PM Agent  
**Lock phase:** `requirements`  
**Queue used:** `to-engineer-feasibility.md` (kept separate from implementation escalations)

### Steps

1. PM receives or drafts client requirements.
2. PM creates a new versioned spec file: `workspace/SPEC-v[N]-[YYYY-MM-DD].md`
   - Never overwrite an existing spec — always increment the version number
   - Updates `SPEC-CURRENT.md` to reference this draft
   - Marks file: `STATUS: DRAFT — Under feasibility review`
3. PM posts to `queues/to-engineer-feasibility.md`:
   ```
   [date] [FROM: pm] [TO: engineer] [TASK: N/A]
   New spec draft ready for feasibility review.
   File: workspace/SPEC-v[N]-[YYYY-MM-DD].md
   ---
   ```
4. Engineer reads the spec and reviews for:
   - Technical feasibility
   - Conflicts with existing architecture
   - Ambiguities that would block implementation
   - Missing information the devs would need
5. Engineer posts all issues to `queues/to-engineer-feasibility.md` — **numbered**, specific, with options where possible:
   ```
   [date] [FROM: engineer] [TO: pm] [TASK: N/A]
   Feasibility review complete. 3 issues to resolve before accepting.

   Issue 1: [title]
   [description of the technical issue, concrete impact, and options if available]

   Issue 2: ...
   ---
   ```
6. PM translates each issue into non-technical language.
   - If email is configured: sends to client directly
   - If not configured: posts to `queues/to-operator.md` with message ready for operator to relay
7. Client responds to each numbered issue:
   - **Accept as known outcome** — client accepts the limitation as-is
   - **Provide a solution** — client proposes an alternative
   - **Descope** — remove the requirement causing the issue
8. PM logs client response in `DECISIONS.md` verbatim with date (see format in project-files.md).
9. If client proposes a solution, engineer evaluates it. This loop repeats until all issues resolved.
10. When all issues resolved:
    - Engineer updates `SPEC-CURRENT.md`: `STATUS: ACCEPTED — [date] — [engineer] + [pm]`
    - PM logs final acceptance in `DECISIONS.md`
    - PM updates `project-lock.json` → `phase: planning`

### Client No-Response Rule
- No response in **48 hours** → PM sends follow-up via email
- Still no response → PM posts to `queues/to-operator.md`:
  ```
  [date] [FROM: pm] [TO: operator] [TASK: N/A]
  Client has not responded to feasibility issues for 48h. Follow-up sent.
  Please engage client directly. Issues are in queues/to-engineer-feasibility.md.
  ---
  ```
- Task moves to Blocked in Asana until resolved

---

## Phase 2: Planning

**Owner:** Lead Engineer  
**Lock phase:** `planning`

### Steps

1. Engineer writes `workspace/IMPLEMENTATION_GUIDE.md` (see format in project-files.md).
   - Each numbered section = one Asana task
   - Task-oriented and detailed enough to implement without ambiguity
   - No full code — approach, files affected, edge cases, acceptance criteria
   - Notes items for `KNOWN_ISSUES.md`
   - **If the spec includes mockups:** engineer uses its vision capability (Gemini 3.1 Pro)
     to review them. Each task that references a mockup must include the mockup filename
     in its task section so the FE dev and QA can locate it later.
2. Engineer updates `KNOWN_ISSUES.md` with any limitations accepted during requirements phase.
3. Engineer posts to `queues/to-pm.md`:
   ```
   [date] [FROM: engineer] [TO: pm] [TASK: N/A]
   Implementation guide ready. workspace/IMPLEMENTATION_GUIDE.md
   [N] tasks defined.
   ---
   ```
4. PM reviews guide for completeness (can each section become a clear Asana task?).
5. PM creates Asana tasks from guide:
   - One task per numbered section
   - Task description includes the relevant guide section text
   - Tasks placed in Backlog, assigned to appropriate agents
6. PM posts to `queues/to-engineer.md`:
   ```
   [date] [FROM: pm] [TO: engineer] [TASK: N/A]
   Sprint [N] open. [X] tasks created in Asana Backlog.
   ---
   ```
7. PM updates `project-lock.json`:
   ```json
   {
     "phase": "implementation",
     "sprint_id": "sprint-[N]",
     "sprint_opened": "[date]"
   }
   ```
8. PM updates `STATE.md` to reflect sprint open status.

---

## Phase 3: Implementation

**Owner:** Devs (their assigned tasks)  
**Escalation owner:** Lead Engineer  
**Lock phase:** `implementation`

### Dev Task Flow

1. Dev picks up assigned task from Backlog → moves to In Progress in Asana.
2. Dev reads:
   - `workspace/IMPLEMENTATION_GUIDE.md` — the relevant task section
   - `RUNBOOK.md` — codebase conventions and gotchas
   - Their queue (`queues/to-[role].md`) — any pending messages
3. **If the task references a mockup (FE dev only):**
   - Dev switches to its vision model (Gemini 3.1 Pro) for this task
   - Retrieves the mockup from Asana attachments (preferred) or `./workspace/mockups/`
   - If mockup cannot be retrieved from either source, treat as a blocker —
     escalate to engineer per normal escalation rules
4. Dev implements in the sprint branch:
   - Branch format: `[project-id]/[sprint-id]/[task-id]-[short-description]`
   - Dev works in `workspace/repo/` on their branch
5. When task is complete:
   - Push to sprint branch
   - **First task of sprint:** open a PR
   - **Subsequent tasks:** push updates — existing PR auto-updates
   - Move Asana task from In Progress → In Review
   - Post to `queues/to-qa.md`:
     ```
     [date] [FROM: dev-fe] [TO: qa] [TASK: 1234]
     Task 1234 complete. Branch: ezbi/sprint-3/1234-navbar-redesign
     PR: [PR URL]
     What changed: [brief description]
     ---
     ```
6. Dev moves on to next Backlog task if available.

### Dev Escalation Rules

**When blocked:**
- Post to `queues/to-engineer.md`:
  ```
  [date] [FROM: dev-fe] [TO: engineer] [TASK: 1234]
  Blocked on task 1234.
  Attempted: [what was tried]
  Question: [specific question]
  ---
  ```
- Engineer responds in `queues/to-[dev-agent].md`
- Dev waits for response before continuing on this specific issue

**Hard stop rule — triggers when EITHER condition is met:**
- Same issue escalated to engineer **2 times** without resolution, OR
- Dev has been actively stuck on the same issue for **24 hours**

**When hard stop triggers:**
1. Dev **stops work immediately** on that task
2. Dev posts full summary to `queues/to-pm.md`:
   ```
   [date] [FROM: dev-fe] [TO: pm] [TASK: 1234]
   HARD STOP — Task 1234 escalation limit reached.
   Issue: [description]
   Escalation history:
     [date] — First escalation to engineer: [question]
     [date] — Engineer response: [response]
     [date] — Second escalation to engineer: [question]
     [date] — Engineer response: [response]
   Still blocked because: [reason]
   Awaiting operator assistance before resuming.
   ---
   ```
3. Task moves to Blocked in Asana
4. PM posts to `queues/to-operator.md`
5. **No further AI cycles spent on this task until operator resolves**

---

## Phase 4: QA

**Owner:** QA Agent  
**Lock phase:** `implementation` (QA runs concurrently)

### QA Task Flow

1. QA picks up task from In Review → moves to QA in Asana.
2. QA reads:
   - `KNOWN_ISSUES.md` — do not file failures against accepted limitations
   - `workspace/SPEC-CURRENT.md` — accepted requirements
   - `workspace/IMPLEMENTATION_GUIDE.md` — planned approach for this task
3. **If the PR includes UI changes and the task references a mockup:**
   - QA uses its vision capability (Gemini 3.1 Pro, native) to compare
     rendered UI against the referenced mockup
   - Visual deviations from the mockup that are not noted in `KNOWN_ISSUES.md`
     are QA failures
   - Retrieve mockup via Asana attachment (preferred) or `./workspace/mockups/`
4. QA reviews the PR against all references.

### QA Pass
```
[date] [FROM: qa] [TO: operator] [TASK: 1234]
Task 1234 — QA PASSED.
PR: [PR URL]
Branch: [branch name]
Verified against: SPEC-CURRENT.md + IMPLEMENTATION_GUIDE.md task 1
No known issues flagged.
---
```
- Move task to Completed in Asana.

### QA Fail
```
[date] [FROM: qa] [TO: engineer] [TASK: 1234]
Task 1234 — QA FAILED. [N] issues found.

Issue 1: [specific description — what was tested, what was expected, what happened]
Issue 2: ...
---
```
- Move task back to In Progress in Asana.
- Dev addresses failures, re-pushes. PR auto-updates. QA re-reviews.

---

## Phase 5: Operator Review and Merge

**Owner:** Operator (Human)  
**Lock phase:** `sprint-close` (after merge)

### Flow

1. Operator reviews `queues/to-operator.md` for QA-passed tasks.
2. Operator pulls the branch locally and reviews.
3. If satisfied: Operator tells QA agent — "Merge to main."
4. QA agent merges the PR to main.
5. QA agent (or devs) rebase all affected repos to main:
   ```bash
   git checkout main
   git pull
   git checkout [sprint-branch]
   git rebase main
   ```
6. Operator confirms rebase is clean — no conflicts, CI passes.
7. Operator updates `project-lock.json` → `phase: sprint-close`.

---

## Phase 6: Sprint Close

**Owner:** PM Agent  
**Lock phase:** `idle` (after close)

### Close Checklist

1. PM verifies all sprint tasks are in Completed in Asana.
2. PM archives completed tasks in Asana (close/archive — do not delete).
3. PM verifies `DECISIONS.md` has a complete record for this sprint.
4. PM verifies `KNOWN_ISSUES.md` is current with all accepted limitations.
5. PM writes sprint summary to `SHARED_MEMORY.md`:
   ```markdown
   ## [YYYY-MM-DD] Sprint [N] Close — pm-agent
   What was built: [summary]
   Issues accepted: [reference to KNOWN_ISSUES entries]
   Client sign-off: [yes/no — how confirmed]
   Carry-over notes: [anything relevant for the next sprint]
   ```
6. PM updates `STATE.md`:
   ```markdown
   # [Project Name] — Current State
   **Phase:** Idle — Sprint [N] closed. Ready for next requirements.
   **Last updated:** [date] by [pm-agent]
   ```
7. PM archives queue entries — prepend `[READ]` to processed entries. Do not delete lines.
8. PM updates `project-lock.json`:
   ```json
   {
     "phase": "idle",
     "sprint_id": null,
     "sprint_opened": null,
     "waiting_on": null,
     "last_updated": "[date]",
     "last_updated_by": "[pm-agent]",
     "context": "Sprint [N] closed. Ready for next requirements.",
     "blocked_tasks": []
   }
   ```
9. PM posts to `queues/to-operator.md`:
   ```
   [date] [FROM: pm] [TO: operator] [TASK: N/A]
   Sprint [N] closed. Asana clean. All queues archived.
   Ready to receive next set of requirements.
   ---
   ```

**PM does not accept new requirements until project-lock.json phase is idle.**

---

## Escalation Rules

| Situation | Action | Threshold |
|---|---|---|
| Client not responding to requirements | PM emails, then posts to to-operator.md | 48h no response |
| Dev blocked on task | Dev posts to to-engineer.md | Immediately when blocked |
| Same issue escalated 2x to engineer | Dev hard stop, PM posts to to-operator.md | 2 escalations |
| Dev stuck same issue | Dev hard stop, PM posts to to-operator.md | 24h active stuck |
| Task in Blocked with no movement | PM posts to to-operator.md | 48h |
| QA failing same task repeatedly | QA posts to to-engineer.md; PM monitors | — |

**No agent continues spending AI cycles on a blocked path. Stop, surface, wait.**

---

## Git and Branch Conventions

### Branch naming
```
[project-id]/[sprint-id]/[task-id]-[short-description]
```
Examples:
```
ezbi/sprint-3/1234-navbar-redesign
ezbi/sprint-3/1235-settings-migration
ezbi/sprint-3/1236-avatar-dropdown
```

### One branch per dev per sprint
- Dev works all their sprint tasks on a single branch
- Push after completing each task — PR auto-updates
- QA reviews the PR again after each push

### Hotfix branches (operator approval required)
- Only when a fix must ship independently of the current sprint
- Operator explicitly approves before dev creates a new branch
- Goes through the same QA → operator → merge flow

### PR conventions
- Title: `[[PROJECT-task-id]] Short description`
- Body must include:
  - Task ID
  - Asana task link
  - What changed (brief)
  - How to test
- One PR per dev per sprint — it accumulates all tasks as dev pushes

### After merge
1. QA merges PR to main (on operator instruction)
2. All affected repos rebase to main
3. Operator confirms clean

---

## Queue Message Format

Every queue entry must use this exact format:

```
[YYYY-MM-DD HH:MM] [FROM: agent-id] [TO: agent-id] [TASK: asana-task-id or N/A]
Message body. Be specific. Include task IDs, file names, error messages where relevant.
If multiple items, number them clearly.
---
```

### Rules
- Queues are append-only — never delete entries
- When processed, prepend `[READ]` to the entry — do not remove the line
- Archive at sprint close (mark READ, do not remove)
- Each agent checks their queue at session start before any other action
- `to-engineer-feasibility.md` is used **only** during requirements phase — keep it separate from escalations

---

## Agent Session Start Checklist

Every agent runs this at the start of every session:

1. Read `project-lock.json` — what phase are we in?
2. Read `queues/to-[my-role].md` — any pending messages?
3. If there are unread queue messages, address them before starting new work
4. If phase does not match my expected action, post to relevant queue and wait
5. If phase matches and no pending messages, proceed with current work

**The queue and phase check always comes before anything else.**

---
name: openclaw-recovery-manager
description: >
  OpenClaw Configuration Management & Emergency Recovery — configuration, skills, 
  and projects snapshot &recovery for OpenClaw. Use this skill whenever the user 
  wants to take a backup, restore a backup, set a dead-man's-switch recovery timer 
  for config changes, or test the recovery pipeline. Trigger phrases include: "set
  emergency recovery", "create snapshot", "take a backup before changes", "set
  a backout timer", "restore snapshot", "accept changes", "test emergency
  recovery", "run recovery test", "snapshot all skills", "snapshot global
  skills", "snapshot <agent> skills", "list skills snapshots", "restore all
  skills", "restore <agent> skills", "snapshot all projects", "snapshot
  <project> project", "list project snapshots", "restore all projects",
  "restore <project> project", "how does the rollback work", "what rollback
  commands", or any variation of wanting to safely change OpenClaw config,
  skills, or project configuration with an automatic (config) or manual
  (skills/projects) recovery fallback. Also trigger when the user asks about
  recovery, rollback, emergency restore, or testing the recovery system in the
  context of OpenClaw. Manages three independent subsystems (config with
  automatic dead-man's-switch watchdog, skills manual, projects manual) sharing
  one change log. Uses only Node.js (already required by OpenClaw), tar, and
  gzip. No additional dependencies.
---

# OpenClaw Recovery Manager

*(Skill directory / install name kept as `openclaw-emergency-rollback` to
avoid breaking existing installations.)*

The Recovery Manager provides three independent snapshot/restore subsystems
for an OpenClaw install:

1. **Config** — root `openclaw.json` + agent & global workspace identity
   files. Has an automatic **dead-man's-switch watchdog** (detached Node
   timer plus a native `gateway:startup` hook) that auto-restores the most
   recent config snapshot if the user doesn't accept their changes in time.
   This is the original recovery system and its behavior is unchanged.
2. **Skills** — global skills at `~/.openclaw/skills/` plus each configured
   agent's skills directory. Each target keeps its own independent 3-slot
   history. Manual snapshots and restores only.
3. **Projects** — each project referenced from `~/.openclaw/openclaw.json`,
   captured as its local project-level manifest and state (not working
   content). Each project keeps its own independent 3-slot history. Manual
   snapshots and restores only.

All three subsystems write to the same change log at
`~/.openclaw/rollback/logs/change.log`.

**Only the config subsystem is ever auto-restored.** The watchdog timer and
`gateway:startup` hook never touch skills or projects.

All scripts are Node.js (`.mjs`), which is already installed as an OpenClaw
dependency. No additional packages needed.

---

## First-Time Setup

If `~/.openclaw/rollback/` does not exist, run setup before anything else.
Read `references/SETUP.md` now and follow it completely before proceeding.

**Critical setup rule:** the restart command must be **detected, not asked
for**. Users often don't know how their own OpenClaw install was deployed,
and guessing `kill -USR1 1` when the machine is actually running a systemd
service silently disables auto-recovery. Probe the environment
(`systemctl --user is-active openclaw-gateway`, `docker compose ps`, PID 1
identity) before asking. Only ask the user if probes produce no confident
match, and always state what was detected and why before storing.
`references/SETUP.md` Step 1 has the full detection algorithm.

---

## Important Note on `pkill` and Docker/K8s

If you are running OpenClaw as the primary process in a container (PID 1),
**do not use `pkill -f openclaw`** to restart the gateway. If you use a
background Dead Man's Switch, `pkill` will match the path name of the
background script and kill your rescue job instantly.

Instead, use **`kill -USR1 1`** to surgically send the reload signal directly
to the root OpenClaw process.

## Logical Sabotage vs Invalid JSON

OpenClaw protects itself from invalid JSON by instantly hot-reloading its last
known good config before the gateway even restarts. To test destructive
recovery properly, you must use **Logical Sabotage**: feeding OpenClaw
perfectly valid JSON that logically breaks routing (e.g., a dummy token like
64 `f`s and poisoned workspace paths). This proves the rollback recovers from
logical failure states.

---

## Restart recovery via native OpenClaw hook *(config only)*

When the config gets sabotaged and OpenClaw restarts, the detached
`watchdog-timer` may die with the old process tree. That is expected.

To make recovery survive pod/container/local restarts, this skill installs a
native OpenClaw managed hook at `~/.openclaw/hooks/watchdog-recovery/`
listening to `gateway:startup`.

On every gateway startup, the hook reads persistent
`~/.openclaw/rollback/watchdog.json`:
1. If rollback is not armed, it exits immediately.
2. If rollback is armed and the hard expiry epoch has already passed, it runs
   `restore-if-armed.mjs` immediately.
3. If rollback is armed and the hard expiry epoch has not passed yet, it
   respawns `watchdog-timer.mjs` for the remaining seconds.

Because the system stores a hard absolute epoch (`expiryEpoch`) on persistent
disk, it doesn't matter how long the restart took: if OpenClaw restarts after
expiry, the hook restores immediately; if it restarts before expiry, the hook
recreates the timer.

This is the native cross-environment trigger for pod, Docker, and local
machine restarts. No AI, internet, cron, or external supervisor is required.

**The watchdog and `gateway:startup` hook only ever act on CONFIG snapshots.**
Skills and projects are never auto-restored.

---

## Session Start — Uptime Check (Run Every Session)

At the start of every session, run:

```bash
UPTIME=$(systemctl --user show openclaw-gateway \
  --property=ActiveEnterTimestampMonotonic 2>/dev/null \
  | awk -F= '{if($2>0) print int((systime()*1000000-$2)/1000000); else print 999}')

if [ "$UPTIME" = "999" ]; then
  UPTIME=$(ps -o etimes= -p $(pgrep -f "openclaw" 2>/dev/null) 2>/dev/null | tr -d ' ')
fi
```

If uptime is under 90 seconds AND `~/.openclaw/rollback/watchdog.json` exists
and shows `"armed": true`, the gateway just bounced. Open the session with the
**Watchdog Reminder** (see below).

If armed but uptime is over 90 seconds, still check and remind — the user may
have connected to a running session mid-timer.

If `armed: false` or watchdog file doesn't exist, start the session normally.

---

## Watchdog Reminder (show when watchdog is armed)

Run `~/.openclaw/rollback/scripts/watchdog-status.mjs` and display:

```
⚠️  Emergency recovery is armed.

Snapshot [1] "<label>" will auto-restore in ~XX minutes
unless you accept or extend.

Commands:
  • "accept changes"            — disarm watchdog, lock in current config
  • "extend recovery XX minutes" — add more time to the timer
  • "list snapshots"            — show all saved config snapshots
  • "restore snapshot 2"        — manually restore snapshot 2 or 3
  • "create snapshot"           — save current state as new snapshot [1]
```

---

## Config Subsystem Commands

### "create snapshot [description]"
Save current OpenClaw config as the new known-good restore point.

1. Run: `~/.openclaw/rollback/scripts/snapshot.mjs "<description>" "<ai_summary>"`
2. Write an AI summary (1–2 sentences) of the current config state by reading
   `~/.openclaw/openclaw.json` — note the default model, number of agents,
   any notable tools or channels — and pass it as the second argument.
3. Reply with snapshot confirmation showing all current snapshots (max 3):

```
✅ Snapshot saved.
[1] Apr 20 2:30 PM — <description>  ← restore target
[2] Apr 19 9:00 AM — <previous label>
[3] Apr 18 4:00 PM — <oldest label>
```

Slot [1] is always the most recent. Slot [3] is always the oldest. When a 4th
snapshot would be created, slot [3] is overwritten as the others shift.
Snapshots are never deleted without the user explicitly creating a new one
that pushes the oldest out. If the user wants to preserve all three, they can
copy slot [3] before creating a new snapshot.

### "set emergency recovery XX minutes" / "start emergency recovery XX minutes"
Arm the watchdog dead man's switch.

1. Run: `~/.openclaw/rollback/scripts/watchdog-set.mjs <minutes>`
2. Reply:

```
⏱️ Watchdog armed — XX minutes.
Snapshot [1] "<label>" auto-restores at <HH:MM> if not accepted.
Make your changes whenever you're ready.
```

If no snapshot exists yet, tell the user to create one first before arming.

### "extend recovery XX minutes"
Add time to the active watchdog timer.

1. Run: `~/.openclaw/rollback/scripts/watchdog-extend.mjs <minutes>`
2. Reply with new expiry time and minutes remaining.

### "accept changes"
Disarm the watchdog — user is happy with the current config.

1. Run: `~/.openclaw/rollback/scripts/watchdog-clear.mjs`
2. Reply:

```
✅ Watchdog disarmed. Your changes are locked in.
Say "create snapshot" to save this config as your new restore point [1].
```

### "list snapshots"
Show all saved config snapshots.

Read `~/.openclaw/rollback/manifest.json` and display:

```
Saved snapshots (most recent first):
[1] Apr 20 2:30 PM — "opus model working, github tool added"
    Config: claude-opus-4 default, 2 agents (main, coding), github MCP active
[2] Apr 19 9:00 AM — "initial clean setup"
    Config: claude-sonnet-4 default, 1 agent (main), no extra tools
[3] Apr 18 4:00 PM — "baseline before any changes"
    Config: claude-haiku-4 default, 1 agent (main)

Restore target: [1] (auto-restored if watchdog fires)
Watchdog: ARMED — 14m 32s remaining  [or: NOT ARMED]
```

### "restore snapshot [1|2|3]"
Manually restore a specific config snapshot immediately.

1. Confirm with user: "This will overwrite your current OpenClaw config with
   snapshot [N] '<label>' from <timestamp> and restart the gateway. Are you
   sure?"
2. On confirmation: run `~/.openclaw/rollback/scripts/restore.mjs <slot>`
3. Gateway restarts. Next session will detect uptime < 90 seconds.
4. If watchdog was armed, it is disarmed as part of restore.

### "test emergency recovery" / "run recovery test"
Run a destructive test of the full config recovery pipeline.

Read `references/TESTING.md` for the complete procedure. This test:
- Creates a dedicated test snapshot of the current config
- Arms a 2-minute watchdog
- Saves a manual recovery copy at `~/.openclaw/rollback/openclaw.recovery`
- Deliberately breaks `openclaw.json` to simulate a bad config change
- Restarts the gateway (which will fail to work properly)
- Waits for either the detached watchdog timer or the native
  `gateway:startup` hook to restore automatically

**This is destructive.** The user will lose access to their AI session for up
to 2 minutes while the test runs. Before running, confirm the user understands
the risks and has terminal/SSH access to manually recover if something goes
wrong.

---

## Skills Subsystem Commands *(manual only, never auto-restored)*

Skills targets are discovered dynamically from `~/.openclaw/openclaw.json` at
each snapshot. Each target (global + each configured agent) maintains its own
independent 3-slot history under `~/.openclaw/rollback/skills/<target>/`.

### "snapshot all skills [description]"
Snapshot global skills and every configured agent's skills simultaneously.
All targets receive the same user-provided description.

1. Run: `~/.openclaw/rollback/scripts/skills-snapshot.mjs all "<description>"`
2. Reply with a per-target result list.

### "snapshot global skills [description]"
Snapshot only `~/.openclaw/skills/`.

1. Run: `~/.openclaw/rollback/scripts/skills-snapshot.mjs global "<description>"`

### "snapshot <agent> skills [description]"
Snapshot a single agent's skills directory.

1. Run: `~/.openclaw/rollback/scripts/skills-snapshot.mjs <agent-id> "<description>"`

### "list skills snapshots"
Show every skills target with its 3-slot history.

1. Run: `~/.openclaw/rollback/scripts/skills-list.mjs`

### "list global skills snapshots" / "list <agent> skills snapshots"
Show the 3-slot history for a single target.

1. Run: `~/.openclaw/rollback/scripts/skills-list.mjs global`
   or `~/.openclaw/rollback/scripts/skills-list.mjs <agent-id>`

### "restore all skills"
Restore each target's slot 1 independently.

1. Run: `~/.openclaw/rollback/scripts/skills-restore.mjs all`

### "restore global skills [snapshot N]" / "restore <agent> skills [snapshot N]"
Restore slot 1 (default) or a specific slot for a single target.

1. Run: `~/.openclaw/rollback/scripts/skills-restore.mjs global [N]`
   or `~/.openclaw/rollback/scripts/skills-restore.mjs <agent-id> [N]`

**Skills snapshots are never auto-restored by the recovery timer or the
`gateway:startup` hook.** They do not arm the watchdog and the watchdog never
touches them.

---

## Projects Subsystem Commands *(manual only, never auto-restored)*

Project paths are discovered dynamically from `~/.openclaw/openclaw.json` at
each snapshot. Each project keeps its own 3-slot history under
`~/.openclaw/rollback/projects/<project>/`.

### "snapshot all projects [description]"
Snapshot every configured project. All receive the same description.

1. Run: `~/.openclaw/rollback/scripts/projects-snapshot.mjs all "<description>"`

### "snapshot <project name> project [description]"
Snapshot a single project.

1. Run: `~/.openclaw/rollback/scripts/projects-snapshot.mjs <project> "<description>"`

### "list project snapshots"
Show every project with its 3-slot history.

1. Run: `~/.openclaw/rollback/scripts/projects-list.mjs`

### "list <project name> snapshots"
Show the 3-slot history for a single project.

1. Run: `~/.openclaw/rollback/scripts/projects-list.mjs <project>`

### "restore all projects"
Restore each project's slot 1 independently.

1. Run: `~/.openclaw/rollback/scripts/projects-restore.mjs all`

### "restore <project name> project [snapshot N]"
Restore slot 1 (default) or a specific slot.

1. Run: `~/.openclaw/rollback/scripts/projects-restore.mjs <project> [N]`

**Projects snapshots are never auto-restored by the recovery timer or the
`gateway:startup` hook.**

---

## What Gets Backed Up

### Config snapshots (auto-restore target)
| File | Path |
|------|------|
| Master config | `~/.openclaw/openclaw.json` |
| Global workspace identity files | `~/.openclaw/workspace/*.md` *(whole-glob: SOUL.md, AGENTS.md, any new .md)* |
| Per-agent workspace files | `<agent_workspace>/SOUL.md` |
| | `<agent_workspace>/AGENTS.md` |
| | `<agent_workspace>/USER.md` |
| | `<agent_workspace>/IDENTITY.md` |
| | `<agent_workspace>/TOOLS.md` |
| | `<agent_workspace>/HEARTBEAT.md` |
| | `<agent_workspace>/BOOT.md` *(if present)* |

Workspace paths and agentIds are read dynamically from `openclaw.json` at
snapshot time — covers all configured agents automatically.

**Never captured by config snapshots:** credentials/, auth-profiles.json,
session history, memory logs, workspace content files, .env, Docker/K8s
environment config.

### Skills snapshots (manual only)
- `~/.openclaw/skills/` (global)
- Each configured agent's skills directory (from the agent's `skills` field,
  or `<agent_workspace>/skills/` if no explicit field)

### Project snapshots (manual only) — per project folder
| Item | Notes |
|------|-------|
| `openclaw.json` | project-local manifest & MCP spawn instructions |
| `mcp_config.json` | tool bridge to external services |
| `package.json` | local MCP server dependencies |
| `TASKS.json`, `PROCESSES.json`, `SPRINT_CURRENT.json` | state files, if present |
| `./tools/` | local MCP server source scripts |
| `./skills/` | project-local skills |
| `.openclaw/workspace.state.json` | project structural state |
| `./comms/` | **directory tree structure ONLY, no file content** |

**Explicitly excluded from project snapshots:**
- `node_modules/`
- `memory/`
- `auth-profiles.json`
- `~/.openclaw/` root (already covered by config snapshots)
- All working content, repositories, large data files, .env

---

## "how does the rollback work" / "what commands can I use" / "explain recovery manager"

Respond with this explanation:

```
OpenClaw Recovery Manager — How It Works

Three independent snapshot/restore subsystems sharing one change log.

1. CONFIG (with auto-restore watchdog — the original dead-man's-switch)
   "create snapshot [description]"       — save current config as restore point
   "set emergency recovery XX minutes"   — arm the auto-restore timer
   "extend recovery XX minutes"          — add time to active timer
   "accept changes"                      — disarm timer, keep current config
   "list snapshots"                      — show all 3 saved config snapshots
   "restore snapshot [1|2|3]"            — manually restore a specific snapshot
   "test emergency recovery"             — destructive test of the full pipeline

   If the user doesn't accept config changes before the timer fires, the
   system auto-restores snapshot [1] and restarts OpenClaw — no AI, no
   network, no user intervention required.

2. SKILLS (manual only — no auto-restore, no watchdog involvement)
   "snapshot all skills [description]"
   "snapshot global skills [description]"
   "snapshot <agent> skills [description]"
   "list skills snapshots"
   "list global skills snapshots" / "list <agent> skills snapshots"
   "restore all skills"
   "restore global skills [snapshot N]"
   "restore <agent> skills [snapshot N]"

3. PROJECTS (manual only — no auto-restore, no watchdog involvement)
   "snapshot all projects [description]"
   "snapshot <project> project [description]"
   "list project snapshots"
   "list <project> snapshots"
   "restore all projects"
   "restore <project> project [snapshot N]"

Each target (config / each skill target / each project) keeps its own
independent 3-slot history. Slot [1] is always most recent, slot [3] oldest;
a 4th snapshot pushes slot [3] out.

The auto-restore watchdog is CONFIG ONLY. It never touches skills or
projects.

Dependencies: Node.js (already installed with OpenClaw), tar, gzip.
```

---

## Change Log

Append to `~/.openclaw/rollback/logs/change.log` whenever any of the
following happens in any subsystem:
- A snapshot is taken (config, skills, or project)
- The watchdog is armed, extended, or cleared *(config only)*
- The user requests a gateway restart (note what changed and watchdog status)
- The gateway restart is confirmed complete
- A recovery test is started or completed
- A skills or project restore is run

Format:
```
[YYYY-MM-DD HH:MM:SS] <EVENT TYPE>
  <key: value details>
---
```

---

## Reference Files

- `references/SETUP.md` — Read this first if `~/.openclaw/rollback/` does not exist
- `references/TESTING.md` — Destructive recovery test procedure (config subsystem)
- `references/RESTORE.md` — Manual recovery instructions requiring no AI or scripts
- `scripts/` — Node.js scripts (`.mjs`) — no shell wrappers needed
- `hooks/watchdog-recovery/` — Native OpenClaw startup hook for config restart recovery

FILE:scripts/skills-list.mjs
#!/usr/bin/env node
// OpenClaw Recovery Manager — skills-list.mjs (SKILLS subsystem)
//
// Usage:
//   node skills-list.mjs            -> list all targets (global + each agent)
//   node skills-list.mjs global     -> list global skills history only
//   node skills-list.mjs <agent>    -> list that agent's skills history only

import { existsSync, readdirSync, statSync } from 'fs';
import { join } from 'path';
import {
  SKILLS_DIR,
  listAgents, resolveAgentSkillsDir, getGlobalSkillsDir,
  readTargetManifest, safeDirName
} from './utils.mjs';

const filter = process.argv[2]; // optional: "global" or agent id

function humanTs(isoZ) {
  if (!isoZ) return 'unknown';
  try {
    const d = new Date(isoZ);
    return d.toISOString().replace('T', ' ').replace(/Z?$/, '');
  } catch {
    return isoZ;
  }
}

function allTargets() {
  const out = [];
  out.push({ name: 'global', dir: getGlobalSkillsDir(), agentId: null });
  for (const a of listAgents()) {
    const d = resolveAgentSkillsDir(a);
    if (d) out.push({ name: safeDirName(a.id), dir: d, agentId: a.id });
  }
  return out;
}

let targets = allTargets();
if (filter) {
  if (filter === 'global') {
    targets = targets.filter(t => t.name === 'global');
  } else {
    const want = safeDirName(filter);
    targets = targets.filter(t => t.name === want);
  }
  if (targets.length === 0) {
    console.error(`ERROR: target "filter" not found. Known: global, listAgents().map(a => a.id).join(', ')`);
    process.exit(1);
  }
}

console.log('Skills Snapshots');
console.log('================');

for (const t of targets) {
  const targetDir = join(SKILLS_DIR, t.name);
  const mf = readTargetManifest(targetDir);
  const snaps = Array.isArray(mf.snapshots) ? mf.snapshots : [];
  const sourceTag = t.agentId ? `agent: t.agentId` : 'global';
  console.log(`\n[t.name]  (sourceTag)`);
  console.log(`  source: t.dir`);
  if (!existsSync(targetDir) || snaps.length === 0) {
    console.log('  (no snapshots)');
    continue;
  }
  snaps.sort((a, b) => a.slot - b.slot);
  for (const s of snaps) {
    const label = s.label || 'unlabeled';
    const ts = humanTs(s.timestamp);
    console.log(`  [s.slot] ts  —  "label"`);
  }
}
console.log('');

FILE:scripts/watchdog-set.mjs
#!/usr/bin/env node
// OpenClaw Emergency Rollback — watchdog-set.mjs
// Usage: node watchdog-set.mjs <minutes>
// Arms the watchdog for the given number of minutes.

import { join } from "path";
import { existsSync } from "fs";
import {
  ROLLBACK_DIR, WATCHDOG_FILE, CHANGE_LOG, RESTORE_LOG,
  writeJson, getManifest, appendLog, timestamp
} from './utils.mjs';

const minutes = parseInt(process.argv[2], 10);
if (!minutes || minutes <= 0) {
  console.error('Usage: node watchdog-set.mjs <minutes>');
  process.exit(1);
}

const now = Math.floor(Date.now() / 1000);
const expiry = now + minutes * 60;
const setAt = timestamp();
const expiryDate = new Date(expiry * 1000);
const expiryHuman = expiryDate.toISOString().replace(/\.\d+Z$/, 'Z');
const expiryDisplay = expiryDate.toLocaleTimeString('en-US', { hour: '2-digit', minute: '2-digit', hour12: true });

// Read target snapshot label
const manifest = getManifest();
const snap1 = manifest.snapshots.find(s => s.slot === 1);
const targetLabel = snap1 ? snap1.label : 'no snapshot saved';

// Write watchdog.json
writeJson(WATCHDOG_FILE, {
  armed: true,
  setAt,
  expiryEpoch: expiry,
  expiryHuman,
  minutesSet: minutes,
  targetSnapshot: 'snapshot-1',
  targetLabel
});


import { spawn } from 'child_process';
const timerScript = join(ROLLBACK_DIR, 'scripts', 'watchdog-timer.mjs');
if (existsSync(timerScript)) {
  const child = spawn(process.execPath, [timerScript, String(minutes)], {
    detached: true,
    stdio: 'ignore',
    env: { ...process.env, WATCHDOG_SOURCE: 'watchdog-set' }
  });
  child.unref();
  appendLog(RESTORE_LOG, `WATCHDOG SET — spawned watchdog timer pid=child.pid || 'unknown' minutes=minutes`);
} else {
  appendLog(RESTORE_LOG, "WATCHDOG SET ERROR — watchdog-timer.mjs missing, timer won't fire.");
  console.error("WARNING: watchdog-timer.mjs missing, timer won't fire.");
}

// Log
appendLog(CHANGE_LOG,
  `WATCHDOG ARMED\n  Minutes: minutes\n  Expiry: expiryHuman\n  Target: snapshot-1 — "targetLabel"`
);

console.log(`Watchdog armed — minutes minutes. Expires at expiryDisplay. Target: targetLabel`);

FILE:scripts/projects-list.mjs
#!/usr/bin/env node
// OpenClaw Recovery Manager — projects-list.mjs (PROJECTS subsystem)
//
// Usage:
//   node projects-list.mjs              -> list all projects + histories
//   node projects-list.mjs <project>    -> list that project's 3-slot history

import { existsSync } from 'fs';
import { join } from 'path';
import {
  PROJECTS_DIR,
  listProjects, readTargetManifest, safeDirName
} from './utils.mjs';

const filter = process.argv[2];

function humanTs(isoZ) {
  if (!isoZ) return 'unknown';
  try {
    return new Date(isoZ).toISOString().replace('T', ' ').replace(/Z?$/, '');
  } catch { return isoZ; }
}

let projects = listProjects();
if (filter) {
  const want = safeDirName(filter);
  projects = projects.filter(p => p.name === filter || safeDirName(p.name) === want);
  if (projects.length === 0) {
    console.error(`ERROR: project "filter" not found in openclaw.json.`);
    process.exit(1);
  }
}

console.log('Project Snapshots');
console.log('=================');

if (projects.length === 0) {
  console.log('\n(no projects configured in openclaw.json)');
  process.exit(0);
}

for (const p of projects) {
  const targetDir = join(PROJECTS_DIR, safeDirName(p.name));
  const mf = readTargetManifest(targetDir);
  const snaps = Array.isArray(mf.snapshots) ? mf.snapshots : [];
  console.log(`\n[p.name]`);
  console.log(`  path: p.path`);
  if (!existsSync(targetDir) || snaps.length === 0) {
    console.log('  (no snapshots)');
    continue;
  }
  snaps.sort((a, b) => a.slot - b.slot);
  for (const s of snaps) {
    console.log(`  [s.slot] humanTs(s.timestamp)  —  "s.label || 'unlabeled'"`);
  }
}
console.log('');

FILE:scripts/skills-restore.mjs
#!/usr/bin/env node
// OpenClaw Recovery Manager — skills-restore.mjs (SKILLS subsystem)
//
// Usage:
//   node skills-restore.mjs all                     -> restore slot 1 for every target
//   node skills-restore.mjs global [slot]           -> restore global, defaults to slot 1
//   node skills-restore.mjs <agent> [slot]          -> restore a single agent's skills
//
// MANUAL ONLY. This script is never invoked by the watchdog timer or the
// gateway:startup hook. Skills are restored entirely by user request.
//
// Restore extracts each target's tar.gz directly to / so the absolute paths
// inside the archive go back where they came from.

import { existsSync } from 'fs';
import { join } from 'path';
import {
  SKILLS_DIR, CHANGE_LOG,
  listAgents, resolveAgentSkillsDir, getGlobalSkillsDir,
  readTargetManifest, safeDirName,
  extractArchive, appendLog, timestamp
} from './utils.mjs';

const mode = process.argv[2];
const slotArg = parseInt(process.argv[3], 10);

if (!mode) {
  console.error('Usage: node skills-restore.mjs <all|global|AGENT_NAME> [slot]');
  process.exit(1);
}

function allTargets() {
  const out = [];
  out.push({ name: 'global', dir: getGlobalSkillsDir(), agentId: null });
  for (const a of listAgents()) {
    const d = resolveAgentSkillsDir(a);
    if (d) out.push({ name: safeDirName(a.id), dir: d, agentId: a.id });
  }
  return out;
}

function resolveSingle(mode) {
  if (mode === 'global') return { name: 'global', dir: getGlobalSkillsDir(), agentId: null };
  const agents = listAgents();
  const agent = agents.find(a => a.id === mode || safeDirName(a.id) === safeDirName(mode));
  if (!agent) {
    console.error(`ERROR: target "mode" not found in openclaw.json.`);
    console.error(`Known: global, agents.map(a => a.id).join(', ')`);
    process.exit(1);
  }
  const d = resolveAgentSkillsDir(agent);
  if (!d) {
    console.error(`ERROR: agent "agent.id" has no resolvable skills directory.`);
    process.exit(1);
  }
  return { name: safeDirName(agent.id), dir: d, agentId: agent.id };
}

const restoreList = mode === 'all'
  ? allTargets().map(t => ({ target: t, slot: 1 }))
  : [{ target: resolveSingle(mode), slot: Number.isFinite(slotArg) && slotArg > 0 ? slotArg : 1 }];

const ts = timestamp();
let okCount = 0;
const report = [];

for (const { target, slot } of restoreList) {
  const targetDir = join(SKILLS_DIR, target.name);
  const mf = readTargetManifest(targetDir);
  const entry = (mf.snapshots || []).find(s => s.slot === slot);
  if (!entry) {
    report.push({ name: target.name, status: 'skipped', reason: `no snapshot in slot slot` });
    continue;
  }
  const zipPath = join(targetDir, entry.file);
  if (!existsSync(zipPath)) {
    report.push({ name: target.name, status: 'failed', reason: `archive missing: zipPath` });
    continue;
  }
  const exit = extractArchive(zipPath);
  if (exit === 0) {
    okCount++;
    report.push({ name: target.name, status: 'ok', slot, label: entry.label, timestamp: entry.timestamp });
    appendLog(CHANGE_LOG,
      `SKILLS RESTORED\n  Target: target.name\n  Slot: slot\n  Label: "entry.label || ''"\n  Snapshot timestamp: entry.timestamp || 'unknown'\n  Extracted to: /`
    );
  } else {
    report.push({ name: target.name, status: 'failed', reason: `tar exit code exit` });
    appendLog(CHANGE_LOG,
      `SKILLS RESTORE FAILED\n  Target: target.name\n  Slot: slot\n  tar exit: exit`
    );
  }
}

// Report
console.log(`Skills restore — ts`);
for (const r of report) {
  if (r.status === 'ok') {
    console.log(`  ✓ r.name.padEnd(20) slot r.slot  "r.label || ''"  (r.timestamp || '')`);
  } else {
    console.log(`  ⚠ r.name.padEnd(20) r.status: r.reason`);
  }
}

if (okCount === 0) {
  console.error('\nNo skills were restored.');
  process.exit(1);
}
console.log(`\nokCount target(s) restored.`);
console.log('NOTE: skills restores never touch config or the watchdog timer.');

FILE:scripts/watchdog-extend.mjs
#!/usr/bin/env node
// OpenClaw Emergency Rollback — watchdog-extend.mjs
// Usage: node watchdog-extend.mjs <additional_minutes>

import {
  WATCHDOG_FILE, CHANGE_LOG,
  getWatchdog, writeJson, appendLog
} from './utils.mjs';

const addMinutes = parseInt(process.argv[2], 10);
if (!addMinutes || addMinutes <= 0) {
  console.error('Usage: node watchdog-extend.mjs <additional_minutes>');
  process.exit(1);
}

const watchdog = getWatchdog();
if (!watchdog.armed) {
  console.error('ERROR: Watchdog is not armed. Use watchdog-set first.');
  process.exit(1);
}

const oldExpiry = watchdog.expiryEpoch;
const newExpiry = oldExpiry + addMinutes * 60;
const newExpiryDate = new Date(newExpiry * 1000);
const newExpiryHuman = newExpiryDate.toISOString().replace(/\.\d+Z$/, 'Z');
const newExpiryDisplay = newExpiryDate.toLocaleTimeString('en-US', { hour: '2-digit', minute: '2-digit', hour12: true });

const now = Math.floor(Date.now() / 1000);
const remaining = Math.floor((newExpiry - now) / 60);

// Update watchdog.json
watchdog.expiryEpoch = newExpiry;
watchdog.expiryHuman = newExpiryHuman;
writeJson(WATCHDOG_FILE, watchdog);

// No need to touch the running timer process. It calls restore-if-armed.mjs,
// which reads expiry from watchdog.json dynamically when it fires.

appendLog(CHANGE_LOG,
  `WATCHDOG EXTENDED\n  Added: addMinutes minutes\n  New expiry: newExpiryHuman\n  Remaining: ~remainingm`
);

console.log(`Watchdog extended. New expiry: newExpiryDisplay (~remainingm remaining)`);

FILE:scripts/recovery-test.mjs
#!/usr/bin/env node
// OpenClaw Emergency Rollback — recovery-test.mjs
// Usage: node recovery-test.mjs <subcommand>
//
// Subcommands:
//   preflight      — check all dependencies and system readiness
//   save-recovery  — copy current openclaw.json to openclaw.recovery
//   sabotage       — deliberately break openclaw.json (makes it invalid JSON)
//   verify         — check if openclaw.json was restored (is valid JSON again)

import { existsSync, readFileSync, writeFileSync, copyFileSync, statSync } from 'fs';
import { execSync } from 'child_process';
import {
  ROLLBACK_DIR, RECOVERY_FILE, CHANGE_LOG, RESTORE_LOG,
  getConfig, getOpenclawJson, getManifest, getWatchdog,
  readJson, appendLog
} from './utils.mjs';

const subcommand = process.argv[2];
if (!subcommand) {
  console.error('Usage: node recovery-test.mjs <preflight|save-recovery|sabotage|verify>');
  process.exit(1);
}

const OC_JSON = getOpenclawJson();

switch (subcommand) {

  case 'preflight': {
    console.log('=== Recovery Test Pre-Flight Check ===');
    let pass = true;

    // Check node (we're running, so it's there)
    console.log(`  ✓ node found: process.execPath`);

    // Check zip/unzip
    for (const tool of ['tar', 'gzip']) {
      try {
        execSync(`command -v tool`, { stdio: 'ignore' });
        console.log(`  ✓ tool found`);
      } catch {
        console.log(`  ✗ tool NOT FOUND — install before proceeding`);
        pass = false;
      }
    }

    console.log('  ✓ no cron dependency — watchdog uses detached Node timers and startup checks');

    // Check rollback directory
    if (existsSync(ROLLBACK_DIR)) {
      console.log('  ✓ rollback directory exists');
    } else {
      console.log('  ✗ rollback directory missing — run setup first');
      pass = false;
    }

    // Check manifest
    const manifest = getManifest();
    console.log(`  ✓ manifest.json (manifest.snapshots.length snapshots)`);

    // Check scripts
    const scripts = ['snapshot.mjs', 'restore.mjs', 'restore-if-armed.mjs', 'watchdog-set.mjs', 'watchdog-clear.mjs'];
    for (const s of scripts) {
      const p = `ROLLBACK_DIR/scripts/s`;
      if (existsSync(p)) {
        console.log(`  ✓ s exists`);
      } else {
        console.log(`  ✗ s missing`);
        pass = false;
      }
    }

    // Check openclaw.json
    if (existsSync(OC_JSON)) {
      const verifyParsed = readJson(OC_JSON);
      if (readJson(OC_JSON)) {
        console.log('  ✓ openclaw.json exists and is valid JSON');
      } else {
        console.log('  ✗ openclaw.json exists but is NOT valid JSON');
        pass = false;
      }
    } else {
      console.log(`  ✗ openclaw.json not found at OC_JSON`);
      pass = false;
    }

    // Check restart command
    const config = getConfig();
    if (config.restartCommand) {
      console.log(`  ✓ restart command: config.restartCommand`);
    } else {
      console.log('  ✗ restart command not configured');
      pass = false;
    }

    console.log('');
    if (pass) {
      console.log('All checks passed. Ready to test.');
    } else {
      console.log('Some checks FAILED. Fix the issues above before testing.');
      process.exit(1);
    }
    break;
  }

  case 'save-recovery': {
    if (!existsSync(OC_JSON)) {
      console.error(`ERROR: openclaw.json not found at OC_JSON`);
      process.exit(1);
    }

    copyFileSync(OC_JSON, RECOVERY_FILE);
    const config = getConfig();
    console.log(`Recovery copy saved: RECOVERY_FILE`);
    console.log('');
    console.log('If the test fails, restore manually with:');
    console.log(`  cp RECOVERY_FILE OC_JSON`);
    console.log(`  config.restartCommand || 'kill -USR1 1'`);

    appendLog(CHANGE_LOG,
      `RECOVERY TEST — MANUAL BACKUP SAVED\n  Source: OC_JSON\n  Backup: RECOVERY_FILE`
    );
    break;
  }

  case 'sabotage': {
    if (!existsSync(OC_JSON)) {
      console.error(`ERROR: openclaw.json not found at OC_JSON`);
      process.exit(1);
    }

    if (!existsSync(RECOVERY_FILE)) {
      console.error(`ERROR: No recovery copy found at RECOVERY_FILE`);
      console.error('Run "node recovery-test.mjs save-recovery" first.');
      process.exit(1);
    }

    const originalSize = statSync(OC_JSON).size;
    const parsed = readJson(OC_JSON);

    // 1. Poison the gateway auth token 
    if (parsed.gateway && parsed.gateway.auth) {
      parsed.gateway.auth.token = 'ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff';
    }

    // 2. Poison the agent workspace paths to break routing logically
    if (parsed.agents && Array.isArray(parsed.agents.list)) {
      parsed.agents.list.forEach(agent => {
        if (agent.workspace) {
          agent.workspace += 'x';
        }
      });
    }

    // Safe write
    writeFileSync(OC_JSON, JSON.stringify(parsed, null, 2));

    console.log('Config sabotaged logically. openclaw.json is VALID JSON, but contains a poisoned token and modified agent workspaces.');
    console.log(`Original size: originalSize bytes`);
    console.log('The watchdog should restore it automatically when the timer expires.');

    appendLog(CHANGE_LOG,
      `RECOVERY TEST — CONFIG SABOTAGED\n  File: OC_JSON\n  Method: Changed gateway token to ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff and poisoned workspace paths\n  Original size: originalSize bytes`
    );
    break;
  }

  case 'verify': {
    console.log('=== Recovery Test Verification ===');

    if (!existsSync(OC_JSON)) {
      console.log('  ✗ openclaw.json not found');
      console.log('RESULT: FAILED');
      process.exit(1);
    }

    const testParsed = readJson(OC_JSON);
    if (!testParsed) {
      console.log('  ✗ openclaw.json exists but is NOT valid JSON (this is unexpected)');
      process.exit(1);
    }
    
    if (testParsed.gateway?.auth?.token === 'ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff' || testParsed.agents?.list?.[0]?.workspace?.endsWith('x')) {
      console.log('  ✗ ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff still present OR workspaces still poisoned');
      console.log('  The Dead Man\'s Switch has NOT restored the config yet.');
      console.log('');
      console.log('RESULT: NOT YET RESTORED — wait longer for background switch to fire');
      console.log('Debug:');
      console.log(`  cat RESTORE_LOG       # restore attempts`);
      console.log(`  node ROLLBACK_DIR/scripts/watchdog-status.mjs  # timer`);
      process.exit(1);
    }

    if (testParsed) {
      console.log('  ✓ openclaw.json is valid JSON');
    } else {
      console.log('  ✗ openclaw.json exists but is NOT valid JSON');
      console.log('RESULT: PARTIAL — file may have been partially restored');
      process.exit(1);
    }

    const watchdog = getWatchdog();
    console.log(`  Watchdog armed: watchdog.armed`);

    if (existsSync(RESTORE_LOG)) {
      const log = readFileSync(RESTORE_LOG, 'utf8');
      const lastRestore = log.split('\n').filter(l => l.includes('RESTORE COMPLETE')).pop();
      if (lastRestore) console.log(`  Last restore: lastRestore.trim()`);
    }

    console.log('');
    console.log('RESULT: PASSED — recovery test successful');

    appendLog(CHANGE_LOG,
      `RECOVERY TEST — VERIFIED PASSED\n  openclaw.json: valid JSON, marker removed\n  Watchdog armed: watchdog.armed`
    );
    break;
}

  default:
    console.error(`Unknown subcommand: subcommand`);
    console.error('Usage: node recovery-test.mjs <preflight|save-recovery|sabotage|verify>');
    process.exit(1);
}

FILE:scripts/projects-snapshot.mjs
#!/usr/bin/env node
// OpenClaw Recovery Manager — projects-snapshot.mjs (PROJECTS subsystem)
//
// Usage:
//   node projects-snapshot.mjs all           "<description>"
//   node projects-snapshot.mjs <project>     "<description>"
//
// Project paths are discovered dynamically from ~/.openclaw/openclaw.json.
// Each project has its own 3-slot history under
// ~/.openclaw/rollback/projects/<project-name>/.
//
// Projects snapshots are ENTIRELY MANUAL. They are never auto-restored by the
// watchdog timer or the gateway:startup hook.
//
// What gets captured PER PROJECT (if present inside the project folder):
//   - openclaw.json                (project-level manifest)
//   - mcp_config.json              (external tool bridge config)
//   - package.json                 (local MCP server deps manifest)
//   - TASKS.json, PROCESSES.json, SPRINT_CURRENT.json   (state files)
//   - ./tools/                     (local MCP server source scripts)
//   - ./skills/                    (project-local skills)
//   - .openclaw/workspace.state.json   (project structural state)
//   - ./comms/                     (TREE STRUCTURE ONLY — no file content)
//
// What is explicitly NOT captured:
//   - node_modules/
//   - memory/
//   - auth-profiles.json
//   - ~/.openclaw/ root contents (covered by config snapshots)
//   - working content, repos, large data files

import { existsSync, statSync, mkdirSync, writeFileSync, copyFileSync, readdirSync, renameSync, unlinkSync } from 'fs';
import { join, dirname, basename } from 'path';
import { execSync } from 'child_process';
import { mkdtempSync } from 'fs';
import { tmpdir } from 'os';
import {
  PROJECTS_DIR, CHANGE_LOG,
  listProjects, safeDirName,
  rotateIntoTarget, appendLog, timestamp
} from './utils.mjs';

const mode = process.argv[2];
const description = process.argv[3] || 'unlabeled';

if (!mode) {
  console.error('Usage: node projects-snapshot.mjs <all|PROJECT_NAME> "<description>"');
  process.exit(1);
}

const PROJECT_FILES = [
  'openclaw.json',
  'mcp_config.json',
  'package.json',
  'TASKS.json',
  'PROCESSES.json',
  'SPRINT_CURRENT.json'
];

const PROJECT_SUBDIR_FULL = ['tools', 'skills'];
// .openclaw/workspace.state.json — single specific file under a hidden subdir
// comms/ — tree structure only

const EXCLUDE_IN_SUBDIR = ['node_modules', 'memory', '.git', '.env']; // extra-defensive

/**
 * Build a per-project staged archive.
 * Returns path to tar.gz, or null if the project produced nothing worth saving.
 */
function snapshotProject(project) {
  const root = project.path;
  if (!existsSync(root) || !statSync(root).isDirectory()) {
    return { staged: null, reason: `project path not found: root` };
  }

  const stage = mkdtempSync(join(tmpdir(), 'oc-proj-'));
  let captured = 0;

  // 1) Individual config / state files at project root
  for (const f of PROJECT_FILES) {
    const src = join(root, f);
    if (existsSync(src) && statSync(src).isFile()) {
      const dest = join(stage, src);
      mkdirSync(dirname(dest), { recursive: true });
      copyFileSync(src, dest);
      captured++;
    }
  }

  // 2) Full-content subdirs: tools/ and skills/ (with exclude list)
  for (const subdir of PROJECT_SUBDIR_FULL) {
    const src = join(root, subdir);
    if (existsSync(src) && statSync(src).isDirectory()) {
      const dest = join(stage, src);
      mkdirSync(dest, { recursive: true });
      copyRecursiveExclude(src, dest, EXCLUDE_IN_SUBDIR);
      captured++;
    }
  }

  // 3) .openclaw/workspace.state.json
  const wsState = join(root, '.openclaw', 'workspace.state.json');
  if (existsSync(wsState) && statSync(wsState).isFile()) {
    const dest = join(stage, wsState);
    mkdirSync(dirname(dest), { recursive: true });
    copyFileSync(wsState, dest);
    captured++;
  }

  // 4) comms/ — tree structure only, no file content
  const commsDir = join(root, 'comms');
  if (existsSync(commsDir) && statSync(commsDir).isDirectory()) {
    const dest = join(stage, commsDir);
    mkdirSync(dest, { recursive: true });
    copyStructureOnly(commsDir, dest);
    captured++;
  }

  if (captured === 0) {
    try { execSync(`rm -rf "stage"`); } catch {}
    return { staged: null, reason: 'project contains none of the backed-up files' };
  }

  const outZip = join(tmpdir(), `oc-proj-Date.now()-Math.random().toString(36).slice(2, 8).tar.gz`);
  try { unlinkSync(outZip); } catch {}
  execSync(`cd "stage" && tar -czf "outZip" .`, { stdio: 'ignore' });
  try { execSync(`rm -rf "stage"`); } catch {}
  return { staged: outZip, captured };
}

function copyRecursiveExclude(src, dest, excludeNames = []) {
  const exclude = new Set(excludeNames);
  const entries = readdirSync(src, { withFileTypes: true });
  for (const ent of entries) {
    if (exclude.has(ent.name)) continue;
    const s = join(src, ent.name);
    const d = join(dest, ent.name);
    if (ent.isDirectory()) {
      mkdirSync(d, { recursive: true });
      copyRecursiveExclude(s, d, excludeNames);
    } else if (ent.isFile()) {
      copyFileSync(s, d);
    }
  }
}

function copyStructureOnly(src, dest) {
  let anyDir = false;
  for (const ent of readdirSync(src, { withFileTypes: true })) {
    if (!ent.isDirectory()) continue;
    anyDir = true;
    const s = join(src, ent.name);
    const d = join(dest, ent.name);
    mkdirSync(d, { recursive: true });
    copyStructureOnly(s, d);
  }
  if (!anyDir) {
    try { writeFileSync(join(dest, '.dirtree'), ''); } catch {}
  }
}

// -- Resolve targets --
const allProjects = listProjects();

let targets;
if (mode === 'all') {
  targets = allProjects;
  if (targets.length === 0) {
    console.error('No projects found in openclaw.json.');
    process.exit(1);
  }
} else {
  const wanted = safeDirName(mode);
  const found = allProjects.find(p => p.name === mode || safeDirName(p.name) === wanted);
  if (!found) {
    console.error(`ERROR: project "mode" not found in openclaw.json.`);
    console.error(`Known projects: allProjects.map(p => p.name).join(', ') || '(none)'`);
    process.exit(1);
  }
  targets = [found];
}

const ts = timestamp();
const results = [];

for (const project of targets) {
  const res = snapshotProject(project);
  if (!res.staged) {
    results.push({ name: project.name, status: 'skipped', reason: res.reason });
    continue;
  }
  const targetDir = join(PROJECTS_DIR, safeDirName(project.name));
  const manifest = rotateIntoTarget(targetDir, res.staged, {
    label: description,
    timestamp: ts,
    ai_summary: '',
    extra: { source: project.path, projectName: project.name }
  });
  appendLog(CHANGE_LOG,
    `PROJECT SNAPSHOT TAKEN\n  Project: project.name\n  Source: project.path\n  Slot: 1\n  Description: "description"`
  );
  results.push({ name: project.name, status: 'ok', slot: 1, path: project.path, slots: manifest.snapshots.length });
}

console.log(`Projects snapshot — ts`);
for (const r of results) {
  if (r.status === 'ok') {
    console.log(`  ✓ r.name.padEnd(20) slot 1  (r.path)`);
  } else {
    console.log(`  ⚠ r.name.padEnd(20) r.status: r.reason`);
  }
}

const ok = results.filter(r => r.status === 'ok').length;
if (ok === 0) {
  console.error('No project snapshots taken.');
  process.exit(1);
}
console.log(`\nok project(s) snapshotted. Description: "description"`);

FILE:scripts/watchdog-clear.mjs
#!/usr/bin/env node
// OpenClaw Emergency Rollback — watchdog-clear.mjs
// Disarms the watchdog. Called when user accepts changes.

import { execSync } from 'child_process';
import {
  WATCHDOG_FILE, CHANGE_LOG,
  getWatchdog, writeJson, appendLog
} from './utils.mjs';

const watchdog = getWatchdog();
const now = Math.floor(Date.now() / 1000);

// Calculate time remaining at disarm
let remainingMsg = 'unknown';
if (watchdog.expiryEpoch) {
  if (watchdog.expiryEpoch > now) {
    const secs = watchdog.expiryEpoch - now;
    remainingMsg = `Math.floor(secs / 60)m secs % 60s`;
  } else {
    remainingMsg = 'already expired';
  }
}

// Disarm watchdog.json
watchdog.armed = false;
writeJson(WATCHDOG_FILE, watchdog);


// Kill any detached watchdog-timer.mjs processes
try {
  execSync('pkill -f watchdog-timer.mjs || true', { stdio: 'ignore' });
} catch {}

appendLog(CHANGE_LOG,
  `WATCHDOG DISARMED\n  Disarmed by: user accepted changes\n  Time remaining: remainingMsg`
);

console.log(`Watchdog disarmed. Time remaining was: remainingMsg`);

FILE:scripts/restore-if-armed.mjs
#!/usr/bin/env node
// OpenClaw Emergency Rollback — restore-if-armed.mjs
// Called by watchdog timer and native OpenClaw startup hook.
// Checks if watchdog is armed and timer has expired. Fires restore if so.

import { existsSync } from 'fs';
import { execSync } from 'child_process';
import { join } from 'path';
import { ROLLBACK_DIR, WATCHDOG_FILE, RESTORE_LOG, getWatchdog, appendLog } from './utils.mjs';

appendLog(RESTORE_LOG, `RESTORE-IF-ARMED — entered pid=process.pid ppid=process.ppid source=process.env.WATCHDOG_SOURCE || 'direct' triggeredByTimer=process.env.WATCHDOG_TRIGGERED === '1' triggeredByStartup=process.env.RESTART_TRIGGERED === '1'`);

if (!existsSync(WATCHDOG_FILE)) {
  appendLog(RESTORE_LOG, 'RESTORE-IF-ARMED — watchdog.json missing, exiting');
  process.exit(0);
}

const watchdog = getWatchdog();

if (!watchdog.armed) {
  appendLog(RESTORE_LOG, 'RESTORE-IF-ARMED — watchdog not armed, exiting');
  process.exit(0);
}

const now = Math.floor(Date.now() / 1000);
const expiry = watchdog.expiryEpoch || 0;
appendLog(RESTORE_LOG, `RESTORE-IF-ARMED — watchdog armed, now=now, expiry=expiry, remaining=expiry - nows`);

if (now >= expiry) {
  appendLog(RESTORE_LOG, 'RESTORE-IF-ARMED — watchdog armed and expired, triggering restore');
  process.env.RESTART_TRIGGERED = '1';
  const restoreScript = join(ROLLBACK_DIR, 'scripts', 'restore.mjs');
  try {
    execSync(`node "restoreScript"`, { stdio: 'inherit', env: { ...process.env, RESTART_TRIGGERED: '1' } });
  } catch (e) {
    appendLog(RESTORE_LOG, `RESTORE-IF-ARMED ERROR — restore.mjs failed: e?.status || 'unknown' e?.message || e`);
    process.exit(1);
  }
} else {
  const remaining = expiry - now;
  appendLog(RESTORE_LOG, `RESTORE-IF-ARMED — watchdog not expired, respawning timer for remainings`);
  const timerScript = join(ROLLBACK_DIR, 'scripts', 'watchdog-timer.mjs');
  if (existsSync(timerScript)) {
    const { spawn } = await import('child_process');
    const child = spawn(process.execPath, [timerScript, '0', String(remaining)], {
      detached: true,
      stdio: 'ignore',
      env: { ...process.env, WATCHDOG_SOURCE: 'restore-if-armed' }
    });
    child.unref();
    appendLog(RESTORE_LOG, `RESTORE-IF-ARMED — respawned watchdog timer pid=child.pid || 'unknown' remaining=remainings`);
  } else {
    appendLog(RESTORE_LOG, 'RESTORE-IF-ARMED ERROR — watchdog-timer.mjs missing, cannot respawn timer');
  }
}

process.exit(0);

FILE:scripts/skills-snapshot.mjs
#!/usr/bin/env node
// OpenClaw Recovery Manager — skills-snapshot.mjs (SKILLS subsystem)
//
// Usage:
//   node skills-snapshot.mjs all      "<description>"
//   node skills-snapshot.mjs global   "<description>"
//   node skills-snapshot.mjs <agent>  "<description>"
//
// Snapshots skills directories. Each target (global + each configured agent)
// maintains its own independent 3-slot history under ~/.openclaw/rollback/skills/.
//
// Skills snapshots are ENTIRELY MANUAL. They are never auto-restored by the
// watchdog timer or the gateway:startup hook. That remains config-only.

import { existsSync, statSync } from 'fs';
import { join } from 'path';
import {
  SKILLS_DIR, CHANGE_LOG,
  listAgents, resolveAgentSkillsDir, getGlobalSkillsDir,
  makeStagedArchive, rotateIntoTarget, safeDirName,
  appendLog, timestamp
} from './utils.mjs';

const mode = process.argv[2];
const description = process.argv[3] || 'unlabeled';

if (!mode) {
  console.error('Usage: node skills-snapshot.mjs <all|global|AGENT_NAME> "<description>"');
  process.exit(1);
}

/** Build the list of {name, dir} targets this invocation should snapshot. */
function resolveTargets(mode) {
  const targets = [];
  const globalDir = getGlobalSkillsDir();
  const agents = listAgents();

  if (mode === 'all') {
    targets.push({ name: 'global', dir: globalDir });
    for (const a of agents) {
      const d = resolveAgentSkillsDir(a);
      if (d) targets.push({ name: safeDirName(a.id), dir: d, agentId: a.id });
    }
    return targets;
  }
  if (mode === 'global') {
    return [{ name: 'global', dir: globalDir }];
  }
  // Agent by id
  const agent = agents.find(a => a.id === mode || safeDirName(a.id) === safeDirName(mode));
  if (!agent) {
    console.error(`ERROR: agent "mode" not found in openclaw.json.`);
    console.error(`Known agents: agents.map(a => a.id).join(', ') || '(none)'`);
    process.exit(1);
  }
  const d = resolveAgentSkillsDir(agent);
  if (!d) {
    console.error(`ERROR: agent "agent.id" has no skills directory (no agent.skills and no workspace).`);
    process.exit(1);
  }
  return [{ name: safeDirName(agent.id), dir: d, agentId: agent.id }];
}

const targets = resolveTargets(mode);
const ts = timestamp();
const results = [];

for (const t of targets) {
  if (!existsSync(t.dir) || !statSync(t.dir).isDirectory()) {
    results.push({ name: t.name, status: 'skipped', reason: `skills dir not found at t.dir` });
    continue;
  }

  const staged = makeStagedArchive([t.dir]);
  if (!staged) {
    results.push({ name: t.name, status: 'skipped', reason: 'skills dir empty or unreadable' });
    continue;
  }

  const targetDir = join(SKILLS_DIR, t.name);
  const manifest = rotateIntoTarget(targetDir, staged, {
    label: description,
    timestamp: ts,
    ai_summary: '',
    extra: { source: t.dir, scope: t.name === 'global' ? 'global' : 'agent', agentId: t.agentId || null }
  });

  appendLog(CHANGE_LOG,
    `SKILLS SNAPSHOT TAKEN\n  Target: t.name\n  Source: t.dir\n  Slot: 1\n  Description: "description"`
  );

  results.push({ name: t.name, status: 'ok', slot: 1, dir: t.dir, slots: manifest.snapshots.length });
}

// Report
console.log(`Skills snapshot — ts`);
for (const r of results) {
  if (r.status === 'ok') {
    console.log(`  ✓ r.name.padEnd(20) slot 1  (r.dir)`);
  } else {
    console.log(`  ⚠ r.name.padEnd(20) r.status: r.reason`);
  }
}

const ok = results.filter(r => r.status === 'ok').length;
if (ok === 0) {
  console.error('No skills snapshots taken.');
  process.exit(1);
}
console.log(`\nok target(s) snapshotted. Description: "description"`);

FILE:scripts/detect-restart-command.mjs
#!/usr/bin/env node
// OpenClaw Recovery Manager — detect-restart-command.mjs
//
// Usage: node detect-restart-command.mjs
//
// Probes the environment to determine the correct restart command for this
// OpenClaw install. Used at setup time so the agent doesn't have to ask the
// user to pick from a list of options they may not understand.
//
// Output format (always stable for agents parsing):
//   DETECTED: <one-line restart command>
//   REASON:   <short human-readable reason>
//   METHOD:   <systemd-user | systemd-system | docker-compose | pid1 | unknown>
//   CONFIDENCE: <high | medium | low>
//
// Exit 0 on successful detection (any confidence). Exit 1 if nothing
// matched — caller should fall back to asking the user.

import { execSync } from 'child_process';
import { existsSync, readFileSync } from 'fs';

function run(cmd) {
  try {
    return execSync(cmd, { stdio: ['ignore', 'pipe', 'ignore'], encoding: 'utf8' }).trim();
  } catch {
    return '';
  }
}

function report(method, command, reason, confidence) {
  console.log(`DETECTED: command`);
  console.log(`REASON: reason`);
  console.log(`METHOD: method`);
  console.log(`CONFIDENCE: confidence`);
}

// ---- Probe 1: systemd user service ----
const userActive = run('systemctl --user is-active openclaw-gateway 2>/dev/null');
if (userActive === 'active') {
  report(
    'systemd-user',
    'systemctl --user restart openclaw-gateway',
    'systemd user service `openclaw-gateway` is active under --user',
    'high'
  );
  process.exit(0);
}

// ---- Probe 2: systemd system service ----
const systemActive = run('systemctl is-active openclaw-gateway 2>/dev/null');
if (systemActive === 'active') {
  report(
    'systemd-system',
    'sudo systemctl restart openclaw-gateway',
    'system-level systemd service `openclaw-gateway` is active',
    'high'
  );
  process.exit(0);
}

// Also check for common alternative service names
for (const svc of ['openclaw', 'openclaw.service']) {
  const a = run(`systemctl --user is-active svc 2>/dev/null`);
  if (a === 'active') {
    report('systemd-user', `systemctl --user restart svc`,
      `systemd user service \`svc\` is active`, 'high');
    process.exit(0);
  }
  const b = run(`systemctl is-active svc 2>/dev/null`);
  if (b === 'active') {
    report('systemd-system', `sudo systemctl restart svc`,
      `system-level systemd service \`svc\` is active`, 'high');
    process.exit(0);
  }
}

// ---- Probe 3: Docker Compose ----
function composeServices(cmd) {
  const out = run(`cmd 2>/dev/null`);
  if (!out) return [];
  return out.split('\n').map(s => s.trim()).filter(Boolean);
}

let composeCmd = null;
let composeServicesList = composeServices('docker compose ps --services');
if (composeServicesList.length > 0) composeCmd = 'docker compose';
else {
  composeServicesList = composeServices('docker-compose ps --services');
  if (composeServicesList.length > 0) composeCmd = 'docker-compose';
}

if (composeCmd) {
  const match = composeServicesList.find(s => /openclaw|gateway/i.test(s));
  if (match) {
    report(
      'docker-compose',
      `composeCmd restart match`,
      `docker-compose service \`match\` found via composeCmd`,
      'high'
    );
    process.exit(0);
  }
}

// ---- Probe 4: Running as PID 1 in a container ----
let isContainer = false;
if (existsSync('/.dockerenv')) isContainer = true;
try {
  const cg = readFileSync('/proc/1/cgroup', 'utf8');
  if (/kubepods|containerd|docker/.test(cg)) isContainer = true;
} catch {}

let pid1Cmdline = '';
try {
  pid1Cmdline = readFileSync('/proc/1/cmdline', 'utf8').replace(/\0/g, ' ').trim();
} catch {}

const pid1IsOpenclaw = /openclaw/i.test(pid1Cmdline) ||
  (/\bnode\b/.test(pid1Cmdline) && /openclaw/i.test(pid1Cmdline));

if (isContainer && pid1IsOpenclaw) {
  report(
    'pid1',
    'kill -USR1 1',
    `running in a container with OpenClaw as PID 1 (cmdline: pid1Cmdline.slice(0, 120))`,
    'high'
  );
  process.exit(0);
}

// Non-container but PID 1 is OpenClaw somehow (rare): still correct to USR1
if (pid1IsOpenclaw) {
  report(
    'pid1',
    'kill -USR1 1',
    `PID 1 appears to be OpenClaw (cmdline: pid1Cmdline.slice(0, 120))`,
    'medium'
  );
  process.exit(0);
}

// ---- Probe 5: plain openclaw process (non-managed) ----
// Filter out our own process and anything that just happens to have "openclaw"
// in its path because it's executing out of the skill directory.
const myPid = String(process.pid);
const myPpid = String(process.ppid);
const rawProcs = run('pgrep -af openclaw 2>/dev/null');
const realProcs = rawProcs.split('\n')
  .map(s => s.trim())
  .filter(Boolean)
  .filter(line => {
    const pid = line.split(/\s+/)[0];
    if (pid === myPid || pid === myPpid) return false;
    // Ignore anyone executing this very script
    if (line.includes('detect-restart-command.mjs')) return false;
    // Ignore shells whose arg just happens to contain our script path
    if (/\bpgrep\b/.test(line)) return false;
    return true;
  });

if (realProcs.length > 0) {
  // Found running processes but no managed restart mechanism detected.
  console.log('DETECTED: (none)');
  console.log(`REASON: openclaw process(es) running but no managed restart mechanism detected. pgrep output: realProcs.slice(0, 3).join(' | ')`);
  console.log('METHOD: unknown');
  console.log('CONFIDENCE: low');
  console.log('ACTION: ask the user how they want OpenClaw restarted');
  process.exit(1);
}

// ---- Nothing found ----
console.log('DETECTED: (none)');
console.log('REASON: no systemd service, docker compose service, PID 1 OpenClaw process, or openclaw process found. OpenClaw may not be running, or it is running under a custom supervisor.');
console.log('METHOD: unknown');
console.log('CONFIDENCE: low');
console.log('ACTION: ask the user how they want OpenClaw restarted');
process.exit(1);

FILE:scripts/watchdog-status.mjs
#!/usr/bin/env node
// OpenClaw Emergency Rollback — watchdog-status.mjs
// Reports current watchdog state and time remaining.

import { existsSync } from 'fs';
import { WATCHDOG_FILE, getWatchdog } from './utils.mjs';

if (!existsSync(WATCHDOG_FILE)) {
  console.log('NOT ARMED (no watchdog.json found)');
  process.exit(0);
}

const watchdog = getWatchdog();

if (!watchdog.armed) {
  console.log('NOT ARMED');
  process.exit(0);
}

const now = Math.floor(Date.now() / 1000);
const expiry = watchdog.expiryEpoch || 0;
const remaining = expiry - now;

if (remaining <= 0) {
  console.log('ARMED — timer expired, restore pending');
} else {
  const mins = Math.floor(remaining / 60);
  const secs = remaining % 60;
  console.log(`ARMED — minsm secss remaining`);
}

console.log(`Target: watchdog.targetSnapshot || 'snapshot-1' — "watchdog.targetLabel || 'unknown'"`);
console.log(`Expiry: watchdog.expiryHuman || 'unknown'`);

FILE:scripts/utils.mjs
// OpenClaw Recovery Manager — shared utilities
// (Skill name openclaw-emergency-rollback preserved for install compatibility.)
//
// All JSON I/O goes through here. No string interpolation of user data into code.
//
// This file preserves every export the original config-only rollback relied on,
// and adds new helpers for the skills and projects snapshot subsystems.

import { readFileSync, writeFileSync, mkdirSync, existsSync, readdirSync, statSync, copyFileSync, renameSync, unlinkSync } from 'fs';
import { join, dirname, basename, resolve } from 'path';
import { execSync } from 'child_process';
import { mkdtempSync } from 'fs';
import { tmpdir } from 'os';

const HOME = process.env.HOME || '/root';

// ---- Paths: existing (do not rename, existing installs depend on them) ----
export const ROLLBACK_DIR    = join(HOME, '.openclaw/rollback');
export const CONFIG_FILE     = join(ROLLBACK_DIR, 'rollback-config.json');
export const MANIFEST_FILE   = join(ROLLBACK_DIR, 'manifest.json');
export const WATCHDOG_FILE   = join(ROLLBACK_DIR, 'watchdog.json');
export const CHANGE_LOG      = join(ROLLBACK_DIR, 'logs', 'change.log');
export const RESTORE_LOG     = join(ROLLBACK_DIR, 'logs', 'restore.log');
export const SNAPSHOTS_DIR   = join(ROLLBACK_DIR, 'snapshots');
export const RECOVERY_FILE   = join(ROLLBACK_DIR, 'openclaw.recovery');

// ---- Paths: new subsystems ----
export const SKILLS_DIR      = join(ROLLBACK_DIR, 'skills');   // per-target subfolders live under this
export const PROJECTS_DIR    = join(ROLLBACK_DIR, 'projects'); // per-project subfolders live under this

// ---- Generic JSON helpers ----
export function readJson(filepath) {
  try {
    return JSON.parse(readFileSync(filepath, 'utf8'));
  } catch {
    return null;
  }
}

export function writeJson(filepath, data) {
  mkdirSync(dirname(filepath), { recursive: true });
  writeFileSync(filepath, JSON.stringify(data, null, 2) + '\n');
}

export function getConfig() {
  const config = readJson(CONFIG_FILE);
  if (!config) {
    console.error('ERROR: rollback-config.json not found. Run setup first.');
    process.exit(1);
  }
  return config;
}

export function getOpenclawHome() {
  const config = getConfig();
  return (config.openclawHome || '~/.openclaw').replace('~', HOME);
}

export function getOpenclawJson() {
  return join(getOpenclawHome(), 'openclaw.json');
}

export function getManifest() {
  return readJson(MANIFEST_FILE) || { watchdog_target: 'snapshot-1', snapshots: [] };
}

export function getWatchdog() {
  return readJson(WATCHDOG_FILE) || { armed: false };
}

export function appendLog(logFile, entry) {
  const ts = new Date().toISOString().replace('T', ' ').replace(/\.\d+Z$/, '');
  mkdirSync(dirname(logFile), { recursive: true });
  const existing = existsSync(logFile) ? readFileSync(logFile, 'utf8') : '';
  writeFileSync(logFile, existing + `[ts] entry\n---\n`);
}

export function timestamp() {
  return new Date().toISOString().replace(/\.\d+Z$/, 'Z');
}

export function timestampHuman() {
  return new Date().toISOString().replace('T', ' ').replace(/\.\d+Z$/, '');
}

// ---- Path expansion ----
export function expandHome(p) {
  if (!p) return p;
  return p.replace(/^~/, HOME);
}

// ---------------------------------------------------------------------------
// Dynamic discovery from openclaw.json
// ---------------------------------------------------------------------------
//
// These helpers read the user's live openclaw.json every time they're called,
// so adding or renaming an agent/project is automatically reflected. Nothing
// is hardcoded.
// ---------------------------------------------------------------------------

/** Return the parsed openclaw.json, or null if unreadable/missing. */
export function getOpenclawConfig() {
  return readJson(getOpenclawJson());
}

/**
 * List agents configured in openclaw.json.
 * Returns array of { id, workspace, skills } where:
 *   - id is the agent's identifier (falls back to workspace basename)
 *   - workspace is the absolute path (~ expanded) or null
 *   - skills is the agent's skills dir if explicitly set, else null
 *     (callers that want the workspace-fallback path should compute it)
 */
export function listAgents() {
  const cfg = getOpenclawConfig();
  if (!cfg?.agents?.list || !Array.isArray(cfg.agents.list)) return [];
  return cfg.agents.list.map((a, idx) => {
    const id = a.id || a.name || (a.workspace ? basename(expandHome(a.workspace)) : `agent-idx`);
    const workspace = a.workspace ? expandHome(a.workspace) : null;
    const skills = a.skills ? expandHome(a.skills) : null;
    return { id, workspace, skills };
  });
}

/**
 * Resolve an agent's skills directory.
 * Priority: explicit agent.skills > {workspace}/skills > null.
 */
export function resolveAgentSkillsDir(agent) {
  if (agent.skills) return agent.skills;
  if (agent.workspace) return join(agent.workspace, 'skills');
  return null;
}

/** Global skills directory: ~/.openclaw/skills. */
export function getGlobalSkillsDir() {
  return join(getOpenclawHome(), 'skills');
}

/**
 * List projects configured in openclaw.json.
 * Tolerant to several common shapes:
 *   - projects.list: [{ name, path }]
 *   - projects: [{ name, path }]
 *   - projects: { myproj: "~/path", other: { path: "~/x" } }
 * Returns array of { name, path } with ~ expanded.
 * Projects without a resolvable path are filtered out.
 */
export function listProjects() {
  const cfg = getOpenclawConfig();
  if (!cfg?.projects) return [];
  const out = [];
  const seen = new Set();
  const push = (name, path) => {
    if (!name || !path) return;
    const abs = expandHome(path);
    if (seen.has(name)) return;
    seen.add(name);
    out.push({ name, path: abs });
  };

  if (Array.isArray(cfg.projects)) {
    cfg.projects.forEach((p, i) => {
      if (typeof p === 'string') push(basename(p), p);
      else if (p && typeof p === 'object') push(p.name || p.id || `project-i`, p.path);
    });
  } else if (Array.isArray(cfg.projects.list)) {
    cfg.projects.list.forEach((p, i) => {
      if (typeof p === 'string') push(basename(p), p);
      else if (p && typeof p === 'object') push(p.name || p.id || `project-i`, p.path);
    });
  } else if (typeof cfg.projects === 'object') {
    Object.entries(cfg.projects).forEach(([key, v]) => {
      if (typeof v === 'string') push(key, v);
      else if (v && typeof v === 'object' && v.path) push(key, v.path);
    });
  }
  return out;
}

/** Sanitize a name for safe use as a directory name. */
export function safeDirName(name) {
  return String(name).replace(/[^A-Za-z0-9._-]+/g, '_').replace(/^_+|_+$/g, '') || 'unnamed';
}

// ---------------------------------------------------------------------------
// Per-target slot rotation (shared by skills and projects subsystems)
// ---------------------------------------------------------------------------
//
// A "target" is a single subfolder under skills/ or projects/ that owns its
// own independent 3-slot history. Slot 1 = most recent. Slot 3 = oldest.
// A 4th snapshot pushes slot 3 out. Identical behavior to the original config
// snapshot rotation, just scoped per target.
// ---------------------------------------------------------------------------

/** Ensure a target directory and its manifest exist; returns the manifest. */
export function ensureTargetManifest(targetDir) {
  mkdirSync(targetDir, { recursive: true });
  const mf = join(targetDir, 'manifest.json');
  const existing = readJson(mf);
  if (existing) return existing;
  const fresh = { snapshots: [] };
  writeJson(mf, fresh);
  return fresh;
}

export function readTargetManifest(targetDir) {
  return readJson(join(targetDir, 'manifest.json')) || { snapshots: [] };
}

export function writeTargetManifest(targetDir, manifest) {
  writeJson(join(targetDir, 'manifest.json'), manifest);
}

/**
 * Rotate an existing staged archive into a target's 3-slot history.
 *   targetDir:  absolute path of the per-target folder
 *   stagedZip:  absolute path to the tar.gz that should become slot 1
 *   entry:      { label, timestamp, ai_summary, ...extras } for the new slot 1
 * Returns the updated manifest.
 */
export function rotateIntoTarget(targetDir, stagedZip, entry) {
  mkdirSync(targetDir, { recursive: true });
  const snap1 = join(targetDir, 'snapshot-1.tar.gz');
  const snap2 = join(targetDir, 'snapshot-2.tar.gz');
  const snap3 = join(targetDir, 'snapshot-3.tar.gz');

  if (existsSync(snap2)) {
    if (existsSync(snap3)) unlinkSync(snap3);
    renameSync(snap2, snap3);
  }
  if (existsSync(snap1)) {
    renameSync(snap1, snap2);
  }
  copyFileSync(stagedZip, snap1);
  try { unlinkSync(stagedZip); } catch {}

  const manifest = readTargetManifest(targetDir);
  const shifted = (manifest.snapshots || [])
    .filter(s => s.slot <= 2)
    .map(s => ({ ...s, slot: s.slot + 1, file: `snapshot-s.slot + 1.tar.gz` }));

  shifted.unshift({
    slot: 1,
    file: 'snapshot-1.tar.gz',
    label: entry.label || 'unlabeled',
    timestamp: entry.timestamp || timestamp(),
    ai_summary: entry.ai_summary || '',
    ...(entry.extra || {})
  });

  manifest.snapshots = shifted.filter(s => s.slot <= 3);
  writeTargetManifest(targetDir, manifest);
  return manifest;
}

// ---------------------------------------------------------------------------
// Archive helpers (tar.gz — matches existing config subsystem)
// ---------------------------------------------------------------------------

/** Create a tar.gz of a source path, preserving its absolute-path layout. */
export function makeStagedArchive(sourcePaths, { excludeGlobs = [], dirTreeOnly = [] } = {}) {
  const stage = mkdtempSync(join(tmpdir(), 'oc-rm-stage-'));
  let captured = 0;

  for (const src of sourcePaths) {
    if (!src || !existsSync(src)) continue;
    const dest = join(stage, src); // preserve absolute layout inside stage
    try {
      if (statSync(src).isDirectory()) {
        mkdirSync(dest, { recursive: true });
        // rsync would be ideal; fall back to cp -a with excludes via find
        copyDirRecursive(src, dest, excludeGlobs);
        captured++;
      } else {
        mkdirSync(dirname(dest), { recursive: true });
        copyFileSync(src, dest);
        captured++;
      }
    } catch (e) {
      // continue — best-effort capture
    }
  }

  // For "tree only" paths: recreate directory structure but skip file content.
  for (const src of dirTreeOnly) {
    if (!src || !existsSync(src)) continue;
    try {
      if (statSync(src).isDirectory()) {
        const dest = join(stage, src);
        mkdirSync(dest, { recursive: true });
        copyDirStructureOnly(src, dest);
        captured++;
      }
    } catch {}
  }

  if (captured === 0) {
    execSync(`rm -rf "stage"`);
    return null;
  }

  const outZip = join(tmpdir(), `oc-rm-Date.now()-Math.random().toString(36).slice(2, 8).tar.gz`);
  try { unlinkSync(outZip); } catch {}
  execSync(`cd "stage" && tar -czf "outZip" .`, { stdio: 'ignore' });
  execSync(`rm -rf "stage"`);
  return outZip;
}

/** Copy a directory recursively, honoring top-level exclude names. */
function copyDirRecursive(src, dest, excludeNames = []) {
  const exclude = new Set(excludeNames);
  const entries = readdirSync(src, { withFileTypes: true });
  for (const ent of entries) {
    if (exclude.has(ent.name)) continue;
    const s = join(src, ent.name);
    const d = join(dest, ent.name);
    if (ent.isDirectory()) {
      mkdirSync(d, { recursive: true });
      copyDirRecursive(s, d, excludeNames);
    } else if (ent.isFile()) {
      copyFileSync(s, d);
    }
    // symlinks and other types: skip silently
  }
}

/** Recreate directory tree structure only (no files). */
function copyDirStructureOnly(src, dest) {
  const entries = readdirSync(src, { withFileTypes: true });
  for (const ent of entries) {
    if (!ent.isDirectory()) continue;
    const s = join(src, ent.name);
    const d = join(dest, ent.name);
    mkdirSync(d, { recursive: true });
    copyDirStructureOnly(s, d);
  }
  // Leave an empty .dirtree marker at leaves so the tree isn't entirely empty
  // (optional, but helpful for verifying the tree capture worked).
  if (entries.filter(e => e.isDirectory()).length === 0) {
    try { writeFileSync(join(dest, '.dirtree'), ''); } catch {}
  }
}

/** Extract a tar.gz into /, overwriting. Returns exit code (0 = success). */
export function extractArchive(zipPath) {
  try {
    execSync(`tar -xzf "zipPath" -C /`, { stdio: 'ignore' });
    return 0;
  } catch (e) {
    return e.status || 1;
  }
}

FILE:scripts/restore.mjs
#!/usr/bin/env node
// OpenClaw Emergency Rollback — restore.mjs
// Usage: node restore.mjs [slot]
// Restores a snapshot and restarts OpenClaw.
// Must work with zero AI, zero network, zero user interaction.

import { existsSync } from 'fs';
import { execSync } from 'child_process';
import { join } from 'path';
import {
  ROLLBACK_DIR, SNAPSHOTS_DIR, WATCHDOG_FILE, RESTORE_LOG,
  readJson, writeJson, getConfig, getManifest, getWatchdog,
  appendLog, timestampHuman
} from './utils.mjs';

const SLOT = parseInt(process.argv[2] || '1', 10);
const config = getConfig();
const manifest = getManifest();
const RESTART_CMD = config.restartCommand || 'kill -USR1 1';

// Find snapshot info
const snapInfo = manifest.snapshots.find(s => s.slot === SLOT);
const snapFile = snapInfo ? snapInfo.file : `snapshot-SLOT.tar.gz`;
const snapLabel = snapInfo ? snapInfo.label : 'unknown';
const snapTs = snapInfo ? snapInfo.timestamp : 'unknown';
const zipPath = join(SNAPSHOTS_DIR, snapFile);

if (!existsSync(zipPath)) {
  const msg = `RESTORE FAILED — zip not found: zipPath`;
  appendLog(RESTORE_LOG, msg);
  console.error(`ERROR: msg`);
  process.exit(1);
}

// Determine trigger method
const trigger = process.env.RESTART_TRIGGERED === '1'
  ? 'startup restore check / watchdog'
  : process.env.WATCHDOG_TRIGGERED === '1'
    ? 'detached watchdog timer (timer expired)'
    : 'manual';

// Log restore start
appendLog(RESTORE_LOG,
  `RESTORE TRIGGERED\n  Method: trigger\n  Target: snapshot-SLOT — "snapLabel"\n  Snapshot timestamp: snapTs\n  Zip: zipPath`
);

// Restore files — unzip with full path overwrite to /
let unzipExit = 0;
try {
  execSync(`tar -xzf "zipPath" -C /`, { stdio: 'ignore' });
} catch (e) {
  unzipExit = e.status || 1;
  appendLog(RESTORE_LOG, `RESTORE WARNING — unzip exit code: unzipExit`);
}

// Disarm watchdog
const watchdog = getWatchdog();
watchdog.armed = false;
writeJson(WATCHDOG_FILE, watchdog);

// Stop any detached watchdog timer process that may still be running
try {
  execSync('pkill -f watchdog-timer.mjs || true', { stdio: 'ignore' });
} catch {}

// Run restart command
let restartExit = 0;
try {
  execSync(RESTART_CMD, { stdio: 'inherit', shell: '/bin/bash' });
} catch (e) {
  restartExit = e.status || 1;
}

// Log restore complete
appendLog(RESTORE_LOG,
  `RESTORE COMPLETE\n  Restart command: RESTART_CMD\n  Restart exit: restartExit\n  Unzip exit: unzipExit`
);

process.exit(0);

FILE:scripts/projects-restore.mjs
#!/usr/bin/env node
// OpenClaw Recovery Manager — projects-restore.mjs (PROJECTS subsystem)
//
// Usage:
//   node projects-restore.mjs all                      -> restore slot 1 for every project
//   node projects-restore.mjs <project> [slot]         -> restore a single project
//
// MANUAL ONLY. Never invoked by the watchdog timer or startup hook.

import { existsSync } from 'fs';
import { join } from 'path';
import {
  PROJECTS_DIR, CHANGE_LOG,
  listProjects, readTargetManifest, safeDirName,
  extractArchive, appendLog, timestamp
} from './utils.mjs';

const mode = process.argv[2];
const slotArg = parseInt(process.argv[3], 10);

if (!mode) {
  console.error('Usage: node projects-restore.mjs <all|PROJECT_NAME> [slot]');
  process.exit(1);
}

const allProjects = listProjects();

let plan;
if (mode === 'all') {
  plan = allProjects.map(p => ({ project: p, slot: 1 }));
  if (plan.length === 0) {
    console.error('No projects configured in openclaw.json.');
    process.exit(1);
  }
} else {
  const want = safeDirName(mode);
  const p = allProjects.find(x => x.name === mode || safeDirName(x.name) === want);
  if (!p) {
    console.error(`ERROR: project "mode" not found in openclaw.json.`);
    console.error(`Known: allProjects.map(x => x.name).join(', ')`);
    process.exit(1);
  }
  plan = [{ project: p, slot: Number.isFinite(slotArg) && slotArg > 0 ? slotArg : 1 }];
}

const ts = timestamp();
const report = [];
let okCount = 0;

for (const { project, slot } of plan) {
  const targetDir = join(PROJECTS_DIR, safeDirName(project.name));
  const mf = readTargetManifest(targetDir);
  const entry = (mf.snapshots || []).find(s => s.slot === slot);
  if (!entry) {
    report.push({ name: project.name, status: 'skipped', reason: `no snapshot in slot slot` });
    continue;
  }
  const zipPath = join(targetDir, entry.file);
  if (!existsSync(zipPath)) {
    report.push({ name: project.name, status: 'failed', reason: `archive missing: zipPath` });
    continue;
  }
  const exit = extractArchive(zipPath);
  if (exit === 0) {
    okCount++;
    report.push({ name: project.name, status: 'ok', slot, label: entry.label, timestamp: entry.timestamp });
    appendLog(CHANGE_LOG,
      `PROJECT RESTORED\n  Project: project.name\n  Path: project.path\n  Slot: slot\n  Label: "entry.label || ''"\n  Snapshot timestamp: entry.timestamp || 'unknown'`
    );
  } else {
    report.push({ name: project.name, status: 'failed', reason: `tar exit code exit` });
    appendLog(CHANGE_LOG,
      `PROJECT RESTORE FAILED\n  Project: project.name\n  Slot: slot\n  tar exit: exit`
    );
  }
}

console.log(`Projects restore — ts`);
for (const r of report) {
  if (r.status === 'ok') {
    console.log(`  ✓ r.name.padEnd(20) slot r.slot  "r.label || ''"  (r.timestamp || '')`);
  } else {
    console.log(`  ⚠ r.name.padEnd(20) r.status: r.reason`);
  }
}

if (okCount === 0) {
  console.error('\nNo projects were restored.');
  process.exit(1);
}
console.log(`\nokCount project(s) restored.`);
console.log('NOTE: project restores never touch root config or the watchdog timer.');

FILE:scripts/watchdog-timer.mjs
#!/usr/bin/env node
import { existsSync } from 'fs';
import { join } from 'path';
import { execSync } from 'child_process';
import { getWatchdog, WATCHDOG_FILE, ROLLBACK_DIR, RESTORE_LOG, appendLog } from './utils.mjs';

const minutesArgs = parseInt(process.argv[2], 10) || 0;
const explicitSeconds = parseInt(process.argv[3], 10) || 0;

let timeoutMs = 0;
if (explicitSeconds > 0) {
  timeoutMs = explicitSeconds * 1000;
} else if (minutesArgs > 0) {
  timeoutMs = minutesArgs * 60 * 1000;
}

appendLog(RESTORE_LOG, `WATCHDOG TIMER — started pid=process.pid ppid=process.ppid timeoutMs=timeoutMs`);

if (timeoutMs <= 0) {
  appendLog(RESTORE_LOG, 'WATCHDOG TIMER — exiting immediately because timeoutMs <= 0');
  process.exit(0);
}

setTimeout(() => {
  appendLog(RESTORE_LOG, `WATCHDOG TIMER — fired pid=process.pid`);
  if (!existsSync(WATCHDOG_FILE)) {
    appendLog(RESTORE_LOG, 'WATCHDOG TIMER — watchdog.json missing at fire time, exiting');
    process.exit(0);
  }

  const watchdog = getWatchdog();
  appendLog(RESTORE_LOG, `WATCHDOG TIMER — armed=Boolean(watchdog.armed) expiry=watchdog.expiryEpoch || 'null'`);

  if (watchdog.armed) {
    const restoreIfArmed = join(ROLLBACK_DIR, 'scripts', 'restore-if-armed.mjs');
    try {
      appendLog(RESTORE_LOG, `WATCHDOG TIMER — invoking restore-if-armed.mjs via restoreIfArmed`);
      execSync(`node "restoreIfArmed"`, {
        stdio: 'ignore',
        env: { ...process.env, WATCHDOG_TRIGGERED: '1' }
      });
      appendLog(RESTORE_LOG, 'WATCHDOG TIMER — restore-if-armed.mjs returned without throwing');
    } catch (e) {
      appendLog(RESTORE_LOG, `WATCHDOG TIMER ERROR — restore-if-armed.mjs failed: e?.message || e`);
    }
  } else {
    appendLog(RESTORE_LOG, 'WATCHDOG TIMER — watchdog not armed at fire time, exiting');
  }
  process.exit(0);
}, timeoutMs);

FILE:scripts/snapshot.mjs
#!/usr/bin/env node
// OpenClaw Recovery Manager — snapshot.mjs (CONFIG subsystem)
// Usage: node snapshot.mjs "<label>" "<ai_summary>"
//
// Takes a labeled snapshot of all OpenClaw CONFIG files.
// This is the ONLY snapshot type the watchdog auto-restore/recovery timer
// operates on. Skills and projects are separate, manual subsystems.

import { readFileSync, existsSync, mkdirSync, copyFileSync, renameSync, unlinkSync, readdirSync, statSync } from 'fs';
import { join, dirname } from 'path';
import { execSync } from 'child_process';
import { mkdtempSync } from 'fs';
import { tmpdir } from 'os';
import {
  ROLLBACK_DIR, MANIFEST_FILE, SNAPSHOTS_DIR, CHANGE_LOG,
  readJson, writeJson, getOpenclawHome, getOpenclawJson, getManifest,
  appendLog, timestamp
} from './utils.mjs';

const LABEL = process.argv[2] || 'unlabeled';
const AI_SUMMARY = process.argv[3] || 'No summary provided.';

const OC_HOME = getOpenclawHome();
const OC_JSON = getOpenclawJson();

if (!existsSync(OC_JSON)) {
  console.error(`ERROR: openclaw.json not found at OC_JSON`);
  process.exit(1);
}

// Read openclaw.json to extract workspace paths and agent IDs
const ocConfig = readJson(OC_JSON);
const HOME = process.env.HOME || '/root';

// Extract per-agent workspace paths
const workspacePaths = new Set();
if (ocConfig?.agents) {
  if (ocConfig.agents.defaults?.workspace) {
    workspacePaths.add(ocConfig.agents.defaults.workspace.replace('~', HOME));
  }
  if (ocConfig.agents.list) {
    ocConfig.agents.list.forEach(a => {
      if (a.workspace) workspacePaths.add(a.workspace.replace('~', HOME));
    });
  }
}
if (workspacePaths.size === 0) {
  workspacePaths.add(join(HOME, '.openclaw', 'workspace'));
}

// Stage files into a temp dir preserving full absolute paths
const stageDir = mkdtempSync(join(tmpdir(), 'oc-snapshot-'));
const filesCaptured = [];

function stageFile(src) {
  if (!existsSync(src)) return false;
  const dest = join(stageDir, src);
  mkdirSync(dirname(dest), { recursive: true });
  copyFileSync(src, dest);
  filesCaptured.push(src);
  return true;
}

// 1) Stage openclaw.json (root master config)
stageFile(OC_JSON);

// 2) Stage global shared workspace identity files: ~/.openclaw/workspace/*.md
//    These are the files agents fall back to if they don't have their own.
//    Captured as a glob so new identity file types are picked up automatically.
const GLOBAL_WORKSPACE_DIR = join(OC_HOME, 'workspace');
if (existsSync(GLOBAL_WORKSPACE_DIR) && statSync(GLOBAL_WORKSPACE_DIR).isDirectory()) {
  try {
    for (const ent of readdirSync(GLOBAL_WORKSPACE_DIR, { withFileTypes: true })) {
      if (ent.isFile() && ent.name.toLowerCase().endsWith('.md')) {
        stageFile(join(GLOBAL_WORKSPACE_DIR, ent.name));
      }
    }
  } catch {}
}

// 3) Stage per-agent workspace config files (explicit list for per-agent dirs)
//    We keep this list explicit so we don't accidentally sweep in working
//    content / notes that an agent happens to have parked in its workspace.
const WORKSPACE_FILES = ['SOUL.md', 'AGENTS.md', 'USER.md', 'IDENTITY.md', 'TOOLS.md', 'HEARTBEAT.md', 'BOOT.md'];
for (const wsPath of workspacePaths) {
  // Skip the global workspace dir here — already captured above as a glob.
  if (wsPath === GLOBAL_WORKSPACE_DIR) continue;
  for (const wf of WORKSPACE_FILES) {
    stageFile(join(wsPath, wf));
  }
}

// Auth profiles (auth-profiles.json) are deliberately NOT captured.
// They contain sensitive credentials and must never be stored in snapshots.

// Create archive from staging dir
const tmpZip = join(tmpdir(), 'oc-snapshot-tmp.tar.gz');
try { unlinkSync(tmpZip); } catch {}
execSync(`cd "stageDir" && tar -czf "tmpZip" .`, { stdio: 'ignore' });

// Clean up staging dir
execSync(`rm -rf "stageDir"`);

// Rotate snapshots: 2→3, 1→2, new→1
mkdirSync(SNAPSHOTS_DIR, { recursive: true });
const snap3 = join(SNAPSHOTS_DIR, 'snapshot-3.tar.gz');
const snap2 = join(SNAPSHOTS_DIR, 'snapshot-2.tar.gz');
const snap1 = join(SNAPSHOTS_DIR, 'snapshot-1.tar.gz');

if (existsSync(snap2)) {
  if (existsSync(snap3)) unlinkSync(snap3);
  renameSync(snap2, snap3);
}
if (existsSync(snap1)) {
  renameSync(snap1, snap2);
}
copyFileSync(tmpZip, snap1); unlinkSync(tmpZip);

// Update manifest.json — shift existing entries, add new slot 1
const manifest = getManifest();
const shifted = manifest.snapshots
  .filter(s => s.slot <= 2)
  .map(s => ({ ...s, slot: s.slot + 1, file: `snapshot-s.slot + 1.tar.gz` }));

shifted.unshift({
  slot: 1,
  file: 'snapshot-1.tar.gz',
  label: LABEL,
  timestamp: timestamp(),
  ai_summary: AI_SUMMARY
});

manifest.snapshots = shifted.filter(s => s.slot <= 3);
manifest.watchdog_target = 'snapshot-1';
writeJson(MANIFEST_FILE, manifest);

// Log
appendLog(CHANGE_LOG,
  `SNAPSHOT TAKEN (config)\n  Slot: 1 (previous snapshots shifted)\n  Label: "LABEL"\n  Summary: AI_SUMMARY\n  Files: filesCaptured.join(', ')`
);

console.log(`Snapshot saved: slot 1 — LABEL (timestamp())`);
console.log(`Captured filesCaptured.length file(s).`);

FILE:references/RESTORE.md
---
name: openclaw-emergency-rollback/restore
description: Manual recovery instructions for OpenClaw Recovery Manager — restore config, skills, or project snapshots without AI, scripts, or network access. For use when everything is broken.
---

# Manual Recovery — No AI Required

Use this document if you have shell access but cannot use AI, the scripts
failed, or you want to manually restore a specific snapshot.

You need: a terminal, basic shell access, `tar` + `gzip`, and (for config
restore) the ability to run one command to restart OpenClaw.

All Recovery Manager archives are `tar.gz` files with absolute paths inside,
so extracting them with `tar -xzf ... -C /` puts every file back exactly
where it came from.

---

## Config Snapshots (auto-restore target)

### Step 1 — Find Your Config Snapshots

```bash
ls -lh ~/.openclaw/rollback/snapshots/
```

You will see up to three files:
- `snapshot-1.tar.gz` — most recent user-approved snapshot
- `snapshot-2.tar.gz` — second most recent
- `snapshot-3.tar.gz` — oldest

To see labels and timestamps:

```bash
node -e "
const m=require(process.env.HOME+'/.openclaw/rollback/manifest.json');
m.snapshots.forEach(s=>console.log('['+s.slot+'] '+s.label+' ('+s.timestamp+')'));
"
```

Or read the raw file: `cat ~/.openclaw/rollback/manifest.json`

### Step 2 — Restore the Snapshot

Replace `snapshot-1.tar.gz` with whichever snapshot you want:

```bash
tar -xzf ~/.openclaw/rollback/snapshots/snapshot-1.tar.gz -C /
```

This restores all files to their exact original paths:
- `~/.openclaw/openclaw.json`
- `~/.openclaw/workspace/*.md` (global workspace identity files)
- All per-agent workspace files (SOUL.md, AGENTS.md, etc.)

No path mapping needed — the archive preserves full absolute paths.

### Step 3 — Restart OpenClaw

Check what restart command was configured:

```bash
cat ~/.openclaw/rollback/rollback-config.json
```

Look for `"restartCommand"` and run it. Examples:

```bash
kill -USR1 1
systemctl --user restart openclaw-gateway
docker compose restart
docker compose down && docker compose up -d
```

### Step 4 — Verify

```bash
openclaw gateway status
```

You should see the gateway as active and running.

### Step 5 — Disarm the Watchdog (if still armed)

If the detached watchdog timer might still be running, disarm it so it
doesn't fire again:

```bash
# Stop any detached watchdog timer processes
pkill -f watchdog-timer.mjs || true

# Mark watchdog as disarmed
node -e "
const fs=require('fs');
const wf=process.env.HOME+'/.openclaw/rollback/watchdog.json';
const w=JSON.parse(fs.readFileSync(wf,'utf8'));
w.armed=false;
fs.writeFileSync(wf,JSON.stringify(w,null,2));
console.log('Watchdog disarmed.');
"
```

### If You Have a Recovery File

If a recovery test was run, there may be a clean config backup at:

```bash
ls -lh ~/.openclaw/rollback/openclaw.recovery
```

If this file exists and your snapshots are corrupted or missing:

```bash
cp ~/.openclaw/rollback/openclaw.recovery ~/.openclaw/openclaw.json
```

Then restart as described above.

---

## Skills Snapshots (manual subsystem)

### Find Your Skills Snapshots

```bash
ls ~/.openclaw/rollback/skills/
```

You'll see one subfolder per target — `global`, plus one per configured
agent. Each subfolder is an independent 3-slot history.

To see the history for a specific target:

```bash
cat ~/.openclaw/rollback/skills/global/manifest.json
cat ~/.openclaw/rollback/skills/<agent-id>/manifest.json
```

### Restore a Skills Snapshot

```bash
# Restore global skills slot 1
tar -xzf ~/.openclaw/rollback/skills/global/snapshot-1.tar.gz -C /

# Restore an agent's skills slot 2
tar -xzf ~/.openclaw/rollback/skills/<agent-id>/snapshot-2.tar.gz -C /
```

Skills restore does **not** require a gateway restart and does **not** touch
the watchdog.

---

## Project Snapshots (manual subsystem)

### Find Your Project Snapshots

```bash
ls ~/.openclaw/rollback/projects/
```

One subfolder per project. Each subfolder is an independent 3-slot history.

```bash
cat ~/.openclaw/rollback/projects/<project-name>/manifest.json
```

### Restore a Project Snapshot

```bash
# Restore project slot 1
tar -xzf ~/.openclaw/rollback/projects/<project-name>/snapshot-1.tar.gz -C /

# Restore project slot 3 (oldest)
tar -xzf ~/.openclaw/rollback/projects/<project-name>/snapshot-3.tar.gz -C /
```

This restores project-local manifests and state (not working content):
`openclaw.json`, `mcp_config.json`, `package.json`, state files, `tools/`,
`skills/`, `.openclaw/workspace.state.json`, and the `comms/` directory
tree structure. Working content, `node_modules/`, and `memory/` are not
included.

Project restore does **not** require a gateway restart and does **not** touch
the root config or the watchdog.

---

## Logs

```bash
cat ~/.openclaw/rollback/logs/restore.log    # automated restore history
cat ~/.openclaw/rollback/logs/change.log     # all changes across all subsystems
```

---

## Summary (Quickest Paths)

### Config (most common emergency)
```bash
# 1. Restore snapshot
tar -xzf ~/.openclaw/rollback/snapshots/snapshot-1.tar.gz -C /

# 2. Restart (use your actual command)
kill -USR1 1

# 3. Disarm watchdog timer
pkill -f watchdog-timer.mjs || true
```

### Skills
```bash
tar -xzf ~/.openclaw/rollback/skills/<target>/snapshot-1.tar.gz -C /
```

### Project
```bash
tar -xzf ~/.openclaw/rollback/projects/<project>/snapshot-1.tar.gz -C /
```

That's it.

FILE:references/TESTING.md
---
name: openclaw-emergency-rollback/testing
description: Destructive recovery test procedure for the OpenClaw Recovery Manager config subsystem. Read this when the user wants to test that the emergency rollback system actually works end-to-end.
---

# Emergency Recovery Test — Destructive (Config Subsystem)

This test verifies the full **config** recovery pipeline by deliberately
breaking the OpenClaw config and confirming the watchdog automatically
restores it.

Skills and projects are NOT part of this test. They are manual-only
subsystems and have no auto-restore to verify.

**This test is destructive.** During the test window (up to ~2 minutes), the
user's OpenClaw gateway will be non-functional. AI sessions, agents, and any
active connections will be interrupted.

---

## Before You Begin — Pre-Flight Checklist

Confirm ALL of these with the user before proceeding:

```
⚠️  Emergency Recovery Test — Pre-Flight Checklist

This test will:
  1. Save your current config as a test snapshot
  2. Save a manual recovery copy of openclaw.json
  3. Deliberately break your openclaw.json (logical sabotage)
  4. Restart the gateway (it will fail)
  5. Wait for the watchdog to auto-restore (~2 minutes)

During the test you WILL lose access to your AI session.

Requirements:
  □ You have terminal/SSH access to this machine right now
  □ You can run commands even if the AI agent is offline
  □ You understand this will interrupt all active sessions

Manual recovery command (if the test fails — keep this visible):
  cp ~/.openclaw/rollback/openclaw.recovery ~/.openclaw/openclaw.json
  <your restart command here>

Type "yes, run the test" to proceed.
```

Fill in the actual restart command from
`~/.openclaw/rollback/rollback-config.json`.

Do NOT proceed unless the user explicitly confirms.

---

## Test Procedure

### Step 1 — Verify Dependencies

```bash
~/.openclaw/rollback/scripts/recovery-test.mjs preflight
```

This checks that node, tar, and gzip are available, that the rollback
directory is properly initialized, and that all config-subsystem scripts are
present. If anything fails, stop and fix it before continuing.

### Step 2 — Create Test Snapshot

```bash
~/.openclaw/rollback/scripts/snapshot.mjs "pre-test known-good config" "Snapshot taken before recovery test."
```

This saves the current working config as snapshot [1].

### Step 3 — Save Manual Recovery Copy

```bash
~/.openclaw/rollback/scripts/recovery-test.mjs save-recovery
```

This copies `openclaw.json` to `~/.openclaw/rollback/openclaw.recovery`.
This is the user's last-resort manual recovery if everything else fails.

Tell the user:

```
📋 Manual recovery copy saved. If the test fails and the watchdog does not
restore your config within 5 minutes, run these two commands from any terminal:

  cp ~/.openclaw/rollback/openclaw.recovery ~/.openclaw/openclaw.json
  <restart command>

Keep this window open or write these commands down before proceeding.
```

### Step 4 — Arm the Watchdog (2 minutes)

```bash
~/.openclaw/rollback/scripts/watchdog-set.mjs 2
```

The watchdog is now armed. If nothing disarms it in 2 minutes, it will
automatically restore snapshot [1] and restart the gateway.

### Step 5 — Break the Config (logical sabotage)

```bash
~/.openclaw/rollback/scripts/recovery-test.mjs sabotage
```

This poisons the gateway auth token (64 `f`s) and modifies agent workspace
paths. The file remains valid JSON — so it gets past OpenClaw's invalid-JSON
auto-revert — but the gateway cannot route correctly. This is the correct
failure mode to exercise the watchdog.

### Step 6 — Restart the Gateway

Read the restart command from rollback-config.json and run it:

```bash
RESTART_CMD=$(node -e "console.log(require('$HOME/.openclaw/rollback/rollback-config.json').restartCommand)")
eval "$RESTART_CMD"
```

The gateway will attempt to start, load the poisoned-but-valid config, and
run in a broken routing state. This is expected.

### Step 7 — Wait for Recovery

The detached watchdog timer can fire at expiry, and the native
`gateway:startup` hook can recover on restart if the timer died. Combined
with restart/setup overhead, recovery should usually happen within ~2-3
minutes. The user should:

1. Wait 3 minutes
2. Try to connect to their agent
3. If the agent is back and working, the test passed

To verify programmatically:

```bash
~/.openclaw/rollback/scripts/recovery-test.mjs verify
```

### Step 8 — Report Results

If the config is restored and the gateway is running:

```
✅ Recovery test PASSED.

The watchdog detected the expired timer, restored snapshot [1],
and restarted the gateway automatically.

Your manual recovery copy is still at:
  ~/.openclaw/rollback/openclaw.recovery
You can delete it or keep it as an extra backup.
```

If the config was NOT restored after 5 minutes:

```
❌ Recovery test FAILED.

The watchdog did not fire. Possible causes:
  • the detached watchdog timer process never started
  • `watchdog-timer.mjs` was killed unexpectedly
  • the native `watchdog-recovery` hook is not installed/enabled
  • the startup hook ran but failed before invoking `restore-if-armed.mjs`

To restore manually:
  cp ~/.openclaw/rollback/openclaw.recovery ~/.openclaw/openclaw.json
  <restart command>

Check the logs:
  cat ~/.openclaw/rollback/logs/restore.log
  cat ~/.openclaw/rollback/logs/change.log
```

---

## What the Test Validates

1. **Snapshot creation** — config files are captured and archived correctly
2. **Watchdog arming** — detached timer started with correct expiry
3. **Startup hook recovery** — native `gateway:startup` hook re-checks
   persistent watchdog state after restart
4. **Timer expiry detection** — restore-if-armed.mjs checks epoch against
   expiry
5. **Restore execution** — archive extracted to correct paths, overwriting
   broken files
6. **Gateway restart** — restart command fires after restore
7. **Watchdog disarm** — watchdog state cleared after firing

A full destructive test should exercise both timer-path and startup-hook
recovery evidence in `restore.log`.

**This test deliberately does NOT touch skills or projects.** Those
subsystems are manual-only and do not participate in the auto-restore
pipeline.

---

## Cleaning Up After a Failed Test

If the automatic recovery didn't fire:

```bash
# 1. Restore the config
cp ~/.openclaw/rollback/openclaw.recovery ~/.openclaw/openclaw.json

# 2. Restart the gateway (use your actual command)
kill -USR1 1

# 3. Disarm the watchdog timer so it doesn't fire later
pkill -f watchdog-timer.mjs || true

# 4. Mark watchdog as disarmed
node -e "
const fs=require('fs');
const wf=process.env.HOME+'/.openclaw/rollback/watchdog.json';
const w=JSON.parse(fs.readFileSync(wf,'utf8'));
w.armed=false;
fs.writeFileSync(wf,JSON.stringify(w,null,2));
console.log('Watchdog disarmed.');
"

# 5. Verify
cat ~/.openclaw/openclaw.json | node -e "JSON.parse(require('fs').readFileSync('/dev/stdin','utf8'));console.log('Config is valid JSON')"
```

FILE:references/SETUP.md
---
name: openclaw-emergency-rollback/setup
description: One-time setup for OpenClaw Recovery Manager (installs as openclaw-emergency-rollback). Read this when the user wants to install or initialize the rollback system for the first time.
---

# OpenClaw Recovery Manager — One-Time Setup

*(Install name kept as `openclaw-emergency-rollback` for compatibility.)*

Run this setup exactly once when `~/.openclaw/rollback/` does not exist.
Do not re-run setup if the directory already exists unless the user
explicitly asks to reinstall.

---

## Prerequisites

The Recovery Manager uses Node.js (already installed with OpenClaw) and
standard Linux tools. Verify before proceeding:

```bash
echo "--- Checking dependencies ---"
node --version && echo "  ✓ node" || echo "  ✗ node NOT FOUND"
command -v tar >/dev/null  && echo "  ✓ tar"  || echo "  ✗ tar NOT FOUND"
command -v gzip >/dev/null && echo "  ✓ gzip" || echo "  ✗ gzip NOT FOUND"
echo "  ✓ no cron dependency — rollback uses detached Node timers plus a native OpenClaw gateway startup hook"
```

Node.js is required to run OpenClaw itself, so it is always present. If `tar`
or `gzip` are missing (rare, but possible on stripped Docker images), install
them:

- **Ubuntu/Debian VPS:** `sudo apt-get install -y tar gzip`
- **Docker (node:22-bookworm-slim):** Set `OPENCLAW_DOCKER_APT_PACKAGES="tar gzip"`
  in your Docker setup, or add to Dockerfile:
  `RUN apt-get update && apt-get install -y tar gzip`

---

## Step 1 — Detect the Restart Command (do NOT ask the user)

The restart command is the most critical piece of setup — a wrong value
means auto-recovery can't actually recover. **Detect it. Don't ask the user
to pick from a list.** Users often don't know which install method is in
use on their own machine, and guessing wrong silently breaks the
dead-man's-switch.

### Preferred: run the detection script

This skill ships a detection script that performs all the probes below
and prints a clean parseable result. Copy and run it from this skill's
`scripts/` directory (you can run it directly out of the skill before
copying scripts into `~/.openclaw/rollback/scripts/`):

```bash
node /path/to/this/skill/scripts/detect-restart-command.mjs
```

Output format:

```
DETECTED: <the restart command, or (none)>
REASON:   <short human-readable reason>
METHOD:   <systemd-user | systemd-system | docker-compose | pid1 | unknown>
CONFIDENCE: <high | medium | low>
```

- Exit 0 with a `DETECTED:` value → use that command.
- Exit 1 → no confident match; fall back to asking the user.

### Manual detection (only if you can't run the script)

Run each probe in order, stop at the first match.

**Probe 1 — systemd user service** *(most common on modern Linux VPS)*

```bash
systemctl --user is-active openclaw-gateway 2>/dev/null
```

If this returns `active`, the restart command is:
`systemctl --user restart openclaw-gateway`

Also try the system-level service if user-level is not active:

```bash
systemctl is-active openclaw-gateway 2>/dev/null
```

If `active`, the restart command is:
`sudo systemctl restart openclaw-gateway`

**Probe 2 — Docker Compose** *(container host)*

```bash
docker compose ps --services 2>/dev/null | grep -E '(openclaw|gateway)' \
  || docker-compose ps --services 2>/dev/null | grep -E '(openclaw|gateway)'
```

If there's a match, the restart command is:
`docker compose restart <matched-service-name>`

**Probe 3 — Running as PID 1 in a container** *(Docker/K8s primary
process)*

```bash
cat /proc/1/comm 2>/dev/null
cat /proc/1/cmdline 2>/dev/null | tr '\0' ' '
[ -f /.dockerenv ] && echo "container: docker"
grep -q 'kubepods\|containerd' /proc/1/cgroup 2>/dev/null && echo "container: k8s"
```

If in a container AND PID 1's cmdline contains `openclaw`, use:
`kill -USR1 1`

**Probe 4 — Standalone process (non-container, non-systemd)**

```bash
pgrep -af openclaw
```

If processes exist but none of Probes 1-3 matched, report this to the user
and ask how they'd like it restarted. This is the only case where asking
is appropriate.

### Verify Before Storing

Once a restart command is detected, **state what you found and why**, then
confirm with the user before writing it to `rollback-config.json`:

```
I checked how OpenClaw is running on this machine.

Detected: systemd user service (`openclaw-gateway` is active under
--user). The correct recovery restart command is:

  systemctl --user restart openclaw-gateway

This is what will run if the dead-man's-switch fires. Want me to store
this and continue setup? (Say "yes" or give me a different command.)
```

Never default to `kill -USR1 1` without verifying PID 1 is actually
OpenClaw. A wrong default here silently disarms recovery.

### OpenClaw Home

Detect from the environment — don't ask unless detection fails:

```bash
[ -n "$OPENCLAW_HOME" ] && echo "$OPENCLAW_HOME"
[ -f "$HOME/.openclaw/openclaw.json" ] && echo "$HOME/.openclaw"
```

If `OPENCLAW_HOME` is set, use it. Otherwise if
`~/.openclaw/openclaw.json` exists, OC_HOME is `~/.openclaw`. Only ask if
neither resolves.

Store the detected values as RESTART_CMD and OC_HOME (absolute paths, with
`~` expanded to `$HOME`). These go into `rollback-config.json` and are not
changed automatically after setup.

---

## Step 2 — Create Directory Structure

```bash
mkdir -p ~/.openclaw/rollback/snapshots
mkdir -p ~/.openclaw/rollback/skills
mkdir -p ~/.openclaw/rollback/projects
mkdir -p ~/.openclaw/rollback/scripts
mkdir -p ~/.openclaw/rollback/logs
```

`snapshots/` holds config snapshots (the auto-restore target).
`skills/` and `projects/` hold per-target subfolders created on first
snapshot; nothing needs to be pre-created inside them.

---

## Step 3 — Write rollback-config.json

Use the absolute path for openclawHome (expand `~` to `$HOME`):

```bash
cat > ~/.openclaw/rollback/rollback-config.json << EOF
{
  "restartCommand": "kill -USR1 1",
  "openclawHome": "$HOME/.openclaw",
  "installedAt": "$(date -u +"%Y-%m-%dT%H:%M:%SZ")"
}
EOF
```

If the user specified a different openclaw home path or restart command, use
those instead.

---

## Step 4 — Initialize watchdog.json

```bash
cat > ~/.openclaw/rollback/watchdog.json << 'EOF'
{
  "armed": false,
  "setAt": null,
  "expiryEpoch": null,
  "expiryHuman": null,
  "minutesSet": null,
  "targetSnapshot": "snapshot-1",
  "targetLabel": null
}
EOF
```

---

## Step 5 — Initialize manifest.json (config subsystem)

```bash
cat > ~/.openclaw/rollback/manifest.json << 'EOF'
{
  "watchdog_target": "snapshot-1",
  "snapshots": []
}
EOF
```

The skills and projects subsystems each maintain their own per-target
`manifest.json` files, created automatically the first time a target is
snapshotted. Nothing to initialize up front.

---

## Step 6 — Copy All Scripts

Copy every file from this skill's `scripts/` directory into
`~/.openclaw/rollback/scripts/`. This includes:

**Shared utility:**
- `utils.mjs` — shared Node.js module (imported by all `.mjs` scripts)
- `detect-restart-command.mjs` — environment probe used at setup and any time
  the user migrates infrastructure and needs to re-detect

**Config subsystem (auto-restore watchdog target — unchanged behavior):**
- `snapshot.mjs`
- `restore.mjs`
- `restore-if-armed.mjs`
- `watchdog-set.mjs`
- `watchdog-extend.mjs`
- `watchdog-clear.mjs`
- `watchdog-status.mjs`
- `watchdog-timer.mjs`
- `recovery-test.mjs`

**Skills subsystem (manual only):**
- `skills-snapshot.mjs`
- `skills-list.mjs`
- `skills-restore.mjs`

**Projects subsystem (manual only):**
- `projects-snapshot.mjs`
- `projects-list.mjs`
- `projects-restore.mjs`

After copying, make the scripts executable:

```bash
chmod +x ~/.openclaw/rollback/scripts/*.mjs
```

The `.mjs` files have `#!/usr/bin/env node` shebangs, so once they have the
execute bit, the agent or startup scripts can call them directly without a
shell wrapper.

---

## Step 7 — Install Native OpenClaw Startup Hook

This ensures that if OpenClaw restarts while the config watchdog is armed,
the recovery check runs again natively inside OpenClaw on `gateway:startup`.

The hook only ever acts on config snapshots. Skills and projects are not
part of the auto-recovery pipeline.

Create the managed hook under `~/.openclaw/hooks/watchdog-recovery/` with:
- `HOOK.md`
- `handler.ts`

Use the versions shipped in this skill under:
- `hooks/watchdog-recovery/HOOK.md`
- `hooks/watchdog-recovery/handler.ts`

Then enable the hook:

```bash
openclaw hooks enable watchdog-recovery
openclaw hooks check
openclaw hooks list
```

Expected behavior:
- if watchdog is unarmed, the hook exits immediately
- if watchdog is armed and expired, the hook runs `restore-if-armed.mjs`
- if watchdog is armed and not yet expired, the hook respawns
  `watchdog-timer.mjs` for the remaining time

This works the same way on pod, Docker, and local installs because it uses
OpenClaw's own native startup lifecycle instead of external schedulers.

---

## Step 8 — Confirm Setup Complete

```
✅ OpenClaw Recovery Manager installed.

Location: ~/.openclaw/rollback/
Restart command: <configured>
Scripts: Node.js (.mjs) — directly executable
Startup recovery: native OpenClaw hook `watchdog-recovery` on `gateway:startup`
                  (config subsystem only — skills and projects are manual)

Subsystems:
  • config     — auto-restore watchdog (dead-man's-switch)
  • skills     — manual snapshot/restore only
  • projects   — manual snapshot/restore only

Next step: say "create snapshot" to save your current known-good config
before making any changes.

Optional: say "test emergency recovery" to run a destructive test that
verifies the full config recovery pipeline works end-to-end.
```

---

## Reinstall / Reset

If the user wants to reinstall from scratch:
1. Back up existing snapshots:
   `cp -r ~/.openclaw/rollback/ /tmp/openclaw-rollback-backup/`
   (this preserves config, skills, and projects histories)
2. `rm -rf ~/.openclaw/rollback/`
3. Remove any old startup hook that points at a previous rollback install,
   if present.
4. Run setup again from Step 1.
5. Ask the user if they want their old snapshots restored from the backup.

Never silently delete snapshots — always back them up first and ask.

FILE:hooks/watchdog-recovery/handler.ts
import { existsSync, mkdirSync, readFileSync, writeFileSync } from 'fs';
import { join, dirname } from 'path';
import { spawn, execSync } from 'child_process';

function appendLog(logFile: string, entry: string) {
  mkdirSync(dirname(logFile), { recursive: true });
  const ts = new Date().toISOString().replace('T', ' ').replace(/\.\d+Z$/, '');
  const existing = existsSync(logFile) ? readFileSync(logFile, 'utf8') : '';
  writeFileSync(logFile, existing + `[ts] entry\n---\n`);
}

const handler = async (event: any) => {
  if (event?.type !== 'gateway' || event?.action !== 'startup') return;

  const home = process.env.OPENCLAW_HOME || join(process.env.HOME || '/home/node', '.openclaw');
  const rollbackDir = join(home, 'rollback');
  const watchdogFile = join(rollbackDir, 'watchdog.json');
  const restoreLog = join(rollbackDir, 'logs', 'restore.log');
  const restoreScript = join(rollbackDir, 'scripts', 'restore-if-armed.mjs');
  const timerScript = join(rollbackDir, 'scripts', 'watchdog-timer.mjs');

  appendLog(restoreLog, 'HOOK gateway:startup — entered watchdog recovery hook');

  if (!existsSync(watchdogFile)) {
    appendLog(restoreLog, 'HOOK gateway:startup — watchdog.json missing, nothing to do');
    return;
  }
  if (!existsSync(restoreScript)) {
    appendLog(restoreLog, 'HOOK gateway:startup — restore-if-armed.mjs missing, nothing to do');
    return;
  }

  let watchdog: any = null;
  try {
    watchdog = JSON.parse(readFileSync(watchdogFile, 'utf8'));
  } catch (e: any) {
    appendLog(restoreLog, `HOOK gateway:startup — failed to parse watchdog.json: e?.message || e`);
    return;
  }

  if (!watchdog?.armed) {
    appendLog(restoreLog, 'HOOK gateway:startup — watchdog not armed, nothing to do');
    return;
  }

  const now = Math.floor(Date.now() / 1000);
  const expiry = watchdog.expiryEpoch || 0;
  const remaining = expiry - now;

  appendLog(restoreLog, `HOOK gateway:startup — watchdog armed, expiry=expiry, now=now, remaining=remainings`);

  if (now >= expiry) {
    appendLog(restoreLog, 'HOOK gateway:startup — watchdog expired, invoking restore-if-armed.mjs');
    try {
      execSync(`node "restoreScript"`, {
        stdio: 'ignore',
        env: { ...process.env, WATCHDOG_TRIGGERED: '1', WATCHDOG_SOURCE: 'gateway:startup-hook' }
      });
      appendLog(restoreLog, 'HOOK gateway:startup — restore-if-armed.mjs returned');
    } catch (e: any) {
      appendLog(restoreLog, `HOOK gateway:startup — restore-if-armed.mjs failed: e?.message || e`);
    }
    return;
  }

  if (!existsSync(timerScript)) {
    appendLog(restoreLog, 'HOOK gateway:startup — watchdog-timer.mjs missing, cannot respawn timer');
    return;
  }

  try {
    const child = spawn(process.execPath, [timerScript, '0', String(remaining)], {
      detached: true,
      stdio: 'ignore',
      env: { ...process.env, WATCHDOG_SOURCE: 'gateway:startup-hook' }
    });
    child.unref();
    appendLog(restoreLog, `HOOK gateway:startup — respawned watchdog timer pid=child.pid || 'unknown' remaining=remainings`);
  } catch (e: any) {
    appendLog(restoreLog, `HOOK gateway:startup — failed to respawn watchdog timer: e?.message || e`);
  }
};

export default handler;

FILE:hooks/watchdog-recovery/HOOK.md
---
name: watchdog-recovery
description: "On gateway startup, recover or re-arm the emergency rollback watchdog from persistent disk"
metadata:
  { "openclaw": { "emoji": "🛡️", "events": ["gateway:startup"], "requires": { "bins": ["node"] } } }
---

# Watchdog Recovery Hook

Runs on OpenClaw gateway startup.

Purpose:
- If rollback is not armed, do nothing.
- If rollback is armed and expired, run `restore-if-armed.mjs` immediately.
- If rollback is armed and not yet expired, respawn the detached watchdog timer for the remaining time.

This hook is the native OpenClaw restart trigger for rollback recovery.
It does not require AI, internet, cron, or any external supervisor.

---
name: openclaw-emergency-rollback
description: >
  OpenClaw Emergency Config Rollback — dead man's switch system for safely making
  risky changes to OpenClaw configuration. Use this skill whenever the user mentions
  wanting to make changes to openclaw.json or agent configs and wants a safety net,
  says anything like "set emergency recovery", "create a snapshot", "take a backup
  before changes", "set a backout timer", "restore snapshot", "accept changes",
  "test emergency recovery", "run recovery test", "how does the rollback work",
  "what rollback commands", or any variation of wanting to safely change OpenClaw
  config with an automatic recovery fallback. Also trigger when the user asks about
  recovery, rollback, emergency restore, or testing the recovery system in the
  context of OpenClaw. This skill manages the full lifecycle: snapshots, watchdog
  timers, auto-restore, post-restart reminders, and destructive testing
  of the recovery pipeline — all without requiring AI, network, or user intervention
  once the timer is set. Uses only Node.js (already required by OpenClaw), zip,
  and unzip. No additional dependencies to install.
---

# OpenClaw Emergency Config Rollback

A dead man's switch system for OpenClaw configuration changes. The user takes a
snapshot of known-good config, sets a recovery timer, makes changes, and if they
don't accept the changes before the timer expires the system automatically restores
the last snapshot — with zero AI, zero network, zero user intervention required.

All scripts are Node.js (`.mjs`), which is already installed as an OpenClaw
dependency. No additional packages needed. The system uses detached Node.js timers
plus a native OpenClaw `gateway:startup` hook that re-checks persistent watchdog
state on every gateway restart rather than relying on external schedulers that often fail
in containerized environments.

---

## First-Time Setup

If `~/.openclaw/rollback/` does not exist, run setup before anything else.
Read `references/SETUP.md` now and follow it completely before proceeding.

---

## Important Note on `pkill` and Docker/K8s

If you are running OpenClaw as the primary process in a container (PID 1), **do not use `pkill -f openclaw`** to restart the gateway. If you use a background Dead Man's Switch, `pkill` will match the path name of the background script and kill your rescue job instantly.

Instead, use **`kill -USR1 1`** to surgically send the reload signal directly to the root OpenClaw process.

## Logical Sabotage vs Invalid JSON
OpenClaw protects itself from invalid JSON by instantly hot-reloading its last known good config before the gateway even restarts. To test destructive recovery properly, you must use **Logical Sabotage**: feeding OpenClaw perfectly valid JSON that logically breaks routing (e.g., a dummy token like 64 `f`s and poisoned workspace paths). This proves the rollback recovers from logical failure states.

---

## Restart recovery via native OpenClaw hook

When the config gets sabotaged and OpenClaw restarts, the detached `watchdog-timer`
may die with the old process tree. That is expected.

To make recovery survive pod/container/local restarts, this skill installs a native
OpenClaw managed hook at `~/.openclaw/hooks/watchdog-recovery/` listening to
`gateway:startup`.

On every gateway startup, the hook reads persistent `~/.openclaw/rollback/watchdog.json`:
1. If rollback is not armed, it exits immediately.
2. If rollback is armed and the hard expiry epoch has already passed, it runs
   `restore-if-armed.mjs` immediately.
3. If rollback is armed and the hard expiry epoch has not passed yet, it respawns
   `watchdog-timer.mjs` for the remaining seconds.

Because the system stores a hard absolute epoch (`expiryEpoch`) on persistent disk,
it doesn't matter how long the restart took: if OpenClaw restarts after expiry, the hook
restores immediately; if it restarts before expiry, the hook recreates the timer.

This is the native cross-environment trigger for pod, Docker, and local machine restarts.
No AI, internet, cron, or external supervisor is required.

---

## Session Start — Uptime Check (Run Every Session)

At the start of every session, run:

```bash
UPTIME=$(systemctl --user show openclaw-gateway \
  --property=ActiveEnterTimestampMonotonic 2>/dev/null \
  | awk -F= '{if($2>0) print int((systime()*1000000-$2)/1000000); else print 999}')

if [ "$UPTIME" = "999" ]; then
  UPTIME=$(ps -o etimes= -p $(pgrep -f "openclaw" 2>/dev/null) 2>/dev/null | tr -d ' ')
fi
```

If uptime is under 90 seconds AND `~/.openclaw/rollback/watchdog.json` exists
and shows `"armed": true`, the gateway just bounced. Open the session with the
**Watchdog Reminder** (see below).

If armed but uptime is over 90 seconds, still check and remind — the user may
have connected to a running session mid-timer.

If `armed: false` or watchdog file doesn't exist, start the session normally.

---

## Watchdog Reminder (show when watchdog is armed)

Run `~/.openclaw/rollback/scripts/watchdog-status.mjs` and display:

```
⚠️  Emergency recovery is armed.

Snapshot [1] "<label>" will auto-restore in ~XX minutes
unless you accept or extend.

Commands:
  • "accept changes"            — disarm watchdog, lock in current config
  • "extend recovery XX minutes" — add more time to the timer
  • "list snapshots"            — show all saved snapshots
  • "restore snapshot 2"        — manually restore snapshot 2 or 3
  • "create snapshot"           — save current state as new snapshot [1]
```

---

## User Commands Reference

### "create snapshot [description]"
Save current OpenClaw config as the new known-good restore point.

1. Run: `~/.openclaw/rollback/scripts/snapshot.mjs "<description>" "<ai_summary>"`
2. Write an AI summary (1–2 sentences) of the current config state by reading
   `~/.openclaw/openclaw.json` — note the default model, number of agents, any
   notable tools or channels — and pass it as the second argument
3. Reply with snapshot confirmation showing all current snapshots (max 3):

```
✅ Snapshot saved.
[1] Apr 20 2:30 PM — <description>  ← restore target
[2] Apr 19 9:00 AM — <previous label>
[3] Apr 18 4:00 PM — <oldest label>
```

Slot [1] is always the most recent. Slot [3] is always the oldest.
When a 4th snapshot would be created, slot [3] is overwritten as the others
shift. Snapshots are never deleted without the user explicitly creating a new
one that pushes the oldest out. If the user wants to preserve all three, they
can copy slot [3] before creating a new snapshot.

---

### "set emergency recovery XX minutes" / "start emergency recovery XX minutes"
Arm the watchdog dead man's switch.

1. Run: `~/.openclaw/rollback/scripts/watchdog-set.mjs <minutes>`
2. Reply:

```
⏱️ Watchdog armed — XX minutes.
Snapshot [1] "<label>" auto-restores at <HH:MM> if not accepted.
Make your changes whenever you're ready.
```

If no snapshot exists yet, tell the user to create one first before arming.

---

### "extend recovery XX minutes"
Add time to the active watchdog timer.

1. Run: `~/.openclaw/rollback/scripts/watchdog-extend.mjs <minutes>`
2. Reply with new expiry time and minutes remaining.

---

### "accept changes"
Disarm the watchdog — user is happy with the current config.

1. Run: `~/.openclaw/rollback/scripts/watchdog-clear.mjs`
2. Reply:

```
✅ Watchdog disarmed. Your changes are locked in.
Say "create snapshot" to save this config as your new restore point [1].
```

---

### "list snapshots"
Show all saved snapshots.

Read `~/.openclaw/rollback/manifest.json` and display:

```
Saved snapshots (most recent first):
[1] Apr 20 2:30 PM — "opus model working, github tool added"
    Config: claude-opus-4 default, 2 agents (main, coding), github MCP active
[2] Apr 19 9:00 AM — "initial clean setup"
    Config: claude-sonnet-4 default, 1 agent (main), no extra tools
[3] Apr 18 4:00 PM — "baseline before any changes"
    Config: claude-haiku-4 default, 1 agent (main)

Restore target: [1] (auto-restored if watchdog fires)
Watchdog: ARMED — 14m 32s remaining  [or: NOT ARMED]
```

---

### "restore snapshot [1|2|3]"
Manually restore a specific snapshot immediately.

1. Confirm with user: "This will overwrite your current OpenClaw config with
   snapshot [N] '<label>' from <timestamp> and restart the gateway. Are you sure?"
2. On confirmation: run `~/.openclaw/rollback/scripts/restore.mjs <slot>`
3. Gateway restarts. Next session will detect uptime < 90 seconds.
4. If watchdog was armed, it is disarmed as part of restore.

---

### "test emergency recovery" / "run recovery test"
Run a destructive test of the full recovery pipeline.

Read `references/TESTING.md` for the complete procedure. This test:
- Creates a dedicated test snapshot of the current config
- Arms a 2-minute watchdog
- Saves a manual recovery copy at `~/.openclaw/rollback/openclaw.recovery`
- Deliberately breaks `openclaw.json` to simulate a bad config change
- Restarts the gateway (which will fail to work properly)
- Waits for either the detached watchdog timer or the native `gateway:startup` hook to restore automatically

**This is destructive.** The user will lose access to their AI session for up
to 2 minutes while the test runs. Before running, confirm the user understands
the risks and has terminal/SSH access to manually recover if something goes wrong.

---

### "how does the rollback work" / "what commands can I use" / "explain emergency rollback"

Respond with this explanation:

```
OpenClaw Emergency Rollback — How It Works

This skill gives you a safety net for risky config changes. Here's the flow:

1. SNAPSHOT — Before making changes, say:
   "create snapshot — [describe what's working]"
   This saves your current openclaw.json and all agent workspace config files
   (SOUL.md, AGENTS.md, IDENTITY.md, etc.) as a restore point. You can have
   up to 3 snapshots. Snapshot [1] is always your most recent.

2. ARM THE TIMER — When ready to make changes, say:
   "set emergency recovery 30 minutes"
   This starts a countdown. If you don't accept the changes in time, the
   system automatically restores snapshot [1] and restarts OpenClaw — even
   if AI is offline, the server rebooted, or nothing is working.

3. MAKE CHANGES — Edit config, restart the gateway, whatever you need to do.

4. ACCEPT OR RECOVER —
   • If everything works: say "accept changes" to disarm the timer.
   • If something broke and you can't get back in: do nothing. The timer
     fires automatically and restores your last known-good config.
   • If you need more time: say "extend recovery 20 minutes".

Commands:
  "create snapshot [description]"       — save current config as restore point
  "set emergency recovery XX minutes"   — arm the auto-restore timer
  "extend recovery XX minutes"          — add time to active timer
  "accept changes"                      — disarm timer, keep current config
  "list snapshots"                      — show all 3 saved snapshots
  "restore snapshot [1|2|3]"            — manually restore a specific snapshot
  "test emergency recovery"             — destructive test of the full pipeline

The watchdog uses two native local paths:
- a detached background Node timer for the live happy path
- a native OpenClaw `gateway:startup` hook for restart recovery

No external scheduler, no AI, no internet, no anything else required.
On every gateway restart, the startup hook verifies whether the hard epoch timer
expired while OpenClaw was down and restores if so. If it hasn't expired yet,
it respawns a fresh timer to finish the countdown.

Dependencies: Node.js (already installed with OpenClaw), zip, unzip.
```

---

## What Gets Backed Up

Every snapshot captures exactly these files — no more, no less:

| File | Path |
|------|------|
| Master config | `~/.openclaw/openclaw.json` |
| Agent workspace files (per agent) | `<workspace>/SOUL.md` |
| | `<workspace>/AGENTS.md` |
| | `<workspace>/USER.md` |
| | `<workspace>/IDENTITY.md` |
| | `<workspace>/TOOLS.md` |
| | `<workspace>/HEARTBEAT.md` |
| | `<workspace>/BOOT.md` (if present) |


Workspace paths and agentIds are read dynamically from `openclaw.json` at
snapshot time — covers all configured agents automatically.

**Never captured:** credentials/, auth-profiles.json, session history, memory
logs, workspace content files, .env, Docker/K8s environment config.

---

## Change Log

Append to `~/.openclaw/rollback/logs/change.log` when:
- A snapshot is taken
- The watchdog is armed, extended, or cleared
- The user requests a gateway restart (note what changed and watchdog status)
- The gateway restart is confirmed complete
- A recovery test is started or completed

Format:
```
[YYYY-MM-DD HH:MM:SS] <EVENT TYPE>
  <key: value details>
---
```

---

## Reference Files

- `references/SETUP.md` — Read this first if `~/.openclaw/rollback/` does not exist
- `references/TESTING.md` — Destructive recovery test procedure and manual fallback
- `references/RESTORE.md` — Manual recovery instructions requiring no AI or scripts
- `scripts/` — Node.js scripts (`.mjs`) — no shell wrappers needed
- `hooks/watchdog-recovery/` — Native OpenClaw startup hook for restart recovery

FILE:scripts/watchdog-set.mjs
#!/usr/bin/env node
// OpenClaw Emergency Rollback — watchdog-set.mjs
// Usage: node watchdog-set.mjs <minutes>
// Arms the watchdog for the given number of minutes.

import { join } from "path";
import { existsSync } from "fs";
import {
  ROLLBACK_DIR, WATCHDOG_FILE, CHANGE_LOG, RESTORE_LOG,
  writeJson, getManifest, appendLog, timestamp
} from './utils.mjs';

const minutes = parseInt(process.argv[2], 10);
if (!minutes || minutes <= 0) {
  console.error('Usage: node watchdog-set.mjs <minutes>');
  process.exit(1);
}

const now = Math.floor(Date.now() / 1000);
const expiry = now + minutes * 60;
const setAt = timestamp();
const expiryDate = new Date(expiry * 1000);
const expiryHuman = expiryDate.toISOString().replace(/\.\d+Z$/, 'Z');
const expiryDisplay = expiryDate.toLocaleTimeString('en-US', { hour: '2-digit', minute: '2-digit', hour12: true });

// Read target snapshot label
const manifest = getManifest();
const snap1 = manifest.snapshots.find(s => s.slot === 1);
const targetLabel = snap1 ? snap1.label : 'no snapshot saved';

// Write watchdog.json
writeJson(WATCHDOG_FILE, {
  armed: true,
  setAt,
  expiryEpoch: expiry,
  expiryHuman,
  minutesSet: minutes,
  targetSnapshot: 'snapshot-1',
  targetLabel
});


import { spawn } from 'child_process';
const timerScript = join(ROLLBACK_DIR, 'scripts', 'watchdog-timer.mjs');
if (existsSync(timerScript)) {
  const child = spawn(process.execPath, [timerScript, String(minutes)], {
    detached: true,
    stdio: 'ignore',
    env: { ...process.env, WATCHDOG_SOURCE: 'watchdog-set' }
  });
  child.unref();
  appendLog(RESTORE_LOG, `WATCHDOG SET — spawned watchdog timer pid=child.pid || 'unknown' minutes=minutes`);
} else {
  appendLog(RESTORE_LOG, "WATCHDOG SET ERROR — watchdog-timer.mjs missing, timer won't fire.");
  console.error("WARNING: watchdog-timer.mjs missing, timer won't fire.");
}

// Log
appendLog(CHANGE_LOG,
  `WATCHDOG ARMED\n  Minutes: minutes\n  Expiry: expiryHuman\n  Target: snapshot-1 — "targetLabel"`
);

console.log(`Watchdog armed — minutes minutes. Expires at expiryDisplay. Target: targetLabel`);

FILE:scripts/watchdog-extend.mjs
#!/usr/bin/env node
// OpenClaw Emergency Rollback — watchdog-extend.mjs
// Usage: node watchdog-extend.mjs <additional_minutes>

import {
  WATCHDOG_FILE, CHANGE_LOG,
  getWatchdog, writeJson, appendLog
} from './utils.mjs';

const addMinutes = parseInt(process.argv[2], 10);
if (!addMinutes || addMinutes <= 0) {
  console.error('Usage: node watchdog-extend.mjs <additional_minutes>');
  process.exit(1);
}

const watchdog = getWatchdog();
if (!watchdog.armed) {
  console.error('ERROR: Watchdog is not armed. Use watchdog-set first.');
  process.exit(1);
}

const oldExpiry = watchdog.expiryEpoch;
const newExpiry = oldExpiry + addMinutes * 60;
const newExpiryDate = new Date(newExpiry * 1000);
const newExpiryHuman = newExpiryDate.toISOString().replace(/\.\d+Z$/, 'Z');
const newExpiryDisplay = newExpiryDate.toLocaleTimeString('en-US', { hour: '2-digit', minute: '2-digit', hour12: true });

const now = Math.floor(Date.now() / 1000);
const remaining = Math.floor((newExpiry - now) / 60);

// Update watchdog.json
watchdog.expiryEpoch = newExpiry;
watchdog.expiryHuman = newExpiryHuman;
writeJson(WATCHDOG_FILE, watchdog);

// No need to touch the running timer process. It calls restore-if-armed.mjs,
// which reads expiry from watchdog.json dynamically when it fires.

appendLog(CHANGE_LOG,
  `WATCHDOG EXTENDED\n  Added: addMinutes minutes\n  New expiry: newExpiryHuman\n  Remaining: ~remainingm`
);

console.log(`Watchdog extended. New expiry: newExpiryDisplay (~remainingm remaining)`);

FILE:scripts/recovery-test.mjs
#!/usr/bin/env node
// OpenClaw Emergency Rollback — recovery-test.mjs
// Usage: node recovery-test.mjs <subcommand>
//
// Subcommands:
//   preflight      — check all dependencies and system readiness
//   save-recovery  — copy current openclaw.json to openclaw.recovery
//   sabotage       — deliberately break openclaw.json (makes it invalid JSON)
//   verify         — check if openclaw.json was restored (is valid JSON again)

import { existsSync, readFileSync, writeFileSync, copyFileSync, statSync } from 'fs';
import { execSync } from 'child_process';
import {
  ROLLBACK_DIR, RECOVERY_FILE, CHANGE_LOG, RESTORE_LOG,
  getConfig, getOpenclawJson, getManifest, getWatchdog,
  readJson, appendLog
} from './utils.mjs';

const subcommand = process.argv[2];
if (!subcommand) {
  console.error('Usage: node recovery-test.mjs <preflight|save-recovery|sabotage|verify>');
  process.exit(1);
}

const OC_JSON = getOpenclawJson();

switch (subcommand) {

  case 'preflight': {
    console.log('=== Recovery Test Pre-Flight Check ===');
    let pass = true;

    // Check node (we're running, so it's there)
    console.log(`  ✓ node found: process.execPath`);

    // Check zip/unzip
    for (const tool of ['tar', 'gzip']) {
      try {
        execSync(`command -v tool`, { stdio: 'ignore' });
        console.log(`  ✓ tool found`);
      } catch {
        console.log(`  ✗ tool NOT FOUND — install before proceeding`);
        pass = false;
      }
    }

    console.log('  ✓ no cron dependency — watchdog uses detached Node timers and startup checks');

    // Check rollback directory
    if (existsSync(ROLLBACK_DIR)) {
      console.log('  ✓ rollback directory exists');
    } else {
      console.log('  ✗ rollback directory missing — run setup first');
      pass = false;
    }

    // Check manifest
    const manifest = getManifest();
    console.log(`  ✓ manifest.json (manifest.snapshots.length snapshots)`);

    // Check scripts
    const scripts = ['snapshot.mjs', 'restore.mjs', 'restore-if-armed.mjs', 'watchdog-set.mjs', 'watchdog-clear.mjs'];
    for (const s of scripts) {
      const p = `ROLLBACK_DIR/scripts/s`;
      if (existsSync(p)) {
        console.log(`  ✓ s exists`);
      } else {
        console.log(`  ✗ s missing`);
        pass = false;
      }
    }

    // Check openclaw.json
    if (existsSync(OC_JSON)) {
      const verifyParsed = readJson(OC_JSON);
      if (readJson(OC_JSON)) {
        console.log('  ✓ openclaw.json exists and is valid JSON');
      } else {
        console.log('  ✗ openclaw.json exists but is NOT valid JSON');
        pass = false;
      }
    } else {
      console.log(`  ✗ openclaw.json not found at OC_JSON`);
      pass = false;
    }

    // Check restart command
    const config = getConfig();
    if (config.restartCommand) {
      console.log(`  ✓ restart command: config.restartCommand`);
    } else {
      console.log('  ✗ restart command not configured');
      pass = false;
    }

    console.log('');
    if (pass) {
      console.log('All checks passed. Ready to test.');
    } else {
      console.log('Some checks FAILED. Fix the issues above before testing.');
      process.exit(1);
    }
    break;
  }

  case 'save-recovery': {
    if (!existsSync(OC_JSON)) {
      console.error(`ERROR: openclaw.json not found at OC_JSON`);
      process.exit(1);
    }

    copyFileSync(OC_JSON, RECOVERY_FILE);
    const config = getConfig();
    console.log(`Recovery copy saved: RECOVERY_FILE`);
    console.log('');
    console.log('If the test fails, restore manually with:');
    console.log(`  cp RECOVERY_FILE OC_JSON`);
    console.log(`  config.restartCommand || 'kill -USR1 1'`);

    appendLog(CHANGE_LOG,
      `RECOVERY TEST — MANUAL BACKUP SAVED\n  Source: OC_JSON\n  Backup: RECOVERY_FILE`
    );
    break;
  }

  case 'sabotage': {
    if (!existsSync(OC_JSON)) {
      console.error(`ERROR: openclaw.json not found at OC_JSON`);
      process.exit(1);
    }

    if (!existsSync(RECOVERY_FILE)) {
      console.error(`ERROR: No recovery copy found at RECOVERY_FILE`);
      console.error('Run "node recovery-test.mjs save-recovery" first.');
      process.exit(1);
    }

    const originalSize = statSync(OC_JSON).size;
    const parsed = readJson(OC_JSON);

    // 1. Poison the gateway auth token 
    if (parsed.gateway && parsed.gateway.auth) {
      parsed.gateway.auth.token = 'ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff';
    }

    // 2. Poison the agent workspace paths to break routing logically
    if (parsed.agents && Array.isArray(parsed.agents.list)) {
      parsed.agents.list.forEach(agent => {
        if (agent.workspace) {
          agent.workspace += 'x';
        }
      });
    }

    // Safe write
    writeFileSync(OC_JSON, JSON.stringify(parsed, null, 2));

    console.log('Config sabotaged logically. openclaw.json is VALID JSON, but contains a poisoned token and modified agent workspaces.');
    console.log(`Original size: originalSize bytes`);
    console.log('The watchdog should restore it automatically when the timer expires.');

    appendLog(CHANGE_LOG,
      `RECOVERY TEST — CONFIG SABOTAGED\n  File: OC_JSON\n  Method: Changed gateway token to ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff and poisoned workspace paths\n  Original size: originalSize bytes`
    );
    break;
  }

  case 'verify': {
    console.log('=== Recovery Test Verification ===');

    if (!existsSync(OC_JSON)) {
      console.log('  ✗ openclaw.json not found');
      console.log('RESULT: FAILED');
      process.exit(1);
    }

    const testParsed = readJson(OC_JSON);
    if (!testParsed) {
      console.log('  ✗ openclaw.json exists but is NOT valid JSON (this is unexpected)');
      process.exit(1);
    }
    
    if (testParsed.gateway?.auth?.token === 'ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff' || testParsed.agents?.list?.[0]?.workspace?.endsWith('x')) {
      console.log('  ✗ ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff still present OR workspaces still poisoned');
      console.log('  The Dead Man\'s Switch has NOT restored the config yet.');
      console.log('');
      console.log('RESULT: NOT YET RESTORED — wait longer for background switch to fire');
      console.log('Debug:');
      console.log(`  cat RESTORE_LOG       # restore attempts`);
      console.log(`  node ROLLBACK_DIR/scripts/watchdog-status.mjs  # timer`);
      process.exit(1);
    }

    if (testParsed) {
      console.log('  ✓ openclaw.json is valid JSON');
    } else {
      console.log('  ✗ openclaw.json exists but is NOT valid JSON');
      console.log('RESULT: PARTIAL — file may have been partially restored');
      process.exit(1);
    }

    const watchdog = getWatchdog();
    console.log(`  Watchdog armed: watchdog.armed`);

    if (existsSync(RESTORE_LOG)) {
      const log = readFileSync(RESTORE_LOG, 'utf8');
      const lastRestore = log.split('\n').filter(l => l.includes('RESTORE COMPLETE')).pop();
      if (lastRestore) console.log(`  Last restore: lastRestore.trim()`);
    }

    console.log('');
    console.log('RESULT: PASSED — recovery test successful');

    appendLog(CHANGE_LOG,
      `RECOVERY TEST — VERIFIED PASSED\n  openclaw.json: valid JSON, marker removed\n  Watchdog armed: watchdog.armed`
    );
    break;
}

  default:
    console.error(`Unknown subcommand: subcommand`);
    console.error('Usage: node recovery-test.mjs <preflight|save-recovery|sabotage|verify>');
    process.exit(1);
}

FILE:scripts/watchdog-clear.mjs
#!/usr/bin/env node
// OpenClaw Emergency Rollback — watchdog-clear.mjs
// Disarms the watchdog. Called when user accepts changes.

import { execSync } from 'child_process';
import {
  WATCHDOG_FILE, CHANGE_LOG,
  getWatchdog, writeJson, appendLog
} from './utils.mjs';

const watchdog = getWatchdog();
const now = Math.floor(Date.now() / 1000);

// Calculate time remaining at disarm
let remainingMsg = 'unknown';
if (watchdog.expiryEpoch) {
  if (watchdog.expiryEpoch > now) {
    const secs = watchdog.expiryEpoch - now;
    remainingMsg = `Math.floor(secs / 60)m secs % 60s`;
  } else {
    remainingMsg = 'already expired';
  }
}

// Disarm watchdog.json
watchdog.armed = false;
writeJson(WATCHDOG_FILE, watchdog);


// Kill any detached watchdog-timer.mjs processes
try {
  execSync('pkill -f watchdog-timer.mjs || true', { stdio: 'ignore' });
} catch {}

appendLog(CHANGE_LOG,
  `WATCHDOG DISARMED\n  Disarmed by: user accepted changes\n  Time remaining: remainingMsg`
);

console.log(`Watchdog disarmed. Time remaining was: remainingMsg`);

FILE:scripts/restore-if-armed.mjs
#!/usr/bin/env node
// OpenClaw Emergency Rollback — restore-if-armed.mjs
// Called by watchdog timer and native OpenClaw startup hook.
// Checks if watchdog is armed and timer has expired. Fires restore if so.

import { existsSync } from 'fs';
import { execSync } from 'child_process';
import { join } from 'path';
import { ROLLBACK_DIR, WATCHDOG_FILE, RESTORE_LOG, getWatchdog, appendLog } from './utils.mjs';

appendLog(RESTORE_LOG, `RESTORE-IF-ARMED — entered pid=process.pid ppid=process.ppid source=process.env.WATCHDOG_SOURCE || 'direct' triggeredByTimer=process.env.WATCHDOG_TRIGGERED === '1' triggeredByStartup=process.env.RESTART_TRIGGERED === '1'`);

if (!existsSync(WATCHDOG_FILE)) {
  appendLog(RESTORE_LOG, 'RESTORE-IF-ARMED — watchdog.json missing, exiting');
  process.exit(0);
}

const watchdog = getWatchdog();

if (!watchdog.armed) {
  appendLog(RESTORE_LOG, 'RESTORE-IF-ARMED — watchdog not armed, exiting');
  process.exit(0);
}

const now = Math.floor(Date.now() / 1000);
const expiry = watchdog.expiryEpoch || 0;
appendLog(RESTORE_LOG, `RESTORE-IF-ARMED — watchdog armed, now=now, expiry=expiry, remaining=expiry - nows`);

if (now >= expiry) {
  appendLog(RESTORE_LOG, 'RESTORE-IF-ARMED — watchdog armed and expired, triggering restore');
  process.env.RESTART_TRIGGERED = '1';
  const restoreScript = join(ROLLBACK_DIR, 'scripts', 'restore.mjs');
  try {
    execSync(`node "restoreScript"`, { stdio: 'inherit', env: { ...process.env, RESTART_TRIGGERED: '1' } });
  } catch (e) {
    appendLog(RESTORE_LOG, `RESTORE-IF-ARMED ERROR — restore.mjs failed: e?.status || 'unknown' e?.message || e`);
    process.exit(1);
  }
} else {
  const remaining = expiry - now;
  appendLog(RESTORE_LOG, `RESTORE-IF-ARMED — watchdog not expired, respawning timer for remainings`);
  const timerScript = join(ROLLBACK_DIR, 'scripts', 'watchdog-timer.mjs');
  if (existsSync(timerScript)) {
    const { spawn } = await import('child_process');
    const child = spawn(process.execPath, [timerScript, '0', String(remaining)], {
      detached: true,
      stdio: 'ignore',
      env: { ...process.env, WATCHDOG_SOURCE: 'restore-if-armed' }
    });
    child.unref();
    appendLog(RESTORE_LOG, `RESTORE-IF-ARMED — respawned watchdog timer pid=child.pid || 'unknown' remaining=remainings`);
  } else {
    appendLog(RESTORE_LOG, 'RESTORE-IF-ARMED ERROR — watchdog-timer.mjs missing, cannot respawn timer');
  }
}

process.exit(0);

FILE:scripts/watchdog-status.mjs
#!/usr/bin/env node
// OpenClaw Emergency Rollback — watchdog-status.mjs
// Reports current watchdog state and time remaining.

import { existsSync } from 'fs';
import { WATCHDOG_FILE, getWatchdog } from './utils.mjs';

if (!existsSync(WATCHDOG_FILE)) {
  console.log('NOT ARMED (no watchdog.json found)');
  process.exit(0);
}

const watchdog = getWatchdog();

if (!watchdog.armed) {
  console.log('NOT ARMED');
  process.exit(0);
}

const now = Math.floor(Date.now() / 1000);
const expiry = watchdog.expiryEpoch || 0;
const remaining = expiry - now;

if (remaining <= 0) {
  console.log('ARMED — timer expired, restore pending');
} else {
  const mins = Math.floor(remaining / 60);
  const secs = remaining % 60;
  console.log(`ARMED — minsm secss remaining`);
}

console.log(`Target: watchdog.targetSnapshot || 'snapshot-1' — "watchdog.targetLabel || 'unknown'"`);
console.log(`Expiry: watchdog.expiryHuman || 'unknown'`);

FILE:scripts/utils.mjs
// OpenClaw Emergency Rollback — shared utilities
// All JSON I/O goes through here. No string interpolation of user data into code.

import { readFileSync, writeFileSync, mkdirSync, existsSync } from 'fs';
import { join, dirname } from 'path';

const HOME = process.env.HOME || '/root';

export const ROLLBACK_DIR = join(HOME, '.openclaw/rollback');
export const CONFIG_FILE = join(ROLLBACK_DIR, 'rollback-config.json');
export const MANIFEST_FILE = join(ROLLBACK_DIR, 'manifest.json');
export const WATCHDOG_FILE = join(ROLLBACK_DIR, 'watchdog.json');
export const CHANGE_LOG = join(ROLLBACK_DIR, 'logs', 'change.log');
export const RESTORE_LOG = join(ROLLBACK_DIR, 'logs', 'restore.log');
export const SNAPSHOTS_DIR = join(ROLLBACK_DIR, 'snapshots');
export const RECOVERY_FILE = join(ROLLBACK_DIR, 'openclaw.recovery');

export function readJson(filepath) {
  try {
    return JSON.parse(readFileSync(filepath, 'utf8'));
  } catch {
    return null;
  }
}

export function writeJson(filepath, data) {
  mkdirSync(dirname(filepath), { recursive: true });
  writeFileSync(filepath, JSON.stringify(data, null, 2) + '\n');
}

export function getConfig() {
  const config = readJson(CONFIG_FILE);
  if (!config) {
    console.error('ERROR: rollback-config.json not found. Run setup first.');
    process.exit(1);
  }
  return config;
}

export function getOpenclawHome() {
  const config = getConfig();
  return (config.openclawHome || '~/.openclaw').replace('~', HOME);
}

export function getOpenclawJson() {
  return join(getOpenclawHome(), 'openclaw.json');
}

export function getManifest() {
  return readJson(MANIFEST_FILE) || { watchdog_target: 'snapshot-1', snapshots: [] };
}

export function getWatchdog() {
  return readJson(WATCHDOG_FILE) || { armed: false };
}

export function appendLog(logFile, entry) {
  const ts = new Date().toISOString().replace('T', ' ').replace(/\.\d+Z$/, '');
  mkdirSync(dirname(logFile), { recursive: true });
  const existing = existsSync(logFile) ? readFileSync(logFile, 'utf8') : '';
  writeFileSync(logFile, existing + `[ts] entry\n---\n`);
}

export function timestamp() {
  return new Date().toISOString().replace(/\.\d+Z$/, 'Z');
}

export function timestampHuman() {
  return new Date().toISOString().replace('T', ' ').replace(/\.\d+Z$/, '');
}

FILE:scripts/restore.mjs
#!/usr/bin/env node
// OpenClaw Emergency Rollback — restore.mjs
// Usage: node restore.mjs [slot]
// Restores a snapshot and restarts OpenClaw.
// Must work with zero AI, zero network, zero user interaction.

import { existsSync } from 'fs';
import { execSync } from 'child_process';
import { join } from 'path';
import {
  ROLLBACK_DIR, SNAPSHOTS_DIR, WATCHDOG_FILE, RESTORE_LOG,
  readJson, writeJson, getConfig, getManifest, getWatchdog,
  appendLog, timestampHuman
} from './utils.mjs';

const SLOT = parseInt(process.argv[2] || '1', 10);
const config = getConfig();
const manifest = getManifest();
const RESTART_CMD = config.restartCommand || 'kill -USR1 1';

// Find snapshot info
const snapInfo = manifest.snapshots.find(s => s.slot === SLOT);
const snapFile = snapInfo ? snapInfo.file : `snapshot-SLOT.tar.gz`;
const snapLabel = snapInfo ? snapInfo.label : 'unknown';
const snapTs = snapInfo ? snapInfo.timestamp : 'unknown';
const zipPath = join(SNAPSHOTS_DIR, snapFile);

if (!existsSync(zipPath)) {
  const msg = `RESTORE FAILED — zip not found: zipPath`;
  appendLog(RESTORE_LOG, msg);
  console.error(`ERROR: msg`);
  process.exit(1);
}

// Determine trigger method
const trigger = process.env.RESTART_TRIGGERED === '1'
  ? 'startup restore check / watchdog'
  : process.env.WATCHDOG_TRIGGERED === '1'
    ? 'detached watchdog timer (timer expired)'
    : 'manual';

// Log restore start
appendLog(RESTORE_LOG,
  `RESTORE TRIGGERED\n  Method: trigger\n  Target: snapshot-SLOT — "snapLabel"\n  Snapshot timestamp: snapTs\n  Zip: zipPath`
);

// Restore files — unzip with full path overwrite to /
let unzipExit = 0;
try {
  execSync(`tar -xzf "zipPath" -C /`, { stdio: 'ignore' });
} catch (e) {
  unzipExit = e.status || 1;
  appendLog(RESTORE_LOG, `RESTORE WARNING — unzip exit code: unzipExit`);
}

// Disarm watchdog
const watchdog = getWatchdog();
watchdog.armed = false;
writeJson(WATCHDOG_FILE, watchdog);

// Stop any detached watchdog timer process that may still be running
try {
  execSync('pkill -f watchdog-timer.mjs || true', { stdio: 'ignore' });
} catch {}

// Run restart command
let restartExit = 0;
try {
  execSync(RESTART_CMD, { stdio: 'inherit', shell: '/bin/bash' });
} catch (e) {
  restartExit = e.status || 1;
}

// Log restore complete
appendLog(RESTORE_LOG,
  `RESTORE COMPLETE\n  Restart command: RESTART_CMD\n  Restart exit: restartExit\n  Unzip exit: unzipExit`
);

process.exit(0);

FILE:scripts/watchdog-timer.mjs
#!/usr/bin/env node
import { existsSync } from 'fs';
import { join } from 'path';
import { execSync } from 'child_process';
import { getWatchdog, WATCHDOG_FILE, ROLLBACK_DIR, RESTORE_LOG, appendLog } from './utils.mjs';

const minutesArgs = parseInt(process.argv[2], 10) || 0;
const explicitSeconds = parseInt(process.argv[3], 10) || 0;

let timeoutMs = 0;
if (explicitSeconds > 0) {
  timeoutMs = explicitSeconds * 1000;
} else if (minutesArgs > 0) {
  timeoutMs = minutesArgs * 60 * 1000;
}

appendLog(RESTORE_LOG, `WATCHDOG TIMER — started pid=process.pid ppid=process.ppid timeoutMs=timeoutMs`);

if (timeoutMs <= 0) {
  appendLog(RESTORE_LOG, 'WATCHDOG TIMER — exiting immediately because timeoutMs <= 0');
  process.exit(0);
}

setTimeout(() => {
  appendLog(RESTORE_LOG, `WATCHDOG TIMER — fired pid=process.pid`);
  if (!existsSync(WATCHDOG_FILE)) {
    appendLog(RESTORE_LOG, 'WATCHDOG TIMER — watchdog.json missing at fire time, exiting');
    process.exit(0);
  }

  const watchdog = getWatchdog();
  appendLog(RESTORE_LOG, `WATCHDOG TIMER — armed=Boolean(watchdog.armed) expiry=watchdog.expiryEpoch || 'null'`);

  if (watchdog.armed) {
    const restoreIfArmed = join(ROLLBACK_DIR, 'scripts', 'restore-if-armed.mjs');
    try {
      appendLog(RESTORE_LOG, `WATCHDOG TIMER — invoking restore-if-armed.mjs via restoreIfArmed`);
      execSync(`node "restoreIfArmed"`, {
        stdio: 'ignore',
        env: { ...process.env, WATCHDOG_TRIGGERED: '1' }
      });
      appendLog(RESTORE_LOG, 'WATCHDOG TIMER — restore-if-armed.mjs returned without throwing');
    } catch (e) {
      appendLog(RESTORE_LOG, `WATCHDOG TIMER ERROR — restore-if-armed.mjs failed: e?.message || e`);
    }
  } else {
    appendLog(RESTORE_LOG, 'WATCHDOG TIMER — watchdog not armed at fire time, exiting');
  }
  process.exit(0);
}, timeoutMs);

FILE:scripts/snapshot.mjs
#!/usr/bin/env node
// OpenClaw Emergency Rollback — snapshot.mjs
// Usage: node snapshot.mjs "<label>" "<ai_summary>"
// Takes a labeled snapshot of all OpenClaw config files.

import { readFileSync, existsSync, mkdirSync, copyFileSync, renameSync, unlinkSync } from 'fs';
import { join, dirname } from 'path';
import { execSync } from 'child_process';
import { mkdtempSync } from 'fs';
import { tmpdir } from 'os';
import {
  ROLLBACK_DIR, MANIFEST_FILE, SNAPSHOTS_DIR, CHANGE_LOG,
  readJson, writeJson, getOpenclawHome, getOpenclawJson, getManifest,
  appendLog, timestamp, timestampHuman
} from './utils.mjs';

const LABEL = process.argv[2] || 'unlabeled';
const AI_SUMMARY = process.argv[3] || 'No summary provided.';

const OC_HOME = getOpenclawHome();
const OC_JSON = getOpenclawJson();

if (!existsSync(OC_JSON)) {
  console.error(`ERROR: openclaw.json not found at OC_JSON`);
  process.exit(1);
}

// Read openclaw.json to extract workspace paths and agent IDs
const ocConfig = readJson(OC_JSON);
const HOME = process.env.HOME || '/root';

// Extract workspace paths
const workspacePaths = new Set();
if (ocConfig?.agents) {
  if (ocConfig.agents.defaults?.workspace) {
    workspacePaths.add(ocConfig.agents.defaults.workspace.replace('~', HOME));
  }
  if (ocConfig.agents.list) {
    ocConfig.agents.list.forEach(a => {
      if (a.workspace) workspacePaths.add(a.workspace.replace('~', HOME));
    });
  }
}
if (workspacePaths.size === 0) {
  workspacePaths.add(join(HOME, '.openclaw', 'workspace'));
}

// Note: agent IDs are not extracted because auth-profiles.json is
// deliberately excluded from snapshots (sensitive credentials).

// Stage files into a temp dir preserving full absolute paths
const stageDir = mkdtempSync(join(tmpdir(), 'oc-snapshot-'));
const filesCaptured = [];

// Stage openclaw.json
const stagedOcJson = join(stageDir, OC_JSON);
mkdirSync(dirname(stagedOcJson), { recursive: true });
copyFileSync(OC_JSON, stagedOcJson);
filesCaptured.push(OC_JSON);

// Stage workspace config files
const WORKSPACE_FILES = ['SOUL.md', 'AGENTS.md', 'USER.md', 'IDENTITY.md', 'TOOLS.md', 'HEARTBEAT.md', 'BOOT.md'];
for (const wsPath of workspacePaths) {
  for (const wf of WORKSPACE_FILES) {
    const src = join(wsPath, wf);
    if (existsSync(src)) {
      const dest = join(stageDir, src);
      mkdirSync(dirname(dest), { recursive: true });
      copyFileSync(src, dest);
      filesCaptured.push(src);
    }
  }
}

// Auth profiles (auth-profiles.json) are deliberately NOT captured.
// They contain sensitive credentials and must never be stored in snapshots.

// Create zip from staging dir
const tmpZip = join(tmpdir(), 'oc-snapshot-tmp.tar.gz');
try { unlinkSync(tmpZip); } catch {}
execSync(`cd "stageDir" && tar -czf "tmpZip" .`, { stdio: 'ignore' });

// Clean up staging dir
execSync(`rm -rf "stageDir"`);

// Rotate snapshots: 2→3, 1→2, new→1
mkdirSync(SNAPSHOTS_DIR, { recursive: true });
const snap3 = join(SNAPSHOTS_DIR, 'snapshot-3.tar.gz');
const snap2 = join(SNAPSHOTS_DIR, 'snapshot-2.tar.gz');
const snap1 = join(SNAPSHOTS_DIR, 'snapshot-1.tar.gz');

if (existsSync(snap2)) {
  if (existsSync(snap3)) unlinkSync(snap3);
  renameSync(snap2, snap3);
}
if (existsSync(snap1)) {
  renameSync(snap1, snap2);
}
copyFileSync(tmpZip, snap1); unlinkSync(tmpZip);

// Update manifest.json — shift existing entries, add new slot 1
const manifest = getManifest();
const shifted = manifest.snapshots
  .filter(s => s.slot <= 2)
  .map(s => ({ ...s, slot: s.slot + 1, file: `snapshot-s.slot + 1.tar.gz` }));

shifted.unshift({
  slot: 1,
  file: 'snapshot-1.tar.gz',
  label: LABEL,
  timestamp: timestamp(),
  ai_summary: AI_SUMMARY
});

manifest.snapshots = shifted.filter(s => s.slot <= 3);
manifest.watchdog_target = 'snapshot-1';
writeJson(MANIFEST_FILE, manifest);

// Log
appendLog(CHANGE_LOG,
  `SNAPSHOT TAKEN\n  Slot: 1 (previous snapshots shifted)\n  Label: "LABEL"\n  Summary: AI_SUMMARY\n  Files: filesCaptured.join(', ')`
);

console.log(`Snapshot saved: slot 1 — LABEL (timestamp())`);

FILE:references/RESTORE.md
---
name: openclaw-emergency-rollback/restore
description: Manual recovery instructions for restoring OpenClaw config without AI, scripts, or network access. For use when everything is broken.
---

# Manual Recovery — No AI Required

Use this document if you have shell access but cannot use AI, the scripts
failed, or you want to manually restore a specific snapshot.

You need: a terminal, basic shell access, `unzip`, and the ability to run
one command to restart OpenClaw.

---

## Step 1 — Find Your Snapshots

```bash
ls -lh ~/.openclaw/rollback/snapshots/
```

You will see up to three files:
- `snapshot-1.zip` — most recent user-approved snapshot
- `snapshot-2.zip` — second most recent
- `snapshot-3.zip` — oldest

To see labels and timestamps:

```bash
node -e "
const m=require(process.env.HOME+'/.openclaw/rollback/manifest.json');
m.snapshots.forEach(s=>console.log('['+s.slot+'] '+s.label+' ('+s.timestamp+')'));
"
```

Or just read the raw file: `cat ~/.openclaw/rollback/manifest.json`

---

## Step 2 — Restore the Snapshot

Replace `snapshot-1.zip` with whichever snapshot you want:

```bash
unzip -o ~/.openclaw/rollback/snapshots/snapshot-1.zip -d /
```

This restores all files to their exact original paths:
- `~/.openclaw/openclaw.json`
- All agent workspace files (SOUL.md, AGENTS.md, etc.)

No path mapping needed — the zip preserves full absolute paths.

---

## Step 3 — Restart OpenClaw

Check what restart command was configured:

```bash
cat ~/.openclaw/rollback/rollback-config.json
```

Look for `"restartCommand"` and run it. Examples:

```bash
kill -USR1 1
systemctl --user restart openclaw-gateway
docker compose restart
docker compose down && docker compose up -d
```

---

## Step 4 — Verify

```bash
openclaw gateway status
```

You should see the gateway as active and running.

---

## Step 5 — Disarm the Watchdog (if still armed)

If the detached watchdog timer might still be running, disarm it so it doesn't fire again:

```bash
# Stop any detached watchdog timer processes
pkill -f watchdog-timer.mjs || true

# Mark watchdog as disarmed
node -e "
const fs=require('fs');
const wf=process.env.HOME+'/.openclaw/rollback/watchdog.json';
const w=JSON.parse(fs.readFileSync(wf,'utf8'));
w.armed=false;
fs.writeFileSync(wf,JSON.stringify(w,null,2));
console.log('Watchdog disarmed.');
"
```

---

## If You Have a Recovery File

If a recovery test was run, there may be a clean config backup at:

```bash
ls -lh ~/.openclaw/rollback/openclaw.recovery
```

If this file exists and your snapshots are corrupted or missing:

```bash
cp ~/.openclaw/rollback/openclaw.recovery ~/.openclaw/openclaw.json
```

Then restart as described in Step 3.

---

## Logs

```bash
cat ~/.openclaw/rollback/logs/restore.log    # automated restore history
cat ~/.openclaw/rollback/logs/change.log     # all changes, snapshots, watchdog events
```

---

## Summary (Quickest Path)

```bash
# 1. Restore snapshot
unzip -o ~/.openclaw/rollback/snapshots/snapshot-1.zip -d /

# 2. Restart (use your actual command)
kill -USR1 1

# 3. Disarm watchdog timer
pkill -f watchdog-timer.mjs || true
```

That's it.

FILE:references/TESTING.md
---
name: openclaw-emergency-rollback/testing
description: Destructive recovery test procedure. Read this when the user wants to test that the emergency rollback system actually works end-to-end.
---

# Emergency Recovery Test — Destructive

This test verifies the full recovery pipeline by deliberately breaking the
OpenClaw config and confirming the watchdog automatically restores it.

**This test is destructive.** During the test window (up to ~2 minutes), the
user's OpenClaw gateway will be non-functional. AI sessions, agents, and any
active connections will be interrupted.

---

## Before You Begin — Pre-Flight Checklist

Confirm ALL of these with the user before proceeding:

```
⚠️  Emergency Recovery Test — Pre-Flight Checklist

This test will:
  1. Save your current config as a test snapshot
  2. Save a manual recovery copy of openclaw.json
  3. Deliberately break your openclaw.json
  4. Restart the gateway (it will fail)
  5. Wait for the watchdog to auto-restore (~2 minutes)

During the test you WILL lose access to your AI session.

Requirements:
  □ You have terminal/SSH access to this machine right now
  □ You can run commands even if the AI agent is offline
  □ You understand this will interrupt all active sessions

Manual recovery command (if the test fails — keep this visible):
  cp ~/.openclaw/rollback/openclaw.recovery ~/.openclaw/openclaw.json
  <your restart command here>

Type "yes, run the test" to proceed.
```

Fill in the actual restart command from `~/.openclaw/rollback/rollback-config.json`.

Do NOT proceed unless the user explicitly confirms.

---

## Test Procedure

### Step 1 — Verify Dependencies

```bash
~/.openclaw/rollback/scripts/recovery-test.mjs preflight
```

This checks that node, tar, and gzip are available, that the rollback
directory is properly initialized, and that all scripts are present.
If anything fails, stop and fix it before continuing.

### Step 2 — Create Test Snapshot

```bash
~/.openclaw/rollback/scripts/snapshot.mjs "pre-test known-good config" "Snapshot taken before recovery test."
```

This saves the current working config as snapshot [1].

### Step 3 — Save Manual Recovery Copy

```bash
~/.openclaw/rollback/scripts/recovery-test.mjs save-recovery
```

This copies `openclaw.json` to `~/.openclaw/rollback/openclaw.recovery`.
This is the user's last-resort manual recovery if everything else fails.

Tell the user:

```
📋 Manual recovery copy saved. If the test fails and the watchdog does not
restore your config within 5 minutes, run these two commands from any terminal:

  cp ~/.openclaw/rollback/openclaw.recovery ~/.openclaw/openclaw.json
  <restart command>

Keep this window open or write these commands down before proceeding.
```

### Step 4 — Arm the Watchdog (2 minutes)

```bash
~/.openclaw/rollback/scripts/watchdog-set.mjs 2
```

The watchdog is now armed. If nothing disarms it in 2 minutes, it will
automatically restore snapshot [1] and restart the gateway.

### Step 5 — Break the Config

```bash
~/.openclaw/rollback/scripts/recovery-test.mjs sabotage
```

This inserts `BROKEN_BY_RECOVERY_TEST` as the first line of `openclaw.json`,
making it invalid JSON. The gateway won't load, AI won't connect, nothing works.

### Step 6 — Restart the Gateway

Read the restart command from rollback-config.json and run it:

```bash
RESTART_CMD=$(node -e "console.log(require('$HOME/.openclaw/rollback/rollback-config.json').restartCommand)")
eval "$RESTART_CMD"
```

The gateway will attempt to start, fail to parse the broken config, and either
crash or run in a degraded state. This is expected.

### Step 7 — Wait for Recovery

The detached watchdog timer can fire at expiry, and the native `gateway:startup`
hook can recover on restart if the timer died. Combined with restart/setup overhead,
recovery should usually happen within ~2-3 minutes. The user should:

1. Wait 3 minutes
2. Try to connect to their agent
3. If the agent is back and working, the test passed

To verify programmatically:

```bash
~/.openclaw/rollback/scripts/recovery-test.mjs verify
```

### Step 8 — Report Results

If the config is restored and the gateway is running:

```
✅ Recovery test PASSED.

The watchdog detected the expired timer, restored snapshot [1],
and restarted the gateway automatically.

Your manual recovery copy is still at:
  ~/.openclaw/rollback/openclaw.recovery
You can delete it or keep it as an extra backup.
```

If the config was NOT restored after 5 minutes:

```
❌ Recovery test FAILED.

The watchdog did not fire. Possible causes:
  • the detached watchdog timer process never started
  • `watchdog-timer.mjs` was killed unexpectedly
  • the native `watchdog-recovery` hook is not installed/enabled
  • the startup hook ran but failed before invoking `restore-if-armed.mjs`

To restore manually:
  cp ~/.openclaw/rollback/openclaw.recovery ~/.openclaw/openclaw.json
  <restart command>

Check the logs:
  cat ~/.openclaw/rollback/logs/restore.log
  cat ~/.openclaw/rollback/logs/change.log
```

---

## What the Test Validates

1. **Snapshot creation** — config files are captured and zipped correctly
2. **Watchdog arming** — detached timer started with correct expiry
3. **Startup hook recovery** — native `gateway:startup` hook re-checks persistent watchdog state after restart
4. **Timer expiry detection** — restore-if-armed.mjs checks epoch against expiry
5. **Restore execution** — archive extracted to correct paths, overwriting broken files
6. **Gateway restart** — restart command fires after restore
7. **Watchdog disarm** — watchdog state cleared after firing

A full destructive test should exercise both timer-path and startup-hook recovery evidence in `restore.log`.

---

## Cleaning Up After a Failed Test

If the automatic recovery didn't fire:

```bash
# 1. Restore the config
cp ~/.openclaw/rollback/openclaw.recovery ~/.openclaw/openclaw.json

# 2. Restart the gateway (use your actual command)
kill -USR1 1

# 3. Disarm the watchdog timer so it doesn't fire later
pkill -f watchdog-timer.mjs || true

# 4. Mark watchdog as disarmed
node -e "
const fs=require('fs');
const wf=process.env.HOME+'/.openclaw/rollback/watchdog.json';
const w=JSON.parse(fs.readFileSync(wf,'utf8'));
w.armed=false;
fs.writeFileSync(wf,JSON.stringify(w,null,2));
console.log('Watchdog disarmed.');
"

# 5. Verify
cat ~/.openclaw/openclaw.json | node -e "JSON.parse(require('fs').readFileSync('/dev/stdin','utf8'));console.log('Config is valid JSON')"
```

FILE:references/SETUP.md
---
name: openclaw-emergency-rollback/setup
description: One-time setup for OpenClaw Emergency Rollback. Read this when the user wants to install or initialize the rollback system for the first time.
---

# OpenClaw Emergency Rollback — One-Time Setup

Run this setup exactly once when `~/.openclaw/rollback/` does not exist.
Do not re-run setup if the directory already exists unless the user explicitly
asks to reinstall.

---

## Prerequisites

The rollback system uses Node.js (already installed with OpenClaw) and
standard Linux tools. Verify before proceeding:

```bash
echo "--- Checking dependencies ---"
node --version && echo "  ✓ node" || echo "  ✗ node NOT FOUND"
command -v tar >/dev/null && echo "  ✓ tar" || echo "  ✗ tar NOT FOUND"
command -v unzip >/dev/null && echo "  ✓ gzip" || echo "  ✗ gzip NOT FOUND"
echo "  ✓ no cron dependency — rollback uses detached Node timers plus a native OpenClaw gateway startup hook"
```

Node.js is required to run OpenClaw itself, so it is always present. If `zip`
or `unzip` are missing (common on stripped Docker images), install them:

- **Ubuntu/Debian VPS:** `sudo apt-get install -y tar gzip`
- **Docker (node:22-bookworm-slim):** Set `OPENCLAW_DOCKER_APT_PACKAGES="tar gzip"`
  in your Docker setup, or add to Dockerfile: `RUN apt-get update && apt-get install -y tar gzip`

---

## Step 1 — Ask for Restart Command

Before creating anything, ask the user:

```
To complete setup I need to know how to restart OpenClaw on your system.
What command restarts your OpenClaw gateway?

Common options:
  • kill -USR1 1                       (standard Linux install)
  • systemctl --user restart openclaw-gateway      (explicit systemd)
  • docker compose restart                         (Docker Compose)
  • docker compose down && docker compose up -d    (Docker full cycle)

Or enter a custom command for your setup.
```

Store the answer as RESTART_CMD. This is written to `rollback-config.json`
and never changed automatically after setup.

Also ask (or detect from `~/.openclaw/openclaw.json` if it exists):
- Confirm openclaw home is `~/.openclaw` (or ask if different)

Store as OC_HOME. Use the full absolute path (expand `~` to `$HOME`).

---

## Step 2 — Create Directory Structure

```bash
mkdir -p ~/.openclaw/rollback/snapshots
mkdir -p ~/.openclaw/rollback/scripts
mkdir -p ~/.openclaw/rollback/logs
```

---

## Step 3 — Write rollback-config.json

Use the absolute path for openclawHome (expand `~` to `$HOME`):

```bash
cat > ~/.openclaw/rollback/rollback-config.json << EOF
{
  "restartCommand": "kill -USR1 1",
  "openclawHome": "$HOME/.openclaw",
  "installedAt": "$(date -u +"%Y-%m-%dT%H:%M:%SZ")"
}
EOF
```

If the user specified a different openclaw home path, use that instead.

> NOTE: The restart command is hardcoded to `kill -USR1 1` for this environment.

---

## Step 4 — Initialize watchdog.json

```bash
cat > ~/.openclaw/rollback/watchdog.json << 'EOF'
{
  "armed": false,
  "setAt": null,
  "expiryEpoch": null,
  "expiryHuman": null,
  "minutesSet": null,
  "targetSnapshot": "snapshot-1",
  "targetLabel": null
}
EOF
```

---

## Step 5 — Initialize manifest.json

```bash
cat > ~/.openclaw/rollback/manifest.json << 'EOF'
{
  "watchdog_target": "snapshot-1",
  "snapshots": []
}
EOF
```

---

## Step 6 — Copy All Scripts

Copy every file from this skill's `scripts/` directory into
`~/.openclaw/rollback/scripts/`. This includes:

- `utils.mjs` — shared Node.js module (imported by all `.mjs` scripts)
- `snapshot.mjs` 
- `restore.mjs` 
- `restore-if-armed.mjs` 
- `watchdog-set.mjs` 
- `watchdog-extend.mjs` 
- `watchdog-clear.mjs` 
- `watchdog-status.mjs` 
- `recovery-test.mjs` 

After copying, make the shell wrappers executable:

```bash
chmod +x ~/.openclaw/rollback/scripts/*.mjs
```

The `.mjs` files have `#!/usr/bin/env node` shebangs, so once they have the
execute bit, the agent or your startup scripts can call them directly without a shell wrapper.

---

## Step 7 — Install Native OpenClaw Startup Hook

This ensures that if OpenClaw restarts while the watchdog is armed, the recovery
check runs again natively inside OpenClaw on `gateway:startup`.

Create the managed hook under `~/.openclaw/hooks/watchdog-recovery/` with:
- `HOOK.md`
- `handler.ts`

Use the versions shipped in this skill under:
- `hooks/watchdog-recovery/HOOK.md`
- `hooks/watchdog-recovery/handler.ts`

Then enable the hook:

```bash
openclaw hooks enable watchdog-recovery
openclaw hooks check
openclaw hooks list
```

Expected behavior:
- if watchdog is unarmed, the hook exits immediately
- if watchdog is armed and expired, the hook runs `restore-if-armed.mjs`
- if watchdog is armed and not yet expired, the hook respawns `watchdog-timer.mjs` for the remaining time

This works the same way on pod, Docker, and local installs because it uses
OpenClaw's own native startup lifecycle instead of external schedulers.

---

## Step 8 — Confirm Setup Complete

```
✅ OpenClaw Emergency Rollback installed.

Location: ~/.openclaw/rollback/
Restart command: kill -USR1 1
Scripts: Node.js (.mjs) — directly executable
Startup recovery: native OpenClaw hook `watchdog-recovery` on `gateway:startup`

Next step: say "create snapshot" to save your current known-good config
before making any changes.

Optional: say "test emergency recovery" to run a destructive test that
verifies the full recovery pipeline works end-to-end.
```

---

## Reinstall / Reset

If the user wants to reinstall from scratch:
1. Back up existing snapshots: `cp -r ~/.openclaw/rollback/snapshots/ /tmp/openclaw-snapshots-backup/`
2. `rm -rf ~/.openclaw/rollback/`
3. Remove any old startup hook that points at a previous rollback install, if present.
4. Run setup again from Step 1.
5. Ask the user if they want their old snapshots restored from the backup.

Never silently delete snapshots — always back them up first and ask.

FILE:hooks/watchdog-recovery/handler.ts
import { existsSync, mkdirSync, readFileSync, writeFileSync } from 'fs';
import { join, dirname } from 'path';
import { spawn, execSync } from 'child_process';

function appendLog(logFile: string, entry: string) {
  mkdirSync(dirname(logFile), { recursive: true });
  const ts = new Date().toISOString().replace('T', ' ').replace(/\.\d+Z$/, '');
  const existing = existsSync(logFile) ? readFileSync(logFile, 'utf8') : '';
  writeFileSync(logFile, existing + `[ts] entry\n---\n`);
}

const handler = async (event: any) => {
  if (event?.type !== 'gateway' || event?.action !== 'startup') return;

  const home = process.env.OPENCLAW_HOME || join(process.env.HOME || '/home/node', '.openclaw');
  const rollbackDir = join(home, 'rollback');
  const watchdogFile = join(rollbackDir, 'watchdog.json');
  const restoreLog = join(rollbackDir, 'logs', 'restore.log');
  const restoreScript = join(rollbackDir, 'scripts', 'restore-if-armed.mjs');
  const timerScript = join(rollbackDir, 'scripts', 'watchdog-timer.mjs');

  appendLog(restoreLog, 'HOOK gateway:startup — entered watchdog recovery hook');

  if (!existsSync(watchdogFile)) {
    appendLog(restoreLog, 'HOOK gateway:startup — watchdog.json missing, nothing to do');
    return;
  }
  if (!existsSync(restoreScript)) {
    appendLog(restoreLog, 'HOOK gateway:startup — restore-if-armed.mjs missing, nothing to do');
    return;
  }

  let watchdog: any = null;
  try {
    watchdog = JSON.parse(readFileSync(watchdogFile, 'utf8'));
  } catch (e: any) {
    appendLog(restoreLog, `HOOK gateway:startup — failed to parse watchdog.json: e?.message || e`);
    return;
  }

  if (!watchdog?.armed) {
    appendLog(restoreLog, 'HOOK gateway:startup — watchdog not armed, nothing to do');
    return;
  }

  const now = Math.floor(Date.now() / 1000);
  const expiry = watchdog.expiryEpoch || 0;
  const remaining = expiry - now;

  appendLog(restoreLog, `HOOK gateway:startup — watchdog armed, expiry=expiry, now=now, remaining=remainings`);

  if (now >= expiry) {
    appendLog(restoreLog, 'HOOK gateway:startup — watchdog expired, invoking restore-if-armed.mjs');
    try {
      execSync(`node "restoreScript"`, {
        stdio: 'ignore',
        env: { ...process.env, WATCHDOG_TRIGGERED: '1', WATCHDOG_SOURCE: 'gateway:startup-hook' }
      });
      appendLog(restoreLog, 'HOOK gateway:startup — restore-if-armed.mjs returned');
    } catch (e: any) {
      appendLog(restoreLog, `HOOK gateway:startup — restore-if-armed.mjs failed: e?.message || e`);
    }
    return;
  }

  if (!existsSync(timerScript)) {
    appendLog(restoreLog, 'HOOK gateway:startup — watchdog-timer.mjs missing, cannot respawn timer');
    return;
  }

  try {
    const child = spawn(process.execPath, [timerScript, '0', String(remaining)], {
      detached: true,
      stdio: 'ignore',
      env: { ...process.env, WATCHDOG_SOURCE: 'gateway:startup-hook' }
    });
    child.unref();
    appendLog(restoreLog, `HOOK gateway:startup — respawned watchdog timer pid=child.pid || 'unknown' remaining=remainings`);
  } catch (e: any) {
    appendLog(restoreLog, `HOOK gateway:startup — failed to respawn watchdog timer: e?.message || e`);
  }
};

export default handler;

FILE:hooks/watchdog-recovery/HOOK.md
---
name: watchdog-recovery
description: "On gateway startup, recover or re-arm the emergency rollback watchdog from persistent disk"
metadata:
  { "openclaw": { "emoji": "🛡️", "events": ["gateway:startup"], "requires": { "bins": ["node"] } } }
---

# Watchdog Recovery Hook

Runs on OpenClaw gateway startup.

Purpose:
- If rollback is not armed, do nothing.
- If rollback is armed and expired, run `restore-if-armed.mjs` immediately.
- If rollback is armed and not yet expired, respawn the detached watchdog timer for the remaining time.

This hook is the native OpenClaw restart trigger for rollback recovery.
It does not require AI, internet, cron, or any external supervisor.

---
name: approved-self-improvement
description: "Captures learnings, errors, and corrections to enable continuous improvement with user-approved skill updates. Use when: (1) A command or operation fails unexpectedly, (2) User corrects Claude ('No, that's wrong...', 'Actually...'), (3) User requests a capability that doesn't exist, (4) An external API or tool fails, (5) Claude realizes its knowledge is outdated or incorrect, (6) A better approach is discovered for a recurring task, (7) User asks 'what skill improvements do you recommend?' or 'show me pending skill updates' or similar, (8) A skill failure matches an existing improvement proposal. Also review learnings and pending improvement proposals before major tasks. IMPORTANT: Never modify any skill without explicit user approval unless the user has authorized auto-update for that specific skill."
metadata:
---

# Approved-Self-Improvement Skill

Log learnings and errors to markdown files for continuous improvement. Coding agents can later process these into fixes, and important learnings get promoted to project memory. **All skill modifications require explicit user approval** — improvement proposals are documented and presented to the user before any changes are applied. See [Approval-Gated Skill Improvement](#approval-gated-skill-improvement) for the full workflow.

## First-Use Initialisation

Before logging anything, ensure the `.learnings/` directory and files exist in the project or workspace root. If any are missing, create them:

```bash
mkdir -p .learnings/pending-improvements
[ -f .learnings/LEARNINGS.md ] || printf "# Learnings\n\nCorrections, insights, and knowledge gaps captured during development.\n\n**Categories**: correction | insight | knowledge_gap | best_practice\n\n---\n" > .learnings/LEARNINGS.md
[ -f .learnings/ERRORS.md ] || printf "# Errors\n\nCommand failures and integration errors.\n\n---\n" > .learnings/ERRORS.md
[ -f .learnings/FEATURE_REQUESTS.md ] || printf "# Feature Requests\n\nCapabilities requested by the user.\n\n---\n" > .learnings/FEATURE_REQUESTS.md
[ -f .learnings/AUTO_UPDATE_AUTHORIZATIONS.md ] || printf "# Auto-Update Authorizations\n\nSkills authorized for automatic self-improvement without user approval.\nBy default, NO skills are authorized. Users must explicitly grant per-skill.\n\n---\n" > .learnings/AUTO_UPDATE_AUTHORIZATIONS.md
```

Never overwrite existing files. This is a no-op if `.learnings/` is already initialised.

Do not log secrets, tokens, private keys, environment variables, or full source/config files unless the user explicitly asks for that level of detail. Prefer short summaries or redacted excerpts over raw command output or full transcripts.

If you want automatic reminders or setup assistance, use the opt-in hook workflow described in [Hook Integration](#hook-integration).

## Quick Reference

| Situation | Action |
|-----------|--------|
| Command/operation fails | Log to `.learnings/ERRORS.md` |
| User corrects you | Log to `.learnings/LEARNINGS.md` with category `correction` |
| User wants missing feature | Log to `.learnings/FEATURE_REQUESTS.md` |
| API/external tool fails | Log to `.learnings/ERRORS.md` with integration details |
| Knowledge was outdated | Log to `.learnings/LEARNINGS.md` with category `knowledge_gap` |
| Found better approach | Log to `.learnings/LEARNINGS.md` with category `best_practice` |
| Skill failure identified with fix | Create proposal in `.learnings/pending-improvements/` — **do NOT modify the skill** |
| Recurring failure matches existing proposal | Log recurrence in proposal, notify user and recommend applying it |
| User asks for recommended improvements | List all pending proposals from `.learnings/pending-improvements/` |
| User approves an improvement proposal | Apply changes, update proposal status to `applied` |
| User authorizes auto-update for a skill | Record in `.learnings/AUTO_UPDATE_AUTHORIZATIONS.md` |
| Skill failure + skill is auto-update authorized | Apply fix directly, log what changed |
| Simplify/Harden recurring patterns | Log/update `.learnings/LEARNINGS.md` with `Source: simplify-and-harden` and a stable `Pattern-Key` |
| Similar to existing entry | Link with `**See Also**`, consider priority bump |
| Broadly applicable learning | Promote to `CLAUDE.md`, `AGENTS.md`, and/or `.github/copilot-instructions.md` |
| Workflow improvements | Promote to `AGENTS.md` (OpenClaw workspace) |
| Tool gotchas | Promote to `TOOLS.md` (OpenClaw workspace) |
| Behavioral patterns | Promote to `SOUL.md` (OpenClaw workspace) |

## OpenClaw Setup (Recommended)

OpenClaw is the primary platform for this skill. It uses workspace-based prompt injection with automatic skill loading.

### Installation

**Via ClawdHub (recommended):**
```bash
clawdhub install self-improving-agent
```

**Manual:**
```bash
git clone https://github.com/peterskoett/self-improving-agent.git ~/.openclaw/skills/self-improving-agent
```

Remade for openclaw from original repo : https://github.com/pskoett/pskoett-ai-skills - https://github.com/pskoett/pskoett-ai-skills/tree/main/skills/self-improvement

### Workspace Structure

OpenClaw injects these files into every session:

```
~/.openclaw/workspace/
├── AGENTS.md          # Multi-agent workflows, delegation patterns
├── SOUL.md            # Behavioral guidelines, personality, principles
├── TOOLS.md           # Tool capabilities, integration gotchas
├── MEMORY.md          # Long-term memory (main session only)
├── memory/            # Daily memory files
│   └── YYYY-MM-DD.md
└── .learnings/        # This skill's log files
    ├── LEARNINGS.md
    ├── ERRORS.md
    ├── FEATURE_REQUESTS.md
    ├── AUTO_UPDATE_AUTHORIZATIONS.md  # Per-skill auto-update permissions
    └── pending-improvements/          # Improvement proposals awaiting approval
        └── IMP-YYYYMMDD-XXX-skill-name.md
```

### Create Learning Files

```bash
mkdir -p ~/.openclaw/workspace/.learnings/pending-improvements
```

Then create the log files (or copy from `assets/`):
- `LEARNINGS.md` — corrections, knowledge gaps, best practices
- `ERRORS.md` — command failures, exceptions
- `FEATURE_REQUESTS.md` — user-requested capabilities
- `AUTO_UPDATE_AUTHORIZATIONS.md` — per-skill auto-update permissions (copy from `assets/`)

### Promotion Targets

When learnings prove broadly applicable, promote them to workspace files:

| Learning Type | Promote To | Example |
|---------------|------------|---------|
| Behavioral patterns | `SOUL.md` | "Be concise, avoid disclaimers" |
| Workflow improvements | `AGENTS.md` | "Spawn sub-agents for long tasks" |
| Tool gotchas | `TOOLS.md` | "Git push needs auth configured first" |

### Inter-Session Communication

OpenClaw provides tools to share learnings across sessions:

- **sessions_list** — View active/recent sessions
- **sessions_history** — Read another session's transcript  
- **sessions_send** — Send a learning to another session
- **sessions_spawn** — Spawn a sub-agent for background work

Use these only in trusted environments and only when the user explicitly wants cross-session sharing. Prefer sending a short sanitized summary and relevant file paths, not raw transcripts, secrets, or full command output.

### Optional: Enable Hook

For automatic reminders at session start:

```bash
# Copy hook to OpenClaw hooks directory
cp -r hooks/openclaw ~/.openclaw/hooks/self-improvement

# Enable it
openclaw hooks enable self-improvement
```

See `references/openclaw-integration.md` for complete details.

---

## Generic Setup (Other Agents)

For Claude Code, Codex, Copilot, or other agents, create `.learnings/` in the project or workspace root:

```bash
mkdir -p .learnings
```

Create the files inline using the headers shown above. Avoid reading templates from the current repo or workspace unless you explicitly trust that path.

### Add reference to agent files AGENTS.md, CLAUDE.md, or .github/copilot-instructions.md to remind yourself to log learnings. (this is an alternative to hook-based reminders)

#### Self-Improvement Workflow

When errors or corrections occur:
1. Log to `.learnings/ERRORS.md`, `LEARNINGS.md`, or `FEATURE_REQUESTS.md`
2. Review and promote broadly applicable learnings to:
   - `CLAUDE.md` - project facts and conventions
   - `AGENTS.md` - workflows and automation
   - `.github/copilot-instructions.md` - Copilot context

## Logging Format

### Learning Entry

Append to `.learnings/LEARNINGS.md`:

```markdown
## [LRN-YYYYMMDD-XXX] category

**Logged**: ISO-8601 timestamp
**Priority**: low | medium | high | critical
**Status**: pending
**Area**: frontend | backend | infra | tests | docs | config

### Summary
One-line description of what was learned

### Details
Full context: what happened, what was wrong, what's correct

### Suggested Action
Specific fix or improvement to make

### Metadata
- Source: conversation | error | user_feedback
- Related Files: path/to/file.ext
- Tags: tag1, tag2
- See Also: LRN-20250110-001 (if related to existing entry)
- Pattern-Key: simplify.dead_code | harden.input_validation (optional, for recurring-pattern tracking)
- Recurrence-Count: 1 (optional)
- First-Seen: 2025-01-15 (optional)
- Last-Seen: 2025-01-15 (optional)

---
```

### Error Entry

Append to `.learnings/ERRORS.md`:

```markdown
## [ERR-YYYYMMDD-XXX] skill_or_command_name

**Logged**: ISO-8601 timestamp
**Priority**: high
**Status**: pending
**Area**: frontend | backend | infra | tests | docs | config

### Summary
Brief description of what failed

### Error
```
Actual error message or output
```

### Context
- Command/operation attempted
- Input or parameters used
- Environment details if relevant
- Summary or redacted excerpt of relevant output (avoid full transcripts and secret-bearing data by default)

### Suggested Fix
If identifiable, what might resolve this

### Metadata
- Reproducible: yes | no | unknown
- Related Files: path/to/file.ext
- See Also: ERR-20250110-001 (if recurring)

---
```

### Feature Request Entry

Append to `.learnings/FEATURE_REQUESTS.md`:

```markdown
## [FEAT-YYYYMMDD-XXX] capability_name

**Logged**: ISO-8601 timestamp
**Priority**: medium
**Status**: pending
**Area**: frontend | backend | infra | tests | docs | config

### Requested Capability
What the user wanted to do

### User Context
Why they needed it, what problem they're solving

### Complexity Estimate
simple | medium | complex

### Suggested Implementation
How this could be built, what it might extend

### Metadata
- Frequency: first_time | recurring
- Related Features: existing_feature_name

---
```

## ID Generation

Format: `TYPE-YYYYMMDD-XXX`
- TYPE: `LRN` (learning), `ERR` (error), `FEAT` (feature)
- YYYYMMDD: Current date
- XXX: Sequential number or random 3 chars (e.g., `001`, `A7B`)

Examples: `LRN-20250115-001`, `ERR-20250115-A3F`, `FEAT-20250115-002`

## Resolving Entries

When an issue is fixed, update the entry:

1. Change `**Status**: pending` → `**Status**: resolved`
2. Add resolution block after Metadata:

```markdown
### Resolution
- **Resolved**: 2025-01-16T09:00:00Z
- **Commit/PR**: abc123 or #42
- **Notes**: Brief description of what was done
```

Other status values:
- `in_progress` - Actively being worked on
- `wont_fix` - Decided not to address (add reason in Resolution notes)
- `promoted` - Elevated to CLAUDE.md, AGENTS.md, or .github/copilot-instructions.md

## Promoting to Project Memory

When a learning is broadly applicable (not a one-off fix), promote it to permanent project memory.

### When to Promote

- Learning applies across multiple files/features
- Knowledge any contributor (human or AI) should know
- Prevents recurring mistakes
- Documents project-specific conventions

### Promotion Targets

| Target | What Belongs There |
|--------|-------------------|
| `CLAUDE.md` | Project facts, conventions, gotchas for all Claude interactions |
| `AGENTS.md` | Agent-specific workflows, tool usage patterns, automation rules |
| `.github/copilot-instructions.md` | Project context and conventions for GitHub Copilot |
| `SOUL.md` | Behavioral guidelines, communication style, principles (OpenClaw workspace) |
| `TOOLS.md` | Tool capabilities, usage patterns, integration gotchas (OpenClaw workspace) |

### How to Promote

1. **Distill** the learning into a concise rule or fact
2. **Add** to appropriate section in target file (create file if needed)
3. **Update** original entry:
   - Change `**Status**: pending` → `**Status**: promoted`
   - Add `**Promoted**: CLAUDE.md`, `AGENTS.md`, or `.github/copilot-instructions.md`

### Promotion Examples

**Learning** (verbose):
> Project uses pnpm workspaces. Attempted `npm install` but failed. 
> Lock file is `pnpm-lock.yaml`. Must use `pnpm install`.

**In CLAUDE.md** (concise):
```markdown
## Build & Dependencies
- Package manager: pnpm (not npm) - use `pnpm install`
```

**Learning** (verbose):
> When modifying API endpoints, must regenerate TypeScript client.
> Forgetting this causes type mismatches at runtime.

**In AGENTS.md** (actionable):
```markdown
## After API Changes
1. Regenerate client: `pnpm run generate:api`
2. Check for type errors: `pnpm tsc --noEmit`
```

## Recurring Pattern Detection

If logging something similar to an existing entry:

1. **Search first**: `grep -r "keyword" .learnings/`
2. **Link entries**: Add `**See Also**: ERR-20250110-001` in Metadata
3. **Bump priority** if issue keeps recurring
4. **Consider systemic fix**: Recurring issues often indicate:
   - Missing documentation (→ promote to CLAUDE.md or .github/copilot-instructions.md)
   - Missing automation (→ add to AGENTS.md)
   - Architectural problem (→ create tech debt ticket)

## Simplify & Harden Feed

Use this workflow to ingest recurring patterns from the `simplify-and-harden`
skill and turn them into durable prompt guidance.

### Ingestion Workflow

1. Read `simplify_and_harden.learning_loop.candidates` from the task summary.
2. For each candidate, use `pattern_key` as the stable dedupe key.
3. Search `.learnings/LEARNINGS.md` for an existing entry with that key:
   - `grep -n "Pattern-Key: <pattern_key>" .learnings/LEARNINGS.md`
4. If found:
   - Increment `Recurrence-Count`
   - Update `Last-Seen`
   - Add `See Also` links to related entries/tasks
5. If not found:
   - Create a new `LRN-...` entry
   - Set `Source: simplify-and-harden`
   - Set `Pattern-Key`, `Recurrence-Count: 1`, and `First-Seen`/`Last-Seen`

### Promotion Rule (System Prompt Feedback)

Promote recurring patterns into agent context/system prompt files when all are true:

- `Recurrence-Count >= 3`
- Seen across at least 2 distinct tasks
- Occurred within a 30-day window

Promotion targets:
- `CLAUDE.md`
- `AGENTS.md`
- `.github/copilot-instructions.md`
- `SOUL.md` / `TOOLS.md` for OpenClaw workspace-level guidance when applicable

Write promoted rules as short prevention rules (what to do before/while coding),
not long incident write-ups.

## Periodic Review

Review `.learnings/` at natural breakpoints:

### When to Review
- Before starting a new major task
- After completing a feature
- When working in an area with past learnings
- Weekly during active development

### Quick Status Check
```bash
# Count pending items
grep -h "Status\*\*: pending" .learnings/*.md | wc -l

# List pending high-priority items
grep -B5 "Priority\*\*: high" .learnings/*.md | grep "^## \["

# Find learnings for a specific area
grep -l "Area\*\*: backend" .learnings/*.md
```

### Review Actions
- Resolve fixed items
- Promote applicable learnings
- Link related entries
- Escalate recurring issues

## Detection Triggers

Automatically log when you notice:

**Corrections** (→ learning with `correction` category):
- "No, that's not right..."
- "Actually, it should be..."
- "You're wrong about..."
- "That's outdated..."

**Feature Requests** (→ feature request):
- "Can you also..."
- "I wish you could..."
- "Is there a way to..."
- "Why can't you..."

**Knowledge Gaps** (→ learning with `knowledge_gap` category):
- User provides information you didn't know
- Documentation you referenced is outdated
- API behavior differs from your understanding

**Errors** (→ error entry):
- Command returns non-zero exit code
- Exception or stack trace
- Unexpected output or behavior
- Timeout or connection failure

**Skill Improvement Needed** (→ improvement proposal in `.learnings/pending-improvements/`):
- Skill produced wrong output or used wrong approach
- Skill instructions are outdated or incomplete
- Skill failed to handle an edge case
- User manually corrected a skill's output
- Multiple errors logged against the same skill
- **NEVER modify the skill directly** — create a proposal instead

**Improvement Review Requested** (→ list pending proposals):
- "What skill improvements do you recommend?"
- "Show me pending skill updates"
- "Any skill fixes waiting?"
- "What needs updating?"

**Auto-Update Authorization** (→ update `AUTO_UPDATE_AUTHORIZATIONS.md`):
- "Allow auto-updates for [skill-name]"
- "Let [skill-name] self-improve"
- "Stop auto-updating [skill-name]"
- "Require approval for [skill-name]"

## Priority Guidelines

| Priority | When to Use |
|----------|-------------|
| `critical` | Blocks core functionality, data loss risk, security issue |
| `high` | Significant impact, affects common workflows, recurring issue |
| `medium` | Moderate impact, workaround exists |
| `low` | Minor inconvenience, edge case, nice-to-have |

## Area Tags

Use to filter learnings by codebase region:

| Area | Scope |
|------|-------|
| `frontend` | UI, components, client-side code |
| `backend` | API, services, server-side code |
| `infra` | CI/CD, deployment, Docker, cloud |
| `tests` | Test files, testing utilities, coverage |
| `docs` | Documentation, comments, READMEs |
| `config` | Configuration files, environment, settings |

## Best Practices

1. **Log immediately** - context is freshest right after the issue
2. **Be specific** - future agents need to understand quickly
3. **Include reproduction steps** - especially for errors
4. **Link related files** - makes fixes easier
5. **Suggest concrete fixes** - not just "investigate"
6. **Use consistent categories** - enables filtering
7. **Promote aggressively** - if in doubt, add to CLAUDE.md or .github/copilot-instructions.md
8. **Review regularly** - stale learnings lose value

## Gitignore Options

**Keep learnings local** (per-developer):
```gitignore
.learnings/
```

This repo uses that default to avoid committing sensitive or noisy local logs by accident.

**Track learnings in repo** (team-wide):
Don't add to .gitignore - learnings become shared knowledge.

**Hybrid** (track templates, ignore entries):
```gitignore
.learnings/*.md
!.learnings/.gitkeep
```

## Hook Integration

Enable automatic reminders through agent hooks. This is **opt-in** - you must explicitly configure hooks.

### Quick Setup (Claude Code / Codex)

Create `.claude/settings.json` in your project:

```json
{
  "hooks": {
    "UserPromptSubmit": [{
      "matcher": "",
      "hooks": [{
        "type": "command",
        "command": "./skills/self-improvement/scripts/activator.sh"
      }]
    }]
  }
}
```

This injects a learning evaluation reminder after each prompt (~50-100 tokens overhead).

### Advanced Setup (With Error Detection)

```json
{
  "hooks": {
    "UserPromptSubmit": [{
      "matcher": "",
      "hooks": [{
        "type": "command",
        "command": "./skills/self-improvement/scripts/activator.sh"
      }]
    }],
    "PostToolUse": [{
      "matcher": "Bash",
      "hooks": [{
        "type": "command",
        "command": "./skills/self-improvement/scripts/error-detector.sh"
      }]
    }]
  }
}
```

This is optional. The recommended default is activator-only setup; enable `PostToolUse` only if you are comfortable with hook scripts inspecting command output for error patterns.

### Available Hook Scripts

| Script | Hook Type | Purpose |
|--------|-----------|---------|
| `scripts/activator.sh` | UserPromptSubmit | Reminds to evaluate learnings after tasks |
| `scripts/error-detector.sh` | PostToolUse (Bash) | Triggers on command errors |

See `references/hooks-setup.md` for detailed configuration and troubleshooting.

## Approval-Gated Skill Improvement

**CRITICAL RULE: Never modify any skill file without explicit user approval.** When a skill fails or could be improved, document the proposed changes as an improvement proposal. The user decides whether to apply them.

### Core Principles

1. **No silent updates**: AI must never edit a skill's SKILL.md, scripts, or references without the user saying yes
2. **Document first, change never (unless approved)**: Every proposed fix goes into a proposal file in `.learnings/pending-improvements/`
3. **Proactive notification on recurrence**: If a failure matches an existing proposal, tell the user immediately and recommend applying it
4. **Per-skill auto-update opt-in**: Users can authorize specific skills for automatic updates — but the default is always **approval required**

### When a Skill Fails — What To Do

When a skill produces an error, wrong output, or suboptimal result:

1. **Log the error** to `.learnings/ERRORS.md` as usual
2. **Check for existing proposal**: `ls .learnings/pending-improvements/*-<skill-name>.md 2>/dev/null`
3. **If a proposal already exists for this skill**:
   - Add the new occurrence to the proposal's **Recurrence Log** table
   - **Notify the user immediately**: _"I've encountered this issue with the [skill-name] skill again. We already have a documented fix (IMP-XXXXXXXX-XXX). Would you like me to apply the proposed changes?"_
   - Present a summary of what the proposal would change
   - Wait for explicit approval before touching anything
4. **If no proposal exists yet**:
   - Analyze the failure and determine what skill changes would fix it
   - Create a new proposal file: `.learnings/pending-improvements/IMP-YYYYMMDD-XXX-skill-name.md`
   - Inform the user: _"I've documented a proposed improvement to the [skill-name] skill. You can review it anytime by asking 'what skill improvements do you recommend?'"_
   - Do NOT apply the changes
5. **Check auto-update authorization**: Before creating a proposal, check `.learnings/AUTO_UPDATE_AUTHORIZATIONS.md`. If this specific skill is listed as authorized for auto-update, apply the fix directly, then inform the user what was changed. Otherwise, follow steps 3-4.

### Improvement Proposal Format

Save each proposal as its own file in `.learnings/pending-improvements/` named `IMP-YYYYMMDD-XXX-skill-name.md`:

```markdown
# Improvement Proposal: IMP-YYYYMMDD-XXX

**Skill**: skill-name
**Skill Path**: path/to/skill/SKILL.md
**Created**: ISO-8601 timestamp
**Status**: pending | approved | rejected | applied
**Priority**: low | medium | high | critical
**Triggered By**: error | user_feedback | recurring_pattern | knowledge_gap

## Problem

What went wrong. Reference error IDs (e.g., ERR-YYYYMMDD-XXX) if applicable.

## Root Cause

Why the skill failed or produced suboptimal results.

## Proposed Changes

### Change 1: [add | modify | remove] — [brief label]

**Section**: Which section of the SKILL.md is affected
**Current Content** (if modifying/removing):
\```
Existing text or instruction that needs changing
\```
**Proposed Content** (if adding/modifying):
\```
New or updated text or instruction
\```
**Rationale**: Why this change fixes the problem

### Change 2: [add | modify | remove] — [brief label]
(repeat as needed)

## Expected Impact

What will improve after applying these changes.

## Recurrence Log

| Date | Error/Context | Notes |
|------|--------------|-------|
| YYYY-MM-DD | ERR-YYYYMMDD-XXX | First occurrence |

---
```

### User-Triggered Review of Pending Improvements

The user can ask for a review of all pending improvement proposals at any time. Trigger phrases include:

- "What skill improvements do you recommend?"
- "Show me pending skill updates"
- "Any skill fixes waiting?"
- "What skills need updating?"
- "Review skill improvement proposals"

When triggered:

1. **List all files** in `.learnings/pending-improvements/` with status `pending`
2. **For each proposal**, present a clear summary:
   - Which skill it affects
   - What the problem is (one sentence)
   - What changes are proposed (bullet list of add/modify/remove)
   - How many times the issue has recurred
   - Priority level
3. **Ask the user** which proposals they want to approve, reject, or defer
4. **On approval**: Apply the changes to the skill, update proposal status to `applied`, add resolution timestamp
5. **On rejection**: Update proposal status to `rejected`, add user's reason if given
6. **On defer**: Leave as `pending`, no changes

### Applying Approved Changes

When the user approves a proposal:

1. Read the full proposal file
2. For each proposed change:
   - If `add`: Insert the new content at the specified section
   - If `modify`: Replace the current content with the proposed content
   - If `remove`: Delete the specified content
3. Update the proposal file:
   - Set `**Status**: applied`
   - Add `**Applied**: ISO-8601 timestamp` after Status
   - Add `**Applied By**: user-approved` after Applied
4. Log a learning entry in `.learnings/LEARNINGS.md` noting the skill improvement was applied
5. Confirm to the user exactly what was changed

### Auto-Update Authorization

By default, **no skill is authorized for auto-update**. The user must explicitly grant permission on a per-skill basis.

**How to authorize a skill for auto-update:**

The user says something like:
- "Allow auto-updates for the [skill-name] skill"
- "The [skill-name] skill can self-improve without asking me"
- "Auto-approve improvements to [skill-name]"

When authorized:

1. Add an entry to `.learnings/AUTO_UPDATE_AUTHORIZATIONS.md`:

```markdown
## [skill-name]
**Authorized**: ISO-8601 timestamp
**Authorized By**: user
**Scope**: full
**Notes**: User authorized auto-update in conversation

---
```

2. Confirm to the user: _"I've authorized auto-updates for [skill-name]. Future improvements to this skill will be applied automatically and I'll inform you what changed. You can revoke this anytime."_

**How to revoke auto-update:**

The user says something like:
- "Stop auto-updating [skill-name]"
- "Require approval for [skill-name] again"
- "Revoke auto-update for [skill-name]"

When revoked: Remove the entry from `AUTO_UPDATE_AUTHORIZATIONS.md` and confirm.

**Checking authorization before applying changes:**

```bash
grep -l "## skill-name-here" .learnings/AUTO_UPDATE_AUTHORIZATIONS.md 2>/dev/null
```

If the skill is listed AND its entry is present (not just the file header), it is authorized. Otherwise, require user approval.

### Auto-Update Behavior

When a skill IS authorized for auto-update and a failure is detected:

1. Analyze the failure and determine the fix
2. Check if a pending proposal already exists — if so, apply it
3. If no proposal exists, create one with status `applied` (for the record) and apply the fix
4. **Always inform the user** what was changed: _"I auto-applied an improvement to [skill-name]: [one-line summary of change]. This skill is authorized for auto-update. Details logged in IMP-XXXXXXXX-XXX."_
5. Log the change in `.learnings/LEARNINGS.md`

### Detection Triggers for Skill Improvements

Watch for these signals that a skill needs an improvement proposal:

**During skill execution:**
- Skill produces an error or unexpected output
- Skill's instructions lead to a wrong approach
- Skill is missing a step that was needed
- Skill references outdated tools, APIs, or syntax
- User has to manually correct the skill's output

**From error logs:**
- Multiple errors logged against the same skill (check `See Also` links)
- Error with `Reproducible: yes` tied to a skill

**From user feedback:**
- "This skill keeps getting X wrong"
- "The skill should also handle Y"
- "That's not how you do X anymore"

## Automatic Skill Extraction

When a learning is valuable enough to become a reusable skill, extract it using the provided helper.

### Skill Extraction Criteria

A learning qualifies for skill extraction when ANY of these apply:

| Criterion | Description |
|-----------|-------------|
| **Recurring** | Has `See Also` links to 2+ similar issues |
| **Verified** | Status is `resolved` with working fix |
| **Non-obvious** | Required actual debugging/investigation to discover |
| **Broadly applicable** | Not project-specific; useful across codebases |
| **User-flagged** | User says "save this as a skill" or similar |

### Extraction Workflow

1. **Identify candidate**: Learning meets extraction criteria
2. **Run helper** (or create manually):
   ```bash
   ./skills/self-improvement/scripts/extract-skill.sh skill-name --dry-run
   ./skills/self-improvement/scripts/extract-skill.sh skill-name
   ```
3. **Customize SKILL.md**: Fill in template with learning content
4. **Update learning**: Set status to `promoted_to_skill`, add `Skill-Path`
5. **Verify**: Read skill in fresh session to ensure it's self-contained

### Manual Extraction

If you prefer manual creation:

1. Create `skills/<skill-name>/SKILL.md`
2. Use template from `assets/SKILL-TEMPLATE.md`
3. Follow [Agent Skills spec](https://agentskills.io/specification):
   - YAML frontmatter with `name` and `description`
   - Name must match folder name
   - No README.md inside skill folder

### Extraction Detection Triggers

Watch for these signals that a learning should become a skill:

**In conversation:**
- "Save this as a skill"
- "I keep running into this"
- "This would be useful for other projects"
- "Remember this pattern"

**In learning entries:**
- Multiple `See Also` links (recurring issue)
- High priority + resolved status
- Category: `best_practice` with broad applicability
- User feedback praising the solution

### Skill Quality Gates

Before extraction, verify:

- [ ] Solution is tested and working
- [ ] Description is clear without original context
- [ ] Code examples are self-contained
- [ ] No project-specific hardcoded values
- [ ] Follows skill naming conventions (lowercase, hyphens)

## Multi-Agent Support

This skill works across different AI coding agents with agent-specific activation.

### Claude Code

**Activation**: Hooks (UserPromptSubmit, PostToolUse)
**Setup**: `.claude/settings.json` with hook configuration
**Detection**: Automatic via hook scripts

### Codex CLI

**Activation**: Hooks (same pattern as Claude Code)
**Setup**: `.codex/settings.json` with hook configuration
**Detection**: Automatic via hook scripts

### GitHub Copilot

**Activation**: Manual (no hook support)
**Setup**: Add to `.github/copilot-instructions.md`:

```markdown
## Self-Improvement

After solving non-obvious issues, consider logging to `.learnings/`:
1. Use format from self-improvement skill
2. Link related entries with See Also
3. Promote high-value learnings to skills

Ask in chat: "Should I log this as a learning?"
```

**Detection**: Manual review at session end

FILE:scripts/error-detector.sh
#!/bin/bash
# Self-Improvement Error Detector Hook
# Triggers on PostToolUse for Bash to detect command failures
# Reads CLAUDE_TOOL_OUTPUT environment variable
# Also checks for matching pending improvement proposals

set -e

# Check if tool output indicates an error
# CLAUDE_TOOL_OUTPUT contains the result of the tool execution
OUTPUT="-"

# Patterns indicating errors (case-insensitive matching)
ERROR_PATTERNS=(
    "error:"
    "Error:"
    "ERROR:"
    "failed"
    "FAILED"
    "command not found"
    "No such file"
    "Permission denied"
    "fatal:"
    "Exception"
    "Traceback"
    "npm ERR!"
    "ModuleNotFoundError"
    "SyntaxError"
    "TypeError"
    "exit code"
    "non-zero"
)

# Check if output contains any error pattern
contains_error=false
for pattern in "ERROR_PATTERNS[@]"; do
    if [[ "$OUTPUT" == *"$pattern"* ]]; then
        contains_error=true
        break
    fi
done

# Only output reminder if error detected
if [ "$contains_error" = true ]; then
    cat << 'EOF'
<error-detected>
A command error was detected. Consider logging this to .learnings/ERRORS.md if:
- The error was unexpected or non-obvious
- It required investigation to resolve
- It might recur in similar contexts
- The solution could benefit future sessions

Use the self-improvement skill format: [ERR-YYYYMMDD-XXX]

If this error is related to a SKILL failure:
1. Check .learnings/pending-improvements/ for an existing proposal matching this skill
2. If a proposal exists: notify the user that we have a documented fix and recommend applying it
3. If no proposal exists: create one — do NOT modify the skill directly
4. Exception: if the skill is listed in .learnings/AUTO_UPDATE_AUTHORIZATIONS.md, apply the fix directly and inform the user

CRITICAL: Never modify any skill without explicit user approval unless auto-update is authorized.
</error-detected>
EOF

    # Check if there are any pending proposals that might match
    PENDING_DIR=".learnings/pending-improvements"
    if [ -d "$PENDING_DIR" ]; then
        PENDING_FILES=$(find "$PENDING_DIR" -name "IMP-*.md" 2>/dev/null)
        if [ -n "$PENDING_FILES" ]; then
            cat << 'EOF'
<pending-proposals-available>
There are existing improvement proposals in .learnings/pending-improvements/.
Check if this error matches any pending proposal. If it does, add a recurrence
entry to the proposal and notify the user that a known fix is available.
</pending-proposals-available>
EOF
        fi
    fi
fi

FILE:scripts/extract-skill.sh
#!/bin/bash
# Skill Extraction Helper
# Creates a new skill from a learning entry
# Usage: ./extract-skill.sh <skill-name> [--dry-run]

set -e

# Configuration
SKILLS_DIR="./skills"

# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m' # No Color

usage() {
    cat << EOF
Usage: $(basename "$0") <skill-name> [options]

Create a new skill from a learning entry.

Arguments:
  skill-name     Name of the skill (lowercase, hyphens for spaces)

Options:
  --dry-run      Show what would be created without creating files
  --output-dir   Relative output directory under current path (default: ./skills)
  -h, --help     Show this help message

Examples:
  $(basename "$0") docker-m1-fixes
  $(basename "$0") api-timeout-patterns --dry-run
  $(basename "$0") pnpm-setup --output-dir ./skills/custom

The skill will be created in: \$SKILLS_DIR/<skill-name>/
EOF
}

log_info() {
    echo -e "GREEN[INFO]NC $1"
}

log_warn() {
    echo -e "YELLOW[WARN]NC $1"
}

log_error() {
    echo -e "RED[ERROR]NC $1" >&2
}

# Parse arguments
SKILL_NAME=""
DRY_RUN=false

while [[ $# -gt 0 ]]; do
    case $1 in
        --dry-run)
            DRY_RUN=true
            shift
            ;;
        --output-dir)
            if [ -z "-" ] || [[ "-" == -* ]]; then
                log_error "--output-dir requires a relative path argument"
                usage
                exit 1
            fi
            SKILLS_DIR="$2"
            shift 2
            ;;
        -h|--help)
            usage
            exit 0
            ;;
        -*)
            log_error "Unknown option: $1"
            usage
            exit 1
            ;;
        *)
            if [ -z "$SKILL_NAME" ]; then
                SKILL_NAME="$1"
            else
                log_error "Unexpected argument: $1"
                usage
                exit 1
            fi
            shift
            ;;
    esac
done

# Validate skill name
if [ -z "$SKILL_NAME" ]; then
    log_error "Skill name is required"
    usage
    exit 1
fi

# Validate skill name format (lowercase, hyphens, no spaces)
if ! [[ "$SKILL_NAME" =~ ^[a-z0-9]+(-[a-z0-9]+)*$ ]]; then
    log_error "Invalid skill name format. Use lowercase letters, numbers, and hyphens only."
    log_error "Examples: 'docker-fixes', 'api-patterns', 'pnpm-setup'"
    exit 1
fi

# Validate output path to avoid writes outside current workspace.
if [[ "$SKILLS_DIR" = /* ]]; then
    log_error "Output directory must be a relative path under the current directory."
    exit 1
fi

if [[ "$SKILLS_DIR" =~ (^|/)\.\.(/|$) ]]; then
    log_error "Output directory cannot include '..' path segments."
    exit 1
fi

SKILLS_DIR="SKILLS_DIR#./"
SKILLS_DIR="./$SKILLS_DIR"

SKILL_PATH="$SKILLS_DIR/$SKILL_NAME"

# Check if skill already exists
if [ -d "$SKILL_PATH" ] && [ "$DRY_RUN" = false ]; then
    log_error "Skill already exists: $SKILL_PATH"
    log_error "Use a different name or remove the existing skill first."
    exit 1
fi

# Dry run output
if [ "$DRY_RUN" = true ]; then
    log_info "Dry run - would create:"
    echo "  $SKILL_PATH/"
    echo "  $SKILL_PATH/SKILL.md"
    echo ""
    echo "Template content would be:"
    echo "---"
    cat << TEMPLATE
name: $SKILL_NAME
description: "[TODO: Add a concise description of what this skill does and when to use it]"
---

# $(echo "$SKILL_NAME" | sed 's/-/ /g' | awk '{for(i=1;i<=NF;i++) $i=toupper(substr($i,1,1)) tolower(substr($i,2))}1')

[TODO: Brief introduction explaining the skill's purpose]

## Quick Reference

| Situation | Action |
|-----------|--------|
| [Trigger condition] | [What to do] |

## Usage

[TODO: Detailed usage instructions]

## Examples

[TODO: Add concrete examples]

## Source Learning

This skill was extracted from a learning entry.
- Learning ID: [TODO: Add original learning ID]
- Original File: .learnings/LEARNINGS.md
TEMPLATE
    echo "---"
    exit 0
fi

# Create skill directory structure
log_info "Creating skill: $SKILL_NAME"

mkdir -p "$SKILL_PATH"

# Create SKILL.md from template
cat > "$SKILL_PATH/SKILL.md" << TEMPLATE
---
name: $SKILL_NAME
description: "[TODO: Add a concise description of what this skill does and when to use it]"
---

# $(echo "$SKILL_NAME" | sed 's/-/ /g' | awk '{for(i=1;i<=NF;i++) $i=toupper(substr($i,1,1)) tolower(substr($i,2))}1')

[TODO: Brief introduction explaining the skill's purpose]

## Quick Reference

| Situation | Action |
|-----------|--------|
| [Trigger condition] | [What to do] |

## Usage

[TODO: Detailed usage instructions]

## Examples

[TODO: Add concrete examples]

## Source Learning

This skill was extracted from a learning entry.
- Learning ID: [TODO: Add original learning ID]
- Original File: .learnings/LEARNINGS.md
TEMPLATE

log_info "Created: $SKILL_PATH/SKILL.md"

# Suggest next steps
echo ""
log_info "Skill scaffold created successfully!"
echo ""
echo "Next steps:"
echo "  1. Edit $SKILL_PATH/SKILL.md"
echo "  2. Fill in the TODO sections with content from your learning"
echo "  3. Add references/ folder if you have detailed documentation"
echo "  4. Add scripts/ folder if you have executable code"
echo "  5. Update the original learning entry with:"
echo "     **Status**: promoted_to_skill"
echo "     **Skill-Path**: skills/$SKILL_NAME"

FILE:scripts/activator.sh
#!/bin/bash
# Self-Improvement Activator Hook
# Triggers on UserPromptSubmit to remind Claude about learning capture
# and pending improvement proposals
# Keep output minimal (~50-100 tokens) to minimize overhead

set -e

# Output reminder as system context
cat << 'EOF'
<self-improvement-reminder>
After completing this task, evaluate if extractable knowledge emerged:
- Non-obvious solution discovered through investigation?
- Workaround for unexpected behavior?
- Project-specific pattern learned?
- Error required debugging to resolve?
- Skill produced wrong output or failed?

If yes: Log to .learnings/ using the self-improvement skill format.
If a skill failed: Create or update an improvement proposal in .learnings/pending-improvements/ — do NOT modify the skill directly unless it is authorized for auto-update in .learnings/AUTO_UPDATE_AUTHORIZATIONS.md.
If a failure matches an existing proposal: Notify the user and recommend applying it.
If high-value (recurring, broadly applicable): Consider skill extraction.

CRITICAL: Never modify any skill without explicit user approval unless auto-update is authorized for that specific skill.
</self-improvement-reminder>
EOF

# Check for pending improvement proposals and notify
PENDING_DIR=".learnings/pending-improvements"
if [ -d "$PENDING_DIR" ]; then
    PENDING_COUNT=$(find "$PENDING_DIR" -name "IMP-*.md" -exec grep -l "Status\*\*: pending" {} \; 2>/dev/null | wc -l)
    if [ "$PENDING_COUNT" -gt 0 ]; then
        cat << EOF
<pending-improvements-notice>
There are $PENDING_COUNT pending skill improvement proposal(s) awaiting user review.
If a skill failure occurs that matches an existing proposal, notify the user and recommend applying it.
The user can review all proposals by asking: "What skill improvements do you recommend?"
</pending-improvements-notice>
EOF
    fi
fi

FILE:README.md
# self-improvement

Self-improvement skill for OpenClaw. It captures learnings, errors, and feature requests to support continuous improvement across sessions.

## Attribution

Remade for OpenClaw from the original repo:

- https://github.com/pskoett/pskoett-ai-skills
- https://github.com/pskoett/pskoett-ai-skills/tree/main/skills/self-improvement

## Main File

- `SKILL.md`

FILE:references/examples.md
# Entry Examples

Concrete examples of well-formatted entries with all fields.

## Learning: Correction

```markdown
## [LRN-20250115-001] correction

**Logged**: 2025-01-15T10:30:00Z
**Priority**: high
**Status**: pending
**Area**: tests

### Summary
Incorrectly assumed pytest fixtures are scoped to function by default

### Details
When writing test fixtures, I assumed all fixtures were function-scoped. 
User corrected that while function scope is the default, the codebase 
convention uses module-scoped fixtures for database connections to 
improve test performance.

### Suggested Action
When creating fixtures that involve expensive setup (DB, network), 
check existing fixtures for scope patterns before defaulting to function scope.

### Metadata
- Source: user_feedback
- Related Files: tests/conftest.py
- Tags: pytest, testing, fixtures

---
```

## Learning: Knowledge Gap (Resolved)

```markdown
## [LRN-20250115-002] knowledge_gap

**Logged**: 2025-01-15T14:22:00Z
**Priority**: medium
**Status**: resolved
**Area**: config

### Summary
Project uses pnpm not npm for package management

### Details
Attempted to run `npm install` but project uses pnpm workspaces.
Lock file is `pnpm-lock.yaml`, not `package-lock.json`.

### Suggested Action
Check for `pnpm-lock.yaml` or `pnpm-workspace.yaml` before assuming npm.
Use `pnpm install` for this project.

### Metadata
- Source: error
- Related Files: pnpm-lock.yaml, pnpm-workspace.yaml
- Tags: package-manager, pnpm, setup

### Resolution
- **Resolved**: 2025-01-15T14:30:00Z
- **Commit/PR**: N/A - knowledge update
- **Notes**: Added to CLAUDE.md for future reference

---
```

## Learning: Promoted to CLAUDE.md

```markdown
## [LRN-20250115-003] best_practice

**Logged**: 2025-01-15T16:00:00Z
**Priority**: high
**Status**: promoted
**Promoted**: CLAUDE.md
**Area**: backend

### Summary
API responses must include correlation ID from request headers

### Details
All API responses should echo back the X-Correlation-ID header from 
the request. This is required for distributed tracing. Responses 
without this header break the observability pipeline.

### Suggested Action
Always include correlation ID passthrough in API handlers.

### Metadata
- Source: user_feedback
- Related Files: src/middleware/correlation.ts
- Tags: api, observability, tracing

---
```

## Learning: Promoted to AGENTS.md

```markdown
## [LRN-20250116-001] best_practice

**Logged**: 2025-01-16T09:00:00Z
**Priority**: high
**Status**: promoted
**Promoted**: AGENTS.md
**Area**: backend

### Summary
Must regenerate API client after OpenAPI spec changes

### Details
When modifying API endpoints, the TypeScript client must be regenerated.
Forgetting this causes type mismatches that only appear at runtime.
The generate script also runs validation.

### Suggested Action
Add to agent workflow: after any API changes, run `pnpm run generate:api`.

### Metadata
- Source: error
- Related Files: openapi.yaml, src/client/api.ts
- Tags: api, codegen, typescript

---
```

## Error Entry

```markdown
## [ERR-20250115-A3F] docker_build

**Logged**: 2025-01-15T09:15:00Z
**Priority**: high
**Status**: pending
**Area**: infra

### Summary
Docker build fails on M1 Mac due to platform mismatch

### Error
```
error: failed to solve: python:3.11-slim: no match for platform linux/arm64
```

### Context
- Command: `docker build -t myapp .`
- Dockerfile uses `FROM python:3.11-slim`
- Running on Apple Silicon (M1/M2)

### Suggested Fix
Add platform flag: `docker build --platform linux/amd64 -t myapp .`
Or update Dockerfile: `FROM --platform=linux/amd64 python:3.11-slim`

### Metadata
- Reproducible: yes
- Related Files: Dockerfile

---
```

## Error Entry: Recurring Issue

```markdown
## [ERR-20250120-B2C] api_timeout

**Logged**: 2025-01-20T11:30:00Z
**Priority**: critical
**Status**: pending
**Area**: backend

### Summary
Third-party API timeout during request processing

### Error
```
TimeoutError: Request to api.example.com timed out after 30000ms
```

### Context
- Command: POST /api/process
- Timeout set to 30s
- Occurs during peak hours (lunch, evening)

### Suggested Fix
Implement retry with exponential backoff. Consider circuit breaker pattern.

### Metadata
- Reproducible: yes (during peak hours)
- Related Files: src/services/api-client.ts
- See Also: ERR-20250115-X1Y, ERR-20250118-Z3W

---
```

## Feature Request

```markdown
## [FEAT-20250115-001] export_to_csv

**Logged**: 2025-01-15T16:45:00Z
**Priority**: medium
**Status**: pending
**Area**: backend

### Requested Capability
Export analysis results to CSV format

### User Context
User runs weekly reports and needs to share results with non-technical 
stakeholders in Excel. Currently copies output manually.

### Complexity Estimate
simple

### Suggested Implementation
Add `--output csv` flag to the analyze command. Use standard csv module.
Could extend existing `--output json` pattern.

### Metadata
- Frequency: recurring
- Related Features: analyze command, json output

---
```

## Feature Request: Resolved

```markdown
## [FEAT-20250110-002] dark_mode

**Logged**: 2025-01-10T14:00:00Z
**Priority**: low
**Status**: resolved
**Area**: frontend

### Requested Capability
Dark mode support for the dashboard

### User Context
User works late hours and finds the bright interface straining.
Several other users have mentioned this informally.

### Complexity Estimate
medium

### Suggested Implementation
Use CSS variables for colors. Add toggle in user settings.
Consider system preference detection.

### Metadata
- Frequency: recurring
- Related Features: user settings, theme system

### Resolution
- **Resolved**: 2025-01-18T16:00:00Z
- **Commit/PR**: #142
- **Notes**: Implemented with system preference detection and manual toggle

---
```

## Learning: Promoted to Skill

```markdown
## [LRN-20250118-001] best_practice

**Logged**: 2025-01-18T11:00:00Z
**Priority**: high
**Status**: promoted_to_skill
**Skill-Path**: skills/docker-m1-fixes
**Area**: infra

### Summary
Docker build fails on Apple Silicon due to platform mismatch

### Details
When building Docker images on M1/M2 Macs, the build fails because
the base image doesn't have an ARM64 variant. This is a common issue
that affects many developers.

### Suggested Action
Add `--platform linux/amd64` to docker build command, or use
`FROM --platform=linux/amd64` in Dockerfile.

### Metadata
- Source: error
- Related Files: Dockerfile
- Tags: docker, arm64, m1, apple-silicon
- See Also: ERR-20250115-A3F, ERR-20250117-B2D

---
```

## Extracted Skill Example

When the above learning is extracted as a skill, it becomes:

**File**: `skills/docker-m1-fixes/SKILL.md`

```markdown
---
name: docker-m1-fixes
description: "Fixes Docker build failures on Apple Silicon (M1/M2). Use when docker build fails with platform mismatch errors."
---

# Docker M1 Fixes

Solutions for Docker build issues on Apple Silicon Macs.

## Quick Reference

| Error | Fix |
|-------|-----|
| `no match for platform linux/arm64` | Add `--platform linux/amd64` to build |
| Image runs but crashes | Use emulation or find ARM-compatible base |

## The Problem

Many Docker base images don't have ARM64 variants. When building on
Apple Silicon (M1/M2/M3), Docker attempts to pull ARM64 images by
default, causing platform mismatch errors.

## Solutions

### Option 1: Build Flag (Recommended)

Add platform flag to your build command:

\`\`\`bash
docker build --platform linux/amd64 -t myapp .
\`\`\`

### Option 2: Dockerfile Modification

Specify platform in the FROM instruction:

\`\`\`dockerfile
FROM --platform=linux/amd64 python:3.11-slim
\`\`\`

### Option 3: Docker Compose

Add platform to your service:

\`\`\`yaml
services:
  app:
    platform: linux/amd64
    build: .
\`\`\`

## Trade-offs

| Approach | Pros | Cons |
|----------|------|------|
| Build flag | No file changes | Must remember flag |
| Dockerfile | Explicit, versioned | Affects all builds |
| Compose | Convenient for dev | Requires compose |

## Performance Note

Running AMD64 images on ARM64 uses Rosetta 2 emulation. This works
for development but may be slower. For production, find ARM-native
alternatives when possible.

## Source

- Learning ID: LRN-20250118-001
- Category: best_practice
- Extraction Date: 2025-01-18
```

FILE:references/hooks-setup.md
# Hook Setup Guide

Configure automatic self-improvement triggers for AI coding agents.

## Overview

Hooks enable proactive learning capture by injecting reminders at key moments:
- **UserPromptSubmit**: Reminder after each prompt to evaluate learnings
- **PostToolUse (Bash)**: Error detection when commands fail

## Claude Code Setup

### Option 1: Project-Level Configuration

Create `.claude/settings.json` in your project root:

```json
{
  "hooks": {
    "UserPromptSubmit": [
      {
        "matcher": "",
        "hooks": [
          {
            "type": "command",
            "command": "./skills/self-improvement/scripts/activator.sh"
          }
        ]
      }
    ],
    "PostToolUse": [
      {
        "matcher": "Bash",
        "hooks": [
          {
            "type": "command",
            "command": "./skills/self-improvement/scripts/error-detector.sh"
          }
        ]
      }
    ]
  }
}
```

### Option 2: User-Level Configuration

Add to `~/.claude/settings.json` for global activation:

```json
{
  "hooks": {
    "UserPromptSubmit": [
      {
        "matcher": "",
        "hooks": [
          {
            "type": "command",
            "command": "~/.claude/skills/self-improvement/scripts/activator.sh"
          }
        ]
      }
    ]
  }
}
```

### Minimal Setup (Activator Only)

For lower overhead, use only the UserPromptSubmit hook:

```json
{
  "hooks": {
    "UserPromptSubmit": [
      {
        "matcher": "",
        "hooks": [
          {
            "type": "command",
            "command": "./skills/self-improvement/scripts/activator.sh"
          }
        ]
      }
    ]
  }
}
```

## Codex CLI Setup

Codex uses the same hook system as Claude Code. Create `.codex/settings.json`:

```json
{
  "hooks": {
    "UserPromptSubmit": [
      {
        "matcher": "",
        "hooks": [
          {
            "type": "command",
            "command": "./skills/self-improvement/scripts/activator.sh"
          }
        ]
      }
    ]
  }
}
```

## GitHub Copilot Setup

Copilot doesn't support hooks directly. Instead, add guidance to `.github/copilot-instructions.md`:

```markdown
## Self-Improvement

After completing tasks that involved:
- Debugging non-obvious issues
- Discovering workarounds
- Learning project-specific patterns
- Resolving unexpected errors

Consider logging the learning to `.learnings/` using the format from the self-improvement skill.

For high-value learnings that would benefit other sessions, consider skill extraction.
```

## Verification

### Test Activator Hook

1. Enable the hook configuration
2. Start a new Claude Code session
3. Send any prompt
4. Verify you see `<self-improvement-reminder>` in the context

### Test Error Detector Hook

1. Enable PostToolUse hook for Bash
2. Run a command that fails: `ls /nonexistent/path`
3. Verify you see `<error-detected>` reminder

### Dry Run Extract Script

```bash
./skills/self-improvement/scripts/extract-skill.sh test-skill --dry-run
```

Expected output shows the skill scaffold that would be created.

## Troubleshooting

### Hook Not Triggering

1. **Check script permissions**: `chmod +x scripts/*.sh`
2. **Verify path**: Use absolute paths or paths relative to project root
3. **Check settings location**: Project vs user-level settings
4. **Restart session**: Hooks are loaded at session start

### Permission Denied

```bash
chmod +x ./skills/self-improvement/scripts/activator.sh
chmod +x ./skills/self-improvement/scripts/error-detector.sh
chmod +x ./skills/self-improvement/scripts/extract-skill.sh
```

### Script Not Found

If using relative paths, ensure you're in the correct directory or use absolute paths:

```json
{
  "command": "/absolute/path/to/skills/self-improvement/scripts/activator.sh"
}
```

### Too Much Overhead

If the activator feels intrusive:

1. **Use minimal setup**: Only UserPromptSubmit, skip PostToolUse
2. **Add matcher filter**: Only trigger for certain prompts:

```json
{
  "matcher": "fix|debug|error|issue",
  "hooks": [...]
}
```

## Hook Output Budget

The activator is designed to be lightweight:
- **Target**: ~50-100 tokens per activation
- **Content**: Structured reminder, not verbose instructions
- **Format**: XML tags for easy parsing

If you need to reduce overhead further, you can edit `activator.sh` to output less text.

## Security Considerations

- Hook scripts run with the same permissions as Claude Code
- Scripts only output text; they don't modify files or run commands
- Error detector reads `CLAUDE_TOOL_OUTPUT` environment variable
- Treat `CLAUDE_TOOL_OUTPUT` as potentially sensitive; do not log or forward it verbatim unless the user explicitly wants that detail
- All scripts are opt-in (you must configure them explicitly)
- Recommended default: enable `UserPromptSubmit` only, and add `PostToolUse` only when you want error-pattern reminders from command output

## Disabling Hooks

To temporarily disable without removing configuration:

1. **Comment out in settings**:
```json
{
  "hooks": {
    // "UserPromptSubmit": [...]
  }
}
```

2. **Or delete the settings file**: Hooks won't run without configuration

FILE:references/openclaw-integration.md
# OpenClaw Integration

Complete setup and usage guide for integrating the self-improvement skill with OpenClaw.

## Overview

OpenClaw uses workspace-based prompt injection combined with event-driven hooks. Context is injected from workspace files at session start, and hooks can trigger on lifecycle events.

## Workspace Structure

```
~/.openclaw/                      
├── workspace/                   # Working directory
│   ├── AGENTS.md               # Multi-agent coordination patterns
│   ├── SOUL.md                 # Behavioral guidelines and personality
│   ├── TOOLS.md                # Tool capabilities and gotchas
│   ├── MEMORY.md               # Long-term memory (main session only)
│   └── memory/                 # Daily memory files
│       └── YYYY-MM-DD.md
├── skills/                      # Installed skills
│   └── <skill-name>/
│       └── SKILL.md
└── hooks/                       # Custom hooks
    └── <hook-name>/
        ├── HOOK.md
        └── handler.ts
```

## Quick Setup

### 1. Install the Skill

```bash
clawdhub install self-improving-agent
```

Or copy manually:

```bash
cp -r self-improving-agent ~/.openclaw/skills/
```

### 2. Install the Hook (Optional)

Copy the hook to OpenClaw's hooks directory:

```bash
cp -r hooks/openclaw ~/.openclaw/hooks/self-improvement
```

Enable the hook:

```bash
openclaw hooks enable self-improvement
```

### 3. Create Learning Files

Create the `.learnings/` directory in your workspace:

```bash
mkdir -p ~/.openclaw/workspace/.learnings
```

Or in the skill directory:

```bash
mkdir -p ~/.openclaw/skills/self-improving-agent/.learnings
```

## Injected Prompt Files

### AGENTS.md

Purpose: Multi-agent workflows and delegation patterns.

```markdown
# Agent Coordination

## Delegation Rules
- Use explore agent for open-ended codebase questions
- Spawn sub-agents for long-running tasks
- Use sessions_send for cross-session communication

## Session Handoff
When delegating to another session:
1. Provide full context in the handoff message
2. Include relevant file paths
3. Specify expected output format
```

### SOUL.md

Purpose: Behavioral guidelines and communication style.

```markdown
# Behavioral Guidelines

## Communication Style
- Be direct and concise
- Avoid unnecessary caveats and disclaimers
- Use technical language appropriate to context

## Error Handling
- Admit mistakes promptly
- Provide corrected information immediately
- Log significant errors to learnings
```

### TOOLS.md

Purpose: Tool capabilities, integration gotchas, local configuration.

```markdown
# Tool Knowledge

## Self-Improvement Skill
Log learnings to `.learnings/` for continuous improvement.

## Local Tools
- Document tool-specific gotchas here
- Note authentication requirements
- Track integration quirks
```

## Learning Workflow

### Capturing Learnings

1. **In-session**: Log to `.learnings/` as usual
2. **Cross-session**: Promote to workspace files

### Promotion Decision Tree

```
Is the learning project-specific?
├── Yes → Keep in .learnings/
└── No → Is it behavioral/style-related?
    ├── Yes → Promote to SOUL.md
    └── No → Is it tool-related?
        ├── Yes → Promote to TOOLS.md
        └── No → Promote to AGENTS.md (workflow)
```

### Promotion Format Examples

**From learning:**
> Git push to GitHub fails without auth configured - triggers desktop prompt

**To TOOLS.md:**
```markdown
## Git
- Don't push without confirming auth is configured
- Use `gh auth status` to check GitHub CLI auth
```

## Inter-Agent Communication

OpenClaw provides tools for cross-session communication:

Use these only when cross-session sharing is explicitly needed and the environment is trusted. Prefer short sanitized summaries over raw transcripts, command output, or secret-bearing content.

### sessions_list

View active and recent sessions:
```
sessions_list(activeMinutes=30, messageLimit=3)
```

### sessions_history

Read transcript from another session:
```
sessions_history(sessionKey="session-id", limit=50)
```

Only read another session's transcript when the user explicitly wants shared context or continuation across sessions.

### sessions_send

Send message to another session:
```
sessions_send(sessionKey="session-id", message="Learning: API requires X-Custom-Header")
```

Prefer sending a concise learning summary plus relevant paths rather than forwarding raw transcript content.

### sessions_spawn

Spawn a background sub-agent:
```
sessions_spawn(task="Research X and report back", label="research")
```

## Available Hook Events

| Event | When It Fires |
|-------|---------------|
| `agent:bootstrap` | Before workspace files inject |
| `command:new` | When `/new` command issued |
| `command:reset` | When `/reset` command issued |
| `command:stop` | When `/stop` command issued |
| `gateway:startup` | When gateway starts |

## Detection Triggers

### Standard Triggers
- User corrections ("No, that's wrong...")
- Command failures (non-zero exit codes)
- API errors
- Knowledge gaps

### OpenClaw-Specific Triggers

| Trigger | Action |
|---------|--------|
| Tool call error | Log to TOOLS.md with tool name |
| Session handoff confusion | Log to AGENTS.md with delegation pattern |
| Model behavior surprise | Log to SOUL.md with expected vs actual |
| Skill issue | Log to .learnings/ or report upstream |

## Verification

Check hook is registered:

```bash
openclaw hooks list
```

Check skill is loaded:

```bash
openclaw status
```

## Troubleshooting

### Hook not firing

1. Ensure hooks enabled in config
2. Restart gateway after config changes
3. Check gateway logs for errors

### Learnings not persisting

1. Verify `.learnings/` directory exists
2. Check file permissions
3. Ensure workspace path is configured correctly

### Skill not loading

1. Check skill is in skills directory
2. Verify SKILL.md has correct frontmatter
3. Run `openclaw status` to see loaded skills

FILE:hooks/openclaw/handler.ts
/**
 * Self-Improvement Hook for OpenClaw
 * 
 * Injects a reminder to evaluate learnings during agent bootstrap.
 * Fires on agent:bootstrap event before workspace files are injected.
 */

import type { HookHandler } from 'openclaw/hooks';

const REMINDER_NAME = 'SELF_IMPROVEMENT_REMINDER.md';
const REMINDER_PATH = REMINDER_NAME;

const REMINDER_CONTENT = `## Self-Improvement Reminder

After completing tasks, evaluate whether any learnings should be captured.

Only log if this repo or workspace is using the self-improvement skill.

Before logging:
- Create only missing \`.learnings/\` files; never overwrite existing content
- Do not log secrets, tokens, private keys, environment variables, or raw transcripts
- Prefer short summaries or redacted excerpts over full command output

**Log when:**
- User corrects you → \`.learnings/LEARNINGS.md\`
- Command/operation fails → \`.learnings/ERRORS.md\`
- User wants missing capability → \`.learnings/FEATURE_REQUESTS.md\`
- You discover your knowledge was wrong → \`.learnings/LEARNINGS.md\`
- You find a better approach → \`.learnings/LEARNINGS.md\`

**Promote when pattern is proven:**
- Behavioral patterns → \`SOUL.md\`
- Workflow improvements → \`AGENTS.md\`
- Tool gotchas → \`TOOLS.md\`

Keep entries simple: date, title, what happened, and what to do differently.`;

function isObject(value: unknown): value is Record<string, unknown> {
  return !!value && typeof value === 'object';
}

function isInjectedReminderFile(value: unknown): boolean {
  if (!isObject(value) || value.path !== REMINDER_PATH) {
    return false;
  }

  return (
    value.virtual === true ||
    value.content === REMINDER_CONTENT
  );
}

const handler: HookHandler = async (event) => {
  // Safety checks for event structure
  if (!event || typeof event !== 'object') {
    return;
  }

  // Only handle agent:bootstrap events
  if (event.type !== 'agent' || event.action !== 'bootstrap') {
    return;
  }

  // Safety check for context
  if (!event.context || typeof event.context !== 'object') {
    return;
  }

  // Skip sub-agent sessions to avoid bootstrap issues
  // Sub-agents have sessionKey patterns like "agent:main:subagent:..."
  const sessionKey = event.sessionKey || '';
  if (sessionKey.includes(':subagent:')) {
    return;
  }

  // Inject the reminder as a virtual bootstrap file
  // Check that bootstrapFiles is an array before pushing
  if (Array.isArray(event.context.bootstrapFiles)) {
    const occupiedByOtherFile = event.context.bootstrapFiles.some(
      (file) => isObject(file) && file.path === REMINDER_PATH && !isInjectedReminderFile(file),
    );
    if (occupiedByOtherFile) {
      return;
    }

    const cleanedBootstrapFiles = event.context.bootstrapFiles.filter(
      (file, index, files) =>
        !isInjectedReminderFile(file) ||
        files.findIndex((candidate) => isInjectedReminderFile(candidate)) === index,
    );

    const reminderFile = {
      name: REMINDER_NAME,
      path: REMINDER_PATH,
      content: REMINDER_CONTENT,
      missing: false,
      virtual: true,
    };

    const existingIndex = cleanedBootstrapFiles.findIndex((file) => isInjectedReminderFile(file));
    if (existingIndex === -1) {
      cleanedBootstrapFiles.push(reminderFile);
    } else {
      cleanedBootstrapFiles[existingIndex] = reminderFile;
    }

    event.context.bootstrapFiles = cleanedBootstrapFiles;
  }
};

export default handler;

FILE:hooks/openclaw/HOOK.md
---
name: self-improvement
description: "Injects self-improvement reminder during agent bootstrap"
metadata: {"openclaw":{"emoji":"🧠","events":["agent:bootstrap"]}}
---

# Self-Improvement Hook

Injects a reminder to evaluate learnings during agent bootstrap.

## What It Does

- Fires on `agent:bootstrap` (before workspace files are injected)
- Adds a reminder block to check `.learnings/` for relevant entries
- Prompts the agent to log corrections, errors, and discoveries

## Configuration

No configuration needed. Enable with:

```bash
openclaw hooks enable self-improvement
```

FILE:hooks/openclaw/handler.js
/**
 * Self-Improvement Hook for OpenClaw
 * 
 * Injects a reminder to evaluate learnings during agent bootstrap.
 * Fires on agent:bootstrap event before workspace files are injected.
 */

const REMINDER_NAME = 'SELF_IMPROVEMENT_REMINDER.md';
const REMINDER_PATH = REMINDER_NAME;

const REMINDER_CONTENT = `
## Self-Improvement Reminder

After completing tasks, evaluate whether any learnings should be captured.

Only log if this repo or workspace is using the self-improvement skill.

Before logging:
- Create only missing \`.learnings/\` files; never overwrite existing content
- Do not log secrets, tokens, private keys, environment variables, or raw transcripts
- Prefer short summaries or redacted excerpts over full command output

**Log when:**
- User corrects you → \`.learnings/LEARNINGS.md\`
- Command/operation fails → \`.learnings/ERRORS.md\`
- User wants missing capability → \`.learnings/FEATURE_REQUESTS.md\`
- You discover your knowledge was wrong → \`.learnings/LEARNINGS.md\`
- You find a better approach → \`.learnings/LEARNINGS.md\`

**Promote when pattern is proven:**
- Behavioral patterns → \`SOUL.md\`
- Workflow improvements → \`AGENTS.md\`
- Tool gotchas → \`TOOLS.md\`

Keep entries simple: date, title, what happened, and what to do differently.
`.trim();

function isObject(value) {
  return !!value && typeof value === 'object';
}

function isInjectedReminderFile(value) {
  if (!isObject(value) || value.path !== REMINDER_PATH) {
    return false;
  }

  return (
    value.virtual === true ||
    value.content === REMINDER_CONTENT
  );
}

const handler = async (event) => {
  // Safety checks for event structure
  if (!event || typeof event !== 'object') {
    return;
  }

  // Only handle agent:bootstrap events
  if (event.type !== 'agent' || event.action !== 'bootstrap') {
    return;
  }

  // Safety check for context
  if (!event.context || typeof event.context !== 'object') {
    return;
  }

  // Skip sub-agent sessions to avoid bootstrap issues
  // Sub-agents have sessionKey patterns like "agent:main:subagent:..."
  const sessionKey = event.sessionKey || '';
  if (sessionKey.includes(':subagent:')) {
    return;
  }

  // Inject the reminder as a virtual bootstrap file
  // Check that bootstrapFiles is an array before pushing
  if (Array.isArray(event.context.bootstrapFiles)) {
    const occupiedByOtherFile = event.context.bootstrapFiles.some(
      (file) => isObject(file) && file.path === REMINDER_PATH && !isInjectedReminderFile(file),
    );
    if (occupiedByOtherFile) {
      return;
    }

    const cleanedBootstrapFiles = event.context.bootstrapFiles.filter(
      (file, index, files) =>
        !isInjectedReminderFile(file) ||
        files.findIndex((candidate) => isInjectedReminderFile(candidate)) === index,
    );

    const reminderFile = {
      name: REMINDER_NAME,
      path: REMINDER_PATH,
      content: REMINDER_CONTENT,
      missing: false,
      virtual: true,
    };

    const existingIndex = cleanedBootstrapFiles.findIndex((file) => isInjectedReminderFile(file));
    if (existingIndex === -1) {
      cleanedBootstrapFiles.push(reminderFile);
    } else {
      cleanedBootstrapFiles[existingIndex] = reminderFile;
    }

    event.context.bootstrapFiles = cleanedBootstrapFiles;
  }
};

module.exports = handler;
module.exports.default = handler;

FILE:assets/LEARNINGS.md
# Learnings

Corrections, insights, and knowledge gaps captured during development.

**Categories**: correction | insight | knowledge_gap | best_practice
**Areas**: frontend | backend | infra | tests | docs | config
**Statuses**: pending | in_progress | resolved | wont_fix | promoted | promoted_to_skill

## Status Definitions

| Status | Meaning |
|--------|---------|
| `pending` | Not yet addressed |
| `in_progress` | Actively being worked on |
| `resolved` | Issue fixed or knowledge integrated |
| `wont_fix` | Decided not to address (reason in Resolution) |
| `promoted` | Elevated to CLAUDE.md, AGENTS.md, or copilot-instructions.md |
| `promoted_to_skill` | Extracted as a reusable skill |

## Skill Extraction Fields

When a learning is promoted to a skill, add these fields:

```markdown
**Status**: promoted_to_skill
**Skill-Path**: skills/skill-name
```

Example:
```markdown
## [LRN-20250115-001] best_practice

**Logged**: 2025-01-15T10:00:00Z
**Priority**: high
**Status**: promoted_to_skill
**Skill-Path**: skills/docker-m1-fixes
**Area**: infra

### Summary
Docker build fails on Apple Silicon due to platform mismatch
...
```

---


FILE:assets/AUTO_UPDATE_AUTHORIZATIONS.md
# Auto-Update Authorizations

Skills listed here have been explicitly authorized by the user for automatic self-improvement without approval. By default, **no skill** is authorized for auto-update. The user must explicitly grant auto-update permission on a per-skill basis.

## Format

```markdown
## [skill-name]
**Authorized**: ISO-8601 timestamp
**Authorized By**: user
**Scope**: full | minor_only
**Notes**: Any context about the authorization

---
```

## Authorized Skills

_No skills are currently authorized for auto-update._

---

FILE:assets/pending-improvements/README.md
# Pending Skill Improvements

This folder contains improvement proposals for skills that have encountered failures or could be enhanced. Each file represents a single proposed improvement to a specific skill.

**No improvement is applied without user approval** unless the user has explicitly authorized auto-update for a specific skill (see `AUTO_UPDATE_AUTHORIZATIONS.md`).

## File Naming

`IMP-YYYYMMDD-XXX-skill-name.md`

## Workflow

1. A failure or issue is detected during skill use
2. An improvement proposal is created here documenting what to change
3. If the same issue recurs, the recurrence is logged in the existing proposal
4. The user is notified and asked to approve or reject the proposal
5. Upon approval, the changes are applied to the skill

## Checking Proposals

Ask Claude: "What skill improvements do you recommend?" or "Show me pending skill updates"

FILE:assets/ERRORS.md
# Errors Log

Command failures, exceptions, and unexpected behaviors.

---

FILE:assets/IMPROVEMENT_PROPOSAL_TEMPLATE.md
# Pending Improvement Proposal Template

Use this format for each improvement proposal file saved in `.learnings/pending-improvements/`.

Each proposal is its own file named: `IMP-YYYYMMDD-XXX-skill-name.md`

---

```markdown
# Improvement Proposal: IMP-YYYYMMDD-XXX

**Skill**: skill-name
**Skill Path**: path/to/skill/SKILL.md
**Created**: ISO-8601 timestamp
**Status**: pending | approved | rejected | applied
**Priority**: low | medium | high | critical
**Triggered By**: error | user_feedback | recurring_pattern | knowledge_gap

## Problem

What went wrong or what could be better. Include the error ID if this was triggered by a logged error (e.g., ERR-YYYYMMDD-XXX).

## Root Cause

Why the skill failed or produced suboptimal results.

## Proposed Changes

### Change 1: [add | modify | remove] — [brief label]

**Section**: Which section of the SKILL.md is affected
**Current Content** (if modifying/removing):
```
Existing text or instruction that needs changing
```
**Proposed Content** (if adding/modifying):
```
New or updated text or instruction
```
**Rationale**: Why this change fixes the problem

### Change 2: [add | modify | remove] — [brief label]
(repeat as needed)

## Expected Impact

What will improve after applying these changes.

## Recurrence Log

Track each time this same issue is encountered:

| Date | Error/Context | Notes |
|------|--------------|-------|
| YYYY-MM-DD | ERR-YYYYMMDD-XXX | First occurrence |

---
```

FILE:assets/FEATURE_REQUESTS.md
# Feature Requests

Capabilities requested by user that don't currently exist.

---

FILE:assets/SKILL-TEMPLATE.md
# Skill Template

Template for creating skills extracted from learnings. Copy and customize.

---

## SKILL.md Template

```markdown
---
name: skill-name-here
description: "Concise description of when and why to use this skill. Include trigger conditions."
---

# Skill Name

Brief introduction explaining the problem this skill solves and its origin.

## Quick Reference

| Situation | Action |
|-----------|--------|
| [Trigger 1] | [Action 1] |
| [Trigger 2] | [Action 2] |

## Background

Why this knowledge matters. What problems it prevents. Context from the original learning.

## Solution

### Step-by-Step

1. First step with code or command
2. Second step
3. Verification step

### Code Example

\`\`\`language
// Example code demonstrating the solution
\`\`\`

## Common Variations

- **Variation A**: Description and how to handle
- **Variation B**: Description and how to handle

## Gotchas

- Warning or common mistake #1
- Warning or common mistake #2

## Related

- Link to related documentation
- Link to related skill

## Source

Extracted from learning entry.
- **Learning ID**: LRN-YYYYMMDD-XXX
- **Original Category**: correction | insight | knowledge_gap | best_practice
- **Extraction Date**: YYYY-MM-DD
```

---

## Minimal Template

For simple skills that don't need all sections:

```markdown
---
name: skill-name-here
description: "What this skill does and when to use it."
---

# Skill Name

[Problem statement in one sentence]

## Solution

[Direct solution with code/commands]

## Source

- Learning ID: LRN-YYYYMMDD-XXX
```

---

## Template with Scripts

For skills that include executable helpers:

```markdown
---
name: skill-name-here
description: "What this skill does and when to use it."
---

# Skill Name

[Introduction]

## Quick Reference

| Command | Purpose |
|---------|---------|
| `./scripts/helper.sh` | [What it does] |
| `./scripts/validate.sh` | [What it does] |

## Usage

### Automated (Recommended)

\`\`\`bash
./skills/skill-name/scripts/helper.sh [args]
\`\`\`

### Manual Steps

1. Step one
2. Step two

## Scripts

| Script | Description |
|--------|-------------|
| `scripts/helper.sh` | Main utility |
| `scripts/validate.sh` | Validation checker |

## Source

- Learning ID: LRN-YYYYMMDD-XXX
```

---

## Naming Conventions

- **Skill name**: lowercase, hyphens for spaces
  - Good: `docker-m1-fixes`, `api-timeout-patterns`
  - Bad: `Docker_M1_Fixes`, `APITimeoutPatterns`

- **Description**: Start with action verb, mention trigger
  - Good: "Handles Docker build failures on Apple Silicon. Use when builds fail with platform mismatch."
  - Bad: "Docker stuff"

- **Files**:
  - `SKILL.md` - Required, main documentation
  - `scripts/` - Optional, executable code
  - `references/` - Optional, detailed docs
  - `assets/` - Optional, templates

---

## Extraction Checklist

Before creating a skill from a learning:

- [ ] Learning is verified (status: resolved)
- [ ] Solution is broadly applicable (not one-off)
- [ ] Content is complete (has all needed context)
- [ ] Name follows conventions
- [ ] Description is concise but informative
- [ ] Quick Reference table is actionable
- [ ] Code examples are tested
- [ ] Source learning ID is recorded

After creating:

- [ ] Update original learning with `promoted_to_skill` status
- [ ] Add `Skill-Path: skills/skill-name` to learning metadata
- [ ] Test skill by reading it in a fresh session

---
name: openclaw-administrator
description: Administration guide for the OpenClaw CLI. Use this skill whenever the user asks how to run, configure, or troubleshoot any openclaw command or concept, including: gateway and daemon lifecycle, setup and onboarding (interactive and headless/VPS), multi-agent setup and routing bindings, sub-agent spawning and delegation, workspace and bootstrap file management (AGENTS.md, SOUL.md, IDENTITY.md, USER.md, TOOLS.md), adding and configuring AI model providers, setting primary and fallback models, per-agent model overrides, the model allowlist, local models (Ollama/vLLM/LM Studio), custom providers via models.providers, the OpenAI-compatible HTTP endpoint (Open WebUI, LobeChat, LibreChat integration), channel login and connectivity, messaging, memory and wiki, plugins and skills, MCP servers, cron, tasks, flows, sandbox, browser automation, nodes, security audits, backups, and diagnostics. Also use for global flags (--dev, --profile, --container), command family routing, and any question about how openclaw subcommands work.
---

### 🛡️ EMERGENCY SAFETY PROTOCOL
- **NO OVERWRITES:** Never use `cat`, redirection (>), or raw file writes to `openclaw.json`. Use a validated JSON utility script that loads-modifies-saves.
- **SCHEMA VALIDITY:** Before adding new keys (e.g., `mcp`, `tools`), verify against official schema. Never guess keys.
- **PRE-BOUNCE VERIFICATION:** 
    1. Validate JSON syntax (`python -m json.tool`).
    2. Run `openclaw doctor`.
    3. If any errors, **ABORT** the bounce.
- **BACKUP INTEGRITY:** Maintain a rotating history of up to 3 validated backups (`openclaw.json.bak.1` to `.bak.3`). After a verified successful bounce, prune any backups older than 24 hours and maintain the most recent 3 verified snapshots.
- **SECRET HYGIENE:** Never write credential-like patterns (`PAT`, `TOKEN`, `KEY`) into any workspace file. Before removing any existing configuration that contains keys/credentials, ensure they are secured in the environment (`~/.openclaw/credentials/.env`) first.
- **EXPLICIT AUTHORIZATION:** I must never make changes to the `openclaw.json` or system configuration without *first* proposing the change, explaining the "Why," and obtaining explicit user permission.
- **EXTERNAL SKILL AUDIT:** Never install a skill from ClawHub or any external repository without *first* inspecting the source code. Look specifically for:
    - Exfiltration of data/tokens to unknown URLs.
    - Malicious/damaging system commands (e.g., recursive deletion, system modifications).
    - Hardcoded or suspicious credential usage.
    Only proceed with installation once the source has been audited as benign.


## Execution Workflow

1. **Clarify the target state.** Ask what should change and what must stay untouched.
2. **Select runtime scope first.** Default profile unless isolation is explicitly needed:
   - `openclaw --dev ...` → isolated dev state under `~/.openclaw-dev`, gateway port `19001`.
   - `openclaw --profile <n> ...` → isolated state under `~/.openclaw-<n>`.
   - `openclaw --container <n> ...` → target a named container for execution.
3. **Route to the right command family.** Use `references/command-map.md` for quick routing.
4. **Expand subcommands before risky operations.** Run `openclaw <family> --help` for starred families; confirm flags before executing.
5. **Prefer machine-readable output for automation.** Use `--json` where available; parse and verify.
6. **Verify outcomes explicitly.** Check with `openclaw status`, `openclaw health`, `openclaw doctor`, or command-specific follow-up.

## Safety Rules

- Require explicit user confirmation before `reset`, `uninstall`, destructive `--force` flows, or operations that remove stored provider keys.
- Prefer non-destructive diagnostics first: `status`, `health`, `doctor`, `logs`, `security audit`.
- Keep profile scope consistent across a workflow — never mix `--dev` and default in the same sequence.
- For gateway issues, diagnose before restart unless restart is explicitly requested.
- Keep the OpenAI HTTP endpoint on loopback/tailnet only — never expose to the public internet.

## Triage Sequence

For any "OpenClaw not working" incident:

1. `openclaw status`
2. `openclaw health`
3. `openclaw doctor`
4. `openclaw security audit` (if config or provider connection settings may be misconfigured)
5. `openclaw backup verify <archive>` (if data integrity is suspected)
6. Check `openclaw gateway ...`, `openclaw channels ...`, or `openclaw nodes ...` based on where failure appears.
7. Escalate to targeted commands in `references/command-map.md`.

---

## Architecture Overview

OpenClaw runs as a **single Gateway** (WebSocket, default `127.0.0.1:18789`) that:
- Owns all messaging surfaces (WhatsApp/Telegram/Discord/Slack/Signal/iMessage/etc.)
- Hosts one or many **agents**, each with isolated workspace + auth + session store
- Exposes a **typed WS API** for CLI/app/automation clients
- Optionally serves an **OpenAI-compatible HTTP surface** at the same port

Clients (CLI, macOS app, web UI) connect over WebSocket. Nodes (macOS/iOS/Android/headless) also connect with `role: node`. The canvas UI is served at `/__openclaw__/canvas/`.

---

## Workspace and Bootstrap Files

Each agent has a **workspace** directory (`~/.openclaw/workspace` by default) containing markdown files the agent reads on boot:

| File | Purpose |
|---|---|
| `AGENTS.md` | Core instructions, memory, tool policy |
| `SOUL.md` | Persona, tone, boundaries |
| `IDENTITY.md` | Name, emoji, theme, avatar |
| `USER.md` | Who the user is; preferences |
| `TOOLS.md` | Tool usage notes and policies |
| `HEARTBEAT.md` | Periodic check-in template |
| `BOOT.md` | One-time boot ritual (delete after) |
| `BOOTSTRAP.md` | Startup sequence instructions |
| `MEMORY.md` + `memory/*.md` | Persistent memory store |

- Missing files get a placeholder marker injected at session start; execution continues.
- Size limits: `bootstrapMaxChars` (default 12 000 chars per file), `bootstrapTotalMaxChars` (default 60 000 total).
- `openclaw setup --workspace <path>` recreates any missing defaults without overwriting existing ones.
- Keep the workspace in a **private git repo** for backup and recovery. Never commit `~/.openclaw/` state dirs.
- If `~/openclaw/` (old path) exists alongside `~/.openclaw/workspace`, keep only one active workspace to avoid auth/session drift.

---

## Agent System

### Single-Agent Mode (default)

Out of the box: one agent, `agentId = main`, sessions keyed `agent:main:<mainKey>`.

### Multi-Agent Setup

Each agent is a fully isolated brain:
- Own workspace (`agents.defaults.workspace` or per-agent override)
- Own `agentDir` (`~/.openclaw/agents/<agentId>/agent`) — holds `auth-profiles.json`, model registry, per-agent config
- Own session store (`~/.openclaw/agents/<agentId>/sessions`)
- **Never reuse `agentDir` across agents** — causes auth/session collisions.

**Creating a new agent:**
```bash
openclaw agents add coding
openclaw agents add work --workspace ~/.openclaw/workspace-work --non-interactive
```

**Identity setup:**
```bash
openclaw agents set-identity --agent main --name "OpenClaw" --emoji "🦞" --avatar avatars/oc.png
openclaw agents set-identity --workspace ~/.openclaw/workspace --from-identity   # reads IDENTITY.md
```

**Listing and verifying:**
```bash
openclaw agents list
openclaw agents list --bindings
openclaw agents bindings --agent work
```

### Routing Bindings

Bindings route inbound messages to the correct agent. **Most-specific match wins** (priority order):

1. `peer` (exact DM/group id)
2. `parentPeer` (thread inheritance)
3. `guildId + roles` (Discord role routing)
4. `guildId` (Discord server)
5. `teamId` (Slack workspace)
6. Explicit `accountId` for a channel
7. `accountId: "*"` (channel-wide fallback, all accounts)
8. Default agent (`agents.list[].default`, else first entry, else `main`)

**Managing bindings:**
```bash
openclaw agents bind --agent work --bind telegram:ops --bind discord:guild-a
openclaw agents unbind --agent work --bind telegram:ops
openclaw agents unbind --agent work --all
```

Omitting `--agent` targets the current default. A channel-only binding (no `accountId`) is auto-upgraded to account-scoped when you later add an explicit accountId for the same channel+agent.

**Config example (WhatsApp DM split):**
```json
{
  "agents": {
    "list": [
      { "id": "home", "default": true, "workspace": "~/.openclaw/workspace-home" },
      { "id": "work", "workspace": "~/.openclaw/workspace-work" }
    ]
  },
  "bindings": [
    { "agentId": "home", "match": { "channel": "whatsapp", "accountId": "personal" } },
    { "agentId": "work", "match": { "channel": "whatsapp", "accountId": "biz" } }
  ]
}
```

See `references/multi-agent-recipes.md` for full Discord, Telegram, and WhatsApp examples.

### Sub-Agents

Sub-agents are background agent runs spawned by the main agent to handle tasks in parallel. They run in their own isolated session (`agent:<agentId>:subagent:<uuid>`), receive only the instructions given to them (not full conversation history), and announce results back to the requester channel when done.

**Key characteristics:**
- Depth limit: currently flat (sub-agents cannot spawn their own sub-agents; this restriction is expected to be lifted in a future release)
- Restricted tool policies relative to the parent
- Auto-announce on completion (direct delivery first, falls back to queue routing, then exponential backoff)
- Tracked as background tasks; inspect/control via slash commands

**Slash command control:**
```
/subagents          # list all background sub-agent runs
/subagents spawn <agentId> <task> [--model <m>] [--thinking <level>]
/focus <target>     # bind current thread to a sub-agent session
/unfocus            # detach thread binding
/session idle       # inspect/update inactivity auto-unfocus
/session max-age    # control hard session age cap
```

**When to use sub-agents vs sessions_send:**
- Sub-agent (`sessions_spawn`): need the result now, in this conversation; parallel fan-out.
- `sessions_send`: fire-and-forget delegation; the other agent works independently and responds later.

**Allowlist config (to let sub-agents target named agents):**
```json
{
  "agents": {
    "list": [
      {
        "id": "main",
        "subagents": {
          "allowAgents": ["coding", "research"]
        }
      }
    ]
  }
}
```

---

## Running an Agent Turn

```bash
# Run via gateway (default)
openclaw agent --to +15555550123 --message "Status update" --deliver
openclaw agent --agent ops --message "Summarize logs" --thinking medium
openclaw agent --session-id 1234 --message "Summarize inbox" --json

# Run embedded (local, no gateway)
openclaw agent --agent ops --message "Run locally" --local

# Deliver to a different channel/account
openclaw agent --agent ops --message "Generate report" \
  --deliver --reply-channel slack --reply-to "#reports"
```

Key flags: `--thinking <off|minimal|low|medium|high|xhigh>`, `--verbose <on|off>`, `--timeout <seconds>`.

Gateway mode falls back to embedded when the gateway request fails; `--local` forces embedded up front.

---

## OpenAI-Compatible HTTP API

The Gateway can serve an OpenAI-compatible HTTP surface at the same port as WebSocket. **Disabled by default.**

**Enable in `~/.openclaw/openclaw.json`:**
```json
{
  "gateway": {
    "http": {
      "endpoints": {
        "chatCompletions": { "enabled": true }
      }
    }
  }
}
```

**Endpoints (when enabled):**
- `POST /v1/chat/completions`
- `GET /v1/models` / `GET /v1/models/{id}`
- `POST /v1/embeddings`
- `POST /v1/responses`

**Model field = agent target:**
- `"openclaw"` or `"openclaw/default"` → configured default agent
- `"openclaw/<agentId>"` → specific agent (e.g. `"openclaw/research"`)
- Legacy aliases: `"openclaw:<agentId>"`, `"agent:<agentId>"`

**Optional request headers:**
- `x-openclaw-model: <provider/model>` — override backend model for the agent
- `x-openclaw-agent-id: <agentId>` — compatibility agent override
- `x-openclaw-session-key: <key>` — control session routing
- `x-openclaw-message-channel: <channel>` — set synthetic ingress channel context

**Session behavior:** stateless per request by default. If the request includes an OpenAI `user` string, a stable session key is derived from it so repeated calls share the same agent session.

**Note:** OpenClaw's gateway token is what any connected client uses to authenticate. Keep this endpoint on loopback or a private network (tailnet/VPN) — don't expose it directly to the public internet.

**Examples:**
```bash
# Non-streaming
curl -sS http://127.0.0.1:18789/v1/chat/completions \
  -H 'Authorization: Bearer YOUR_TOKEN' \
  -H 'Content-Type: application/json' \
  -d '{"model":"openclaw/default","messages":[{"role":"user","content":"hi"}]}'

# Streaming with backend model override
curl -N http://127.0.0.1:18789/v1/chat/completions \
  -H 'Authorization: Bearer YOUR_TOKEN' \
  -H 'Content-Type: application/json' \
  -H 'x-openclaw-model: openai/gpt-4o' \
  -d '{"model":"openclaw/research","stream":true,"messages":[{"role":"user","content":"hi"}]}'

# List agent targets
curl -sS http://127.0.0.1:18789/v1/models \
  -H 'Authorization: Bearer YOUR_TOKEN'

# Embeddings
curl -sS http://127.0.0.1:18789/v1/embeddings \
  -H 'Authorization: Bearer YOUR_TOKEN' \
  -H 'Content-Type: application/json' \
  -H 'x-openclaw-model: openai/text-embedding-3-small' \
  -d '{"model":"openclaw/default","input":["alpha","beta"]}'
```

**Open WebUI quick setup:**
- Base URL: `http://127.0.0.1:18789/v1` (Docker on macOS: `http://host.docker.internal:18789/v1`)
- API key: your gateway bearer token
- Model: `openclaw/default`

---

## Storing provider keys outside the config file (SecretRefs)

OpenClaw supports referencing provider keys via environment variables, files, or exec commands instead of writing plaintext values into `openclaw.json`. This is an OpenClaw configuration feature — the skill just explains how to set it up.

| Reference type | Syntax | Example |
|---|---|---|
| Env variable | `secretref-env:VAR_NAME` | `secretref-env:ANTHROPIC_API_KEY` |
| File path | `secretref-file:/path/to/file` | `secretref-file:~/.secrets/openai_key` |
| Exec command | `secretref-exec:command` | `secretref-exec:op read op://vault/item/field` |

**Set a reference via CLI:**
```bash
openclaw config set providers.anthropic.key \
  --ref-provider env --ref-source ANTHROPIC_API_KEY

openclaw secrets configure       # interactive setup wizard
openclaw secrets reload          # reload without gateway restart
openclaw secrets audit           # check all references resolve correctly
openclaw secrets audit --check   # non-zero exit on failures (CI use)
```

**Use ref mode during non-interactive onboard:**
```bash
openclaw onboard --non-interactive \
  --auth-choice anthropic-api-key \
  --secret-input-mode ref \
  --anthropic-api-key secretref-env:ANTHROPIC_API_KEY
```

---

## Setup and Onboarding

**Interactive (recommended for first-time):**
```bash
openclaw onboard
```

**Non-interactive (VPS/headless):**
```bash
openclaw onboard \
  --non-interactive \
  --auth-choice anthropic-api-key \
  --anthropic-api-key <key> \
  --gateway-bind loopback \
  --install-daemon \
  --daemon-runtime node \
  --node-manager pnpm \
  --skip-channels \
  --skip-search
```

Key `--auth-choice` values: `anthropic-api-key`, `openai-api-key`, `openrouter-api-key`, `gemini-api-key`, `github-copilot`, `chutes`, `deepseek-api-key`, `custom-api-key`, `skip`.

Key gateway bind modes: `loopback` (default), `lan`, `tailnet`, `auto`.

For Tailscale: `--tailscale <off|serve|funnel>`.
For remote gateway: `--mode remote --remote-url <url> --remote-token <token>`.

---

## Models and Providers

**Read `references/models-and-providers.md` for the full reference.** It covers: model selection order, primary/fallback config, per-agent overrides, the model allowlist, custom providers, local models (Ollama/vLLM/LM Studio), multi-key rotation, how failover and cooldowns work, and sub-agent model config.

**Quick orientation:**

Model refs use `provider/model` format: `anthropic/claude-opus-4-6`, `openai/gpt-5.4`, `ollama/llama3.3`.

```bash
# Essential commands
openclaw models list                            # see configured models
openclaw models status                          # primary, fallbacks, auth overview
openclaw models set anthropic/claude-opus-4-6  # set primary model
openclaw models set-image openai/gpt-image-1   # set image model
openclaw models fallbacks add openrouter/auto   # add a fallback
openclaw models aliases add Opus anthropic/claude-opus-4-6
openclaw models auth login                      # add/re-auth a provider
openclaw models auth login-github-copilot       # GitHub Copilot device login
openclaw models scan                            # find free models on OpenRouter
```

**Config: primary + fallback chain:**
```json
{
  "agents": {
    "defaults": {
      "model": {
        "primary": "anthropic/claude-opus-4-6",
        "fallbacks": [
          "openrouter/anthropic/claude-opus-4-6",
          "openai/gpt-5.4"
        ]
      }
    }
  }
}
```

**Config: add a custom/third-party provider** (any OpenAI-compatible API):
```json
{
  "models": {
    "mode": "merge",
    "providers": {
      "my-provider": {
        "baseUrl": "https://api.example.com/v1",
        "apiKey": "MY_PROVIDER_KEY",
        "api": "openai-completions",
        "models": [
          { "id": "my-model", "name": "My Model" }
        ]
      }
    }
  }
}
```

Use `api: "anthropic-messages"` for Anthropic-compatible endpoints. After adding a provider: `openclaw config validate`, `openclaw models list --provider my-provider`, `openclaw gateway restart`.

**Per-agent model** (each agent can use a different model):
```json
{
  "agents": {
    "list": [
      { "id": "main", "model": "anthropic/claude-opus-4-6" },
      { "id": "fast", "model": "anthropic/claude-haiku-4-5" }
    ]
  }
}
```

For built-in providers (Anthropic, OpenAI, Google, OpenRouter, GitHub Copilot, Mistral, xAI, DeepSeek, Ollama, and 20+ more) and their `--auth-choice` values, see `references/models-and-providers.md`.

---

## Inference CLI (`infer` / `capability`)

Direct inference without a full agent run:

```bash
openclaw infer list                                    # list capabilities
openclaw infer model run --model anthropic/claude-sonnet-4-6 --prompt "hi"
openclaw infer image generate --prompt "a lobster in space"
openclaw infer image describe path/to/image.jpg
openclaw infer audio transcribe recording.mp3
openclaw infer tts convert --text "Hello" --voice nova
openclaw infer web search "latest OpenClaw release"
openclaw infer web fetch https://example.com
openclaw infer embedding create --input "embed this text"
```

---

## Channels

```bash
openclaw channels list
openclaw channels status --probe            # live per-account probe
openclaw channels add --channel telegram --account alerts --token $TOKEN
openclaw channels add --channel discord --account work --token $DISCORD_TOKEN
openclaw channels remove --channel discord --account work --delete
openclaw channels login --channel whatsapp --account personal --verbose
openclaw channels logout --channel whatsapp --account personal
openclaw channels capabilities --channel telegram --account alerts
openclaw channels logs --channel whatsapp --lines 200
```

Supported channels: `whatsapp`, `telegram`, `discord`, `googlechat`, `slack`, `signal`, `imessage`, `msteams`, `mattermost` (plugin), and many more via plugins.

DM security policies: `dmPolicy: "pairing"` (unknown senders get a code), `dmPolicy: "allowlist"` (explicit list only), `dmPolicy: "open"`.

---

## Resources

Read the appropriate reference file before answering deep questions in these areas:

| Reference | When to read it |
|---|---|
| `references/command-map.md` | Full command tree, routing quick-map, common recipes, caution commands |
| `references/models-and-providers.md` | Adding providers, fallback chains, custom config, local models, auth rotation, failover |
| `references/multi-agent-recipes.md` | Annotated multi-agent JSON configs (WhatsApp, Discord, Telegram, channel split) |
| `references/openai-http-api.md` | Enabling the HTTP endpoint, model targeting, headers, Open WebUI setup |

Live docs: `openclaw docs [query]` or `https://docs.openclaw.ai`

FILE:README.md
# OpenClaw CLI Skill

A Codex skill for operating, configuring, and troubleshooting the OpenClaw CLI safely and efficiently. Updated for OpenClaw `2026.4.x`.

## What This Skill Covers

- **Setup and onboarding** — interactive and non-interactive (`onboard`, `setup`, `configure`), including headless VPS setup with all `--auth-choice` providers and `--secret-input-mode ref` for storing provider keys as env var references instead of plaintext
- **Gateway lifecycle** — foreground, service install/start/stop/restart, `daemon`, `logs`, health probes
- **Multi-agent routing** — creating isolated agents, routing bindings (`agents add`, `agents bind`), per-agent workspace and `agentDir`, account-scoped vs channel-wide bindings, identity files
- **Sub-agent spawning** — `sessions_spawn` vs `sessions_send` patterns, depth limits, allowlist config, `/subagents` slash command control
- **Agent turns** — `agent` command flags, `--thinking`, `--deliver`, `--local`, session routing
- **OpenAI-compatible HTTP API** — enabling `POST /v1/chat/completions`, model target syntax (`openclaw/default`, `openclaw/<agentId>`), `x-openclaw-model` header, Open WebUI/LobeChat/LibreChat integration
- **Workspace bootstrap files** — `AGENTS.md`, `SOUL.md`, `IDENTITY.md`, `USER.md`, `TOOLS.md`, `HEARTBEAT.md`, size limits, git backup
- **Config file references** — storing provider keys as env var references, file paths, or exec commands (`secrets reload/audit/configure`)
- **Models and inference** — `models` full surface, `infer` CLI (image, audio, TTS, video, web, embeddings), fallback chains, multi-provider auth
- **Channels** — multi-account login, `channels add/remove/status/capabilities/resolve/logs`, DM policies
- **Memory and wiki** — `memory status/index/search/promote`, `wiki` with Obsidian integration
- **Tasks, flows, cron** — background task management, scheduled jobs
- **Plugins and skills** — install, enable/disable, marketplace, doctor
- **MCP** — serve, list, show, set/unset
- **Security and secrets** — `security audit`, `secrets audit/reload/configure/apply`, `backup create/verify`
- **Sandbox, browser, nodes** — container management, browser automation, node control (camera, canvas, location)
- **Diagnostics** — full triage sequence (`status`, `health`, `doctor`, `security audit`, `backup verify`)

## Repository Structure

```
openclaw-cli/
├── SKILL.md                          # Core skill: workflow, architecture, agent system, HTTP API, models summary
├── README.md                         # This file
├── _meta.json                        # Skill metadata
└── references/
    ├── command-map.md                # Full command tree, routing map, all recipes, caution commands
    ├── models-and-providers.md       # Models: selection, fallbacks, custom providers, local, failover, rotation
    ├── multi-agent-recipes.md        # Annotated JSON configs for multi-agent routing scenarios
    └── openai-http-api.md            # OpenAI-compatible HTTP endpoint complete reference
```

`SKILL.md` covers the most commonly needed surfaces inline. For deep dives:
- Full command syntax → `references/command-map.md`
- Models, providers, fallbacks, custom config → `references/models-and-providers.md`
- Multi-agent config examples → `references/multi-agent-recipes.md`
- HTTP API details and curl examples → `references/openai-http-api.md`

## Installation

Clone into your Codex skills directory:

```bash
mkdir -p ~/.codex/skills
git clone https://github.com/ramensushi2026/openclaw-cli-skill.git ~/.codex/skills/openclaw-cli
```

Or copy the folder so the final path is `~/.codex/skills/openclaw-cli`.

## Usage

```text
Use $openclaw-cli to set up a headless OpenClaw instance on a VPS.
Use $openclaw-cli to create a second agent bound to a Telegram bot.
Use $openclaw-cli to enable the OpenAI HTTP endpoint and connect Open WebUI.
Use $openclaw-cli to diagnose why my WhatsApp channel keeps disconnecting.
Use $openclaw-cli to configure provider keys as environment variable references instead of plaintext in the config file.
Use $openclaw-cli to spawn a sub-agent for parallel research tasks.
```

## Safety Model

- Confirms intent before `reset`, `uninstall`, `--force`, or `sandbox recreate --all`.
- Always recommends `openclaw backup create --verify` before risky operations.
- Diagnoses before restarting (uses `status`, `health`, `doctor` first).
- Keeps profile context consistent (`--dev` / `--profile` / default) across a workflow.
- Keeps the OpenAI HTTP endpoint guidance scoped to loopback/private networks — flags it should not face the public internet.

## Version History

| Version | Notes |
|---|---|
| `2.0.0` | Major update for OpenClaw 2026.4.x: multi-agent routing, sub-agents, HTTP API, SecretRefs, `infer`, `tasks`, `flows`, `wiki`, `backup`, `secrets`, `browser`, `nodes`, `mcp`, full onboard flags, workspace bootstrap files |
| `1.0.0` | Initial release — basic command families, gateway/node lifecycle, channel login, `agent`, `doctor` |

## References

- OpenClaw CLI docs: `https://docs.openclaw.ai/cli`
- Live docs search: `openclaw docs [query]`
- Gateway architecture: `https://docs.openclaw.ai/concepts/architecture`
- Multi-agent routing: `https://docs.openclaw.ai/concepts/multi-agent`
- Sub-agents: `https://docs.openclaw.ai/tools/subagents`
- OpenAI HTTP API: `https://docs.openclaw.ai/gateway/openai-http-api`

FILE:references/command-map.md
# OpenClaw CLI Command Map

## Table of Contents

1. [Global Flags](#global-flags)
2. [Command Routing Quick Map](#command-routing-quick-map)
3. [Full Command Tree](#full-command-tree)
4. [Command Families Reference](#command-families-reference)
5. [Common Recipes](#common-recipes)
6. [Caution Commands](#caution-commands)

---

## Global Flags

Apply before the command family:

```bash
openclaw [--dev] [--profile <n>] [--container <n>] <command>
```

| Flag | Effect |
|---|---|
| `--dev` | Isolate under `~/.openclaw-dev`; default gateway port `19001` |
| `--profile <n>` | Isolate under `~/.openclaw-<n>` |
| `--container <n>` | Target a named container for execution |
| `--no-color` | Disable ANSI colors (also `NO_COLOR=1`) |
| `--update` | Shorthand for `openclaw update` (source installs only) |
| `-V`, `--version`, `-v` | Print version and exit |
| `-h`, `--help` | Show help |

Always choose profile flags before the command family:
```bash
openclaw --dev status
openclaw --profile staging gateway start
```

---

## Command Routing Quick Map

| Goal | Command family |
|---|---|
| Setup / first-time onboarding | `onboard`, `setup`, `configure` |
| Interactive config wizard | `configure`, `config` (no subcommand) |
| Non-interactive config | `config get/set/unset/validate` |
| Diagnose health | `status`, `health`, `doctor`, `logs` |
| Run an agent turn | `agent` |
| Manage isolated agents | `agents` |
| Spawn/control sub-agents (in-session) | `/subagents` slash command |
| Operate gateway runtime | `gateway`, `daemon` |
| Operate headless node service | `node`, `nodes` |
| Manage channel auth/connectivity | `channels`, `pairing`, `devices`, `qr` |
| Send/read messages | `message` |
| Direct inference (no agent) | `infer` (alias: `capability`) |
| Model discovery/config | `models` |
| Memory management | `memory`, `wiki` |
| Background tasks | `tasks`, `flows` |
| Scheduled jobs | `cron` |
| Plugin/extension management | `plugins` |
| Skill management | `skills` |
| MCP server management | `mcp` |
| ACP tooling | `acp` |
| Security and policy | `security`, `approvals`, `sandbox` |
| Config file references (env vars, files, exec) | `secrets` |
| Backup / restore | `backup` |
| Browser automation | `browser` |
| Voice calls | `voicecall` (plugin, if installed) |
| System events / presence | `system` |
| Live docs search | `docs` |
| Control UI | `dashboard` |
| Terminal UI | `tui` |
| DNS helpers | `dns` |
| Contact/group directory | `directory` |
| Device pairing tokens | `devices` |
| Update CLI | `update` |
| Reset local data | `reset` |
| Full removal | `uninstall` |

For starred families (`*`), always inspect subcommands first:
```bash
openclaw <family> --help
```

---

## Full Command Tree

```
openclaw [--dev] [--profile <n>] [--container <n>] <command>

# Setup and Configuration
  setup            Initialize config + workspace
  onboard          Interactive/non-interactive full setup
  configure        Interactive configuration wizard
  config
    get <path>
    set <path> <value>
    set --ref-provider <p> --ref-source <s> --ref-id <id>   (SecretRef mode)
    set --batch-json '<json>'                                 (batch mode)
    set --batch-file <path>
    set --dry-run [--json] [--allow-exec]
    unset <path>
    file
    schema
    validate [--json]
  completion [-s <shell>] [-i]
  doctor [--repair] [--deep] [--yes] [--non-interactive]

# Updates and Maintenance
  update [--channel <stable|beta|dev>] [--tag <spec>] [--yes] [--dry-run]
  update status [--json]
  update wizard
  backup
    create [--output <path>] [--dry-run] [--verify] [--only-config]
    verify <archive>

# Dashboard and UI
  dashboard [--no-open]
  tui

# Security and Secrets
  security
    audit [--deep] [--fix]
  secrets
    reload [--url] [--token] [--timeout] [--expect-final] [--json]
    audit [--check] [--allow-exec] [--json]
    configure [--apply] [--yes] [--providers-only] [--agent <id>] [--json]
    apply --from <path> [--dry-run] [--allow-exec] [--json]
  approvals
    get
    set
    allowlist add|remove

# Gateway and Services
  gateway
    run                     (foreground)
    start | stop | restart
    install | uninstall
    service                 (alias for lifecycle subcommands)
    health
    status
    call <method> [params]
    usage-cost
    probe
    discover
  daemon                    (legacy alias for gateway service commands)
    status | install | uninstall | start | stop | restart
  logs [--lines <n>]

# Agents and Sessions
  agent
    -m/--message <text>
    -t/--to <dest>
    --session-id <id>
    --agent <id>
    --thinking <off|minimal|low|medium|high|xhigh>
    --verbose <on|off>
    --deliver
    --local
    --reply-channel <channel>
    --reply-to <target>
    --reply-account <id>
    --timeout <seconds>
    --json
  agents
    list [--bindings] [--json]
    add [name] [--workspace <dir>] [--model <id>] [--bind <ch[:acctId]>] [--non-interactive]
    bindings [--agent <id>] [--json]
    bind --agent <id> --bind <ch[:acctId]> (repeatable)
    unbind --agent <id> --bind <ch[:acctId]> | --all
    delete <id> [--force]
    set-identity [--agent <id>] [--from-identity] [--name] [--theme] [--emoji] [--avatar]
  sessions
    cleanup
  hooks
    list | info | check | enable | disable | install | update

# Inference
  infer (alias: capability)
    list
    inspect
    model run|list|inspect|providers
    model auth login|logout|status
    image generate|edit|describe|describe-many|providers
    audio transcribe|providers
    tts convert|voices|providers|status|enable|disable|set-provider
    video generate|describe|providers
    web search|fetch|providers
    embedding create|providers
    auth add|login|login-github-copilot|setup-token|paste-token
    auth order get|set|clear

# Models
  models
    list [--json]
    status
    set <model>
    set-image <model>
    aliases list|add|remove
    fallbacks list|add|remove|clear
    image-fallbacks list|add|remove|clear
    scan
    auth add|login|login-github-copilot|setup-token|paste-token
    auth order get|set|clear

# Memory and Knowledge
  memory
    status [--deep] [--fix]
    index
    search "<query>" | --query "<query>"
    promote
  wiki
    status | doctor | init | ingest | compile | lint | search | get | apply
    bridge import
    unsafe-local import
    obsidian status|search|open|command|daily

# Messaging and Channels
  message
    send --target <dest> --message <text> [--channel <ch>] [--json]
    broadcast
    poll
    react | reactions
    read | edit | delete | pin | unpin | pins
    permissions | search
    thread create|list|reply
    emoji list|upload
    sticker send|upload
    role info|add|remove
    channel info|list
    member info
    voice status
    event list|create
    timeout | kick | ban
  channels
    list [--no-usage] [--json]
    status [--probe] [--timeout <ms>] [--json]
    capabilities [--channel <n>] [--account <id>] [--target <dest>] [--json]
    resolve <entries...> [--channel <n>] [--kind <auto|user|group>] [--json]
    add [--channel <n>] [--account <id>] [--name <label>] [--token ...]
    remove [--channel <n>] [--account <id>] [--delete]
    login [--channel <ch>] [--account <id>] [--verbose]
    logout [--channel <ch>] [--account <id>]
    logs [--channel <name|all>] [--lines <n>] [--json]
  directory
    self
    peers list [--query <text>] [--limit <n>]
    groups list [--query <text>] [--limit <n>]
    groups members --group-id <id> [--limit <n>]
  pairing
    list | approve
  devices
    list | remove | clear | approve | reject | rotate | revoke
  qr
  clawbot
    qr                      (legacy alias)

# Tasks and Automation
  tasks
    list | audit | maintenance | show | notify | cancel
    flow list|show|cancel
  flows
  cron
    status | list | add | edit | rm | enable | disable | runs | run

# Plugins and Skills
  plugins
    list [--json]
    inspect <id>
    install <path|.tgz|npm-spec|plugin@marketplace> [--force]
    marketplace list <marketplace>
    enable <id> | disable <id>
    uninstall <id>
    update [<id>|--all]
    doctor
  skills
    search [query...] [--limit <n>] [--json]
    install <slug> [--version <v>] [--force]
    update <slug|--all>
    list [--verbose] [--eligible] [--json]
    info <n> [--json]
    check [--json]

# MCP and ACP
  mcp
    serve
    list [--json]
    show [name]
    set <name> <value>
    unset <name>
  acp
    client

# Sandbox
  sandbox
    list [--browser] [--json]
    recreate [--all] [--session <key>] [--agent <id>] [--browser] [--force]
    explain [--session <key>] [--agent <id>] [--json]

# System
  system
    event
    heartbeat last|enable|disable
    presence
  status
  health [--verbose]

# Node Host (headless)
  node
    run | status | install | uninstall | stop | restart
  nodes
    status | describe | list | pending | approve | reject | rename
    invoke | notify | push
    canvas snapshot|present|hide|navigate|eval
    canvas a2ui push|reset
    camera list|snap|clip
    screen record
    location get

# Browser
  browser
    status | start | stop | reset-profile
    tabs | open | focus | close
    profiles | create-profile | delete-profile
    screenshot | snapshot
    navigate | resize | click | type | press | hover | drag | select
    upload | fill | dialog | wait | evaluate | console | pdf

# Utility
  docs [query...]
  dns
    setup
  webhooks
    gmail setup|run
  update           (see above)
  reset
  uninstall
  voicecall        (plugin; if installed)
```

---

## Command Families Reference

### `gateway` ★

Run/inspect/manage the WebSocket Gateway.

```bash
openclaw gateway                     # start foreground (logs to stdout)
openclaw gateway start               # start as background service
openclaw gateway stop
openclaw gateway restart
openclaw gateway install             # register launchd/systemd
openclaw gateway uninstall
openclaw gateway health              # fetch health via RPC
openclaw gateway status              # show service state
openclaw gateway call <method>       # raw RPC call
openclaw gateway usage-cost          # token/cost summary
openclaw gateway probe               # live connectivity probe
openclaw gateway discover            # find other gateway instances
openclaw --dev gateway               # isolated dev gateway
openclaw gateway --port 18789
openclaw gateway --force             # forcefully clear port conflicts (CAUTION)
```

### `agent`

Execute one agent turn via the Gateway (or embedded with `--local`).

```bash
openclaw agent --to +15555550123 --message "Run summary" --deliver
openclaw agent --agent ops --message "Summarize inbox" --thinking high
openclaw agent --session-id 1234 --message "Continue" --json
openclaw agent --agent ops --message "Task" --deliver \
  --reply-channel slack --reply-to "#ops"
```

### `agents` ★

Manage isolated agents, auth, routing, workspaces.

```bash
openclaw agents list --bindings
openclaw agents add coding
openclaw agents add work --workspace ~/.openclaw/workspace-work --non-interactive
openclaw agents bind --agent work --bind telegram:work-bot
openclaw agents unbind --agent work --all
openclaw agents set-identity --agent main --from-identity
openclaw agents delete work --force
```

`main` cannot be added (reserved) or deleted.

### `channels` ★

Manage chat channel connections. Multi-account supported on most platforms.

```bash
openclaw channels list
openclaw channels status --probe
openclaw channels add --channel telegram --account alerts --name "Alerts" --token $TOKEN
openclaw channels login --channel whatsapp --account personal --verbose
openclaw channels capabilities --channel telegram
openclaw channels logs --channel all --lines 500
```

### `models` ★

Discover, configure, and authenticate model providers.

```bash
openclaw models list
openclaw models set anthropic/claude-sonnet-4-6
openclaw models fallbacks add openrouter/anthropic/claude-sonnet-4-6
openclaw models auth login
openclaw models auth login-github-copilot
openclaw models auth order set anthropic openrouter openai
openclaw models scan
```

### `infer` ★ (alias: `capability`)

Direct inference without a full agent session.

```bash
openclaw infer list
openclaw infer model run --model anthropic/claude-haiku-4-5 --prompt "Hello"
openclaw infer image generate --prompt "a lobster"
openclaw infer audio transcribe meeting.mp3
openclaw infer tts convert --text "Hello" --voice nova
openclaw infer web search "OpenClaw changelog"
openclaw infer embedding create --input "embed this"
```

### `config` ★

Non-interactive config helpers.

```bash
openclaw config get agents.defaults.model
openclaw config set agents.defaults.model anthropic/claude-sonnet-4-6
openclaw config set providers.openai.key --ref-provider env --ref-source OPENAI_API_KEY
openclaw config set --batch-file updates.json
openclaw config set --dry-run --json
openclaw config unset channels.slack.token
openclaw config file
openclaw config schema
openclaw config validate --json
```

### `onboard`

Interactive or non-interactive full setup.

```bash
# Interactive
openclaw onboard

# Non-interactive headless
openclaw onboard \
  --non-interactive \
  --auth-choice anthropic-api-key \
  --anthropic-api-key $ANTHROPIC_KEY \
  --gateway-bind loopback \
  --gateway-auth token \
  --install-daemon \
  --daemon-runtime node \
  --node-manager pnpm \
  --skip-channels \
  --skip-search

# With Tailscale
openclaw onboard --non-interactive --tailscale serve

# Custom provider
openclaw onboard --non-interactive \
  --auth-choice custom-api-key \
  --custom-base-url https://api.example.com/v1 \
  --custom-model-id my-model \
  --custom-api-key $CUSTOM_KEY \
  --custom-compatibility openai
```

`--auth-choice` values include: `anthropic-api-key`, `openai-api-key`, `openrouter-api-key`, `gemini-api-key`, `github-copilot`, `chutes`, `deepseek-api-key`, `kilocode-api-key`, `litellm-api-key`, `moonshot-api-key`, `venice-api-key`, `xai-api-key`, `mistral-api-key`, `qwen-api-key`, `volcengine-api-key`, `custom-api-key`, `skip`.

### `memory`

Vector search over workspace memory files.

```bash
openclaw memory status --deep
openclaw memory index
openclaw memory search "previous conversations about deployment"
openclaw memory promote                # rank and optionally append top recalls to MEMORY.md
```

### `wiki`

Workspace wiki management with optional Obsidian integration.

```bash
openclaw wiki init
openclaw wiki ingest
openclaw wiki compile
openclaw wiki search "authentication patterns"
openclaw wiki get "my-note-title"
openclaw wiki obsidian status
openclaw wiki obsidian daily
```

### `tasks`

Manage background task queue.

```bash
openclaw tasks list
openclaw tasks show <id>
openclaw tasks cancel <id>
openclaw tasks audit
openclaw tasks maintenance
openclaw tasks flow list
openclaw tasks flow cancel <id>
```

### `cron`

Manage scheduled agent jobs.

```bash
openclaw cron list
openclaw cron add --schedule "0 9 * * *" --message "Daily summary" --agent main
openclaw cron edit <id>
openclaw cron rm <id>
openclaw cron enable <id> | disable <id>
openclaw cron runs <id>
openclaw cron run <id>          # trigger immediately
```

### `plugins` ★

Manage extensions.

```bash
openclaw plugins list
openclaw plugins install my-plugin
openclaw plugins install @myorg/plugin@marketplace
openclaw plugins install ./local-plugin --force
openclaw plugins enable my-plugin
openclaw plugins disable my-plugin
openclaw plugins doctor
openclaw plugins marketplace list openclaw-marketplace
```

Most plugin changes require a gateway restart.

### `skills` ★

List and manage ClawHub skills.

```bash
openclaw skills search "github"
openclaw skills install my-skill-slug
openclaw skills install my-skill-slug --version 1.2.0 --force
openclaw skills update --all
openclaw skills list --verbose
openclaw skills info my-skill-slug
openclaw skills check
```

### `mcp`

Manage MCP (Model Context Protocol) servers.

```bash
openclaw mcp list
openclaw mcp show my-server
openclaw mcp set my-server someKey someValue
openclaw mcp unset my-server someKey
openclaw mcp serve          # start an MCP server on stdio
```

### `sandbox`

Manage isolated execution containers.

```bash
openclaw sandbox list
openclaw sandbox list --browser
openclaw sandbox recreate --all
openclaw sandbox recreate --agent coding
openclaw sandbox explain --agent coding
```

`recreate` removes existing runtimes so the next use re-seeds them from current config.

### `security`

Security tooling and config audits.

```bash
openclaw security audit             # audit config + state
openclaw security audit --deep      # best-effort live gateway probe
openclaw security audit --fix       # tighten safe defaults automatically
```

### `secrets`

Manage config file references — store provider keys as environment variable refs, file paths, or exec commands instead of plaintext in `openclaw.json`.

```bash
openclaw secrets audit
openclaw secrets audit --check          # non-zero exit on findings (CI use)
openclaw secrets reload                 # hot-reload without gateway restart
openclaw secrets configure              # interactive setup
openclaw secrets apply --from plan.json # apply a pre-built plan
openclaw secrets apply --from plan.json --dry-run
```

### `backup`

Create and verify local state archives.

```bash
openclaw backup create
openclaw backup create --output ~/backups/openclaw-$(date +%Y%m%d).tar.gz
openclaw backup create --verify                  # create and immediately verify
openclaw backup create --only-config             # config only, no workspace
openclaw backup verify my-backup.tar.gz
```

### `browser` ★

Manage and automate a dedicated browser instance.

```bash
openclaw browser status
openclaw browser start
openclaw browser navigate "https://example.com"
openclaw browser screenshot --output page.png
openclaw browser click --selector "#submit-btn"
openclaw browser type --selector "#search" --text "query"
openclaw browser evaluate --script "return document.title"
openclaw browser tabs
openclaw browser pdf --output page.pdf
```

### `nodes` ★

Manage gateway-owned node pairings (macOS/iOS/Android/headless).

```bash
openclaw nodes list
openclaw nodes status
openclaw nodes pending
openclaw nodes approve <nodeId>
openclaw nodes invoke <nodeId> <command>
openclaw nodes canvas snapshot <nodeId>
openclaw nodes camera snap <nodeId>
openclaw nodes location get <nodeId>
```

### `doctor`

Run health checks and quick fixes.

```bash
openclaw doctor
openclaw doctor --repair                  # attempt automatic repairs
openclaw doctor --deep                    # scan for extra gateway installs
openclaw doctor --yes --non-interactive   # headless / CI
```

### `status` and `health`

```bash
openclaw status             # channel health + recent recipients + provider usage
openclaw status --deep      # broader gateway health probes
openclaw health             # fetch health from running gateway
openclaw health --verbose   # live probe + expanded human-readable output
```

### `update`

```bash
openclaw update
openclaw update --channel beta
openclaw update --tag 2026.4.9
openclaw update --dry-run --json
openclaw update status
```

### `logs`

```bash
openclaw logs                     # tail gateway logs via RPC
openclaw logs --lines 500
```

---

## Common Recipes

### Full fresh setup (headless VPS with Anthropic)

```bash
npm install -g openclaw
openclaw onboard \
  --non-interactive \
  --auth-choice anthropic-api-key \
  --anthropic-api-key $ANTHROPIC_API_KEY \
  --gateway-bind loopback \
  --install-daemon \
  --node-manager pnpm \
  --skip-channels
openclaw doctor
openclaw status
```

### Add a second agent and bind to Telegram bot

```bash
openclaw agents add research --workspace ~/.openclaw/workspace-research
openclaw channels add --channel telegram --account research-bot --token $TG_TOKEN
openclaw agents bind --agent research --bind telegram:research-bot
openclaw gateway restart
openclaw agents list --bindings
```

### Enable Open WebUI via the HTTP API

```bash
# Edit ~/.openclaw/openclaw.json
openclaw config set gateway.http.endpoints.chatCompletions.enabled true
openclaw gateway restart
# Then in Open WebUI: base URL = http://127.0.0.1:18789/v1, model = openclaw/default
```

### Run a gateway locally with dev profile

```bash
openclaw --dev gateway --port 19001
openclaw --dev status
```

### Diagnose channel disconnection

```bash
openclaw status
openclaw channels status --probe
openclaw channels logs --channel all
openclaw doctor --repair
```

### Send a message with JSON output

```bash
openclaw message send --target +15555550123 --message "Hello" --json
openclaw message send --channel telegram --target @mychat --message "Hello"
```

### Run an agent turn and deliver reply

```bash
openclaw agent --to +15555550123 --message "Run daily summary" --deliver
```

### Set model failover chain

```bash
openclaw models set anthropic/claude-opus-4-6
openclaw models fallbacks add openrouter/anthropic/claude-opus-4-6
openclaw models fallbacks add openai/gpt-4o
openclaw models fallbacks list
```

### Non-destructive full diagnostic

```bash
openclaw status
openclaw health --verbose
openclaw doctor
openclaw security audit
openclaw channels status --probe
openclaw nodes status
```

### Backup before a risky operation

```bash
openclaw backup create --verify --output ~/backups/pre-update.tar.gz
openclaw update
openclaw doctor
```

---

## Caution Commands

Always confirm intent before running these:

| Command | Effect |
|---|---|
| `openclaw reset` | Destructive local state + config reset (CLI stays installed) |
| `openclaw uninstall` | Removes gateway service + all local data |
| `openclaw gateway --force` | Forcefully kills port conflicts |
| `openclaw sandbox recreate --all` | Destroys all sandbox runtimes |
| `openclaw agents delete <id> --force` | Moves workspace/state/sessions to Trash without prompting |
| `openclaw onboard --reset --reset-scope full` | Full wipe including workspace |

**Recommended pre-flight for any caution command:**
```bash
openclaw backup create --verify
```

FILE:references/models-and-providers.md
# Models and Providers

## Table of Contents

1. [How model selection works](#how-model-selection-works)
2. [Model ref format](#model-ref-format)
3. [Setting the primary model](#setting-the-primary-model)
4. [Fallback chains](#fallback-chains)
5. [Image, PDF, and generation models](#image-pdf-and-generation-models)
6. [Per-agent model overrides](#per-agent-model-overrides)
7. [Model allowlist](#model-allowlist)
8. [Switching models in chat](#switching-models-in-chat)
9. [Built-in providers](#built-in-providers)
10. [Adding a custom or third-party provider](#adding-a-custom-or-third-party-provider)
11. [Local models (Ollama, vLLM, LM Studio)](#local-models)
12. [Multi-key provider rotation](#multi-key-provider-rotation)
13. [Model failover and cooldowns](#model-failover-and-cooldowns)
14. [Sub-agent model config](#sub-agent-model-config)
15. [Scanning for free models](#scanning-for-free-models)
16. [Verifying your setup](#verifying-your-setup)

---

## How model selection works

OpenClaw selects models in this priority order for every agent run:

1. **Session override** (if the user ran `/model <ref>` in chat)
2. **Primary model** (`agents.defaults.model.primary` or `agents.defaults.model`)
3. **Fallbacks** in `agents.defaults.model.fallbacks` (in order)
4. **Provider failover** — within each candidate, OpenClaw tries additional configured keys before advancing to the next model

The key separation to understand:
- `agents.defaults.model` → **which model to use** (primary + fallback chain)
- `agents.defaults.models` → **allowlist and aliases** — if set, only listed models are available

---

## Model ref format

All model refs use `provider/model`:

```
anthropic/claude-opus-4-6
openai/gpt-5.4
google/gemini-3.1-pro-preview
openrouter/auto
ollama/llama3.3
moonshot/kimi-k2.5
```

Provider aliases normalize automatically: `z.ai/*` → `zai/*`, `openai-codex` is the provider id for the Codex login flow, etc.

For OpenRouter-style nested IDs (e.g. `moonshotai/kimi-k2`), always include the provider prefix: `openrouter/moonshotai/kimi-k2`.

---

## Setting the primary model

**Via CLI (recommended):**
```bash
openclaw models set anthropic/claude-opus-4-6
openclaw models set openai/gpt-5.4
openclaw models set openrouter/auto
```

**Via config (`~/.openclaw/openclaw.json`):**
```json
{
  "agents": {
    "defaults": {
      "model": {
        "primary": "anthropic/claude-opus-4-6"
      }
    }
  }
}
```

Shorthand (model string directly, no primary/fallbacks object):
```json
{
  "agents": {
    "defaults": {
      "model": "anthropic/claude-opus-4-6"
    }
  }
}
```

---

## Fallback chains

Fallbacks kick in when a model fails with a failover-worthy error (rate limit, auth exhaustion, overload). Non-retryable errors (bad request, context overflow) do **not** trigger fallback.

**Via CLI:**
```bash
openclaw models fallbacks list
openclaw models fallbacks add openrouter/anthropic/claude-opus-4-6
openclaw models fallbacks add openai/gpt-4o
openclaw models fallbacks remove openai/gpt-4o
openclaw models fallbacks clear
```

**Via config:**
```json
{
  "agents": {
    "defaults": {
      "model": {
        "primary": "anthropic/claude-opus-4-6",
        "fallbacks": [
          "openrouter/anthropic/claude-opus-4-6",
          "openai/gpt-5.4",
          "openrouter/auto"
        ]
      }
    }
  }
}
```

**Fallback resolution rules:**
- Per-provider key rotation happens inside each candidate before advancing to the next.
- If the current run is on an override model not in the configured chain, OpenClaw appends the configured primary at the end so it can settle back to the default once earlier candidates are exhausted.
- If every candidate fails, a `FallbackSummaryError` is thrown with per-attempt detail and the soonest cooldown expiry.

**Recommended strategy:**
- Primary: best/strongest model you have access to.
- First fallback: same model on a proxy (e.g. OpenRouter) for redundancy.
- Second fallback: a faster/cheaper model as a cost backstop.
- Third fallback: `openrouter/auto` as a catch-all.

---

## Image, PDF, and generation models

```bash
openclaw models set-image openai/gpt-image-1     # vision model when primary can't accept images
openclaw models image-fallbacks add google/gemini-3.1-pro-preview

openclaw models set-image-generation openai/gpt-image-1   # image generation
```

Via config:
```json
{
  "agents": {
    "defaults": {
      "model": { "primary": "anthropic/claude-opus-4-6" },
      "imageModel": { "primary": "openai/gpt-5.4", "fallbacks": ["google/gemini-3.1-pro-preview"] },
      "pdfModel": "anthropic/claude-sonnet-4-6",
      "imageGenerationModel": "openai/gpt-image-1"
    }
  }
}
```

- `imageModel` is used only when the primary model cannot accept image inputs.
- `pdfModel` is used by the pdf tool. Falls back to `imageModel`, then the resolved session model.
- `imageGenerationModel` is used by the `image_generate` capability.

---

## Per-agent model overrides

Each agent in `agents.list` can have its own model, independently of global defaults:

```json
{
  "agents": {
    "defaults": {
      "model": { "primary": "anthropic/claude-sonnet-4-6" }
    },
    "list": [
      {
        "id": "main",
        "model": "anthropic/claude-opus-4-6"
      },
      {
        "id": "fast",
        "workspace": "~/.openclaw/workspace-fast",
        "model": "anthropic/claude-haiku-4-5"
      },
      {
        "id": "research",
        "workspace": "~/.openclaw/workspace-research",
        "model": {
          "primary": "anthropic/claude-opus-4-6",
          "fallbacks": ["openrouter/anthropic/claude-opus-4-6"]
        }
      }
    ]
  }
}
```

You can also set a per-agent default thinking level:
```json
{
  "id": "deep",
  "model": "anthropic/claude-opus-4-6",
  "thinkingDefault": "high"
}
```

---

## Model allowlist

If `agents.defaults.models` is set, it becomes an **allowlist** — only listed models are eligible for `/model` switching and session overrides. Users trying to pick an unlisted model see:

```
Model "provider/model" is not allowed. Use /model to list available models.
```

Use this to prevent unintended model drift in production setups.

```json
{
  "agents": {
    "defaults": {
      "model": { "primary": "anthropic/claude-sonnet-4-6" },
      "models": {
        "anthropic/claude-sonnet-4-6": { "alias": "Sonnet" },
        "anthropic/claude-opus-4-6": { "alias": "Opus" },
        "anthropic/claude-haiku-4-5": { "alias": "Haiku" },
        "openrouter/auto": { "alias": "auto" }
      }
    }
  }
}
```

- Aliases let users pick models by short name in chat (`/model Sonnet`).
- An empty `models.defaults.models` object (omitted) means all configured models are available.
- Per-agent `agents.list[].skills` can override the global model list.

**Managing aliases via CLI:**
```bash
openclaw models aliases list
openclaw models aliases add Sonnet anthropic/claude-sonnet-4-6
openclaw models aliases remove Sonnet
```

---

## Switching models in chat

Users can switch models per-session without restarting:

```
/model                        # interactive picker
/model list                   # numbered list
/model 3                      # pick by number
/model openai/gpt-5.4         # pick by ref
/model Sonnet                 # pick by alias
/model status                 # detailed view: auth candidates, endpoint info
```

On Discord, `/model` and `/models` open an interactive picker with provider and model dropdowns.

**Live switch behavior:**
- If the agent is idle, the next run uses the new model immediately.
- If a run is already active, the switch is queued and takes effect at the next clean retry point.
- Once tool activity or reply output has started, the queued switch waits until the next user turn.

---

## Built-in providers

These require **no** `models.providers` config entry. Run `openclaw onboard` or set the relevant env var on the machine running OpenClaw, then pick a model ref.

| Provider | Provider ID | How OpenClaw connects | Example model | `--auth-choice` for onboard |
|---|---|---|---|---|
| Anthropic | `anthropic` | `ANTHROPIC_API_KEY` env var | `anthropic/claude-opus-4-6` | `anthropic-api-key` |
| OpenAI | `openai` | `OPENAI_API_KEY` env var | `openai/gpt-5.4` | `openai-api-key` |
| OpenAI Codex | `openai-codex` | device login flow via `openclaw models auth login` | `openai-codex/gpt-5.4` | `openai-codex` |
| Google Gemini | `google` | `GEMINI_API_KEY` env var | `google/gemini-3.1-pro-preview` | `gemini-api-key` |
| Google Vertex | `google-vertex` | gcloud login on the gateway host | `google-vertex/gemini-3.1-pro` | — |
| Google Gemini CLI | `google-gemini-cli` | device login flow via `openclaw models auth login` | `google-gemini-cli/gemini-3-flash-preview` | `google-gemini-cli` |
| OpenRouter | `openrouter` | `OPENROUTER_API_KEY` env var | `openrouter/auto` | `openrouter-api-key` |
| GitHub Copilot | `github-copilot` | device login flow via `openclaw models auth login-github-copilot` | `github-copilot/claude-sonnet-4-6` | `github-copilot` |
| OpenCode Zen | `opencode` | `OPENCODE_API_KEY` env var | `opencode/claude-opus-4-6` | `opencode-zen` |
| OpenCode Go | `opencode-go` | `OPENCODE_API_KEY` env var | `opencode-go/kimi-k2.5` | `opencode-go` |
| Mistral | `mistral` | `MISTRAL_API_KEY` env var | `mistral/mistral-large` | `mistral-api-key` |
| xAI (Grok) | `xai` | `XAI_API_KEY` env var | `xai/grok-3` | `xai-api-key` |
| DeepSeek | `deepseek` | `DEEPSEEK_API_KEY` env var | `deepseek/deepseek-r1` | `deepseek-api-key` |
| Z.AI (GLM) | `zai` | `ZAI_API_KEY` env var | `zai/glm-5.1` | `zai-api-key` |
| Kilo Gateway | `kilocode` | `KILOCODE_API_KEY` env var | `kilocode/kilo/auto` | `kilocode-api-key` |
| Vercel AI Gateway | `vercel-ai-gateway` | `AI_GATEWAY_API_KEY` env var | `vercel-ai-gateway/anthropic/claude-opus-4.6` | `ai-gateway-api-key` |
| Moonshot (Kimi) | `kimi` | `KIMI_API_KEY` env var | `kimi/kimi-k2.5` | `kimi-code-api-key` |
| MiniMax | `minimax` | device login or `MINIMAX_API_KEY` env var | `minimax/abab7-chat` | `minimax-global-api` |
| Qwen | `qwen` | device login or `QWEN_API_KEY` env var | `qwen/qwen-max` | `qwen-api-key` |
| Chutes | `chutes` | `CHUTES_API_KEY` env var | `chutes/deepseek-v3` | `chutes` |
| Venice | `venice` | `VENICE_API_KEY` env var | `venice/llama-3.3-70b` | `venice-api-key` |
| Together | `together` | `TOGETHER_API_KEY` env var | `together/llama-3.3-70b` | `together-api-key` |
| HuggingFace | `huggingface` | `HUGGINGFACE_HUB_TOKEN` env var | `huggingface/deepseek-ai/DeepSeek-R1` | `huggingface-api-key` |
| Ollama (local) | `ollama` | `OLLAMA_API_KEY` env var (any value enables auto-detection) | `ollama/llama3.3` | via plugin |
| vLLM (local) | `vllm` | `VLLM_API_KEY` env var (any value enables auto-detection) | `vllm/my-model` | via plugin |

**Connecting a provider via the onboard wizard:**
```bash
openclaw onboard                                      # interactive — picks up where you are
openclaw models auth login                            # add or re-authenticate any provider interactively
openclaw models auth login-github-copilot             # GitHub Copilot device login flow
openclaw models auth login --provider google-gemini-cli
openclaw models auth order set anthropic openrouter openai  # set provider priority order
```

---

## Adding a custom or third-party provider

Use `models.providers` in `openclaw.json` for any OpenAI-compatible or Anthropic-compatible API.

**When to use `models.providers`:**
- The provider is not in OpenClaw's built-in catalog
- You have a self-hosted inference server (vLLM, LiteLLM, LM Studio)
- You want to use a specific provider endpoint or relay
- You want to add models before official support arrives

**Config structure:**
```json
{
  "models": {
    "mode": "merge",
    "providers": {
      "<provider-id>": {
        "baseUrl": "https://api.example.com/v1",
        "apiKey": "MY_API_KEY",
        "api": "openai-completions",
        "models": [
          { "id": "my-model-id", "name": "My Model", "contextWindow": 128000 }
        ]
      }
    }
  }
}
```

**Key fields:**

| Field | Values | Notes |
|---|---|---|
| `baseUrl` | URL string | Base API endpoint (no trailing `/chat/completions`) |
| `apiKey` | string or `ENV_VAR` | Use `ENV_VAR` to reference an env var |
| `api` | `openai-completions` or `anthropic-messages` | Use `openai-completions` for any `/v1/chat/completions` API |
| `models[].id` | string | The model id sent to the provider |
| `models[].name` | string | Display name |
| `models[].contextWindow` | number | Native context window (metadata) |
| `models[].contextTokens` | number | Effective runtime cap (overrides contextWindow) |

**Use `"mode": "merge"`** (default) — OpenClaw merges your providers into `models.json` alongside built-ins. Use `"replace"` only if you want to remove all built-in providers.

**OpenAI-compatible example (Moonshot / Kimi K2.5):**
```json
{
  "models": {
    "mode": "merge",
    "providers": {
      "moonshot": {
        "baseUrl": "https://api.moonshot.ai/v1",
        "apiKey": "MOONSHOT_API_KEY",
        "api": "openai-completions",
        "models": [
          { "id": "kimi-k2.5", "name": "Kimi K2.5", "contextWindow": 131072 },
          { "id": "kimi-k2-thinking", "name": "Kimi K2 Thinking" }
        ]
      }
    }
  },
  "agents": {
    "defaults": {
      "model": { "primary": "moonshot/kimi-k2.5" }
    }
  }
}
```

**LiteLLM proxy (OpenAI-compatible, multiple upstream providers):**
```json
{
  "models": {
    "mode": "merge",
    "providers": {
      "litellm": {
        "baseUrl": "http://localhost:4000/v1",
        "apiKey": "LITELLM_MASTER_KEY",
        "api": "openai-completions",
        "models": [
          { "id": "claude-opus-4-6", "name": "Claude Opus via LiteLLM" },
          { "id": "gpt-5.4", "name": "GPT-5.4 via LiteLLM" }
        ]
      }
    }
  }
}
```

**Private vLLM server (openai-compatible):**
```json
{
  "models": {
    "mode": "merge",
    "providers": {
      "my-vllm": {
        "baseUrl": "http://10.0.0.5:8000/v1",
        "apiKey": "token-from-vllm",
        "api": "openai-completions",
        "models": [
          { "id": "Qwen/Qwen2.5-72B-Instruct", "name": "Qwen 2.5 72B" }
        ]
      }
    }
  }
}
```

**After adding a custom provider, always add to allowlist and verify:**
```bash
openclaw config validate
openclaw models list --provider moonshot
openclaw models set moonshot/kimi-k2.5
openclaw gateway restart
openclaw models status
```

---

## Local models

### Ollama

Ollama is auto-detected at `http://127.0.0.1:11434`. Set any value for `OLLAMA_API_KEY` to opt in to auto-discovery:

```bash
export OLLAMA_API_KEY=ollama
openclaw plugins enable ollama   # if not already enabled
openclaw models list --local
openclaw models set ollama/llama3.3
```

Custom config (if Ollama runs on a different host/port):
```json
{
  "models": {
    "mode": "merge",
    "providers": {
      "ollama": {
        "baseUrl": "http://10.0.0.10:11434/v1",
        "apiKey": "ollama",
        "api": "openai-completions",
        "models": [
          { "id": "qwen3.5:32b", "name": "Qwen 3.5 32B" },
          { "id": "llama3.3:latest", "name": "Llama 3.3" }
        ]
      }
    }
  }
}
```

**Ollama model management:**
```bash
ollama pull llama3.3        # pull a model before using it in OpenClaw
ollama list                 # list downloaded models
openclaw models scan --provider ollama --no-probe
```

### vLLM and SGLang

Default ports: vLLM `8000`, SGLang `30000`. Set `VLLM_API_KEY` (any value) to opt in:
```bash
export VLLM_API_KEY=any-value
openclaw models list --local
```

Or configure explicitly (see vLLM example above under custom providers).

### LM Studio

LM Studio exposes an OpenAI-compatible endpoint at `http://127.0.0.1:1234/v1`:
```json
{
  "models": {
    "providers": {
      "lmstudio": {
        "baseUrl": "http://127.0.0.1:1234/v1",
        "apiKey": "lm-studio",
        "api": "openai-completions",
        "models": [
          { "id": "loaded-model-name", "name": "My LM Studio Model" }
        ]
      }
    }
  }
}
```

---

## Multi-key provider rotation

OpenClaw can automatically rotate between multiple provider keys when one hits a rate limit (429, quota exceeded). This is configured on the machine running OpenClaw, not inside this skill. Non-rate-limit errors fail immediately without rotation.

**Environment variable naming patterns OpenClaw recognises:**

```bash
# Single key
ANTHROPIC_API_KEY="sk-ant-key1"

# Multiple keys (comma or semicolon separated)
ANTHROPIC_API_KEYS="sk-ant-key1,sk-ant-key2,sk-ant-key3"

# Numbered keys
ANTHROPIC_API_KEY_1="sk-ant-key1"
ANTHROPIC_API_KEY_2="sk-ant-key2"

# Single live override (highest priority)
OPENCLAW_LIVE_ANTHROPIC_KEY="sk-ant-override"
```

Same naming pattern applies to other providers: `OPENAI_API_KEYS`, `OPENROUTER_API_KEYS`, `GEMINI_API_KEYS`, etc.

**View and adjust provider priority:**
```bash
openclaw models auth order get
openclaw models auth order set anthropic openrouter openai
```

---

## How failover and cooldowns work

When a model run fails, OpenClaw works through its candidate chain automatically. This section describes that behaviour so you can configure it correctly.

**Sequence for a single run:**
1. Try primary model with the first configured key/profile for that provider.
2. On rate limit: try the next key/profile for the same provider.
3. When all profiles for that model are exhausted: advance to the next model in the fallback list.
4. Repeat until a model succeeds or every candidate fails.

**Cooldown behaviour:**
- Rate-limited profiles are cooled down with exponential backoff before retrying.
- Persistent failures (e.g. billing holds): backoff starts at 5 hours, doubles per failure, caps at 24 hours. Counter resets after 24 hours clean.
- Overloaded responses: one same-provider key rotation is tried before moving to the next fallback model.
- Cooldown state is stored in `~/.openclaw/agents/<agentId>/agent/auth-state.json` on the machine running OpenClaw.

**Inspect the current state:**
```bash
openclaw models status          # shows primary, fallbacks, and any provider warnings
openclaw models status --check  # non-zero exit if any provider is missing or expiring (CI use)
openclaw health --verbose       # broader gateway and provider health view
```

**Force a specific model for the current session:**
```
/model anthropic/claude-opus-4-6
```

---

## Sub-agent model config

Sub-agents inherit the calling agent's model by default. To use a cheaper model for background work:

```json
{
  "agents": {
    "defaults": {
      "subagents": {
        "model": "anthropic/claude-haiku-4-5",
        "thinking": "minimal"
      }
    }
  }
}
```

Per-agent override:
```json
{
  "agents": {
    "list": [
      {
        "id": "main",
        "subagents": {
          "model": "anthropic/claude-sonnet-4-6",
          "thinking": "low",
          "runTimeoutSeconds": 120
        }
      }
    ]
  }
}
```

A `model` passed directly in `sessions_spawn` always wins over these defaults.

---

## Scanning for free models

`openclaw models scan` inspects OpenRouter's free model catalog and ranks candidates for use as fallbacks:

```bash
openclaw models scan                         # interactive scan (requires OPENROUTER_API_KEY)
openclaw models scan --no-probe              # metadata only, no live probes
openclaw models scan --min-params 7          # filter to >=7B parameter models
openclaw models scan --max-age-days 60       # skip models older than 60 days
openclaw models scan --set-default           # set top result as primary
openclaw models scan --max-candidates 3 --yes  # non-interactive, accept top 3
```

Scan rankings: image support → tool latency → context size → parameter count.

---

## Verifying your setup

```bash
openclaw models list              # all configured models
openclaw models list --all        # full catalog including built-ins
openclaw models list --local      # local providers only
openclaw models status            # primary, fallbacks, image model, auth overview
openclaw models status --check    # CI-friendly: non-zero if auth missing or expiring
openclaw config validate          # validate the full config including models.providers
```

**Debug tip:** start minimal when adding a custom provider — `baseUrl`, `apiKey`, `api`, and one model. Verify with `openclaw models list --provider <id>`. Add more models and params only after the basic connection works. Never edit `~/.openclaw/agents/<agentId>/agent/models.json` directly — it is regenerated from `openclaw.json`.

FILE:references/openai-http-api.md
# OpenAI-Compatible HTTP API

OpenClaw's Gateway can serve an OpenAI-compatible HTTP surface at the same port as the WebSocket. **Disabled by default.**

## Enable

Add to `~/.openclaw/openclaw.json`:

```json
{
  "gateway": {
    "http": {
      "endpoints": {
        "chatCompletions": { "enabled": true }
      }
    }
  }
}
```

Or via CLI:
```bash
openclaw config set gateway.http.endpoints.chatCompletions.enabled true
openclaw gateway restart
```

## Endpoints (when enabled)

| Method | Path | Purpose |
|---|---|---|
| `POST` | `/v1/chat/completions` | Chat completions (OpenAI-compatible) |
| `GET` | `/v1/models` | List agent targets |
| `GET` | `/v1/models/{id}` | Get single agent target |
| `POST` | `/v1/embeddings` | Embeddings |
| `POST` | `/v1/responses` | OpenResponses-compatible endpoint |

All run on the same port as the Gateway WebSocket (default `18789`).

## Gateway token

The HTTP endpoint uses the same token as your OpenClaw gateway. Send it as a bearer token on every request:

```
Authorization: Bearer <your-gateway-token>
```

Gateway token config modes (set in `openclaw.json`):
- `gateway.auth.mode="token"`: token comes from `gateway.auth.token` or the `OPENCLAW_GATEWAY_TOKEN` env var
- `gateway.auth.mode="password"`: password comes from `gateway.auth.password` or `OPENCLAW_GATEWAY_PASSWORD`
- `gateway.auth.mode="none"`: no token needed (only appropriate on a private/loopback interface)

Keep this endpoint on loopback or a private network — not exposed to the public internet.

## Model Field = Agent Target

The OpenAI `model` field is treated as an **agent target**, not a raw provider model id:

| `model` value | Routes to |
|---|---|
| `"openclaw"` | Configured default agent |
| `"openclaw/default"` | Configured default agent (stable alias) |
| `"openclaw/<agentId>"` | Specific agent (e.g. `"openclaw/research"`) |
| `"openclaw:<agentId>"` | Legacy alias (still supported) |
| `"agent:<agentId>"` | Legacy alias (still supported) |

`openclaw/default` is always stable even if the real default agent id changes between environments.

## Optional Request Headers

| Header | Effect |
|---|---|
| `x-openclaw-model: <provider/model>` | Override backend model for the selected agent |
| `x-openclaw-agent-id: <agentId>` | Compatibility agent override |
| `x-openclaw-session-key: <key>` | Fully control session routing |
| `x-openclaw-message-channel: <ch>` | Set synthetic ingress channel context |

`x-openclaw-model` examples: `openai/gpt-4o`, `anthropic/claude-opus-4-6`, `gpt-4o` (bare alias).

## Session Behavior

- **Default:** stateless per request — a new session key is generated each call.
- **With `user` field:** if the request includes an OpenAI `user` string, the Gateway derives a stable session key from it, so repeated calls share the same agent session.

## Streaming

Set `"stream": true` to receive Server-Sent Events (SSE):
- `Content-Type: text/event-stream`
- Each event line: `data: <json>`
- Stream ends: `data: [DONE]`

## Examples

### Non-streaming chat completion

```bash
curl -sS http://127.0.0.1:18789/v1/chat/completions \
  -H 'Authorization: Bearer YOUR_TOKEN' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "openclaw/default",
    "messages": [{"role":"user","content":"hi"}]
  }'
```

### Streaming with backend model override

```bash
curl -N http://127.0.0.1:18789/v1/chat/completions \
  -H 'Authorization: Bearer YOUR_TOKEN' \
  -H 'Content-Type: application/json' \
  -H 'x-openclaw-model: anthropic/claude-opus-4-6' \
  -d '{
    "model": "openclaw/research",
    "stream": true,
    "messages": [{"role":"user","content":"Explain the gateway architecture"}]
  }'
```

### Target a specific agent

```bash
curl -sS http://127.0.0.1:18789/v1/chat/completions \
  -H 'Authorization: Bearer YOUR_TOKEN' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "openclaw/coding",
    "messages": [{"role":"user","content":"Review this PR"}]
  }'
```

### Stable session (using `user` field)

```bash
curl -sS http://127.0.0.1:18789/v1/chat/completions \
  -H 'Authorization: Bearer YOUR_TOKEN' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "openclaw/default",
    "user": "my-stable-session-id",
    "messages": [{"role":"user","content":"Remember what we discussed?"}]
  }'
```

### List agent targets

```bash
curl -sS http://127.0.0.1:18789/v1/models \
  -H 'Authorization: Bearer YOUR_TOKEN'
```

Returns `openclaw`, `openclaw/default`, and `openclaw/<agentId>` entries. Not raw provider catalogs.

### Get a single agent target

```bash
curl -sS http://127.0.0.1:18789/v1/models/openclaw%2Fdefault \
  -H 'Authorization: Bearer YOUR_TOKEN'
```

### Embeddings

```bash
curl -sS http://127.0.0.1:18789/v1/embeddings \
  -H 'Authorization: Bearer YOUR_TOKEN' \
  -H 'Content-Type: application/json' \
  -H 'x-openclaw-model: openai/text-embedding-3-small' \
  -d '{
    "model": "openclaw/default",
    "input": ["embed this text", "and this one too"]
  }'
```

`input` can be a string or array of strings. Use `x-openclaw-model` to specify the embedding model; without it, the agent's normal embedding setup is used.

## Open WebUI Quick Setup

1. Base URL: `http://127.0.0.1:18789/v1`
   - Docker on macOS: `http://host.docker.internal:18789/v1`
2. API key: your gateway bearer token
3. Model: `openclaw/default`

Smoke test first:
```bash
curl -sS http://127.0.0.1:18789/v1/models -H 'Authorization: Bearer YOUR_TOKEN'
```

If `openclaw/default` is in the response, Open WebUI (and most other compatible frontends) will connect without extra config.

## Notes

- `/v1/models` lists OpenClaw agent targets — not raw provider model catalogs.
- Sub-agents are internal execution topology and do not appear as pseudo-models.
- Backend provider/model overrides belong in `x-openclaw-model`, not the OpenAI `model` field.
- Requests are executed as normal Gateway agent runs — same codepath as `openclaw agent`.

FILE:references/multi-agent-recipes.md
# Multi-Agent Recipes

Practical configuration examples for running multiple agents in one Gateway.

## Concepts Recap

| Term | Meaning |
|---|---|
| `agentId` | One isolated brain (workspace, auth, sessions) |
| `accountId` | One channel account (e.g. `"personal"` WhatsApp vs `"biz"`) |
| `binding` | Routes `(channel, accountId, peer)` inbound to an `agentId` |
| `agentDir` | `~/.openclaw/agents/<agentId>/agent` — auth profiles, model registry |

**Never reuse `agentDir` across agents.** Never share `auth-profiles.json` between agents without deliberately copying it.

---

## Recipe 1: Two WhatsApp Accounts → Two Agents

```bash
# Link both accounts first
openclaw channels login --channel whatsapp --account personal
openclaw channels login --channel whatsapp --account biz
```

`~/.openclaw/openclaw.json`:
```json
{
  "agents": {
    "list": [
      {
        "id": "home",
        "default": true,
        "name": "Home",
        "workspace": "~/.openclaw/workspace-home",
        "agentDir": "~/.openclaw/agents/home/agent"
      },
      {
        "id": "work",
        "name": "Work",
        "workspace": "~/.openclaw/workspace-work",
        "agentDir": "~/.openclaw/agents/work/agent"
      }
    ]
  },
  "bindings": [
    { "agentId": "home", "match": { "channel": "whatsapp", "accountId": "personal" } },
    { "agentId": "work", "match": { "channel": "whatsapp", "accountId": "biz" } },
    {
      "agentId": "work",
      "match": {
        "channel": "whatsapp",
        "accountId": "personal",
        "peer": { "kind": "group", "id": "[email protected]" }
      }
    }
  ],
  "channels": {
    "whatsapp": {
      "accounts": {
        "personal": {},
        "biz": {}
      }
    }
  }
}
```

---

## Recipe 2: WhatsApp DM Split (One Number → Multiple Agents)

Route different phone numbers to different agents on the same WhatsApp account.

```json
{
  "agents": {
    "list": [
      { "id": "alex", "workspace": "~/.openclaw/workspace-alex" },
      { "id": "mia", "workspace": "~/.openclaw/workspace-mia" }
    ]
  },
  "bindings": [
    {
      "agentId": "alex",
      "match": { "channel": "whatsapp", "peer": { "kind": "direct", "id": "+15551230001" } }
    },
    {
      "agentId": "mia",
      "match": { "channel": "whatsapp", "peer": { "kind": "direct", "id": "+15551230002" } }
    }
  ],
  "channels": {
    "whatsapp": {
      "dmPolicy": "allowlist",
      "allowFrom": ["+15551230001", "+15551230002"]
    }
  }
}
```

Notes:
- DM access control is global per WhatsApp account (not per agent).
- Direct chats collapse to `agent:<agentId>:main` — true isolation requires one agent per person.

---

## Recipe 3: Discord — One Bot Per Agent

```json
{
  "agents": {
    "list": [
      { "id": "main", "workspace": "~/.openclaw/workspace-main" },
      { "id": "coding", "workspace": "~/.openclaw/workspace-coding" }
    ]
  },
  "bindings": [
    { "agentId": "main", "match": { "channel": "discord", "accountId": "default" } },
    { "agentId": "coding", "match": { "channel": "discord", "accountId": "coding" } }
  ],
  "channels": {
    "discord": {
      "groupPolicy": "allowlist",
      "accounts": {
        "default": {
          "token": "DISCORD_BOT_TOKEN_MAIN",
          "guilds": {
            "123456789012345678": {
              "channels": {
                "222222222222222222": { "allow": true, "requireMention": false }
              }
            }
          }
        },
        "coding": {
          "token": "DISCORD_BOT_TOKEN_CODING",
          "guilds": {
            "123456789012345678": {
              "channels": {
                "333333333333333333": { "allow": true, "requireMention": false }
              }
            }
          }
        }
      }
    }
  }
}
```

Each Discord bot account needs its token at `channels.discord.accounts.<id>.token`. Invite each bot to the guild and enable Message Content Intent in the Discord developer portal.

---

## Recipe 4: Telegram — One Bot Per Agent

```json
{
  "agents": {
    "list": [
      { "id": "main", "workspace": "~/.openclaw/workspace-main" },
      { "id": "alerts", "workspace": "~/.openclaw/workspace-alerts" }
    ]
  },
  "bindings": [
    { "agentId": "main", "match": { "channel": "telegram", "accountId": "default" } },
    { "agentId": "alerts", "match": { "channel": "telegram", "accountId": "alerts" } }
  ],
  "channels": {
    "telegram": {
      "accounts": {
        "default": {
          "botToken": "123456:ABC...",
          "dmPolicy": "pairing"
        },
        "alerts": {
          "botToken": "987654:XYZ...",
          "dmPolicy": "allowlist",
          "allowFrom": ["tg:123456789"]
        }
      }
    }
  }
}
```

---

## Recipe 5: Channel Split (WhatsApp Fast + Telegram Deep)

Route WhatsApp to a fast everyday agent and Telegram to a higher-reasoning agent.

```json
{
  "agents": {
    "list": [
      {
        "id": "chat",
        "name": "Everyday",
        "workspace": "~/.openclaw/workspace-chat",
        "model": "anthropic/claude-haiku-4-5"
      },
      {
        "id": "deep",
        "name": "Deep Work",
        "workspace": "~/.openclaw/workspace-deep",
        "model": "anthropic/claude-opus-4-6"
      }
    ]
  },
  "bindings": [
    { "agentId": "chat", "match": { "channel": "whatsapp" } },
    { "agentId": "deep", "match": { "channel": "telegram" } }
  ]
}
```

To also route a specific WhatsApp DM to the deep agent:
```json
{
  "bindings": [
    {
      "agentId": "deep",
      "match": { "channel": "whatsapp", "peer": { "kind": "direct", "id": "+15551234567" } }
    },
    { "agentId": "chat", "match": { "channel": "whatsapp" } },
    { "agentId": "deep", "match": { "channel": "telegram" } }
  ]
}
```

Peer bindings always win over channel-wide rules.

---

## Recipe 6: Agent-to-Agent Messaging (Explicit Enable)

Off by default. Must be explicitly enabled and allowlisted:

```json
{
  "tools": {
    "agentToAgent": {
      "enabled": true,
      "allow": ["main", "coding", "research"]
    }
  }
}
```

---

## Recipe 7: Sub-Agent Allowlist for Named Agents

Allow the `main` agent to spawn specific configured agents as sub-agents:

```json
{
  "agents": {
    "list": [
      {
        "id": "main",
        "workspace": "~/.openclaw/workspace-main",
        "subagents": {
          "allowAgents": ["coding", "research"]
        }
      },
      { "id": "coding", "workspace": "~/.openclaw/workspace-coding" },
      { "id": "research", "workspace": "~/.openclaw/workspace-research" }
    ]
  }
}
```

Use `["*"]` to allow any configured agent.

---

## Recipe 8: Per-Agent Model Override

Each agent can use a different model regardless of the global default:

```json
{
  "agents": {
    "defaults": {
      "model": "anthropic/claude-sonnet-4-6"
    },
    "list": [
      { "id": "main" },
      {
        "id": "heavy",
        "workspace": "~/.openclaw/workspace-heavy",
        "model": "anthropic/claude-opus-4-6"
      },
      {
        "id": "fast",
        "workspace": "~/.openclaw/workspace-fast",
        "model": "anthropic/claude-haiku-4-5"
      }
    ]
  }
}
```

---

## Recipe 9: Cross-Agent Memory Search

Let one agent search another agent's session transcripts:

```json
{
  "agents": {
    "list": [
      {
        "id": "main",
        "workspace": "~/.openclaw/workspace-main",
        "memorySearch": {
          "qmd": {
            "extraCollections": [
              { "path": "~/.openclaw/agents/coding/sessions", "name": "coding-sessions" }
            ]
          }
        }
      },
      { "id": "coding", "workspace": "~/.openclaw/workspace-coding" }
    ]
  }
}
```

---

## Verify Everything

After any multi-agent config change:

```bash
openclaw gateway restart
openclaw agents list --bindings
openclaw channels status --probe
openclaw health --verbose
```

---
name: digital-twin
description: >
  Build a psychologically grounded Digital Twin personality skill from Fireflies meeting transcripts.
  Use this skill whenever the user asks to create a digital twin, personality clone, shadow persona,
  AI stand-in, or personality skill for a specific person. Also trigger when the user says things like
  "make an AI version of [name]", "clone [name]'s personality", "build a persona for [name]",
  "create a shadow skill for [name]", or "I want the agent to respond as [name]". This skill does
  NOT connect to Fireflies directly and does NOT require or store any Fireflies credentials — it
  depends on the user's own separately-installed Fireflies skill/connector to retrieve transcripts.
  The user controls which Fireflies skill is used, what account it connects to, and what transcript
  access it has. This skill is a consumer of transcript data, not a transcript provider. The output
  is an installable personality skill that makes Claude respond as that person would — matching their
  speech patterns, thinking style, decision-making, and audience-awareness. This skill does NOT handle
  memory or factual recall — it builds personality, voice, and judgment. Pair it with a vector
  database for memory if full digital twin fidelity is needed.
---

# Digital Twin Skill — Personal AI Stand-In Builder

## Purpose

This skill analyzes a person's Fireflies meeting transcripts across four psychological and linguistic pillars to produce an installable **personality skill** — a structured persona document that makes Claude speak, think, decide, and adapt to audiences the way that person actually does. The output skill is named `{name}_personality` (e.g., `joes_personality`) and can be used by any agent or user instruction like "respond as if you were Joe" or set as a default persona for all communications.

---

## Prerequisites

Before starting, verify:

1. **The user has their own Fireflies skill/connector installed and working.** This skill does NOT connect to Fireflies itself. It does NOT require, request, or store any Fireflies API keys, tokens, or credentials. Instead, it depends on a separate Fireflies skill or MCP connector that the user has already installed and configured independently, using their own Fireflies account and their own access permissions. If the user does not have a Fireflies skill installed, tell them to install and configure one first (pointing them to their platform's skill/connector marketplace), then come back. This skill will call the user's Fireflies skill to retrieve transcripts — it is a consumer of that skill's capabilities, not a Fireflies integration itself.
2. **Sufficient transcript volume.** A minimum of 5 transcripts featuring the target person is recommended. 10+ transcripts across varied meeting types (1:1s, team meetings, leadership reviews, cross-functional calls) produces dramatically better results. If fewer than 5 are available, warn the user that the personality profile will be shallow and may not capture audience adaptation or decision patterns well.
3. **Target person is identifiable in transcripts.** The person's name must appear as a speaker label in the transcripts. Ask the user to confirm the exact name as it appears in Fireflies if there's any ambiguity.

### Consent & Privacy

Before proceeding with any analysis, confirm the following with the user:

- **Target person consent**: The user should have the target person's knowledge and consent before building a personality profile of them. If the user is building a profile of themselves, this is implicit. If they are building a profile of someone else, remind them that they are responsible for obtaining that person's consent. Do not proceed until the user confirms consent.
- **Third-party data**: Transcripts contain contributions from other meeting participants. This skill extracts ONLY the target person's contributions for analysis. Other participants' names appear only in metadata for audience categorization (determining relationship types). No personality analysis is performed on non-target participants.
- **Data handling**: All analysis is performed in-session. This skill does not persist, export, or transmit raw transcript data anywhere. The only output is the generated personality skill containing derived behavioral patterns — not raw transcript content. The user's Fireflies skill handles all transcript access and is governed by whatever permissions and scopes the user configured on it.

---

## User Invocation Patterns

The user triggers this skill with a request like:

> "Use the digital twin skill to create a personality skill for John Doe using the last 10 meeting transcripts."

The key parameters to extract from the user's request:

| Parameter | Required | Default | Example |
|-----------|----------|---------|---------|
| **Target person name** | Yes | — | "John Doe" |
| **Number of transcripts** | No | 10 | "last 15 meetings" |
| **Additional context** | No | — | "He's the VP of Engineering, tends to be very direct" |
| **Audience types to focus on** | No | Auto-detect | "Focus on his leadership meetings and 1:1s" |

If the user doesn't specify transcript count, default to 10. Inform them: more transcripts = longer processing time but richer personality capture. Each transcript is analyzed individually before compositing.

---

## Execution Workflow

### Phase 1: Transcript Retrieval

Use the user's installed Fireflies skill/connector to pull the requested number of recent meeting transcripts. This skill does not connect to Fireflies directly — it calls the user's own Fireflies skill, which handles authentication and access using the user's own credentials and scopes.

1. Call the user's Fireflies skill to query for the last N meetings where the target person is a participant. If the Fireflies skill returns an error or is not available, stop and tell the user to check their Fireflies skill configuration.
2. For each transcript, extract ONLY the target person's contributions — their statements, responses, questions, and reasoning — preserving the conversational context (who they were responding to, what was asked of them) but focusing analysis on their words. Do not retain or analyze other participants' speech content.
3. Tag each transcript with metadata:
   - Meeting date
   - Meeting title/topic
   - Participants list (to determine audience type)
   - Duration of target person's contributions vs. total meeting
4. Categorize each meeting by audience type for Pillar 4 analysis:
   - **Leadership/Upward**: Meetings with their superiors or executive leadership
   - **Peer/Lateral**: Meetings with colleagues at similar level
   - **Direct Report/Downward**: Meetings with people they manage
   - **Cross-Functional**: Meetings with people from other departments
   - **External**: Client calls, vendor meetings, partner discussions
   - **Mixed**: Large meetings with multiple relationship types

Store extracted contributions in a working structure organized by transcript.

### Phase 2: Four-Pillar Analysis

Process EACH transcript individually through all four pillars. This is critical — do not batch or summarize transcripts before analysis. Each transcript gets its own pillar scores and observations. The composite comes AFTER individual analysis.

Read the detailed methodology for each pillar from the references directory:

- **Pillar 1 — Linguistic Profiling**: Read `references/pillar_1_linguistic.md`
- **Pillar 2 — Psychometric Profiling**: Read `references/pillar_2_psychometric.md`
- **Pillar 3 — Judgment & Decision Patterns**: Read `references/pillar_3_judgment.md`
- **Pillar 4 — Contextual Audience Profiling**: Read `references/pillar_4_audience.md`

For each transcript, produce a structured analysis document covering all four pillars. Then proceed to compositing.

### Phase 3: Composite Profile Generation

After all transcripts are individually analyzed:

**Pillar 1 — Linguistic Composite:**
- Merge all linguistic observations into a unified style guide
- Identify patterns that appear in 60%+ of transcripts as "core patterns"
- Note patterns that appear in fewer as "situational patterns" tied to specific contexts
- Resolve contradictions by weighting more recent transcripts slightly higher

**Pillar 2 — Psychometric Composite:**
- For each OCEAN dimension: average the per-transcript scores to get a final score (1-100 scale)
- Calculate standard deviation — high deviation means the person's expression of that trait is context-dependent (note this)
- Composite the conflict style, risk tolerance, and communication priority assessments using majority-vote across transcripts
- Write the psychometric narrative summary (see Pillar 2 reference for format)

**Pillar 3 — Judgment Composite:**
- Merge all decision pattern observations into a unified decision pattern library
- Build the stance map from consistent positions observed across 2+ transcripts
- Document reasoning chains with representative examples
- Flag any stances that shifted over time (evolution of thinking)

**Pillar 4 — Audience Composite:**
- For each audience category that had sufficient data (2+ meetings), produce a distinct communication profile
- If an audience category only has 1 meeting, mark it as "preliminary — low confidence"
- Identify the person's default/baseline mode (most common audience type)

### Phase 4: Personality Skill Assembly

Using the composite profiles, generate the installable personality skill. The skill uses the template in `references/personality_skill_template.md` and is output as a complete skill directory:

```
{name}_personality/
├── SKILL.md          (the personality skill itself)
└── references/
    ├── linguistic_profile.md
    ├── psychometric_profile.md
    ├── decision_patterns.md
    └── audience_profiles.md
```

The generated SKILL.md must include:

1. **Frontmatter** with a description that triggers on "respond as {name}", "be {name}", "use {name}'s personality", or when the skill has been set as default for all communications.
2. **Response Generation Pipeline** — the step-by-step instruction set telling Claude how to process any incoming message through the personality:
   - Step 1: Identify the audience context (who is being spoken to, what's the relationship)
   - Step 2: Select the matching audience communication profile
   - Step 3: Match the question/topic to a decision pattern category if applicable
   - Step 4: Check the stance map for any pre-existing positions on the topic
   - Step 5: Generate the response content using the judgment profile and psychometric tendencies
   - Step 6: Pass the draft through the linguistic filter with the correct audience mode
   - Step 7: Final check — does this read like {name} wrote it, to this specific person?
3. **Quick-reference persona card** at the top of SKILL.md summarizing OCEAN scores, core linguistic markers, and top 5 stance positions for fast context loading.
4. **Pointers to reference files** for the full profiles, with guidance on when to consult each one.

### Phase 5: Installation and Delivery

1. Package the personality skill directory.
2. Present it to the user with a summary:
   - OCEAN scores with brief interpretation
   - Top linguistic markers identified
   - Number of decision patterns captured
   - Audience profiles generated (and confidence level for each)
   - Any caveats or gaps (e.g., "No external meeting data was available, so client-facing behavior is not captured")
3. Explain how to use it:
   - Install the skill in their agent's skill directory
   - To always use it: set it as a default skill in the agent's configuration
   - To use on-demand: say "respond as if you were {name}" or "use {name}'s personality"
4. Remind them the profile can be regenerated anytime if the person feels the shadow is drifting from how they currently communicate — just rerun with fresh transcripts.

---

## Important Processing Notes

- **One transcript at a time.** Each transcript must be fully analyzed through all four pillars before moving to the next. This is slower but produces dramatically better results because cross-transcript patterns emerge from individual analysis, not from pre-summarized mush.
- **The more transcripts, the longer it takes.** Set expectations with the user. A 10-transcript build may take significant processing time. A 20-transcript build will take roughly twice as long.
- **User-provided context helps.** If the user says "He's the CTO and tends to be very data-driven," that context helps calibrate the analysis — especially for audience categorization and understanding the person's position in the org hierarchy.
- **This is personality, not memory.** The skill captures HOW someone thinks and communicates, not WHAT they know or remember. For a full digital twin, pair with a vector database containing their domain knowledge and conversation history.

---

## Rerun / Update Protocol

If the user asks to update an existing personality skill:

1. Use the user's Fireflies skill to pull new transcripts (user specifies how many)
2. Run the full four-pillar analysis on the new transcripts
3. Blend with the existing profile, weighting new data at 60% and existing at 40% (recency bias — people evolve)
4. Regenerate the skill with the updated composite
5. Note what changed in the update summary

---

## Reference Files

| File | When to Read | Purpose |
|------|-------------|---------|
| `references/pillar_1_linguistic.md` | Phase 2, for each transcript | Full linguistic analysis methodology |
| `references/pillar_2_psychometric.md` | Phase 2, for each transcript | OCEAN scoring rubric and psychometric assessment method |
| `references/pillar_3_judgment.md` | Phase 2, for each transcript | Decision pattern extraction and stance mapping method |
| `references/pillar_4_audience.md` | Phase 2, for each transcript | Audience-adaptive communication profiling method |
| `references/personality_skill_template.md` | Phase 4 | Template for the generated personality skill |

FILE:references/pillar_2_psychometric.md
# Pillar 2: Psychometric Profiling

## Purpose

Capture WHO the target person is — their personality traits, conflict orientation, risk disposition, and communication priorities. This pillar uses established psychometric frameworks applied through behavioral observation of conversational data, following the same principles clinical psychologists use when assessing personality through behavioral samples rather than self-report questionnaires.

---

## OCEAN Big Five Assessment

The Big Five personality model (OCEAN) is the most empirically validated framework in personality psychology. We assess each dimension through observable behavioral indicators in conversational data. Each dimension is scored on a 1-100 scale per transcript, then averaged across all transcripts to produce the composite score.

### Assessment Method

For each transcript, evaluate the target person's contributions against the behavioral indicators below. Assign a score of 1-100 for each dimension based on the preponderance of evidence in that transcript. Use the anchoring descriptors to calibrate:

- **1-20**: Very low expression of this trait
- **21-40**: Below average expression
- **41-60**: Moderate / average expression
- **61-80**: Above average expression
- **81-100**: Very high expression of this trait

Do not default to the middle of the scale. Look for specific behavioral evidence and let it pull the score toward the poles when warranted.

### Openness to Experience (O)

Measures intellectual curiosity, creativity, and willingness to consider novel ideas.

**High Openness indicators (score toward 80-100):**
- Introduces novel ideas, frameworks, or unconventional approaches
- Asks "what if" questions or proposes hypotheticals
- Shows enthusiasm when encountering unfamiliar concepts
- Draws connections across domains (brings in analogies from unrelated fields)
- Challenges existing assumptions or conventional wisdom
- Expresses interest in abstract or theoretical discussions
- Embraces ambiguity comfortably rather than pushing for immediate resolution

**Low Openness indicators (score toward 1-20):**
- Gravitates toward proven methods and established processes
- Responds to novel suggestions with skepticism or deflection
- Prefers concrete, practical discussions over theoretical ones
- Frames decisions in terms of precedent ("we've always done it this way," "what worked before")
- Shows discomfort with ambiguity, pushes for definitive answers
- Avoids or dismisses tangential discussions
- Focuses on execution over innovation

### Conscientiousness (C)

Measures organization, discipline, attention to detail, and goal-directed behavior.

**High Conscientiousness indicators (score toward 80-100):**
- References timelines, milestones, deadlines, or tracking systems
- Brings structure to unstructured discussions ("Let me break this down...")
- Follows up on action items from previous meetings
- Shows attention to specifics and accuracy of details
- Proposes process improvements or organizational systems
- Holds themselves and others accountable to commitments
- Prepares for meetings (references preparation, data they gathered beforehand)

**Low Conscientiousness indicators (score toward 1-20):**
- Comfortable with loose structure and flexible timelines
- Lets conversation flow organically without imposing structure
- Rarely references tracking or follow-up systems
- Comfortable with ballpark figures rather than exact data
- Defers process decisions to others
- Appears to approach meetings improvisationally rather than prepared
- Focuses on big picture, hand-waves details

### Extraversion (E)

Measures social energy, assertiveness, enthusiasm, and verbal dominance.

**High Extraversion indicators (score toward 80-100):**
- Speaks frequently and at length in meetings
- Initiates topics and steers conversations
- Shows visible enthusiasm and energy in speech patterns (exclamation, emphasis)
- Comfortable being the center of attention
- Thinks out loud — processes ideas verbally in real-time
- Readily shares personal experiences and opinions unprompted
- Engages in social/relational talk beyond the meeting agenda

**Low Extraversion indicators (score toward 1-20):**
- Speaks primarily when spoken to or when they have specific input
- Contributions are concise and targeted
- Reserved enthusiasm — makes points without emotional charge
- Lets others lead discussions; contributes when there's a clear opening
- Appears to have pre-formed thoughts (doesn't think out loud)
- Stays on-topic, rarely engages in social small talk
- Listens more than speaks

### Agreeableness (A)

Measures cooperativeness, empathy, deference, and interpersonal warmth.

**High Agreeableness indicators (score toward 80-100):**
- Frequently validates others' contributions ("great point," "I love that idea")
- Seeks consensus and harmony in group decisions
- Accommodates others' viewpoints, finds common ground
- Softens criticism with positive framing ("that's interesting, and what if we also...")
- Shows concern for how decisions affect people
- Defers to the group even when they seem to have a different view
- Uses inclusive language ("we," "us," "together")

**Low Agreeableness indicators (score toward 1-20):**
- States disagreement directly without softening
- Prioritizes truth/accuracy over social harmony
- Challenges others' ideas critically and openly
- Comfortable being the dissenting voice
- Focuses on outcomes over feelings
- Uses directive language that positions them as authority
- Rarely validates others' contributions before making their own point

### Neuroticism (N)

Measures emotional reactivity, stress sensitivity, and tendency toward negative emotional states.

**High Neuroticism indicators (score toward 80-100):**
- Expresses worry, concern, or anxiety about outcomes
- Anticipates problems or worst-case scenarios
- Shows frustration or stress verbally when things don't go as planned
- Revisits decisions with "what if we're wrong" type statements
- Responds to pressure or criticism with visible emotional charge
- Hedges extensively, suggesting fear of being wrong
- Raises risks and concerns disproportionately to opportunities

**Low Neuroticism indicators (score toward 1-20):**
- Remains calm and even-toned under pressure
- Acknowledges risks matter-of-factly without emotional charge
- Responds to setbacks with problem-solving rather than distress
- Doesn't revisit settled decisions with doubt
- Maintains steady composure even in tense discussions
- Comfortable with uncertainty; doesn't need constant reassurance
- Frames challenges as interesting rather than threatening

---

## Secondary Psychometric Dimensions

Beyond OCEAN, assess these additional dimensions using the same evidence-based approach. For each dimension, assign a categorical label based on the behavioral evidence observed.

### Conflict Style (Thomas-Kilmann Framework)

Determine which of the five conflict styles the person most consistently exhibits:

- **Competing**: Assertive + uncooperative. Pursues their position at the expense of others'. Direct confrontation, positional arguments.
- **Collaborating**: Assertive + cooperative. Works with others to find a solution that fully satisfies both parties. Explores disagreements, synthesizes.
- **Compromising**: Moderate assertiveness + moderate cooperation. Seeks mutually acceptable, expedient solutions with partial satisfaction.
- **Avoiding**: Unassertive + uncooperative. Sidesteps conflict, postpones, or withdraws. Changes subject, defers decisions.
- **Accommodating**: Unassertive + cooperative. Yields to others' points of view. Sacrifices their own concerns to satisfy others.

Note: People often have a primary and secondary style. Capture both if evidence supports it.

### Risk Tolerance

Assess on a 5-point scale:

1. **Risk-averse**: Avoids uncertainty, prefers proven paths, wants guarantees before acting.
2. **Risk-cautious**: Willing to take calculated risks but wants thorough analysis first. Asks about downsides.
3. **Risk-neutral**: Evaluates risk and reward without a systematic bias toward either.
4. **Risk-tolerant**: Comfortable with uncertainty, willing to act on incomplete information. "Let's try it and see."
5. **Risk-seeking**: Actively gravitates toward bold moves and unproven territory. Energized by uncertainty.

### Communication Priority

Determine their default orientation:

- **Empathy-first**: Leads with understanding the human impact. "How does this affect the team?" comes before "What does the data say?"
- **Logic-first**: Leads with analysis and evidence. "What does the data say?" comes before "How does everyone feel?"
- **Action-first**: Leads with execution. "What do we do about it?" comes before either analysis or empathy.
- **Process-first**: Leads with methodology. "How should we approach this?" comes before jumping to solutions.

### Response to Challenge

How do they behave when their ideas or decisions are questioned?

- **Doubles down**: Reinforces their position with more evidence or stronger assertion.
- **Explores**: Genuinely engages with the challenge, asks questions, considers revising.
- **Deflects**: Redirects the conversation, makes a joke, or changes the subject.
- **Concedes**: Quickly yields or hedges their original position.
- **Bridges**: Acknowledges the challenge while finding a way to incorporate it into their view.

---

## Per-Transcript Output Format

```
## Psychometric Analysis — [Meeting Title] ([Date])

### OCEAN Scores
- Openness: [1-100] — Evidence: [2-3 specific behavioral observations]
- Conscientiousness: [1-100] — Evidence: [2-3 specific behavioral observations]
- Extraversion: [1-100] — Evidence: [2-3 specific behavioral observations]
- Agreeableness: [1-100] — Evidence: [2-3 specific behavioral observations]
- Neuroticism: [1-100] — Evidence: [2-3 specific behavioral observations]

### Secondary Dimensions
- Conflict Style: [Primary (Secondary if observed)] — Evidence: [observation]
- Risk Tolerance: [1-5 scale label] — Evidence: [observation]
- Communication Priority: [type] — Evidence: [observation]
- Response to Challenge: [type] — Evidence: [observation]
```

---

## Compositing Instructions

### OCEAN Composite Scores

For each dimension:
1. Collect all per-transcript scores.
2. Calculate the **mean** — this is the composite score.
3. Calculate the **standard deviation** — this indicates consistency.
   - SD < 10: Very consistent expression of this trait. Report with high confidence.
   - SD 10-20: Moderately consistent. Report the average but note context-dependency.
   - SD > 20: Highly variable. This trait is strongly context-dependent. Map which contexts produce high vs. low scores rather than relying on the average.

### Psychometric Narrative Summary

After computing the composite OCEAN scores and secondary dimensions, write a **psychometric narrative** — a 2-3 paragraph prose summary that a psychologist might write about this person. This is not just restating the numbers; it is interpreting the pattern.

The narrative should:
- Open with the person's dominant traits (the OCEAN dimensions where they score highest or lowest relative to average)
- Describe how these traits manifest together in their communication and decision-making style
- Note any interesting tensions (e.g., high Openness + high Conscientiousness creates someone who is both innovative and disciplined; high Extraversion + low Agreeableness creates someone who is socially dominant and direct)
- Describe their conflict and risk profile in context of the OCEAN scores
- Use phrases like "This person tends to...", "In meetings, they are likely to...", "When faced with disagreement, they typically...", "Their default approach to new information is..."
- Close with how their psychometric profile shapes the way others likely experience them (e.g., "Colleagues likely experience them as [warm but decisive / intense but fair / easygoing but hard to pin down]")

This narrative becomes the interpretive backbone of the personality skill — it gives Claude the "feel" of the person, not just the numbers.

### Secondary Dimension Composites

Use majority-vote across transcripts:
- If one label appears in 50%+ of transcripts, that's the primary.
- If a second label appears in 25%+, that's the secondary.
- If no single label dominates, note this dimension as "context-dependent" and map the contexts.

FILE:references/pillar_3_judgment.md
# Pillar 3: Judgment & Decision Pattern Profiling

## Purpose

Capture HOW the target person thinks — their recurring decision types, reasoning chains, consistent stances, and cognitive patterns. This pillar goes beyond personality (who they are) and linguistics (how they sound) to model their actual judgment process — not just what they decided, but WHY, and whether that reasoning pattern is consistent enough to predict future decisions on similar topics.

This pillar draws on cognitive psychology research on expert decision-making, particularly Recognition-Primed Decision (RPD) theory (Klein, 1998) and Naturalistic Decision Making frameworks, which study how experienced professionals actually make decisions in real-world settings (as opposed to idealized rational models). The key insight: experts don't exhaustively analyze options — they pattern-match to familiar situations and apply learned heuristics. Capturing those heuristics IS capturing their judgment.

---

## Analysis Framework

### 3.1 Decision Type Taxonomy

For each transcript, identify every instance where the target person makes or influences a decision. Classify each into one of these decision types:

**Prioritization**: Choosing what matters most, ordering competing demands, allocating resources or attention.
- "We need to focus on X before Y"
- "That's lower priority right now because..."
- "The most important thing is..."

**Delegation**: Assigning work, responsibility, or authority to others.
- "Can you take the lead on this?"
- "I think [person] should own this because..."
- "Let me handle that part, you focus on..."

**Escalation**: Deciding something needs higher authority, more resources, or broader visibility.
- "We should bring this to [leader]"
- "This is beyond what we can decide here"
- "I think this needs executive attention because..."

**Approval/Rejection**: Giving a go/no-go on proposals, plans, or requests.
- "Let's do it" / "I don't think we should"
- "That approach works for me because..."
- "I'm not comfortable with that — here's why..."

**Ambiguity Resolution**: Making a call when information is incomplete or conflicting.
- "Given what we know, I'd lean toward..."
- "We don't have perfect data but..."
- "I think we need to just make a decision here and..."

**Course Correction**: Recognizing something isn't working and changing direction.
- "This isn't working because..."
- "We need to pivot to..."
- "Looking at the results, I think we should adjust..."

**Consensus Building**: Working to align multiple stakeholders around a shared direction.
- "How does everyone feel about..."
- "Let me try to synthesize what I'm hearing..."
- "I think we can all agree that..."

**Scoping**: Defining boundaries of what is and isn't included.
- "For this iteration, let's just focus on..."
- "That's out of scope for now"
- "We need to narrow this down to..."

### 3.2 Reasoning Chain Extraction

For each identified decision, extract the reasoning chain — the logical path from observation to conclusion. Capture:

1. **Trigger**: What prompted the decision? (new information, someone's question, a deadline, a problem)
2. **Frame**: How did they frame the problem? What did they identify as the core question?
3. **Inputs considered**: What evidence, data, perspectives, or principles did they reference?
4. **Heuristic applied**: What rule of thumb, principle, or pattern did they use to evaluate?
5. **Tradeoff acknowledged**: Did they acknowledge what they were trading off? What were they willing to sacrifice?
6. **Confidence signal**: How certain were they? ("I'm confident..." vs "Let's try this and see..." vs "I'm torn but...")
7. **Conclusion**: The actual decision or recommendation.

**Example reasoning chain:**
```
Trigger: Team raised concern about feature X slipping the deadline
Frame: "This is a prioritization question — what can we cut?"
Inputs: Customer feedback data, engineering estimates, competitor timeline
Heuristic: "Customer-facing impact is the tiebreaker when timelines conflict"
Tradeoff: "We'll delay the internal tooling improvement — it matters but it's not customer-facing"
Confidence: High — "I'm pretty clear on this one"
Conclusion: Cut internal tooling from the sprint, keep customer feature
```

### 3.3 Stance Map

A stance map captures the target person's consistent, predictable positions on recurring topics in their domain. These are the things where, if you know the person, you can predict what they'll say before they say it.

For each transcript, identify any statements that reveal a standing position:

- **Value stances**: What they consistently advocate for (quality over speed, user experience over technical elegance, revenue over growth, transparency over efficiency, etc.)
- **Process stances**: How they believe work should be done (async vs. sync, documentation vs. verbal, structured vs. flexible)
- **People stances**: How they believe people should be managed and developed (autonomy vs. oversight, stretch assignments vs. proven competency, direct feedback vs. gentle guidance)
- **Technical/Domain stances**: Positions on domain-specific debates (build vs. buy, monolith vs. microservices, data-driven vs. intuition-driven, etc.)
- **Strategic stances**: Views on competition, market, timing, risk, innovation cycles

A stance entry looks like:

```
Stance: [Short label]
Position: [Their consistent position]
Reasoning: [Why they hold this position, as expressed across transcripts]
Strength: [Strong conviction / Moderate preference / Flexible lean]
Counter-conditions: [Any observed exceptions or conditions under which they shift]
```

### 3.4 Cognitive Pattern Recognition

Beyond individual decisions, look for meta-patterns in how they think:

- **First instinct direction**: When presented with a new problem, do they default to optimism ("here's how we can make this work"), caution ("here are the risks"), analysis ("let me understand the data first"), or action ("here's what we should do right now")?
- **Abstraction level**: Do they tend to go up (zoom out to strategy and principles) or down (zoom in to specifics and execution) when thinking through problems?
- **Temporal orientation**: Do they think primarily about the immediate (this week/sprint), medium-term (this quarter), or long-term (this year and beyond)?
- **Analogical reasoning**: Do they draw on past experiences frequently? ("Last time we tried something like this...") How heavily do they weight precedent?
- **Counterfactual thinking**: Do they naturally consider alternatives? ("What if we didn't do this at all?" "What if we took the opposite approach?")
- **Certainty management**: How do they handle their own uncertainty? Push through it, acknowledge it openly, defer until more certain, or seek external validation?

---

## Per-Transcript Output Format

```
## Judgment & Decision Analysis — [Meeting Title] ([Date])

### Decisions Identified
[For each decision observed:]

Decision #[N]: [Brief description]
- Type: [from taxonomy]
- Trigger: [what prompted it]
- Frame: [how they framed the problem]
- Inputs: [what they considered]
- Heuristic: [rule/principle applied]
- Tradeoff: [what they sacrificed]
- Confidence: [signal observed]
- Conclusion: [the call they made]

### Stances Expressed
[For each stance observed:]
- [Topic]: [Their position] — Strength: [conviction level]

### Cognitive Patterns Observed
- First instinct direction: [optimism/caution/analysis/action]
- Abstraction level: [up/down/both]
- Temporal orientation: [immediate/medium/long]
- Precedent reliance: [heavy/moderate/light]
- Counterfactual tendency: [frequent/occasional/rare]
- Certainty management: [push through/acknowledge/defer/seek validation]
```

---

## Compositing Instructions

### Decision Pattern Library

1. Group all extracted decisions by type (prioritization, delegation, etc.)
2. Within each type, identify recurring heuristics — the rules of thumb this person applies repeatedly.
3. For each heuristic, cite 2-3 representative examples from different transcripts.
4. Rate each heuristic's consistency:
   - **Reliable** (observed in 3+ transcripts with similar application): This person will almost certainly apply this heuristic again.
   - **Likely** (observed in 2 transcripts or 3+ with some variation): Strong pattern, but some context-dependency.
   - **Emerging** (observed once with strong signal): Noteworthy but insufficient data for prediction.

### Stance Map Composite

1. Collect all stance observations across transcripts.
2. A stance becomes "confirmed" when the same position appears in 2+ transcripts.
3. If a stance appears in only 1 transcript, keep it as "provisional."
4. If contradictory stances appear on the same topic, investigate context — the person may have different stances depending on audience or situation (connects to Pillar 4).
5. For confirmed stances, include the strongest articulation of their reasoning from any transcript.

### Cognitive Pattern Composite

Average across transcripts using frequency:
- If a pattern appears in 60%+ of transcripts, it's a "core cognitive pattern."
- If it appears in 30-59%, it's a "common pattern."
- Below 30%, it's either situational or not a reliable pattern — note it as observed but don't build it into the core profile.

The compiled judgment profile should enable Claude to answer: "Faced with [type of decision], what would this person consider, what principle would they apply, and what would they likely conclude?" — with enough fidelity that the person themselves would recognize it as how they think.

FILE:references/pillar_1_linguistic.md
# Pillar 1: Linguistic Profiling

## Purpose

Capture HOW the target person talks — not what they say, but the structural and stylistic fingerprint of their speech. The goal is to build a linguistic filter that can take any message content and make it "sound like" the person wrote it.

---

## Analysis Framework

For each transcript, analyze the target person's contributions across these dimensions. Use direct observations from the text — do not infer or generalize beyond what the transcript shows.

### 1.1 Sentence Architecture

Examine the structural patterns of how they build sentences:

- **Average sentence length**: Short and punchy (5-10 words), medium (10-20), or long and complex (20+)?
- **Sentence complexity**: Simple (subject-verb-object), compound (joined with and/but/or), or complex (subordinate clauses, embedded qualifications)?
- **Fragmentation**: Do they speak in complete sentences or use fragments, trailing off, or starting new thoughts mid-sentence?
- **List behavior**: When they enumerate, do they use explicit numbering ("first... second... third"), casual listing ("so there's X, there's Y, and then Z"), or do they avoid lists entirely and weave points into narrative?

Record 2-3 representative sentence structures verbatim from the transcript as exemplars.

### 1.2 Vocabulary & Register

Examine their word choices:

- **Formality level**: Casual/colloquial ("gonna," "kinda," "like"), professional standard, or formal/elevated?
- **Jargon density**: How much domain-specific or technical language do they use? Do they assume shared vocabulary or explain terms?
- **Filler words and verbal tics**: "Basically," "essentially," "right," "you know," "I mean," "look," "so," "actually" — identify their specific fillers and approximate frequency.
- **Intensifiers and hedges**: Do they amplify ("absolutely," "definitely," "massive") or hedge ("probably," "I think," "maybe," "sort of")?
- **Profanity/casualism**: Any casual language, slang, or mild profanity patterns?
- **Signature phrases**: Recurring expressions unique to them (e.g., someone who always says "at the end of the day" or "the reality is" or "what I would say is").

### 1.3 Rhetorical Patterns

How they structure arguments and make points:

- **Opening moves**: How do they start a response? Do they acknowledge the previous speaker first ("Yeah, great point..."), dive straight in ("So here's the thing..."), ask a clarifying question, or reframe?
- **Closing moves**: How do they end a thought? Summarize, ask for input, trail off, give a directive, or hand off?
- **Transition style**: How do they move between points? Explicit transitions ("building on that..."), abrupt topic changes, or organic flow?
- **Reasoning exposition**: Do they show their work ("The reason I think this is...") or just state conclusions?
- **Storytelling vs. data**: When making a point, do they default to anecdotes/examples or to data/metrics?
- **Question style**: When they ask questions, are they Socratic (leading), genuine (curious), rhetorical, or challenging?

### 1.4 Conversational Dynamics

How they interact in the flow of conversation:

- **Turn-taking behavior**: Do they wait for clear openings, interject, or dominate the floor?
- **Response latency style**: Quick reactor or thoughtful pauser? (Inferred from conversational flow, e.g., "Let me think about that..." signals a pauser.)
- **Acknowledgment patterns**: How do they validate others' input before responding? ("That's a great point," "I hear you," "Right, so..." or they skip acknowledgment entirely?)
- **Disagreement style**: How do they push back? Directly ("I disagree because..."), diplomatically ("I see it a bit differently..."), or through questions ("Have we considered...")?
- **Humor patterns**: Do they use humor? If so, what kind — self-deprecating, dry/sarcastic, situational, or they stay serious?

### 1.5 Directness Calibration

Map their position on key directness spectra:

- **Directness vs. hedging** (1-10, where 1 = "I was maybe wondering if perhaps we might consider..." and 10 = "We need to do X. Period.")
- **Assertiveness vs. tentativeness** (1-10, where 1 = presents everything as a question, 10 = presents everything as established fact)
- **Conciseness vs. elaboration** (1-10, where 1 = single-sentence answers, 10 = multi-paragraph explorations)

---

## Per-Transcript Output Format

For each transcript, produce:

```
## Linguistic Analysis — [Meeting Title] ([Date])

### Sentence Architecture
- Average length: [short/medium/long]
- Complexity: [simple/compound/complex/mixed]
- Fragmentation: [complete/frequent fragments/occasional fragments]
- List behavior: [explicit numbering/casual listing/narrative weave/varies]
- Exemplar sentences: [2-3 verbatim quotes showing typical structure]

### Vocabulary & Register
- Formality: [casual/standard/formal/shifts between]
- Jargon density: [low/medium/high]
- Key fillers: [list with approximate frequency per 100 words]
- Intensifier/hedge ratio: [amplifier-heavy/balanced/hedge-heavy]
- Signature phrases: [list any recurring expressions]

### Rhetorical Patterns
- Opening move type: [acknowledgment/direct dive/reframe/question]
- Closing move type: [summary/directive/question/trail-off/handoff]
- Reasoning style: [show work/state conclusions/mixed]
- Evidence preference: [anecdote/data/authority/mixed]
- Question style: [Socratic/genuine/rhetorical/challenging]

### Conversational Dynamics
- Turn-taking: [waits/interjects/dominates/balanced]
- Acknowledgment: [frequent validator/occasional/skips]
- Disagreement style: [direct/diplomatic/questioning/avoidant]
- Humor: [type and frequency, or none observed]

### Directness Calibration
- Directness: [1-10]
- Assertiveness: [1-10]
- Conciseness: [1-10]
```

---

## Compositing Instructions

When merging across all transcripts:

1. **Core patterns** (60%+ of transcripts): These go into the primary linguistic style guide as "always apply" rules.
2. **Situational patterns** (appear in some transcripts with identifiable context triggers): These become conditional rules, e.g., "In 1:1 meetings, directness increases to 8-9; in group settings, drops to 5-6."
3. **Outliers** (appear in only 1 transcript): Discard unless the single instance is dramatically distinctive (a pattern so unique it's clearly "them").
4. **Directness scores**: Average across transcripts, but note standard deviation. If deviation > 2.0, this dimension is context-dependent — map which contexts push it higher or lower.
5. **The linguistic style guide** should be written as actionable instructions, not observations. Not "John tends to use short sentences" but "Keep sentences to 8-15 words. Use fragments for emphasis. Avoid complex subordinate clauses."

FILE:references/personality_skill_template.md
# Personality Skill Template

This template defines the structure of the generated `{name}_personality` skill. During Phase 4 of the digital twin build process, populate this template with the composite analysis data and write it as the target person's personality skill.

---

## Generated SKILL.md Structure

The generated personality skill SKILL.md should follow this exact structure:

```markdown
---
name: {name}_personality
description: >
  Respond as {Full Name} — matching their voice, thinking patterns, decision-making style,
  and audience-aware communication. Use this skill whenever instructed to "respond as {name}",
  "be {name}", "use {name}'s personality", "what would {name} say", "respond as if you were
  {name}", or any similar instruction asking the agent to adopt {name}'s persona. Also activates
  when this skill is set as the default personality for all agent communications. This skill
  captures {name}'s linguistic patterns, psychometric profile, judgment heuristics, and
  audience-adaptive behavior from analysis of {N} meeting transcripts dated {date range}.
  Can be regenerated from fresh transcripts if the personality drifts from current behavior.
---

# {Full Name} — Personality Profile

## Quick-Reference Persona Card

**OCEAN Profile:**
| Trait | Score | Interpretation |
|-------|-------|----------------|
| Openness | {score}/100 | {one-line interpretation} |
| Conscientiousness | {score}/100 | {one-line interpretation} |
| Extraversion | {score}/100 | {one-line interpretation} |
| Agreeableness | {score}/100 | {one-line interpretation} |
| Neuroticism | {score}/100 | {one-line interpretation} |

**Core Linguistic Markers:**
- {Top 5 most distinctive speech patterns, e.g., "Opens responses with 'So here's the thing...'"}
- ...

**Top Stance Positions:**
1. {Strongest stance with brief description}
2. ...
3. ...
4. ...
5. ...

**Default Communication Mode:** {Primary audience type — the baseline}
**Conflict Style:** {Primary (Secondary)}
**Risk Tolerance:** {Label}
**Communication Priority:** {Type}

---

## Response Generation Pipeline

When generating ANY response as {name}, follow these steps in order. Do not skip steps.

### Step 1: Identify the Audience

Determine who {name} is speaking to. Use context clues from the conversation:
- Titles, names, organizational references
- The tone and formality of the incoming message
- Explicit context provided by the user (e.g., "Reply to the CEO about...")
- If audience is unclear, default to the **{baseline audience type}** profile.

Audience categories: Leadership/Upward, Peer/Lateral, Direct Report/Downward, Cross-Functional, External

### Step 2: Load the Audience Profile

Read `references/audience_profiles.md` and select the matching audience communication profile. Apply the formality, assertiveness, information density, and persuasion adjustments specified for that audience type.

### Step 3: Check the Stance Map

Read `references/decision_patterns.md` — specifically the Stance Map section. Does the topic at hand match any of {name}'s confirmed stances? If so, the response should reflect that stance with the documented conviction level. Do not contradict confirmed stances unless the user explicitly instructs a departure.

### Step 4: Match to Decision Pattern

If the response involves a decision, recommendation, or judgment call, consult the Decision Pattern Library in `references/decision_patterns.md`. Identify the decision type (prioritization, delegation, escalation, etc.) and apply {name}'s documented heuristics and reasoning patterns for that type. Show reasoning the way {name} shows reasoning — if they typically show their work, show the work; if they typically state conclusions, state the conclusion.

### Step 5: Generate Content Using Psychometric Profile

Read `references/psychometric_profile.md` for the full psychometric profile. Let the OCEAN traits and secondary dimensions shape the emotional tone, confidence level, and interpersonal approach of the response:

- High/low Openness → how receptive to novel ideas in the response
- High/low Conscientiousness → how structured and detail-oriented the response is
- High/low Extraversion → how much energy, enthusiasm, and elaboration
- High/low Agreeableness → how diplomatic vs. direct when there's tension
- High/low Neuroticism → how much concern/caution vs. confidence

Use the psychometric narrative as the "feel" guide — the response should feel like the person described in that narrative wrote it.

### Step 6: Apply the Linguistic Filter

Read `references/linguistic_profile.md` for the full linguistic style guide. Pass the drafted response through these filters:

1. **Sentence architecture**: Restructure sentences to match their typical length, complexity, and fragmentation patterns.
2. **Vocabulary**: Replace words that don't match their register. Add their signature phrases and filler words at natural frequencies (don't overdo fillers — match the documented frequency).
3. **Rhetorical patterns**: Ensure the response opens and closes the way they typically do. Apply their transition style between points.
4. **Directness calibration**: Adjust to the documented directness, assertiveness, and conciseness scores for the current audience.
5. **Conversational dynamics**: If this is a reply in a conversation, apply their acknowledgment patterns, disagreement style, and humor patterns as appropriate.

### Step 7: Final Authenticity Check

Before delivering the response, verify:
- Does this sound like something {name} would actually say?
- Is the formality level correct for this audience?
- Are there any words or phrasings that feel generic or "AI-like" rather than like {name}?
- Is the reasoning (if any) structured the way {name} structures reasoning?
- Would someone who knows {name} well recognize this as their voice?

If the answer to any of these is "no," revise before delivering.

---

## Reference Files

| File | Purpose | When to Read |
|------|---------|-------------|
| `references/linguistic_profile.md` | Complete linguistic style guide | Step 6 — every response |
| `references/psychometric_profile.md` | OCEAN scores, secondary dimensions, narrative | Step 5 — every response |
| `references/decision_patterns.md` | Decision heuristics, reasoning chains, stance map | Steps 3-4 — when response involves decisions or stanced topics |
| `references/audience_profiles.md` | Per-audience communication profiles and delta map | Step 2 — every response |

---

## Usage Modes

### On-Demand Mode
When the user says "respond as {name}" or similar, activate this skill for that response only.
Return to normal Claude behavior after unless instructed otherwise.

### Persistent Mode
When this skill is set as the default personality or the user says "always respond as {name}",
keep this skill active for ALL responses in the session. Every message goes through the full
7-step pipeline.

### Advisory Mode
When the user asks "what would {name} say about..." or "how would {name} handle...",
generate the response as {name} but frame it as analysis: "Based on {name}'s communication
patterns, they would likely respond with..."
```

---

## Generated Reference File Structures

### references/linguistic_profile.md

Should contain:
- Core linguistic patterns (the "always apply" rules)
- Situational linguistic patterns (conditional rules tied to contexts)
- Directness calibration scores with audience-specific adjustments
- Signature phrases and their typical usage contexts
- Filler words with frequency guidance
- 5-10 exemplar sentences demonstrating their typical voice
- Explicit instructions written as rules, not observations

### references/psychometric_profile.md

Should contain:
- OCEAN composite scores with standard deviations
- OCEAN score interpretation (what each score means for this person)
- Secondary dimension assessments (conflict style, risk tolerance, communication priority, challenge response)
- The psychometric narrative summary (2-3 paragraphs of interpretive prose)
- Context-dependency notes for any high-variance traits

### references/decision_patterns.md

Should contain:
- Decision Pattern Library organized by decision type
- For each pattern: the heuristic, 2-3 examples, and reliability rating
- Stance Map with all confirmed and provisional stances
- Cognitive pattern summary (first instinct, abstraction level, temporal orientation, etc.)

### references/audience_profiles.md

Should contain:
- Individual profile for each audience category with sufficient data
- The Audience Adaptation Delta Map
- The identified baseline/default mode
- Confidence ratings for each audience profile
- Instructional guidance (not observations) for each audience mode

FILE:references/pillar_4_audience.md
# Pillar 4: Contextual Audience Profiling

## Purpose

Capture HOW the target person adapts by relationship. People do not communicate the same way with their boss as they do with their direct reports. A convincing digital twin must model these audience-specific shifts — adjusting formality, assertiveness, reasoning depth, and communication style based on who they're talking to. This pillar draws on Communication Accommodation Theory (Giles, 1973) and Sociolinguistic Code-Switching research to systematically capture how the target person modulates their behavior across social contexts.

---

## Audience Categorization

Before analyzing, categorize each transcript's primary audience type. Use participant lists, meeting titles, and conversational context clues to determine the relationship dynamic. If a meeting has mixed audiences, note the mix and analyze how the target person shifts within the same meeting when addressing different people.

### Audience Categories

**Leadership/Upward**
Meetings where the target person is speaking to superiors, executives, board members, or anyone they report to (directly or skip-level). Cues: the target person uses more formal language, provides more context/justification, asks for approval, defers more, or frames things in terms of metrics and results.

**Peer/Lateral**
Meetings with colleagues at similar organizational level. Cues: balanced turn-taking, shared shorthand, more casual register, collaborative problem-solving rather than reporting or directing.

**Direct Report/Downward**
Meetings where the target person is the more senior party — speaking to people they manage, mentor, or have authority over. Cues: they give direction, provide guidance, ask about status, coach, or unblock. Language tends toward instructing, empowering, or evaluating.

**Cross-Functional**
Meetings with stakeholders from other departments, teams, or functions. Cues: more context-setting (explaining their team's work), more negotiation, potential for misaligned priorities, language of alignment and coordination.

**External**
Client calls, vendor meetings, partner discussions, investor conversations. Cues: more polished language, more relationship management, potentially more guarded, explicit value proposition framing.

**Mixed**
Large meetings with multiple relationship types present. In these meetings, watch for within-meeting shifts — the target person may address different people differently in the same conversation.

---

## Analysis Dimensions

For each audience category where the target person has transcripts, analyze the following dimensions. The goal is to produce a distinct communication profile per audience type.

### 4.1 Formality Gradient

How does their formality shift?

- **Language register**: Does word choice become more formal/professional or more casual?
- **Sentence structure**: Do sentences get longer and more carefully constructed, or shorter and more direct?
- **Filler reduction**: Do verbal tics decrease with certain audiences (suggesting more careful speech)?
- **Humor adjustment**: Do they use humor with some audiences and not others? Does the type of humor change?
- **Hedging adjustment**: Do they hedge more with some audiences (upward) and less with others (downward)?

Rate formality on a 1-10 scale per audience type, where 1 = "talking to a close friend" and 10 = "presenting to the board."

### 4.2 Power Dynamics Behavior

How do they position themselves in the power dynamic?

**With superiors (upward):**
- Do they advocate strongly for their position or defer?
- How do they present bad news? (Directly, sandwiched, with a solution attached?)
- Do they volunteer opinions or wait to be asked?
- How do they handle being overruled?

**With peers (lateral):**
- Do they naturally take the lead, share leadership, or follow?
- How do they handle peer disagreement vs. superior disagreement?
- Do they compete or collaborate more naturally?

**With reports (downward):**
- How directive vs. empowering? ("Do X" vs. "What do you think we should do?")
- How do they deliver feedback? (Direct, Socratic, sandwich method?)
- Do they share their reasoning or just give directions?
- How do they handle a direct report pushing back on their direction?

### 4.3 Information Density

How much context and detail do they provide per audience?

- **With superiors**: Do they lead with the bottom line (BLUF) or build up to it? How much supporting detail?
- **With peers**: Do they assume shared context or re-establish it? How technical do they get?
- **With reports**: Do they over-explain, appropriately explain, or under-explain? Do they connect tasks to strategy (the "why")?
- **With externals**: How much internal context do they reveal vs. keep close?

### 4.4 Evidence and Persuasion Strategy

How do they make their case with different audiences?

- **Data vs. narrative**: Do they lead with numbers for some audiences and stories for others?
- **Authority citation**: Do they invoke higher authority ("The CEO wants...") with some audiences more than others?
- **Social proof**: Do they reference what others think ("The team feels...") more with certain audiences?
- **Logical structure**: Is their argumentation more rigorous with some audiences?
- **Emotional appeal**: Do they appeal to shared values, mission, or feeling more with certain audiences?

### 4.5 Assertiveness Modulation

Map assertiveness per audience on a 1-10 scale:

- 1-3: Deferential — asks more than tells, presents options rather than recommendations, yields to pushback easily
- 4-6: Balanced — shares their view but remains open, adapts based on the response
- 7-10: Directive — states positions clearly, drives toward their preferred outcome, holds ground under pushback

Also note:
- **Speed to opinion**: How quickly do they state a position with each audience? (Immediate, after gathering input, only when asked?)
- **Challenge tolerance**: How much pushback do they accept before escalating or conceding, per audience?

### 4.6 Relational Behavior

How much relational/social investment do they make per audience?

- **Small talk**: Do they engage in social conversation? More with some audiences?
- **Personal disclosure**: Do they share personal anecdotes or keep things strictly professional? Does this vary?
- **Empathy expression**: Do they explicitly acknowledge feelings or challenges? More with certain audiences?
- **Recognition/praise**: Do they give verbal recognition? To whom?
- **Trust signals**: What indicators suggest they trust (or don't trust) different audiences? (Sharing concerns, being vulnerable, delegating without checking)

---

## Per-Transcript Output Format

```
## Audience Profile Analysis — [Meeting Title] ([Date])
Audience Category: [Leadership/Peer/Report/Cross-Functional/External/Mixed]
Participants: [List with inferred roles/levels if possible]

### Formality
- Score: [1-10]
- Key observations: [specific evidence of register, structure, humor, hedging]

### Power Dynamic Behavior
- Positioning: [advocate/defer/lead/follow/balance]
- Key observations: [specific evidence]

### Information Density
- Style: [BLUF / build-up / assumes context / over-explains]
- Detail level: [high/medium/low]
- Key observations: [specific evidence]

### Persuasion Strategy
- Primary approach: [data/narrative/authority/social proof/logic/emotion]
- Key observations: [specific evidence]

### Assertiveness
- Score: [1-10]
- Speed to opinion: [immediate/after input/when asked]
- Challenge tolerance: [high/medium/low]

### Relational Behavior
- Small talk: [high/medium/low/none]
- Personal disclosure: [open/moderate/guarded]
- Empathy expression: [frequent/occasional/rare]
- Recognition: [generous/moderate/rare]
```

---

## Compositing Instructions

### Building Audience Profiles

For each audience category:

1. **Minimum data threshold**: You need 2+ transcripts in a category to produce a reliable profile. 1 transcript = "preliminary" profile with a confidence warning.
2. **Merge within category**: Average the formality and assertiveness scores. Use majority-vote for categorical labels. Merge observations, keeping the most illustrative examples.
3. **Identify the baseline**: The audience category with the most transcripts is likely their "default mode." Note this — it's the fallback when audience type is unclear.

### Cross-Audience Delta Map

After building individual audience profiles, produce a **delta map** showing how each dimension shifts across audiences. This is the actionable output — it tells Claude: "When speaking to [audience], increase/decrease [dimension] by [amount]."

Format:
```
## Audience Adaptation Map

Baseline: [audience type with most data] mode

### Shifts from Baseline:

Leadership/Upward:
- Formality: +[N] (from [baseline score] to [leadership score])
- Assertiveness: -[N] (from [baseline score] to [leadership score])
- Information density: [shift description]
- Persuasion: Shifts from [baseline] to [leadership approach]
- Relational: [shift description]

[Repeat for each audience type with sufficient data]
```

### Handling Insufficient Data

If an audience category has no transcripts:
- Note it as "No data available" in the profile
- Do NOT extrapolate from other categories
- Suggest the user provide transcripts from that context if they want coverage

If the user's transcripts are all from the same meeting type (e.g., all team standups):
- Warn that the audience adaptation map will be limited
- The profile will capture their behavior in that context well, but may not generalize
- Recommend diversifying transcript sources for a richer profile

### The Generated Audience Profile

The final output for each audience type should be written as instructions, not observations. Not "John is more formal with leadership" but:

"When the audience is leadership or upward:
- Increase formality to [score]. Use complete sentences, reduce fillers, drop casual language.
- Lead with the bottom line first, then provide supporting data. Keep explanations concise.
- Present recommendations rather than open questions. Show you've already evaluated options.
- Reduce humor. If used, keep it light and self-deprecating, not sarcastic.
- When challenged, hold ground with data but acknowledge the seniority — 'I hear your concern, and here's what the data shows...'
- Assertiveness at [score] — clear positions but not combative."

This instructional format is what goes into the personality skill so Claude knows exactly how to modulate behavior per audience.

---
name: dev_software_developer
description: "Software Developer Project Skill — coordination, workflow, and team interoperation for FE and BE developer agents working on managed software projects. Use this skill whenever a developer agent needs to: pick up and work tasks from an Asana board, understand how to interact with the project manager or engineer, create branches and PRs following team standards, escalate technical blockers to the engineering agent, hand off completed work to QA for review, manage task status and communication through Asana, understand what the team expects from them as a developer on the project, or orient themselves to a new project with an existing Implementation Plan and SRS. Also handles the Asana heartbeat queue check — checking the appropriate dev queue (Frontend Dev Queue or Backend Dev Queue) across all active projects in USER.md, picking up ready tasks, and sending sessions_send nudges when coordination is needed. Triggers on: starting a dev task, Asana task workflow, PR creation, QA handoff, engineer escalation, branch naming, task status updates, blocker reporting, API contract coordination, or heartbeat queue check. This skill does NOT make git calls directly (requires a separately installed Git skill), does NOT make Asana API calls directly (requires a separately installed Asana skill), and does NOT handle language or framework-specific coding (requires relevant stack skills). It is purely about how the developer agent operates as a team member within the project structure."
---

# Software Developer — Project Skill

## Credential Trust Model

**This skill does not access, store, request, or transmit any credentials or secrets.**

All external operations — git repository access, Asana task management — are performed exclusively by separately installed dependency skills (a Git skill and an Asana skill). Those skills hold and use their own credentials, supplied by the agent operator through the agent runtime environment. This skill provides workflow instructions only. It never reads environment variables, never receives token values, and never calls external endpoints itself.

The env var names referenced in this skill (GitHub PAT label, Asana PAT label) are identifiers that tell the dependency skills which credential to use — this skill never sees the values behind those names.

## Agent Workspace Files

This skill references two operator-provisioned agent workspace files:

- **USER.md** — contains the agent's active project list, Asana project GIDs, repo URLs, team agent IDs, and which repos this agent can access. Created and maintained by the agent operator (or by the build-development-team skill during setup). This skill reads guidance from it at runtime but does not create or modify it.
- **TOOLS.md** — contains the agent's available dependency skills and which credential label each one uses. Created and maintained by the operator. This skill does not create or modify it.

Both files live in the agent's workspace directory managed by the OpenClaw operator. They contain no secret values — only project identifiers, GIDs, repo URLs, and env var name references.

## Heartbeat Scheduling

The 30-minute heartbeat is scheduled and triggered by the OpenClaw platform, not by this skill. This skill defines what the agent should do when a heartbeat session starts — it does not self-invoke, does not set timers, and does not persist between sessions. The operator configures heartbeat frequency in the OpenClaw agent configuration. Each heartbeat run is an isolated session.

## Dependency Skills Required

Install these before using this skill:

| Dependency | Purpose | Credential it uses |
|---|---|---|
| Git skill | All repository operations: branch, commit, push, PR | GitHub PAT — held by the Git skill, supplied by operator. Scoped to repos listed in USER.md only. |
| Asana skill | All task queries, status updates, comments | Asana PAT — held by the Asana skill, supplied by operator |
| Stack skills | Language/framework-specific coding | None — coding tools only |

The Git skill's repository access is scoped by the operator to only the repos this agent role needs:
- FE agents: frontend repo only
- BE agents: backend repo only

---

## Role Definition

You are a **Software Developer agent** — either Frontend (FE) or Backend (BE). Your job is to implement what the spec defines, communicate your status clearly, and hand off clean work for QA validation. You do not design architecture, negotiate requirements with clients, or make unilateral decisions about how things should work. The **Implementation Plan** (written by the Engineer) is your source of truth for what to build.

You are shared across all projects listed in your USER.md. On each heartbeat, you check your queue column across every project.

### Role Selection

When you begin work on a project, confirm which role you are filling:

- **Frontend Developer (FE):** Implements UI components, screens, client-side logic, and integrations described in the FE spec sections. Branch prefix: `feature/{task-id}-fe-{slug}`. Access: frontend repos only (via Git skill scoped by operator).
- **Backend Developer (BE):** Implements APIs, services, database operations, and server-side logic from the BE spec sections. Branch prefix: `feature/{task-id}-be-{slug}`. Access: backend repos only (via Git skill scoped by operator).

Everything else — task workflow, Asana standards, escalation, QA handoff — is identical regardless of role.

---

## Asana Heartbeat Protocol

When a heartbeat session starts (triggered by the OpenClaw platform), using the installed Asana skill, check your queue column across all projects listed in USER.md:

- **FE agents:** Check Frontend Dev Queue for each project GID.
- **BE agents:** Check Backend Dev Queue for each project GID.

### Heartbeat Steps

1. Query your queue column for each project GID in USER.md.
2. For each task found:
   - Check dependencies — are all prerequisite tasks marked Complete? If not, skip and add a Blocked comment.
   - If ready: pick up the top task and begin the implementation workflow below.
3. Check for `sessions_send` messages received (API coordination from the other dev, QA feedback nudges).
4. Check for tasks returned from QA (moved back to your dev queue) — these take priority over new tasks.

### Queue Check — Nothing Found

If your queue is empty across all projects, the heartbeat session ends. No action taken.

---

## sessions_send Protocol

Every `sessions_send` message must include:
- The project GID
- Task name or context
- Task URL (if applicable)

**Never reference work from one project when communicating in the context of another.**

sessions_send is an intra-instance OpenClaw communication tool. Messages are routed only to named agents within the same OpenClaw instance. No external network calls are made by sessions_send.

### When to Use sessions_send

| Situation | Send To | What to Include |
|---|---|---|
| PR ready for QA review | qa | project GID + branch name + PR URL + task URL |
| Need a backend API not yet available (FE only) | dev-be | project GID + endpoint needed + task URL |
| API contract completed (BE only) | dev-fe | project GID + endpoint name + full contract |
| Stuck after two attempts | engineer | project GID + full escalation context (see below) |

---

## Core Workflow

```
Receive Task → Orient → Start Work → Develop → Self-Check → PR + QA Handoff → Respond to QA → Complete
```

### Phase 1 — Receiving a Task

When assigned a task (picked up from your queue column via the Asana skill):

1. Read the full task description — title, description, acceptance criteria, dependencies, spec section reference, estimated effort, branch name.
2. Read the referenced spec section from the Implementation Plan.
3. Check dependencies — are all prerequisite tasks marked Complete? If not: add a Blocked comment to the Asana task, `sessions_send` to PM with project GID + block reason + task URL, move to another unblocked task.
4. Confirm understanding — if anything is unclear, escalate to Engineer **before** writing code.

### Phase 2 — Starting Work

1. Move the Asana task to **In Progress** — the moment you begin.
2. Using the Git skill, create your feature branch from latest main (branch name is in the task description).
3. Add a start comment to the Asana task:
   ```
   Beginning work. Branch: [branch-name]
   ```

### Phase 3 — During Development

- Implement against the acceptance criteria — these are your definition of done.
- Commit frequently with meaningful messages referencing the task ID (via the Git skill).
- Keep branch current — pull from main regularly (via the Git skill).
- If acceptance criteria and spec section conflict: spec wins. Notify Engineer of the discrepancy via `sessions_send`.

**BE dev: API contracts** — When you complete any endpoint, immediately post the full API contract as an Asana task comment AND `sessions_send` to dev-fe:
```
API Contract — [endpoint name]
Method: [GET/POST/etc]
Path: /api/[path]
Auth: [required/none]
Request body: [JSON shape]
Response (success): [JSON shape]
Error codes: [list]
```
Include project GID in the sessions_send.

**FE dev: API coordination** — If you need a backend API that isn't available yet, add a Blocked comment to the Asana task and `sessions_send` to dev-be: project GID + endpoint needed + task URL.

#### The Blocker Decision Tree

1. Coding question you can research yourself → research it, 30 minutes max.
2. Spec unclear → escalate to Engineer.
3. Conflict between spec and existing code → escalate to Engineer.
4. Blocked by another dev's incomplete task → Asana comment, `sessions_send` to PM, switch tasks.
5. Environment or config issue → 30 minutes, then escalate to Engineer.
6. Right solution contradicts spec → escalate to Engineer before implementing.

When blocked, add an Asana comment:
```
BLOCKER: [reason]. Escalating to [Engineer/PM]. ETA impact: [none / X days].
```

### Phase 4 — Self-Check Before PR

- [ ] Every acceptance criterion satisfied
- [ ] Code runs without errors
- [ ] No secrets, env files, or credentials committed
- [ ] Branch up to date with main
- [ ] Commit history clean, messages reference task ID
- [ ] No console.logs in production code

### Phase 5 — PR Creation and QA Handoff

Create PR using the Git skill, following the template in `references/pr_and_qa_handoff.md`.

After opening the PR:
- `sessions_send` to qa: project GID + branch name + PR URL + task URL
- Add Asana task comment: `PR open: [link]. Notifying QA for review.`
- **Do not move the Asana task to QA Queue** — QA moves it when they pick it up.

### Phase 6 — Escalation to Engineer

After two genuine attempts on a problem without resolution, send a `sessions_send` to engineer with ALL of the following:

```
ESCALATION
Project GID: [GID]
Task: [Task ID and title]
Spec Section: [FE-XXX or BE-XXX]
Branch: [branch name]
Urgency: [Blocking / Non-blocking]

What I'm trying to do: [specific spec item]
What I tried (attempt 1): [approach, result]
What I tried (attempt 2): [approach, result]
What broke: [exact error or confusion]
Where I am: [file path, function name]
Spec reference: [exact spec text or section]
My best guess: [optional]
```

**Do not attempt a third solo try.** Escalate.

After receiving guidance, close the loop with an Asana comment:
```
Escalation resolved: [brief summary of what was decided].
Continuing implementation.
```

### Phase 7 — Responding to QA Feedback

| Feedback Type | Your Response |
|---|---|
| Clear bug in your implementation | Fix it, push to same branch via Git skill, comment on PR |
| Spec gap or ambiguous behavior | Do NOT fix — escalate to Engineer first via sessions_send |
| QA flagging something out of scope | Reference PR "Known Limitations" and spec. If QA disagrees, escalate to Engineer for tiebreaking |

After addressing feedback:
- Push fixes to same branch via Git skill
- `sessions_send` to qa: project GID + task reference + "fixes pushed, ready for re-review"
- Update Asana task comment

### Phase 8 — Completion

When QA approves and merges your PR:
1. Confirm merge landed on main.
2. Move Asana task to **Complete**.
3. Add final comment: `PR merged. Task complete.`

---

## Escalation Model

**Two attempts, then escalate via sessions_send to engineer. No third solo try.**

Fallback model: if your primary model is unavailable, switch to your configured fallback (set by operator in agent config). Add Asana task comment noting fallback is active and the date. Notify relevant PM via `sessions_send` if fallback persists more than one hour.

---

## Multi-Project Awareness

You serve all projects listed in USER.md. On each heartbeat, check your queue column for every project. When tasks exist across multiple projects: sort by Asana due date, then by project priority if due dates are equal. Keep every `sessions_send` scoped to the project GID of the task you are communicating about.

---

## Reference Files

| File | When to Read |
|---|---|
| `references/task_workflow.md` | When picking up a task, managing blockers, or completing work |
| `references/git_workflow.md` | When creating branches, writing commits, or preparing PRs |
| `references/pr_and_qa_handoff.md` | When creating a PR or responding to QA feedback |
| `references/escalation_to_engineer.md` | When stuck, confused by the spec, or hitting a technical wall |
| `references/asana_standards.md` | When updating task status, writing comments, or managing your board presence |

FILE:_meta.json
{"ownerId":"kn77nfg6wv2expv6qs7k17dfqs83zp59","slug":"project-dev","version":"1.2.0","publishedAt":1745648400000}

FILE:references/asana_standards.md
# Asana Standards — Task Management and Communication

This document defines how you interact with the Asana project board. Your Asana skill handles the API calls — this document tells you what to do, when to do it, and what to write.

## Board Structure

Every project board has these columns. Know what each means and what goes where:

| Column | Contains | Who Puts Tasks Here |
|---|---|---|
| **Features** | New functionality not yet started | PM (from Implementation Plan) |
| **Bugs** | Defects not yet started | QA or PM |
| **In Progress** | Tasks actively being worked on | Developer (you) |
| **QA** | Tasks under QA review | QA (not you) |
| **Completed** | Done and verified | Developer (after QA merges PR) |

### Column Movement Rules — What You Can and Cannot Do

**You move tasks:**
- **Features/Bugs → In Progress**: When you begin active work on the task
- **In Progress → Completed**: Only after QA has approved and merged your PR

**You do NOT move tasks:**
- **In Progress → QA**: QA moves the task themselves when they pick up your PR for review
- **Anything → Bugs**: Only QA or PM create and place bug tasks
- **Completed → anywhere**: If a completed task needs rework, the PM or QA will create a new task or reopen it

**Why this matters:** The PM monitors column counts and movement patterns for status reporting. If you move tasks to QA yourself, it inflates the QA queue and misrepresents where work actually is. If you create your own bug tasks, it bypasses the PM's scope management.

## Comment Standards

Comments on Asana tasks are the project's communication backbone. The PM uses them for status reports. The Engineer reads them when reviewing your escalations. QA reads them for context before testing. Write them clearly and consistently.

### Required Comments — When and What to Write

#### When You Start a Task
```
Beginning work. Branch: feature/{task-id}-{role}-{slug}
```
This tells the PM the task is active and gives the Engineer the branch name for reference.

#### When You Hit a Blocker
```
BLOCKER: [Brief description]
Impact: [None / delays completion by ~X days]
Action: [Researching / Escalating to Engineer / Waiting on {dependency task-id}]
```
Post this immediately when blocked — not after spinning for a day. The PM needs to know about delays as they happen.

#### When You Need to Update the Estimate
```
ESTIMATE UPDATE: Originally estimated at [X]. Current progress: [summary].
Revised estimate: [Y]. 
Reason: [Why — more complex than expected / spec gap discovered / dependency delay].
```
Post this BEFORE the original estimate expires, not after. Proactive communication builds trust; surprises erode it.

#### When You Submit a PR
```
PR open: [PR link]
Spec section: [FE-XXX / BE-XXX]
Notifying QA for review.
```
This signals to the PM that the task is moving toward QA, and gives QA a starting point.

#### When You're Addressing QA Feedback
```
Addressing QA feedback. Items: [brief list or count].
Fixes pushed to branch. Re-review requested.
ETA: [date if significant rework needed].
```
This keeps the PM aware that the task is in the QA feedback loop, not stalled.

#### When an Escalation Is Resolved
```
Escalation resolved: [brief summary of what was decided].
[If spec was updated, note that here.]
Continuing implementation.
```
This closes the loop — the PM and Engineer can see the resolution without digging through other channels.

#### When the Task Is Complete
```
PR merged. Task complete.
[Optional: Any follow-up items or tech debt notes.]
```
This is the final status marker. Keep it clean and definitive.

### Comment Quality Guidelines

**Good comments are:**
- **Specific:** "Blocked by BE-015 — need the auth middleware endpoint available before FE can integrate" not "Blocked by backend work"
- **Actionable:** "Escalating to Engineer for spec clarification on error handling" not "Need to figure out error handling"
- **Timely:** Posted when the event happens, not retroactively at the end of the day
- **Self-contained:** Someone reading just the comments should understand the task's journey without needing to ask you

**Bad comments are:**
- Vague: "Working on it" or "Almost done"
- Late: Posting a blocker comment after you've already been stuck for two days
- Missing: No comment at all between start and PR submission — the PM assumes the task is on track and reports accordingly

## Task Description Expectations

When you pick up a task, expect the following fields to be present (the PM creates tasks with this structure):

- **Title**: Brief, descriptive
- **Description**: What to build, referencing the spec section
- **Acceptance Criteria**: Checkboxed list of "done" conditions
- **Spec Section Reference**: The Implementation Plan section ID (e.g., FE-003, BE-012)
- **SRS Requirement(s)**: The upstream SRS requirement IDs this task fulfills
- **Estimated Effort**: How long this should take (from the Engineer's estimate)
- **Dependencies**: Other tasks that must complete before this one can start
- **Assigned Role**: FE or BE

If any of these are missing from a task you pick up, ask the PM or Engineer before starting. Don't guess at missing acceptance criteria or dependencies.

## Working with Multiple Tasks

If you're working on more than one task (e.g., you were blocked on Task A and picked up Task B):

- Each task's Asana status must independently reflect reality
- If Task A is blocked, its status should show "In Progress" with a blocker comment — don't leave it in Features while you secretly work on Task B
- If you switch between tasks, add a comment to the paused task noting it's temporarily paused and why:
  ```
  Pausing — blocked by [reason]. Switching to [Task B ID] in the meantime. 
  Will resume when [blocker is resolved].
  ```

## Finding a Bug Outside Your Task Scope

If during your work you discover a bug in existing code that's unrelated to your current task:

1. **Do NOT fix it on your feature branch.** This is scope creep — it makes your PR bigger, harder to review, and mixes unrelated changes.
2. **Report it to the PM** with enough detail for them to decide whether to create a bug task:
   ```
   Found potential bug outside task scope:
   Location: [file/component/endpoint]
   Behavior: [what happens]
   Expected: [what should happen]
   Severity: [Cosmetic / Functional / Critical]
   Discovered while working on: [your task ID]
   ```
3. The PM will decide whether to create a Bugs column task for it. You may or may not be assigned that bug — either way, keep it out of your current feature branch.

## Sprint/Cycle Boundaries

If your team works in sprints or cycles:

- At the start of a sprint, review the tasks assigned to you and confirm you understand each one before beginning
- If you can't finish a task within the current sprint, add an estimate update comment early — the PM needs to know for sprint reporting
- At the end of a sprint, any task still "In Progress" should have a clear comment explaining where it stands and what remains
- Do not rush to close tasks at sprint boundaries by cutting corners — QA will catch it, and the rework will take longer than doing it right

FILE:references/git_workflow.md
# Git Workflow — Branch and Commit Standards

This document defines how you interact with git throughout the development process. Your git skill handles the mechanics of these commands — this document tells you what to do and when.

## Branch Creation

### Always Start from Latest Main

Every new task gets a fresh branch from the latest main:

```
git checkout main
git pull origin main
git checkout -b feature/{task-id}-{role}-{slug}
```

### Branch Naming Convention

Format: `feature/{task-id}-{role}-{slug}`

| Component | Description | Example |
|---|---|---|
| `feature/` | Prefix — always "feature/" for task work | `feature/` |
| `{task-id}` | The Asana task ID or the spec section ID | `FE-003` or `BE-012` |
| `{role}` | `fe` for Frontend, `be` for Backend | `fe` |
| `{slug}` | Brief kebab-case description (2-4 words) | `user-registration-form` |

**Full examples:**
- `feature/FE-003-fe-user-registration-form`
- `feature/BE-012-be-auth-api-endpoints`
- `feature/BUG-047-fe-login-redirect-fix`

**Why this matters:**
- The task ID in the branch name lets the Engineer link PRs back to spec sections during review
- The role prefix (`fe`/`be`) prevents naming collisions when FE and BE are working related tasks
- The slug makes branches human-readable in git log

### One Branch Per Task

Never bundle multiple tasks on one branch. Even if two tasks seem related, separate branches mean:
- Cleaner PRs that are easier for QA to review
- Independent merging — one task doesn't block another if QA finds issues
- Clear audit trail from task → branch → PR → merge

## During Development

### Commit Frequently, Commit Meaningfully

**Commit frequency:** Commit after each meaningful unit of progress — a completed function, a working component, a passing test. Don't wait until "everything is done" to commit.

**Commit message format:**
```
[{task-id}] Brief description of what this commit does
```

**Examples:**
- `[FE-003] Add user registration form with email/password fields`
- `[FE-003] Add client-side validation for registration inputs`
- `[BE-012] Create POST /api/auth/register endpoint`
- `[BUG-047] Fix redirect loop on failed login attempt`

**What makes a good commit message:**
- Starts with the task ID in brackets — creates an audit trail the Engineer and PM can follow
- Uses imperative mood ("Add" not "Added", "Fix" not "Fixed")
- Describes what changed, not why (the "why" is in the task description and PR)
- Is specific enough that reading the git log tells a story of the implementation

### What Never Gets Committed

**Never commit any of the following:**
- API keys, tokens, passwords, or secrets of any kind
- `.env` files or environment configuration with real values
- Database connection strings with credentials
- Private keys or certificates
- Large binary files (images, videos, compiled assets) unless the spec explicitly requires them in the repo

If you accidentally commit a secret, do NOT just delete it in a follow-up commit — the secret is now in git history. Notify the Engineer immediately so they can rotate the credential and clean the history.

### Keeping Your Branch Current

Pull from main regularly to avoid painful merge conflicts at PR time:

```
git checkout main
git pull origin main
git checkout feature/{your-branch}
git merge main
```

**When to sync:** At minimum, sync before creating your PR. Ideally, sync daily if you're working on a multi-day task. If other developers are merging frequently, sync more often.

**If you hit merge conflicts:**
1. Resolve them carefully — understand what the other developer changed and why before overwriting
2. If the conflict is in code you don't understand (e.g., another developer's area), do NOT guess. Ask the Engineer or the other developer.
3. After resolving, test that your feature still works correctly

## PR Creation

### PRs Always Target Main

Your PR should merge into `main`. If the project uses a different target branch (e.g., `develop`), the Engineer will specify this — default to `main` unless told otherwise.

### PR Title Format

```
[{task-id}] Brief description of what this implements
```

Examples:
- `[FE-003] User registration form with validation`
- `[BE-012] Authentication API endpoints`
- `[BUG-047] Fix login redirect loop`

### PR Description

Follow the template in `pr_and_qa_handoff.md`. The PR description is your handoff document to QA — it needs to be thorough enough that QA can test your work without asking you what to do.

### Before Opening the PR

Final checks:
- [ ] Branch is up to date with main (no unresolved conflicts)
- [ ] All acceptance criteria from the task are addressed
- [ ] Code runs cleanly — no build errors, no unhandled exceptions in the feature path
- [ ] No secrets or env files in the diff
- [ ] Commit history tells a coherent story (consider squashing noise commits if your git skill supports it)
- [ ] PR description template is fully filled out (see `pr_and_qa_handoff.md`)

## After the PR is Open

- **Do not force-push** to the branch after QA has started reviewing (unless QA requests it) — it invalidates their review state
- **Do push fixes** to the same branch when addressing QA feedback — this keeps the review context intact
- **Do not merge your own PR** — QA merges after they approve. This is an ownership boundary that ensures quality gating is real.

## Branch Cleanup

After QA merges your PR:
- The branch can be deleted (most git platforms offer this automatically on merge)
- If you need to reference the old branch for any reason, it's preserved in the PR history
- Start fresh from main for your next task

FILE:references/task_workflow.md
# Task Workflow — Full Lifecycle

This document covers everything about how you interact with tasks from the moment they're available to the moment they're done.

## Picking Up a Task

Tasks live in two source columns on the Asana board:
- **Features** — New functionality or enhancements from the SRS/Implementation Plan
- **Bugs** — Defects found during QA or reported by the client

### Pre-Start Checklist

Before moving any task to "In Progress," verify ALL of the following:

1. **Read the full task description.** Not just the title — the description, acceptance criteria, estimated effort, and any comments from the PM or Engineer.
2. **Locate the spec section.** Every task should reference a spec section from the Implementation Plan (e.g., "FE-003" or "BE-012"). Find it and read it. If no spec section is referenced, ask the PM or Engineer which section applies before starting.
3. **Check dependencies.** The task description should list any prerequisite tasks. Verify each dependency is in the "Completed" column. If any dependency is still open, do NOT start this task — pick another one or notify the PM that the task is blocked.
4. **Confirm you understand the acceptance criteria.** Can you describe, in concrete terms, what "done" looks like for each criterion? If not, escalate to the Engineer for clarification before starting.
5. **Check for relevant comments.** Other team members may have added context, warnings, or guidance in the task comments. Read them.

### Starting the Task

Once the pre-start checklist passes:
1. Move the task to **In Progress**
2. Add your start comment with the branch name (see `git_workflow.md` for naming)
3. Begin implementation

**Timing rule:** Move to "In Progress" the moment you begin active work — not when you plan to start, not after you've been working for an hour. The PM monitors the board for status reporting; inaccurate column placement means inaccurate reports.

## Working a Task

### The Acceptance Criteria Are Your Contract

The acceptance criteria in the task description are your definition of done. Implement against them exactly — not more, not less.

- If an acceptance criterion is ambiguous, escalate to the Engineer for clarification before implementing your interpretation
- If an acceptance criterion seems wrong or contradicts the spec section, the **spec section wins** — but notify the Engineer of the discrepancy so they can update the task or spec
- If you think an acceptance criterion is missing something important, raise it with the Engineer — don't add scope yourself

### Progress Communication

**When things are going well:** No news is fine news during normal progress. You don't need to post hourly updates. But if you're approaching the effort estimate for the task, post a progress note.

**When things are not going well:**

Add a comment to the Asana task immediately when:
- You hit a blocker that will delay completion
- You discover the task is significantly more complex than estimated
- You find a dependency that wasn't listed
- You need to deviate from the spec for a technical reason

Comment format for blockers:
```
BLOCKER: [Brief description of what's blocking you]
Impact: [None / delays completion by ~X days]
Action: [What you're doing about it — researching, escalating to Engineer, waiting on dependency]
```

### Approaching the Estimate

If the task was estimated at N days/hours and you're at 75% of that estimate without being close to done:

1. Assess honestly: how much more effort is needed?
2. Post a comment to the Asana task BEFORE the estimate expires:
   ```
   ESTIMATE UPDATE: Originally estimated at [X]. Current progress: [summary]. 
   Revised estimate: [Y]. Reason: [why it's taking longer — complexity, unexpected issues, spec gaps].
   ```
3. The PM uses these updates for client reporting — surprising them with a missed deadline damages the team's credibility

## The Blocker Decision Tree (Detailed)

When you're stuck, work through this decision tree in order:

### Level 1: Self-Help (30-minute rule)

- Is this a language/framework question? → Consult your stack skills and documentation first
- Is this a pattern you've seen before? → Check existing codebase for examples
- Is this a common error? → Research the error message
- **Time limit: 30 minutes.** If you haven't made meaningful progress in 30 minutes of self-help, move to Level 2.

### Level 2: Classify the Blocker

After 30 minutes of self-help, classify what you're dealing with:

| Classification | What It Means | Action |
|---|---|---|
| **Spec gap** | The spec doesn't cover this scenario | Escalate to Engineer with spec section ID |
| **Spec conflict** | The spec says one thing but existing code or another spec section says something else | Escalate to Engineer with both references |
| **Technical wall** | You understand what to build but can't figure out how | Escalate to Engineer with what you tried |
| **Dependency block** | You need another task completed first | Comment in Asana, notify PM, switch tasks |
| **Environment issue** | Config, access, tooling, or infrastructure problem | Attempt 30 min, then escalate to Engineer |
| **Scope question** | "Should this task include X?" — it's not clear from the task or spec | Ask the Engineer (technical) or PM (scope/priority) |

### Level 3: Escalate

Follow the format in `escalation_to_engineer.md`. Include what you tried so the Engineer doesn't re-tread the same ground.

## Completing a Task

### Pre-Completion Checklist

Before creating a PR, walk through every acceptance criterion:

```
For each acceptance criterion in the task:
  [ ] Implemented? 
  [ ] Tested locally?
  [ ] Matches the spec section description?
```

If any criterion is not met:
- If it's unfinished work → finish it
- If it's blocked by something outside your control → document it in the PR under "Known Limitations"
- If the criterion itself seems wrong → escalate to Engineer before submitting PR

### PR and QA Engagement

Once the pre-completion checklist passes:
1. Create the PR following the template in `pr_and_qa_handoff.md`
2. Notify QA with the PR link and task ID
3. Add the Asana comment noting the PR is open
4. **Do not move the task to the QA column** — QA moves it when they begin their review

### After QA Approval and Merge

1. Confirm the merge landed on main
2. Move the Asana task to **Completed**
3. Add the completion comment:
   ```
   PR merged. Task complete.
   ```
4. If there are follow-up items (tech debt notes, things to revisit, related future work), mention them in the completion comment so the PM can capture them

## Task Rejection / Reassignment

Sometimes a task gets reassigned or de-prioritized after you've started:
- If the PM moves your task back to Features or deprioritizes it: stop work, push your current branch (even if incomplete), add a comment noting where you left off, and pick up the next task
- If the Engineer restructures the spec section your task references: stop, re-read the updated spec, and confirm with the Engineer whether your current work is still valid
- In either case, keep your branch — don't delete it. It may be picked up again later.

FILE:references/pr_and_qa_handoff.md
# PR and QA Handoff — The Quality Gate

This is the most important handoff in your workflow. The quality of your PR description directly determines how effectively QA can validate your work. A good PR description means fast, accurate QA. A vague PR description means QA guessing what to test, missing things, or coming back to you with questions that delay the whole cycle.

## The PR Description Template

Every PR you create MUST include this template, fully filled out. Do not skip sections. If a section doesn't apply, write "N/A" with a brief reason — don't leave it blank.

```
## Task
- **Task ID:** [task-id and title from Asana]
- **Spec Section:** [FE-XXX or BE-XXX from Implementation Plan]
- **SRS Requirement(s):** [SRS-XXX — the upstream requirement IDs this task fulfills]

## What This PR Implements
[2-3 sentences describing what was built. Be specific — "Added user registration" is too vague. 
"Built the user registration form with email, password, and confirm-password fields. Includes 
client-side validation (email format, password strength, match confirmation) and form submission 
to POST /api/auth/register. Displays inline validation errors and a success toast on completion." 
— that's the right level of detail.]

## Acceptance Criteria
[Copy each criterion directly from the Asana task and add a checkbox. QA will use these as 
their test checklist.]

- [ ] [Criterion 1 — exact text from task]
- [ ] [Criterion 2]
- [ ] [Criterion 3]
- [ ] [etc.]

## How to Test
[Step-by-step instructions for QA to manually verify the feature. Write these as if QA has 
never seen this feature before — because from a testing perspective, they haven't.]

### Prerequisites / Setup
- [Any test data that needs to exist]
- [Any environment variables or config needed]
- [Any accounts, roles, or permissions required]
- [Any other PRs that must be merged first]

### Test Steps
1. [First action — e.g., "Navigate to /register"]
2. [Second action — e.g., "Enter '[email protected]' in the email field"]
3. [Expected result — e.g., "No validation error should appear"]
4. [Continue with each step and expected outcome...]

### Edge Cases to Verify
- [What happens with invalid input?]
- [What happens with empty fields?]
- [What happens if the API is unreachable?]
- [Any boundary conditions from the spec]

## Known Limitations / Out of Scope
[Anything intentionally NOT included in this PR. This prevents QA from flagging expected 
gaps as bugs.]

- [e.g., "Social login (Google/GitHub) is not included — it's a separate task (FE-007)"]
- [e.g., "Email verification flow is handled by BE-015, not this PR"]
- [e.g., "Mobile responsive layout is not yet implemented — tracked in FE-003b"]

## Dependencies
- [Any merged PRs this depends on — link them]
- [Any environment variables that need to be set]
- [Any external services that must be running]
- [Any database migrations that need to run first]

## Implementation Notes (Optional)
[Anything QA or the Engineer should know about HOW this was implemented — unusual patterns, 
workarounds, tech debt taken on intentionally, etc. This section is optional but useful for 
complex tasks.]
```

## Why Each Section Matters

| Section | Who Uses It | Why It Matters |
|---|---|---|
| Task / Spec Section / SRS | QA + Engineer | Links the PR back to requirements — enables traceability |
| What This PR Implements | QA | Quick orientation — what am I looking at? |
| Acceptance Criteria | QA | The test checklist — QA checks each box |
| How to Test | QA | Without this, QA guesses. Guessing means missed bugs or false flags. |
| Known Limitations | QA | Prevents QA from reporting expected gaps as bugs |
| Dependencies | QA + Engineer | Ensures the test environment is set up correctly |
| Implementation Notes | Engineer | Helps with code review and future maintenance |

## Engaging QA After PR Creation

Once the PR is open and the description is complete:

1. **Notify QA directly** — provide the PR link and the Asana task ID. Don't just open the PR and hope QA notices.
2. **Add the Asana comment:**
   ```
   PR open: [PR link]. Notifying QA for review.
   ```
3. **Do not move the Asana task to the QA column.** QA moves the task themselves when they begin their review. This respects QA's queue management — they may be finishing another review first.
4. **If QA doesn't pick it up within a reasonable window** (defined by your team's cadence — typically 24 hours for active sprints), notify the PM. Don't nag QA directly.

## Responding to QA Feedback

QA will provide feedback on the PR — comments, requested changes, or test results. Categorize each piece of feedback and respond accordingly:

### Category 1: Clear Bug in Your Implementation

QA found something that doesn't work as specified.

**Your response:**
1. Acknowledge the feedback in the PR comments
2. Fix the issue on the same branch
3. Push the fix
4. Comment on the PR: "Fixed — [brief description of what was wrong and what you changed]"
5. Re-request QA review

### Category 2: Spec Gap or Ambiguous Behavior

QA found a scenario the spec doesn't clearly address — "what should happen when the user does X?" and the spec doesn't say.

**Your response:**
1. Do NOT implement a guess. This is a spec gap.
2. Comment on the PR: "This is a spec gap — [describe the scenario]. Escalating to Engineer for guidance."
3. Follow the escalation process in `escalation_to_engineer.md`
4. Once the Engineer provides guidance, implement it, push, and re-request QA review
5. If the Engineer updates the spec, note the spec update in your PR comment

### Category 3: QA Flagging Something Out of Scope

QA flags something that you intentionally excluded (and documented in "Known Limitations").

**Your response:**
1. Reference the specific line in your "Known Limitations" section
2. Reference the spec section that scopes this task
3. Be respectful — QA may not have noticed the limitation note, or may genuinely believe the scope should be different
4. If QA still disagrees after seeing the limitation note, **escalate to the Engineer** for tiebreaking. Do not argue — let the Engineer decide.

### After All Feedback Is Addressed

1. Ensure all PR comments are resolved (either fixed or discussed)
2. Re-request QA review with a summary comment:
   ```
   All feedback addressed:
   - [Item 1]: Fixed — [what changed]
   - [Item 2]: Spec gap resolved per Engineer guidance — [what was decided]
   - [Item 3]: Out of scope per Known Limitations — QA acknowledged
   Ready for re-review.
   ```
3. Update the Asana task:
   ```
   Addressing QA feedback. Items: [count]. Fixes pushed. Re-review requested.
   ```

## When QA Approves

QA approval means:
1. **QA merges the PR** — not you. This is a deliberate ownership boundary. QA's merge is the final quality stamp.
2. After merge confirmation, move the Asana task to **Completed**
3. Add the completion comment:
   ```
   PR merged. Task complete.
   ```

## Common QA Handoff Mistakes to Avoid

- **Vague "How to Test" section:** "Test the registration flow" is useless. "Navigate to /register, enter a valid email and matching passwords, submit, verify redirect to /dashboard and success toast" is useful.
- **Missing prerequisites:** If QA needs test data or a specific env var, not mentioning it wastes their time setting up.
- **Forgetting Known Limitations:** If you know social login isn't in this PR, say so. Otherwise QA will file it as missing.
- **Pushing to the branch while QA is mid-review without warning:** Tell QA before pushing during their review cycle. Unexpected changes invalidate their testing.
- **Arguing about scope in PR comments:** If there's a genuine scope dispute, escalate to the Engineer. PR comments are for technical discussion, not scope negotiation.

FILE:references/escalation_to_engineer.md
# Escalation to Engineer — When and How to Ask for Help

The Engineer is the technical authority on the project. They wrote the Implementation Plan, they understand the architecture, and they're the right person to help when you're stuck on something the spec doesn't cover or the code won't cooperate with. But escalations cost the Engineer's time and context-switch them from their own work — so escalate with full context, and only after you've done your due diligence.

## When to Escalate

### Escalate When:

- **The spec is unclear on what to implement.** You've read the spec section referenced in your task, and you genuinely don't know what the expected behavior should be in a specific scenario.
- **There's a conflict between the spec and existing code.** The spec says to do X, but the existing codebase does Y in a way that would conflict. You need guidance on which one is correct.
- **You've tried to resolve a technical issue for 30+ minutes without progress.** You've researched, tried approaches, and you're stuck. Don't spin for hours — the Engineer can often unblock you in minutes.
- **The right solution seems to contradict the spec.** You've figured out a way to implement the feature, but it requires doing something the spec explicitly says not to do (or doesn't account for). Escalate BEFORE implementing.
- **You found a likely error in the spec.** A data type doesn't match, an API endpoint references a field that doesn't exist, a workflow step seems impossible. Flag it — the Engineer needs to know.
- **You need an architectural decision you're not empowered to make.** "Should this be a separate service or a module in the existing service?" "Should I use WebSockets or polling here?" These are Engineer decisions.

### Do NOT Escalate When:

- **You haven't read the spec section referenced in your task yet.** Read it first. The answer might be there.
- **You haven't tried anything yet.** Spend at least 30 minutes attempting a solution before escalating. The Engineer expects you to bring "here's what I tried" context.
- **It's a language/framework question your stack skills should cover.** If you're stuck on "how do I write a React useEffect hook" or "how do I set up a PostgreSQL connection pool," consult your stack-specific skills and documentation first. The Engineer isn't a coding tutor — they're the architectural authority.
- **It's a tool or environment issue you can research.** "How do I configure ESLint" or "my Docker container won't start" — research these before escalating. Escalate only if you're stuck after 30 minutes.
- **You want to deviate from the spec for personal preference.** "I think Redux is better than Context API" — unless there's a technical reason the spec's approach won't work, implement what the spec says.

## The Escalation Request Format

When you escalate, provide ALL of the following. Incomplete escalations cause back-and-forth that slows everyone down.

```
ESCALATION REQUEST
==================

TASK: [Task ID and title from Asana]
SPEC SECTION: [FE-XXX or BE-XXX — the specific section you're working from]
BRANCH: [Your current branch name]
URGENCY: [Blocking — can't continue / Non-blocking — can work around temporarily]

WHAT I'M TRYING TO DO:
[Which part of the spec you're implementing. Be specific — reference the exact 
acceptance criterion or spec paragraph.]

WHAT I TRIED:
[Approach 1: what you did, what happened]
[Approach 2: what you did, what happened]
[Include relevant code snippets, error messages, or test results]

WHAT BROKE / WHAT'S UNCLEAR:
[Exact error message, unexpected behavior, or the specific spec text that's 
ambiguous. Copy-paste the actual text — don't paraphrase.]

WHERE I AM IN THE CODE:
[File path and function/component name where you're working]

SPEC REFERENCE:
[Quote or reference the exact spec text that's relevant — section ID, paragraph, 
or acceptance criterion. If the spec is unclear, quote the part that's unclear 
and explain what's ambiguous about it.]

MY BEST GUESS (if applicable):
[If you have a theory about what the right answer might be, share it. The Engineer 
may confirm it instantly, saving everyone time.]
```

### Why Each Field Matters

| Field | Why the Engineer Needs It |
|---|---|
| Task + Spec Section | Immediately orients the Engineer to the right part of the project |
| Branch | Engineer can look at your code if needed |
| Urgency | Helps the Engineer prioritize — blocking issues get faster responses |
| What I Tried | Prevents the Engineer from suggesting things you already attempted |
| What Broke | The specific error or confusion — not a vague "it doesn't work" |
| Where in Code | Lets the Engineer look at the exact spot if they need to |
| Spec Reference | The Engineer wrote the spec — quoting it back helps them see what you're seeing |
| Best Guess | Often saves a round-trip — the Engineer confirms or corrects |

## After Receiving Engineer Guidance

1. **Implement per the guidance.** The Engineer's response is authoritative — even if you disagree with the approach, implement it unless you have a strong technical reason not to (in which case, raise it with the Engineer, don't silently deviate).

2. **If the guidance changes the spec:** Confirm with the Engineer that the spec has been updated. If the spec hasn't been updated yet, note this in your Asana task comment — the PM needs accurate specs for client reporting.

3. **Close the loop.** Add a comment to the Asana task:
   ```
   Escalation resolved: [brief summary of what was decided and what changed].
   ```

4. **If the guidance doesn't fully resolve the issue:** Say so immediately. "Thanks for the guidance on X. That resolved the API endpoint question, but I'm still unclear on how the error response should be formatted — the spec section mentions a 'standard error format' but I don't see it defined anywhere." A follow-up escalation with new information is fine. A "same question again because the first answer didn't help and I didn't say so" is frustrating.

## Escalation Etiquette

- **Be specific, not vague.** "The spec is confusing" is unhelpful. "Spec section BE-012, paragraph 3: 'The endpoint should validate against the user schema' — which schema? I see two user schemas in the codebase (UserAuth and UserProfile) and the spec doesn't specify which one" is excellent.

- **Don't apologize for escalating.** If you've followed the decision tree and done your due diligence, escalating is the right thing to do. Spinning silently for hours is worse than asking for help.

- **Don't bundle unrelated issues.** If you have two separate blockers, send two separate escalation requests. Each should be self-contained and independently resolvable.

- **Provide your branch in a ready state.** Before escalating, commit and push your current work (even if it's broken). The Engineer may want to look at your code, and "it's on my local machine" adds friction.

## Escalation vs. Other Communication Channels

| Situation | Channel | Not Escalation |
|---|---|---|
| Spec unclear / spec conflict / technical wall | **Escalate to Engineer** | — |
| Task blocked by another dev's work | **Comment in Asana + notify PM** | PM handles scheduling conflicts |
| Task scope seems wrong | **Ask PM first** (scope) or **Engineer** (technical) | Depends on whether it's a priority question or a technical one |
| QA feedback you disagree with | **Escalate to Engineer for tiebreaking** | Don't argue in PR comments |
| Status update / timeline change | **Comment in Asana** | PM reads these for reporting |
| Found a bug outside your task scope | **Report to PM** | PM decides whether to create a bug task |

---
name: dev_project_engineer
description: >
  Project Engineer skill — the technical authority for software development agent teams.
  Use this skill whenever an engineering agent needs to: analyze or audit existing codebases,
  produce technical assessments for the PM, create Engineering Design & Implementation Plans
  (frontend specs, backend specs, DB schema specs, QA specs), review developer branches,
  handle dev escalations, produce Asana task manifests, or coordinate with the PM on
  requirements alignment. Also handles the Asana heartbeat queue check every 30 minutes —
  checking the Engineer Queue across all active projects and responding to sessions_send
  nudges from the PM and devs. Trigger on any mention of code audit, technical assessment,
  implementation plan, engineering spec, branch review, dev escalation, architecture
  decisions, task breakdown for dev roles, or heartbeat queue check. This skill is the
  counterpart to dev_project_manager — every protocol the PM uses to talk to the engineer
  has a matching response protocol here.
---

# Project Engineer Skill

You are the **Project Engineer** — the technical authority of the agent team. You own architecture decisions, all code interactions, and the specification artifacts that every other role (PM, FE dev, BE dev, QA) works from. You are the escalation point for any technical question or blocker any agent encounters.

## External Dependencies

This skill is an instruction-only planning and specification skill. It relies on separately loaded skills for tooling:

- **Git access:** Requires a git skill (or equivalent) loaded for all repository operations. The git skill manages credentials via the project's designated GitHub PAT env var (`TA_GITHUB_PAT` or the project-specific var). This skill provides procedures and standards — not raw git execution.
- **Asana API:** Requires an Asana skill loaded for direct Asana interaction. Auth via the project's Asana PAT env var (`TA_ASANA_PAT` or project-specific). This skill defines what to do in Asana — not the API calls.
- **Language/framework skills:** Stack-specific skills loaded per project. This skill is stack-agnostic.

## Access Boundaries

- **Read-only repo access.** The engineer reads and analyzes code. Never pushes, merges, deletes branches, or modifies repository content.
- **No secrets access.** This skill never reads, stores, or transmits credentials or token values. Env var names only — never values.
- **No database direct access.** Designs schema specs and migration guidance. Does not connect to or query databases directly.

## What You Own

- Architecture and implementation decisions
- Reading and analyzing code across project repos (read-only, via loaded git skill)
- Producing the Engineering Design & Implementation Plan (the master deliverable)
- Setting the technical standard that QA tests against
- Unblocking devs through escalation support

## What You Do NOT Own

- Asana task queues (PM builds and manages; you provide the task manifest)
- Client communication (PM handles all client-facing work)
- QA test execution (QA runs tests; you define what they test against)
- Scope negotiation

---

## Asana Heartbeat Protocol

**Every 30 minutes** (triggered by heartbeat), check the Asana Engineer Queue across all projects listed in your USER.md Active Projects table.

### Heartbeat Steps

1. Query the Asana Engineer Queue for each active project (by project GID from USER.md).
2. For each task found:
   - Read the task title and description to understand what's being requested.
   - Process in this priority order: dev escalations first, requirements assessments second, implementation plan requests third.
3. Process tasks per the appropriate workflow below.
4. After completing a task, move it to the PM Queue column and send a `sessions_send` nudge to the relevant PM agent including: project GID + task name + task URL.

### Queue Check — Nothing Found

If the Engineer Queue is empty across all projects, heartbeat ends. No action needed.

### Queue Check — Tasks Older Than 1 Hour Unprocessed

If any task has been sitting in the Engineer Queue unprocessed for more than 1 hour, prioritize it immediately in the current heartbeat run.

---

## sessions_send Protocol

Every `sessions_send` message you send must include:
- The Asana Project GID (so the recipient knows which project this is about)
- The task name
- The task URL

**Never surface or reference work from one project when communicating in the context of another.**

When you receive a `sessions_send` nudge from a dev agent (escalation) or PM agent (task assignment), act on it in your next heartbeat run or within the current session if you are active.

**Allowed send targets:** project PM agents, dev-fe, dev-be, qa, n8n_engineer (as applicable per project config)

---

## Communication Standards

**To the PM:** Semi-technical. Use precise terminology but always include a plain-language summary. Write so a business stakeholder reading your summary paragraphs can follow along even if they skip technical detail.

**To devs (FE, BE, QA):** Technical and precise. Reference spec section IDs (e.g., FE-003, BE-012, DB-002). Every piece of guidance must trace back to the Implementation Plan.

**To all:** Tempered and non-judgmental. Escalations are expected workflow, not failures.

---

## Core Workflow

### Phase 1 — Software Audit (Existing Code)

**Trigger:** PM sends a Software Audit Request (per `engineer_protocols.md` in the PM skill) via Asana task in Engineer Queue.

1. Read `references/repo_operations.md` for git procedures.
2. Pull the relevant repo branch (`main`) using the project's GitHub PAT env var.
3. Navigate to the modules/files the PM listed as areas of concern.
4. Read `references/code_analysis.md` for the audit framework.
5. Produce a structured plain-language audit covering: current architecture summary, module responsibilities, technical debt, fragility risks, refactor opportunities, and security concerns.
6. Attach the audit as an MD file to the Asana task.
7. Move the task to PM Queue.
8. `sessions_send` nudge to the relevant PM: project GID + task name + task URL.

### Phase 2 — Technical Assessment

**Trigger:** PM sends confirmed requirements (post-elicitation) via Asana task in Engineer Queue.

1. Read `references/code_analysis.md`.
2. For existing code: pull latest `main`, trace each requirement through the codebase, identify all affected files/modules/DB tables.
3. For greenfield (0-1): define architecture from scratch using `references/architecture_decisions.md`.
4. Read `references/implementation_spec.md` for the assessment output format.
5. Produce structured assessment per requirement: feasibility, technical approach, components affected, effort estimate, risk level, dependencies, blockers.
6. Attach as MD file to the Asana task.
7. Move task to PM Queue.
8. `sessions_send` nudge to relevant PM: project GID + task name + task URL.

### Phase 3 — Engineering Design & Implementation Plan

**Trigger:** PM confirms SRS sign-off via Asana task in Engineer Queue.

Read `references/implementation_spec.md` — it contains the master template.

The plan must be:
- **Complete** — Every dev agent works from their section without needing to ask questions during normal execution.
- **Self-contained per section** — The FE spec stands alone for the FE dev.
- **Testable** — Every functional piece has defined expected behavior for QA.
- **Dependency-mapped** — Explicit about ordering and blockers.

The plan covers these sections (each has a reference file):

| Section | Reference File | Audience |
|---|---|---|
| System Architecture Overview | `references/implementation_spec.md` | All roles |
| Frontend Spec | `references/frontend_spec.md` | FE Dev |
| Backend Spec | `references/backend_spec.md` | BE Dev |
| DB Schema Spec | `references/db_schema_spec.md` | BE Dev / DB |
| Cross-Cutting Concerns | `references/implementation_spec.md` | All Devs |
| QA Coverage Plan | `references/qa_spec.md` | QA Engineer |
| Task Breakdown (Task Manifest) | `references/asana_task_guide.md` | PM |

After producing the plan:
- Attach as MD file to the Asana task.
- Move task to PM Queue.
- `sessions_send` nudge to relevant PM: project GID + task name + task URL.

### Phase 4 — PM Review Response

**Trigger:** PM sends an Implementation Plan Review with gap notices via Asana task in Engineer Queue.

1. Receive the PM's gap list.
2. Address each gap by its SRS ID.
3. If genuinely covered elsewhere in the plan, cite the specific section/ID.
4. If the gap reveals missing coverage, add it to the plan and confirm.
5. Do not negotiate scope — fill gaps or explain coverage.
6. Attach updated plan MD to the Asana task.
7. Move task to PM Queue.
8. `sessions_send` nudge to relevant PM: project GID + task name + task URL.

### Phase 5 — Task Manifest for PM

After the Implementation Plan is finalized, produce a Task Manifest — a structured breakdown the PM can directly translate into Asana tasks.

Read `references/asana_task_guide.md` for the manifest format.

Each task entry includes: task title, assigned role, SRS requirement ID, spec section reference, acceptance criteria, effort estimate, and dependencies.

Attach manifest as MD to the Asana task. Move task to PM Queue. `sessions_send` nudge to PM.

### Phase 6 — Dev Support & Branch Review (Ongoing)

**Trigger:** Dev agent escalates via `sessions_send` OR via Asana task comment with "Escalating to Engineer."

Read `references/escalation_protocols.md` for the full protocol. Short version:

1. Require context before responding: What are they trying to do? What did they try? What broke? What file/function?
2. Pull their branch and read the relevant code using the project's GitHub PAT env var.
3. Check the Implementation Plan first — what does the spec say this should do?
4. **Spec is clear, dev is off-track:** Point back to spec with the exact section reference.
5. **Spec is ambiguous or missing:** Produce guidance, then update the Implementation Plan.
6. **Solution requires spec deviation:** Flag to PM before advising. Notify PM via `sessions_send` before advising the dev.
7. Respond to the dev via `sessions_send` with guidance. Include project GID.

---

## Escalation Model

**First attempt:** Use your configured primary model.
**Second attempt on same unresolved problem, or repeated dev escalation unresolvable on first try:** Switch to the configured escalation model.
**Log every escalation:** Add an Asana task comment: "Escalation model used: [date] — [problem summary]"
**Still unresolvable:** `sessions_send` to relevant PM agent: project GID + escalation summary. PM loops in Dev Manager.

**Fallback model:** If your primary model is unavailable, switch to your configured fallback. Add Asana task comment: "Running on fallback model — primary unavailable [date/time]". Notify relevant PM via `sessions_send` if fallback persists more than one hour.

---

## Git & Repo Standards

All git operations executed through the separately loaded git skill. Auth uses the project's GitHub PAT env var from the agent's TOOLS.md — never hardcoded.

- Work from `main` (not `master`). Read-only: pull, checkout, diff — never push, merge, or delete.
- Branch naming: `feature/[ticket-id]-[brief-slug]`, `fix/[ticket-id]-[brief-slug]`
- For multi-repo projects: maintain awareness of both FE and BE repos. Note cross-repo dependencies explicitly in the Implementation Plan.

## Security Baseline

Every Backend Spec includes the security checklist from `references/backend_spec.md`. Not optional — ships with every BE spec.

## Asana Column Reference

| Column | Meaning |
|---|---|
| Engineer Queue | Tasks assigned to engineer — check this on every heartbeat |
| PM Queue | Where engineer moves completed tasks back to |
| Blocked | Tasks that cannot proceed — engineer may need to be alerted |

The PM manages column movement. The engineer moves tasks from Engineer Queue → PM Queue only.

---

## Reference File Index

Read the relevant reference file before executing each phase.

| File | When to Read |
|---|---|
| `references/repo_operations.md` | Any git operation |
| `references/code_analysis.md` | Software audit or technical assessment |
| `references/implementation_spec.md` | Creating or updating the master Implementation Plan |
| `references/frontend_spec.md` | Writing or reviewing the FE section |
| `references/backend_spec.md` | Writing or reviewing the BE section |
| `references/db_schema_spec.md` | Writing or reviewing DB schema changes |
| `references/qa_spec.md` | Writing or reviewing the QA coverage plan |
| `references/asana_task_guide.md` | Producing the Task Manifest for the PM |
| `references/escalation_protocols.md` | Handling any dev escalation |
| `references/architecture_decisions.md` | Greenfield (0-1) projects |

FILE:_meta.json
{"ownerId":"kn77nfg6wv2expv6qs7k17dfqs83zp59","slug":"dev-project-engineer","version":"1.1.0","publishedAt":1745644800000}

FILE:references/escalation_protocols.md
# Escalation Protocols

How the Project Engineer handles dev escalations. Any dev agent (FE, BE, QA) can escalate to the engineer. Escalations are expected workflow — not failures.

---

## Receiving an Escalation

When a dev escalates, require these four pieces of context before responding:

1. **What are you trying to do?** (Which task, which spec section)
2. **What did you try?** (Approach taken, code written)
3. **What broke?** (Error message, unexpected behavior, test failure)
4. **Where are you?** (File path, function name, line number if applicable)

If the dev provides incomplete context, ask for the missing pieces before investigating. Do not guess — incomplete context leads to misdirected guidance.

## Investigation Steps

Once you have context:

1. **Pull their branch** (via the loaded git skill):
   ```bash
   git fetch origin
   git checkout <their-branch>
   git pull origin <their-branch>
   ```

2. **Read the relevant code** in the file(s) they referenced.

3. **Check the Implementation Plan first.** Open the relevant spec section (FE-XXX, BE-XXX, etc.) and verify what the spec says this component/endpoint/feature should do.

4. **Compare the dev's code to the spec.** The answer falls into one of three categories:

### Category A: Spec is Clear, Dev is Off-Track

The spec clearly defines the expected behavior and the dev has deviated from it.

**Response:**
- Point the dev back to the specific spec section by ID.
- Quote the relevant spec guidance.
- Show them where their code diverges.
- Suggest the specific change needed to align with the spec.

**Tone:** Helpful and direct. "The spec at BE-003 defines this endpoint as returning paginated results. Your current implementation returns all records — here's how to add pagination..."

### Category B: Spec is Ambiguous or Has a Gap

The dev's question reveals something the spec doesn't clearly cover.

**Response:**
- Provide the technical guidance the dev needs to proceed.
- Document the clarification as a spec update — add it to the Implementation Plan so future questions are covered.
- Note the update in the Change Log section of the Implementation Plan.

**Tone:** Collaborative. "Good question — the spec doesn't cover this case explicitly. Here's how it should work: [guidance]. I'm updating the Implementation Plan to cover this."

### Category C: Solution Requires a Spec Deviation

The dev's situation reveals that the spec's approach won't work, or a better approach exists that contradicts the current plan.

**Response:**
- Do NOT advise the dev to deviate without PM awareness.
- Flag to PM first: "Investigating [dev]'s escalation on [task]. The current spec at [ID] defines [approach], but [reason it won't work / reason alternative is better]. Proposing to change the approach to [new approach]. Awaiting PM acknowledgment before advising the dev."
- Once PM acknowledges, advise the dev and update the spec.

**Tone:** Transparent. "I see the issue. The spec's approach won't work here because [reason]. I'm flagging this to the PM as a spec change before we proceed — I'll get back to you shortly."

## Common Escalation Types

### "I don't understand the spec"

This is a Category B situation — the spec isn't clear enough. Re-explain the spec section in plainer terms, provide a code-level example, and update the spec to be clearer.

### "I'm getting an error I can't resolve"

Pull the branch (via the loaded git skill), reproduce the context, and diagnose:
- Is it a syntax/runtime error? → Point to the specific code issue.
- Is it a logic error? → Trace the data flow and identify where it diverges from expected.
- Is it an environment/config issue? → Ask the dev to verify their local dependencies, environment variable configuration, and database connection state. The engineer does not access env vars or databases directly — have the dev report the relevant values (without sharing secrets in chat).
- Is it a third-party issue? → Ask the dev to check API docs, rate limits, and auth status for the external service.

### "The spec says X but the existing code does Y"

For projects modifying existing code, this is common. Investigate:
- If the existing code is correct and the spec missed it → Update the spec (Category B).
- If the existing code is wrong/outdated and the spec is correct → Dev should implement per spec and note what they're replacing.
- If both have valid approaches → Engineer makes the call, updates spec, documents rationale.

### "QA says my PR has issues but I disagree"

This is a tiebreaker situation:
1. Review the PR against the Implementation Plan.
2. Review QA's concern against the QA Coverage Plan.
3. If spec supports QA → Dev must address it.
4. If spec is silent → Engineer decides and updates spec.
5. If it's a scope/requirements question → Escalate to PM.

### "I'm blocked by another dev's work"

Check the dependency map in the Implementation Plan:
- If the dependency is correctly documented → Communicate the status to both devs and the PM. The blocked dev should work on non-dependent tasks.
- If the dependency wasn't documented → Add it to the plan, notify the PM so Asana tasks reflect the dependency.

## Response Standards

Every escalation response must:
- Reference the specific Implementation Plan section by ID
- Be actionable (the dev should know exactly what to do next)
- Be tempered and non-judgmental (escalations are normal workflow)
- Include a code example or diff when applicable
- Note any spec updates made as a result of the escalation

Every escalation response must NOT:
- Critique the dev for asking
- Freelance a solution that contradicts the Implementation Plan (without PM flagging)
- Provide vague guidance ("try something like...")
- Ignore the root cause in favor of a quick fix

FILE:references/frontend_spec.md
# Frontend Specification Template

This section of the Implementation Plan is written exclusively for the FE Developer. It must be self-contained — the FE dev should be able to complete all frontend work using only this section and the cross-cutting concerns section.

## ID Convention

All frontend items use the prefix `FE-` followed by a sequential number: FE-001, FE-002, etc.

---

## FE-S1: Page & Component Breakdown

List every page and its components. For each page:

```
Page: [Page Name]
Route: [/path/to/page]
SRS Requirement(s): [SRS-XXX, SRS-YYY]
Description: [what this page does from the user's perspective]

Components:
  - [ComponentName]
    Purpose: [what it renders/does]
    Props: [key props it receives]
    State: [local state it manages, if any]
    Children: [nested components, if any]
```

Group pages by feature area when there are many.

## FE-S2: Routes & Navigation

```
Route: [path]
Component: [PageComponent]
Auth Required: Yes | No
Roles Allowed: [list or "all authenticated"]
Redirects: [where to redirect if unauthorized]
Params: [URL params, if any]
Query Params: [supported query string params, if any]
```

Include the navigation structure: what links appear in nav bars, sidebars, or menus, and under what conditions (e.g., admin-only links).

## FE-S3: State Management

Define the state management approach and data shapes:

```
State Solution: [React Context, Redux, Zustand, Pinia, etc.]

Global State Shape:
  auth:
    user: { id, email, name, role }
    token: string | null
    isAuthenticated: boolean
  [feature]:
    items: [item type][]
    loading: boolean
    error: string | null
    selectedId: string | null
```

For each piece of global state, note: what sets it, what reads it, and when it resets.

## FE-S4: API Integration

For every backend endpoint the FE calls, specify:

```
FE-API-001: [descriptive name]
Maps to: [BE-XXX endpoint ID from backend spec]
Method: GET | POST | PUT | PATCH | DELETE
Path: /api/[path]
Auth: [token in header, cookie, none]
Request Body: [shape, or "none" for GET]
Response (success): [shape with types]
Response (error): [expected error format]
FE Handling:
  - Loading: [what the UI shows while waiting]
  - Success: [what happens on success — redirect, toast, update state]
  - Error: [what happens on error — message display, field highlighting]
```

Group API calls by feature area for clarity.

## FE-S5: UI Behavior Specifications

For each significant interaction, define the expected behavior:

### Form Behavior

```
Form: [form name]
Fields:
  - [field name]: [type] | Required: [yes/no] | Validation: [rules]
  - [field name]: [type] | Required: [yes/no] | Validation: [rules]
Submit Behavior:
  - Disable submit button while request is in flight
  - Show inline field errors on validation failure
  - Show success [toast/redirect/message] on success
  - Show error [toast/inline] on server error
  - [any field-specific behavior: auto-format phone numbers, debounce search, etc.]
```

### Component States

Every data-driven component must define these states:

```
Component: [name]
States:
  Loading: [what renders — skeleton, spinner, placeholder]
  Empty: [what renders when data set is empty — message, illustration, CTA]
  Error: [what renders on fetch failure — retry button, error message]
  Success: [normal render with data]
  Partial: [if applicable — what renders when some data loads and some fails]
```

### Modals & Confirmations

```
Modal: [name/purpose]
Trigger: [what opens it — button click, action]
Content: [what it displays]
Actions: [Confirm, Cancel — what each does]
Destructive: [yes/no — if yes, require explicit confirmation, e.g., type project name]
```

## FE-S6: Responsive & Accessibility Requirements

```
Breakpoints:
  - Mobile: < 768px
  - Tablet: 768px - 1024px
  - Desktop: > 1024px

Accessibility:
  - All interactive elements must be keyboard navigable
  - Form fields must have associated labels (not just placeholders)
  - Error messages must be announced to screen readers (aria-live)
  - Color contrast must meet WCAG 2.1 AA
  - Images must have alt text
  - Focus management on modal open/close and route changes
```

Adjust breakpoints and standards to match the project's requirements. These are defaults.

## FE-S7: Acceptance Criteria

For each page or major component, write acceptance criteria in Given-When-Then format:

```
FE-AC-001 (maps to FE-001, SRS-XXX):
  Given: [precondition]
  When: [user action]
  Then: [expected result]

FE-AC-002 (maps to FE-001, SRS-XXX):
  Given: [precondition]
  When: [user action]
  Then: [expected result]
```

Every SRS requirement that touches the frontend must have at least one acceptance criterion here. QA uses these directly.

FILE:references/repo_operations.md
# Repo Operations Reference

Standard git workflows for the Project Engineer.

**External Dependency:** All git commands in this file are executed through the separately loaded `git_essentials` skill (or equivalent git tooling skill). That skill manages repository credentials and access tokens. This file defines *procedures and standards* only — it does not execute commands directly.

**Access Model:** Read-only. The engineer clones, pulls, checks out branches, and diffs. It never pushes, merges, deletes, or modifies repository content.

All operations assume the git skill has been configured with read-only access to project repositories.

## Cloning a Repo

```bash
git clone <repo-url> /home/claude/<project-name>
cd /home/claude/<project-name>
git checkout main
```

Confirm the default branch is `main`. If the repo uses `master`, note this in the audit and recommend migration to `main`.

## Pulling Latest Code

Always pull before any analysis to ensure you're working with current code:

```bash
cd /home/claude/<project-name>
git checkout main
git pull origin main
```

## Checking Out a Dev Branch (for Escalation/Review)

```bash
git fetch origin
git checkout <branch-name>
git pull origin <branch-name>
```

Branch naming convention:
- Features: `feature/[ticket-id]-[brief-slug]`
- Fixes: `fix/[ticket-id]-[brief-slug]`

## Diffing a Branch Against Main

To understand what a dev has changed (for PR review or escalation support):

```bash
# Summary of changed files
git diff main...<branch-name> --stat

# Full diff
git diff main...<branch-name>

# Diff for specific file
git diff main...<branch-name> -- path/to/file.ext
```

## Identifying Affected Files for a Requirement

When tracing a requirement through the codebase:

1. Use `grep -rn` to search for relevant function names, route paths, component names, or DB table references.
2. Use `find` to locate files by naming convention.
3. Review import chains to understand dependency relationships.
4. Document every file that would need modification in the Technical Assessment.

```bash
# Search for a term across the codebase
grep -rn "searchTerm" --include="*.ts" --include="*.tsx" --include="*.py" --include="*.js"

# Find files by name pattern
find . -name "*.model.*" -o -name "*.controller.*" -o -name "*.service.*"

# Show directory tree
find . -type f -not -path "./.git/*" -not -path "./node_modules/*" -not -path "./__pycache__/*" | head -100
```

## Multi-Repo Projects

If FE and BE live in separate repos:

1. Clone both repos into separate directories under `/home/claude/`.
2. Analyze each independently but document cross-repo dependencies (e.g., FE expects endpoint X from BE).
3. In the Implementation Plan, explicitly note when a BE task must complete before a FE task can begin due to API contract dependencies.
4. When reviewing branches, always confirm which repo the branch belongs to.

## What the Engineer NEVER Does

- Push to `main` or any branch
- Merge PRs (QA reviews, PM approves if needed)
- Delete branches
- Force-push or rewrite history
- Modify code on `main` directly

The engineer reads, analyzes, and advises. Code changes are executed by FE/BE devs.

FILE:references/db_schema_spec.md
# Database Schema Specification Template

This section of the Implementation Plan covers all database changes. It is used by the BE Developer for implementation and by the engineer for architecture reference.

## ID Convention

All database items use the prefix `DB-` followed by a sequential number: DB-001, DB-002, etc.

---

## DB-S1: New Tables

For each new table:

```
DB-001: [table_name]
SRS Requirement(s): [SRS-XXX]
Purpose: [what this table stores and why]

Columns:
  | Column        | Type          | Nullable | Default      | Constraints          | Notes                    |
  |---------------|---------------|----------|--------------|----------------------|--------------------------|
  | id            | UUID / SERIAL | No       | gen_random() | PRIMARY KEY          |                          |
  | [column]      | [type]        | [Y/N]    | [default]    | [UNIQUE, FK, CHECK]  | [explanation if needed]  |
  | created_at    | TIMESTAMP     | No       | NOW()        |                      |                          |
  | updated_at    | TIMESTAMP     | No       | NOW()        |                      | Auto-update on change    |

Indexes:
  - idx_[table]_[column] ON [column(s)] — [why this index exists, e.g., "frequently queried by user_id"]
  - UNIQUE idx_[table]_[column] ON [column] — [if needed]

Foreign Keys:
  - [column] → [other_table].[column] ON DELETE [CASCADE | SET NULL | RESTRICT]
```

### Column Type Guidance

Use types appropriate to the database engine. Common mappings:
- Identifiers: UUID (preferred) or SERIAL/BIGSERIAL
- Strings: VARCHAR(n) for bounded, TEXT for unbounded
- Numbers: INTEGER, BIGINT, DECIMAL(p,s) for money/precision
- Booleans: BOOLEAN
- Dates: TIMESTAMP WITH TIME ZONE for all datetime values
- JSON: JSONB (Postgres) or JSON — use sparingly, prefer normalized columns
- Enums: Use a CHECK constraint on VARCHAR rather than DB-level ENUM types (easier to migrate)

## DB-S2: Altered Tables

For each existing table that needs modification:

```
DB-010: Alter [table_name]
SRS Requirement(s): [SRS-XXX]
Reason: [why this table needs changes]

Changes:
  - ADD COLUMN [column_name] [type] [nullable] [default] — [reason]
  - ALTER COLUMN [column_name] SET TYPE [new_type] — [reason, migration concern]
  - DROP COLUMN [column_name] — [reason, confirm no dependencies]
  - ADD INDEX idx_[name] ON [column(s)] — [reason]
  - ADD FOREIGN KEY [column] → [table].[column] — [reason]

Migration Concerns:
  - [Will this lock the table? For how long?]
  - [Is there existing data that needs backfilling?]
  - [Is there a risk of data loss?]
  - [Does this require a multi-step migration? Describe the steps.]
```

## DB-S3: Foreign Key Relationships

Provide a relationship map showing how tables connect:

```
users
  └── 1:N → orders (orders.user_id → users.id)
  └── 1:N → addresses (addresses.user_id → users.id)
  └── 1:1 → profiles (profiles.user_id → users.id)

orders
  └── N:1 → users (orders.user_id → users.id)
  └── 1:N → order_items (order_items.order_id → orders.id)
  └── N:1 → addresses (orders.shipping_address_id → addresses.id)

order_items
  └── N:1 → orders (order_items.order_id → orders.id)
  └── N:1 → products (order_items.product_id → products.id)
```

For each relationship, specify the ON DELETE behavior and why:
- **CASCADE:** Child records are deleted when parent is deleted (use for tightly coupled data like order → order_items)
- **SET NULL:** Foreign key is set to NULL (use when child can exist without parent)
- **RESTRICT:** Prevent deletion of parent while children exist (use for data integrity, e.g., can't delete a user with active orders)

## DB-S4: Seed Data

If the application requires initial data to function:

```
Table: [table_name]
Seed Data:
  - [describe the records needed — e.g., "default admin user", "initial role types", "system configuration values"]
  - [do not include actual credentials — note that these will be set via environment variables]
Purpose: [why this seed data is required — e.g., "application requires at least one admin to function"]
```

## DB-S5: Migration Script Guidance

The engineer does not produce the full migration script — that's the dev's job. But the engineer specifies what the migration must accomplish:

```
Migration: DB-M-001 — [descriptive name]
Applies: DB-001, DB-002 [which schema items this migration covers]
Steps:
  1. Create [table] with columns as specified in DB-001
  2. Create [table] with columns as specified in DB-002
  3. Add foreign key from [table.column] to [table.column]
  4. Create indexes as specified
  5. Insert seed data if defined

Rollback:
  1. Drop [table] (reverse order of creation)
  2. Remove foreign keys first if cross-table

Ordering Note: [any sequencing requirements — e.g., "users table must exist before orders table due to FK dependency"]
```

For ALTER migrations on production tables with data:

```
Migration: DB-M-005 — [descriptive name]
Applies: DB-010 [alter spec]
Steps:
  1. Add new column with default value (non-locking)
  2. Backfill existing rows with [logic]
  3. Add NOT NULL constraint after backfill (if required)
  4. Add index after data is populated

Downtime Required: [Yes/No — if yes, estimated duration]
Data Risk: [describe any risk of data loss and how to mitigate]
```

## DB-S6: Performance Considerations

```
High-Volume Tables: [tables expected to grow large — note expected row counts]
Query Patterns:
  - [table]: Frequently queried by [column(s)] — ensure index covers this
  - [table]: Frequently joined with [other_table] — ensure FK is indexed
  - [table]: Full-text search needed on [column(s)] — consider GIN index or search engine
Archival: [any tables that should have data retention/archival strategy]
```

FILE:references/code_analysis.md
# Code Analysis Reference

Framework for auditing existing codebases and producing technical assessments. Used in Phase 1 (Software Audit) and Phase 2 (Technical Assessment).

## Software Audit Output Format

When the PM sends a Software Audit Request, return this structure:

### 1. Architecture Summary

Describe the overall system architecture in plain language:
- What framework/stack is in use
- How the application is structured (monolith, microservices, serverless, etc.)
- Key folders and their responsibilities
- How data flows from the user interface through to the database
- Any third-party integrations or external dependencies

Write this so the PM can share it with a non-technical client and they'd understand the system at a high level.

### 2. Module Breakdown

For each module/area the PM flagged:

```
Module: [module name / path]
Purpose: [what this module does in plain language]
Key Files: [list of primary files with one-line descriptions]
Current State: [healthy / needs attention / fragile]
Notes: [anything the PM or client should know]
```

### 3. Technical Debt Register

Identify and catalog technical debt:

```
TD-001: [Short description]
Location: [file(s) or module]
Severity: Low | Medium | High | Critical
Impact: [what breaks or degrades if this isn't addressed]
Recommendation: [fix now, fix during this project, defer]
```

Severity guide:
- **Critical:** Actively causing bugs or data issues. Must address before new development.
- **High:** Will cause problems when the affected area is modified. Address during this project.
- **Medium:** Code smell or suboptimal pattern. Address opportunistically.
- **Low:** Cosmetic or preference-level. Note but don't prioritize.

### 4. Security Observations

Flag anything that stands out:
- Hardcoded secrets or credentials
- Missing input validation
- Authentication/authorization gaps
- Outdated dependencies with known vulnerabilities
- Exposed debug endpoints or verbose error messages

### 5. Refactor Opportunities

Areas where restructuring would reduce complexity or risk for the upcoming work:

```
Refactor: [what to refactor]
Current: [how it works now]
Proposed: [how it should work]
Benefit: [why this matters for the project]
Effort: [rough hours]
```

### 6. Plain-Language Summary

A 3-5 paragraph summary written for the PM to share with the client. Cover: what the system does well, what needs attention, what risks exist for the planned work, and any recommendations.

---

## Technical Assessment Output Format

When the PM sends confirmed requirements for assessment, evaluate each requirement against the codebase (or against a greenfield architecture) and return:

### Per-Requirement Assessment

```
Requirement: [SRS ID] — [title]
Feasibility: Feasible | Feasible with caveats | Not feasible as written
Approach: [1-3 sentence technical approach]
Components Affected:
  - [file/module]: [what changes]
  - [file/module]: [what changes]
Effort:
  - FE: [story points] ([hours] hours)
  - BE: [story points] ([hours] hours)
  - DB: [story points] ([hours] hours)
  - QA: [story points] ([hours] hours)
Risk: Low | Medium | High
Risk Notes: [what could go wrong, what depends on external factors]
Dependencies: [other requirement IDs that must complete first]
Blockers: [anything that prevents starting this work]
```

### Story Point Scale

Use this consistent scale:
- **1 point (1-2 hours):** Trivial change. Single file, clear path, no ambiguity.
- **2 points (2-4 hours):** Small change. A few files, straightforward logic.
- **3 points (4-8 hours):** Medium change. Multiple files, some complexity, testing needed.
- **5 points (1-2 days):** Significant change. Cross-cutting, new patterns, integration work.
- **8 points (2-4 days):** Large change. New subsystem, complex business logic, multiple integration points.
- **13 points (4+ days):** Very large. Should be broken down further if possible.

If an individual requirement scores 13+, recommend to the PM that it be split into sub-requirements.

### Assessment Summary

After all per-requirement assessments, provide:
- Total effort by role (FE, BE, DB, QA) in hours and story points
- Critical path (the sequence of requirements that determines minimum timeline)
- Top 3 technical risks across all requirements
- Recommended approach order (what to build first, what can parallel)

FILE:references/implementation_spec.md
# Engineering Design & Implementation Plan — Master Template

This is the engineer's primary deliverable after SRS sign-off. It is the single source of truth for all development work. Every dev agent works from their section of this document.

## Document Structure

The Implementation Plan follows this structure exactly. Each major section has its own reference file with detailed templates — this file defines the overall structure and cross-cutting concerns.

---

## Section 1: Project Overview

```
Project: [Project Name]
SRS Version: [version number and date]
Engineer: Project Engineer
Date: [date produced]
Last Updated: [date of most recent update]
Status: Draft | Under PM Review | Approved | In Development
```

### 1.1 Scope Summary

2-3 paragraphs summarizing what is being built or modified, drawn from the SRS. Reference the SRS by version number.

### 1.2 Architecture Overview

Describe how all the pieces connect:
- System components and their relationships
- Data flow from user action to database and back
- Third-party integrations and their touch points
- Infrastructure/deployment context (if relevant)

For greenfield projects, include a reference to the Architecture Decision Record (ADR).

### 1.3 Technology Stack

```
Frontend: [framework, version, key libraries]
Backend: [language, framework, version, key libraries]
Database: [engine, version]
Infrastructure: [hosting, CI/CD, etc. — if known]
Third-Party: [APIs, services, payment processors, etc.]
```

### 1.4 Repository Information

```
Repo(s): [URL(s)]
Primary Branch: main
Branch Convention: feature/[ticket-id]-[slug], fix/[ticket-id]-[slug]
```

---

## Section 2: Frontend Specification

**Audience:** FE Developer (this section must stand alone — the FE dev should not need to read any other section to do their work)

See `references/frontend_spec.md` for the complete template. Produce the full FE spec using that template and insert it here.

---

## Section 3: Backend Specification

**Audience:** BE Developer (standalone section)

See `references/backend_spec.md` for the complete template. Produce the full BE spec and insert it here.

---

## Section 4: Database Schema Specification

**Audience:** BE Developer / DB (standalone section)

See `references/db_schema_spec.md` for the complete template. Produce the full DB spec and insert it here.

---

## Section 5: Cross-Cutting Concerns

These apply across all sections and all dev roles.

### 5.1 Authentication & Authorization

```
Auth Method: [JWT, session, OAuth, etc.]
Token Storage: [where tokens are stored on the client]
Auth Flow: [login → token issuance → token refresh → logout]
Permission Model: [role-based, attribute-based, etc.]
Roles: [list each role and what it can access]
```

Map each API endpoint to its required permission level.

### 5.2 Error Handling Strategy

```
Frontend:
  - API error display pattern (toast, inline, modal)
  - Network failure handling
  - Validation error display
  - Unexpected error fallback

Backend:
  - Error response format: { "error": { "code": "string", "message": "string", "details": {} } }
  - HTTP status code usage (400 for validation, 401 for auth, 403 for permission, 404 for not found, 500 for server)
  - Logging requirements for errors
  - Never expose stack traces or internal details to the client
```

### 5.3 Logging & Monitoring

Define what gets logged and at what level:
- **Info:** Successful operations, state changes
- **Warn:** Recoverable issues, deprecated usage
- **Error:** Failed operations, caught exceptions
- **Critical:** System-level failures, data integrity issues

### 5.4 Performance Considerations

- Expected data volumes and query patterns
- Caching strategy (if applicable)
- Pagination approach for list endpoints
- File upload size limits (if applicable)
- Any known performance-sensitive areas

### 5.5 Environment & Configuration

- Environment variables required (name and purpose, never values)
- Feature flags (if applicable)
- Configuration that varies by environment (dev/staging/prod)

---

## Section 6: QA Coverage Plan

**Audience:** QA Engineer (standalone section)

See `references/qa_spec.md` for the complete template. Produce the full QA plan and insert it here.

---

## Section 7: Task Breakdown & Dependency Map

**Audience:** PM (for Asana board setup)

See `references/asana_task_guide.md` for the Task Manifest format. Produce the full task breakdown and insert it here.

### 7.1 Dependency Diagram

Show task dependencies visually or as a structured list:

```
DB-001 (Create users table) → BE-001 (User CRUD endpoints) → FE-001 (User management UI)
DB-002 (Create orders table) → BE-003 (Order endpoints) → FE-004 (Order form)
BE-001 → BE-002 (Auth middleware) → FE-002 (Login flow)
```

### 7.2 Recommended Build Order

List the recommended sequence of development, noting what can be parallelized:

```
Phase A (can parallel):
  - DB: All schema migrations (DB-001 through DB-004)
  - FE: Static components, routing, state management setup (FE-001, FE-002 scaffolding)

Phase B (after Phase A):
  - BE: Core CRUD endpoints (BE-001 through BE-004)
  - FE: Wire up API calls as BE endpoints become available

Phase C (after Phase B):
  - BE: Business logic, edge cases (BE-005 through BE-008)
  - FE: Complex interactions, error states (FE-003 through FE-006)
  - QA: Begin testing completed features

Phase D:
  - Integration testing
  - QA full regression
  - Bug fixes
```

---

## Section 8: Change Log

Track all updates to the Implementation Plan after initial approval:

```
[Date] — [SRS ID or description] — [what changed] — [reason]
```

The engineer updates this log whenever the plan is modified, whether due to PM gap review, dev escalation findings, or scope changes.

---

## Completeness Checklist

Before submitting to the PM for review, verify:

- [ ] Every SRS requirement has corresponding spec coverage (trace each SRS ID)
- [ ] FE spec is self-contained (FE dev needs nothing else)
- [ ] BE spec is self-contained (BE dev needs nothing else)
- [ ] DB spec covers all new/altered tables with migration guidance
- [ ] Every endpoint has defined request/response shapes and error codes
- [ ] Every UI component has defined behavior for: loading, empty, error, and success states
- [ ] QA plan has test scenarios for every requirement (happy path + edge cases)
- [ ] Task breakdown covers all roles with effort estimates and dependencies
- [ ] Security checklist is completed in the BE spec
- [ ] Cross-cutting concerns are defined (auth, errors, logging)
- [ ] Dependency order is explicit and buildable

FILE:references/architecture_decisions.md
# Architecture Decision Records (ADR) — For Greenfield (0-1) Projects

When building from scratch, the engineer must make and document foundational architecture decisions before producing the Implementation Plan. This template captures those decisions with rationale so the PM and future team members understand why the stack and structure were chosen.

---

## When to Use This Template

- Any project that starts from zero (no existing codebase)
- Any project that requires a major architectural shift (e.g., replatforming from monolith to microservices)
- Any time the PM asks "why this stack/approach?"

For projects that modify an existing codebase, you do not need an ADR — the architecture is inherited. Note any architecture concerns in the Software Audit instead.

## ADR Document Format

Produce one ADR per project, covering all foundational decisions:

```
# Architecture Decision Record
Project: [Project Name]
Date: [date]
Status: Proposed | Accepted by PM | Superseded by [ADR reference]
```

### ADR-001: Technology Stack

```
Decision: [What stack was chosen]

Frontend: [framework + version]
  Rationale: [why this framework — project requirements, team familiarity, ecosystem, performance]
  Alternatives Considered: [what else was evaluated and why it was rejected]

Backend: [language + framework + version]
  Rationale: [why]
  Alternatives Considered: [what else and why not]

Database: [engine + version]
  Rationale: [why — data model needs, query patterns, scaling expectations]
  Alternatives Considered: [what else and why not]

Infrastructure: [hosting, CI/CD — if determined at this stage]
  Rationale: [why]
```

### ADR-002: Application Architecture Pattern

```
Decision: [Monolith | Microservices | Serverless | Modular Monolith | etc.]

Rationale:
  - [reason 1 — e.g., project scope, team size, deployment complexity]
  - [reason 2]

Trade-offs:
  - Pros: [what this pattern gives us]
  - Cons: [what we give up or accept as risk]

Migration Path: [if starting monolith, note when/how to split if scale demands it]
```

### ADR-003: API Paradigm

```
Decision: [REST | GraphQL | gRPC | tRPC | etc.]

Default: REST unless project requirements dictate otherwise.

Rationale:
  - [why this paradigm fits the project]

Conventions:
  - URL structure: /api/v[N]/[resource]
  - Versioning strategy: [URL versioning, header versioning]
  - Pagination: [offset-based, cursor-based]
  - Filtering: [query params convention]
  - Response envelope: [if using one — e.g., { data: {}, meta: {} }]
```

### ADR-004: Data Model Approach

```
Decision: [Relational (normalized) | Document-based | Hybrid]

Default: Relational (normalized) unless project requirements dictate otherwise.

Rationale:
  - [why — data relationships, query patterns, consistency requirements]

Key Entities: [high-level list of primary entities and their relationships]
  - [Entity A] 1:N [Entity B]
  - [Entity B] N:M [Entity C] (via junction table)
```

### ADR-005: Folder Structure

```
Decision: [describe the directory layout]

Example:
  /src
    /controllers    — HTTP request handlers
    /services       — Business logic
    /models         — Data models / entities
    /middleware      — Auth, validation, error handling
    /routes         — Route definitions
    /utils          — Shared utilities
    /config         — Environment and app configuration
    /migrations     — Database migrations
    /tests          — Test files mirroring src structure

Rationale: [why this structure — separation of concerns, convention for the framework, scalability]
```

### ADR-006: Authentication Strategy

```
Decision: [JWT stateless | Session-based | OAuth2 | Auth provider (Auth0, Clerk, etc.)]

Rationale:
  - [why this approach]
  - [token storage strategy — httpOnly cookie, localStorage (with XSS caveat), etc.]
  - [refresh token strategy if applicable]

Trade-offs:
  - [what this gives us vs. what we accept]
```

## Defaults

When project requirements don't specify a preference, the engineer uses these defaults and notes them in the ADR:

- **API:** REST with JSON
- **Database:** Relational (PostgreSQL preferred)
- **Architecture:** Monolith (modular)
- **Auth:** JWT with httpOnly cookie storage
- **Folder structure:** Standard MVC / service-layer pattern for the chosen framework
- **Naming:** snake_case for DB columns, camelCase for API request/response fields, PascalCase for components

Any deviation from defaults requires explicit rationale in the ADR.

## PM Review

The ADR is submitted to the PM for review before the full Implementation Plan is produced. The PM may share the ADR with the client for awareness. Write the "Rationale" sections so a non-technical stakeholder can understand the reasoning even if they skip the technical details.

If the PM or client pushes back on a decision, the engineer addresses the concern with technical justification. If the concern is valid, update the ADR and adjust the approach. The engineer has technical authority but the PM has project authority — if there's a genuine business reason to override a technical preference, document the trade-off and proceed.

FILE:references/backend_spec.md
# Backend Specification Template

This section of the Implementation Plan is written exclusively for the BE Developer. It must be self-contained — the BE dev should be able to complete all backend work using only this section, the DB schema spec, and the cross-cutting concerns section.

## ID Convention

All backend items use the prefix `BE-` followed by a sequential number: BE-001, BE-002, etc.

---

## BE-S1: API Endpoint Definitions

For every endpoint the backend must expose:

```
BE-001: [Descriptive Name]
SRS Requirement(s): [SRS-XXX]
Method: GET | POST | PUT | PATCH | DELETE
Path: /api/v1/[resource]/[path]
Auth Required: Yes | No
Required Role(s): [admin, user, etc. — or "any authenticated"]
Rate Limited: [Yes — X requests per Y seconds | No]

Request:
  Headers:
    Authorization: Bearer <token>
    Content-Type: application/json
  URL Params: [id, slug, etc.]
  Query Params:
    - page (integer, optional, default: 1)
    - limit (integer, optional, default: 20, max: 100)
    - sort (string, optional, values: [created_at, name])
    - [other filters]
  Body:
    {
      "field_name": "type — required | optional — description",
      "field_name": "type — required | optional — description"
    }

Response (200/201):
    {
      "field_name": "type — description",
      "field_name": "type — description"
    }

Error Responses:
    400: { "error": { "code": "VALIDATION_ERROR", "message": "...", "details": { "field": "reason" } } }
    401: { "error": { "code": "UNAUTHORIZED", "message": "Authentication required" } }
    403: { "error": { "code": "FORBIDDEN", "message": "Insufficient permissions" } }
    404: { "error": { "code": "NOT_FOUND", "message": "[Resource] not found" } }
    409: { "error": { "code": "CONFLICT", "message": "..." } }
    500: { "error": { "code": "INTERNAL_ERROR", "message": "An unexpected error occurred" } }
```

Not every endpoint will use every error code — include only the ones that apply.

## BE-S2: Business Logic Rules

For each endpoint or feature that involves business logic beyond simple CRUD:

```
Rule: BE-BL-001 — [descriptive name]
Applies to: [BE-XXX endpoint(s)]
SRS Requirement: [SRS-XXX]
Logic:
  1. [step-by-step description of the business rule]
  2. [next step]
  3. [next step]
Edge Cases:
  - [what happens if X]
  - [what happens if Y]
Validation:
  - [field]: [rule — e.g., must be positive integer, must be unique, must reference existing record]
```

## BE-S3: Service / Controller / Model Structure

Define the code organization:

```
Structure:
  controllers/
    [resource]Controller — Handles HTTP request/response for [resource]
  services/
    [resource]Service — Business logic for [resource]
  models/
    [Resource] — Database model/entity for [resource]
  middleware/
    authMiddleware — Validates token, attaches user to request
    validationMiddleware — Runs request validation schemas
    errorHandler — Catches and formats all errors
  utils/
    [utility files as needed]
```

For each service, note its public methods and what they do:

```
[Resource]Service:
  - create(data) → [Resource] — creates a new record, validates [rules]
  - getById(id) → [Resource] | null — retrieves by primary key
  - list(filters, pagination) → { items: [Resource][], total: number }
  - update(id, data) → [Resource] — partial update, validates [rules]
  - delete(id) → void — [soft delete / hard delete], checks [conditions]
```

## BE-S4: Authentication & Permission Implementation

```
Auth Flow:
  1. Login: POST /api/v1/auth/login → validates credentials → returns token
  2. Token Type: [JWT / session / OAuth token]
  3. Token Expiry: [duration]
  4. Refresh: [how token refresh works, if applicable]
  5. Logout: [what happens server-side — blacklist token, destroy session, etc.]

Permission Checks:
  - Middleware extracts user from token
  - Each endpoint specifies required role(s)
  - Middleware rejects with 403 if role insufficient
  - Resource-level permissions: [e.g., users can only edit their own records unless admin]
```

## BE-S5: Data Validation Rules

For each endpoint's request body, define validation:

```
Endpoint: BE-001
Validation:
  - email: required, valid email format, max 255 chars, unique in users table
  - password: required, min 8 chars, must contain uppercase + number
  - name: required, string, max 100 chars, trimmed
  - role: optional, must be one of ["admin", "user", "viewer"], default "user"
```

Note where validation happens (middleware vs. service layer) and how errors are returned (per the cross-cutting error format).

## BE-S6: Integration Points

For each third-party service the backend communicates with:

```
Integration: [service name]
Purpose: [what it's used for]
Endpoint(s): [API URLs or SDK methods]
Auth: [API key, OAuth, etc.]
Data Sent: [what data goes out]
Data Received: [what comes back]
Error Handling: [what happens if the service is down or returns an error]
Fallback: [retry strategy, graceful degradation, etc.]
```

## BE-S7: Security Checklist

This checklist is mandatory for every backend spec. Complete it before submitting the Implementation Plan.

```
[ ] Input Validation: All user inputs are validated and sanitized before processing
[ ] Authentication: All non-public endpoints require valid authentication
[ ] Authorization: Endpoints enforce role-based access; users cannot access other users' data without explicit permission
[ ] Secrets Management: No hardcoded secrets; all sensitive values come from environment variables
[ ] SQL Injection: All database queries use parameterized queries or ORM methods (never string concatenation)
[ ] Rate Limiting: Authentication endpoints (login, register, password reset) have rate limits
[ ] CORS: CORS is configured to allow only known origins
[ ] Headers: Security headers set (X-Content-Type-Options, X-Frame-Options, etc.)
[ ] Error Messages: Error responses never expose stack traces, internal paths, or database details
[ ] Logging: Sensitive data (passwords, tokens, PII) is never written to logs
[ ] Dependencies: No known critical CVEs in production dependencies
[ ] File Uploads: If applicable — file type validation, size limits, malware considerations
```

If an item does not apply, mark it N/A with a reason.

## BE-S8: Acceptance Criteria

For each endpoint or business logic rule, write acceptance criteria:

```
BE-AC-001 (maps to BE-001, SRS-XXX):
  Given: [precondition]
  When: [API call with specific inputs]
  Then: [expected response status and body]

BE-AC-002 (maps to BE-001, SRS-XXX):
  Given: [precondition — e.g., user is not authenticated]
  When: [API call]
  Then: [401 response]
```

Cover: happy path, authentication failure, authorization failure, validation failure, and any business-logic edge cases. QA uses these directly.

FILE:references/qa_spec.md
# QA Coverage Plan Template

This section of the Implementation Plan is written for the QA Engineer. It defines what to test, expected outcomes, and the definition of "done" for each feature. QA should be able to execute a full test pass using only this section and the acceptance criteria from the FE/BE specs.

## ID Convention

All QA items use the prefix `QA-` followed by a sequential number: QA-001, QA-002, etc.

---

## QA-S1: Test Scenarios by Requirement

For each SRS requirement, define test scenarios:

```
QA-001: [Test Scenario Name]
SRS Requirement: [SRS-XXX]
Spec Reference: [FE-XXX / BE-XXX — which spec section defines this behavior]
Type: Happy Path | Edge Case | Error Case | Security | Performance
Priority: P1 (must pass for release) | P2 (should pass) | P3 (nice to have)

Preconditions:
  - [system state required before test — e.g., "user is logged in as admin"]
  - [test data required — e.g., "at least 3 orders exist in the database"]

Steps:
  1. [action]
  2. [action]
  3. [action]

Expected Result:
  - [what should happen — be specific about UI state, API response, DB state]
  - [what should NOT happen — if relevant]

Acceptance Criteria Reference: [FE-AC-XXX / BE-AC-XXX]
```

### Coverage Requirements

Every SRS requirement must have at minimum:
- 1 happy path scenario (normal use, valid inputs, expected outcome)
- 1 error/edge case scenario (invalid input, unauthorized access, boundary condition)

High-complexity requirements (effort 5+ story points) should have:
- 2+ happy path scenarios covering different valid inputs
- 2+ error scenarios covering different failure modes
- 1 boundary/edge case scenario

## QA-S2: Test Categories

Organize test scenarios into these categories:

### Functional Tests
Tests that verify features work as specified. These are the bulk of the test plan.

### Integration Tests
Tests that verify components work together:
- FE calls BE endpoint and correctly handles the response
- BE writes to DB and returns correct data
- Third-party integrations send/receive data correctly

### Authentication & Authorization Tests
Verify access control:
```
QA-AUTH-001: Unauthenticated access to protected endpoint
  Steps: Call [endpoint] without a token
  Expected: 401 response

QA-AUTH-002: Insufficient role access
  Steps: Call [admin endpoint] with a user-role token
  Expected: 403 response

QA-AUTH-003: Expired token
  Steps: Call [endpoint] with an expired token
  Expected: 401 response, clear error message

QA-AUTH-004: Cross-user data access
  Steps: User A tries to access User B's [resource] by ID
  Expected: 403 or 404 (depending on spec — never return the data)
```

### Data Integrity Tests
Verify the database remains consistent:
- Required fields cannot be null
- Unique constraints are enforced
- Foreign key relationships are maintained
- Cascade deletes work as specified
- Concurrent operations don't corrupt data (where applicable)

### UI State Tests
Verify all component states render correctly:
- Loading state displays properly
- Empty state displays when no data exists
- Error state displays on API failure
- Data renders correctly in normal state
- Forms validate correctly on submit

## QA-S3: Manual vs. Automated Testing

Specify which tests should be manual and which should be automated:

```
Automated (run on every PR):
  - All API endpoint tests (status codes, response shapes, validation errors)
  - Authentication/authorization tests
  - Data integrity tests
  - Unit tests for business logic

Manual (run before release):
  - Full user flow walkthroughs
  - Visual/layout verification across breakpoints
  - Accessibility spot checks
  - Edge cases that require complex state setup
  - Third-party integration verification
```

Adjust this split based on project maturity and tooling availability.

## QA-S4: Test Data Requirements

Define what test data QA needs:

```
Users:
  - Admin account: [email/password — use test credentials]
  - Standard user account: [email/password]
  - User with no data (empty state testing)
  - User with large data set (pagination testing)

[Resource]:
  - Minimum: [N] records for list/pagination testing
  - Edge cases: [record with max-length fields, record with special characters, etc.]
  - Relationships: [records that reference each other for FK/cascade testing]
```

Note whether test data should be seeded automatically or created manually during testing.

## QA-S5: Definition of Done

A feature is considered "done" when:

1. All P1 test scenarios for the feature pass
2. All P2 test scenarios pass (or failures are documented and accepted by PM)
3. No critical or high-severity bugs remain open against the feature
4. The feature matches the acceptance criteria in the FE/BE spec
5. Code has been reviewed (PR approved)
6. No regressions introduced in existing features (regression test pass)

## QA-S6: Bug Report Format

When QA finds a bug, report it in this format:

```
Title: [Short, descriptive — what's wrong, not what was expected]
Severity: Critical | High | Medium | Low
SRS Requirement: [SRS-XXX]
Spec Reference: [FE-XXX / BE-XXX]
Environment: [browser, OS, or API client]

Steps to Reproduce:
  1. [exact steps]
  2. [exact steps]
  3. [exact steps]

Expected: [what should happen per the spec]
Actual: [what actually happens]
Evidence: [screenshot, error log, network trace]

Notes: [any additional context — does it happen consistently? Only on certain data?]
```

Severity guide:
- **Critical:** Feature is completely broken, data loss, security vulnerability. Blocks release.
- **High:** Feature partially broken, workaround exists but is unacceptable. Should block release.
- **Medium:** Feature works but with noticeable issues. Fix before release if time permits.
- **Low:** Cosmetic, minor UX issue. Can be deferred to next cycle.

## QA-S7: PR Review Checklist

When QA reviews a PR before merge:

```
[ ] Code changes match the spec section referenced in the PR description
[ ] No unrelated changes included (scope creep in the PR)
[ ] Error handling is implemented per the cross-cutting concerns spec
[ ] No hardcoded values that should be configurable
[ ] No console.log / debug statements left in production code
[ ] Tests are included for new functionality
[ ] Existing tests still pass
[ ] Acceptance criteria from the spec are met
```

If QA and the dev disagree on a PR issue, escalate to the Project Engineer. The engineer reviews against the Implementation Plan and makes the call.

FILE:references/asana_task_guide.md
# Asana Task Guide — Task Manifest Format

The engineer produces a Task Manifest after the Implementation Plan is finalized. The PM uses this manifest to create the Asana board. The engineer does not create Asana tasks directly — this document is the handoff artifact.

## Task Manifest Structure

The manifest is a structured list organized by role. Each task maps directly to the PM's Asana task format (SRS ID, complexity, effort, acceptance criteria, dependencies).

---

## Task Entry Format

For each task:

```
Task ID: [ROLE]-TASK-[NNN] (e.g., FE-TASK-001, BE-TASK-001, DB-TASK-001, QA-TASK-001)
Title: [concise task title — verb + noun, e.g., "Implement user registration form"]
Assigned Role: FE Dev | BE Dev | DB / BE Dev | QA Engineer
SRS Requirement(s): [SRS-XXX, SRS-YYY]
Spec Section: [FE-001, BE-003, DB-002 — the Implementation Plan section this task implements]
Complexity: Low | Medium | High
Effort: [story points] ([hours] hours)
Sprint/Phase: [which development phase this belongs to]

Description:
  [2-4 sentences describing what this task accomplishes. Reference the spec section for full detail.]

Acceptance Criteria:
  - [ ] [criterion 1 — matches the spec's acceptance criteria]
  - [ ] [criterion 2]
  - [ ] [criterion 3]

Dependencies:
  - Blocked by: [ROLE-TASK-NNN — task(s) that must complete before this one can start]
  - Blocks: [ROLE-TASK-NNN — task(s) that cannot start until this one completes]
```

## Organizing by Role

Group tasks under role headers so the PM can assign them in bulk:

### Database Tasks
DB tasks generally come first — schema must exist before BE can build on it.

### Backend Tasks
BE tasks follow DB tasks. Group by feature area, not by endpoint.

### Frontend Tasks
FE tasks can begin in parallel (scaffolding, routing, static components) but API-dependent work follows BE tasks.

### QA Tasks
QA tasks map to features, not to individual FE/BE tasks. QA tests the integrated feature, not isolated endpoints or components.

### Engineering Tasks
Tasks the engineer owns (rare — usually just "Code Review" and "Produce Implementation Plan"):

```
ENG-TASK-001: Produce Engineering Design & Implementation Plan
Assigned Role: Project Engineer
SRS Requirement(s): All
Complexity: High
Effort: [estimate based on project scope]
Dependencies: Blocked by PM completion of SRS sign-off
```

## Dependency Identification Rules

Use these patterns to identify dependencies (aligned with the PM's dependency framework):

1. **Schema before service:** Any BE task that reads/writes a table depends on the DB task that creates/modifies that table.
2. **Service before consumer:** Any FE task that calls an API endpoint depends on the BE task that implements that endpoint.
3. **Auth before protected:** Any task involving protected resources depends on the auth implementation task.
4. **Core before extension:** Base CRUD depends on table creation. Advanced business logic depends on base CRUD.
5. **Integration before consumption:** Third-party integration setup must precede any task that uses that integration.

## Task Sizing Guidance

Aim for tasks that are completable in 1-3 days. If a task estimates at 8+ story points or 2+ days, consider splitting it:

- Split by sub-feature (e.g., "User CRUD" → "Create user endpoint" + "List users endpoint" + "Update user endpoint")
- Split by layer if a single feature spans FE+BE (each gets its own task)
- Never split artificially — a task should be a coherent unit of work

## Manifest Summary

At the end of the manifest, provide a summary the PM can use for sprint planning:

```
Total Tasks: [count]
  FE: [count] ([total story points] points, [total hours] hours)
  BE: [count] ([total story points] points, [total hours] hours)
  DB: [count] ([total story points] points, [total hours] hours)
  QA: [count] ([total story points] points, [total hours] hours)
  ENG: [count] ([total story points] points, [total hours] hours)

Critical Path: [list the sequence of dependent tasks that determines minimum project duration]
Parallelizable: [which role tracks can run simultaneously]
Recommended Sprint Breakdown: [if applicable — which tasks fit in sprint 1 vs. 2 vs. 3]
```

---
name: dev_project_manager
description: "Comprehensive AI Project Manager skill for software development. Use this skill whenever the PM agent needs to: engage with clients about new or existing software requirements, conduct requirements elicitation, request and review technical assessments from engineers, create or update Software Requirements Specifications (SRS) documents, classify change impacts, estimate effort/cost/AI-vs-human comparisons, manage scope creep and change requests, build or update Asana project boards and tasks, provide client status updates, review engineering implementation plans against SRS, render UI mockup comparisons, or coordinate between clients and engineering agents. Also handles the Asana heartbeat queue check — checking the PM Queue for each project and sending sessions_send nudges to the appropriate agents when work is ready. Triggers on any mention of: client requirements, SRS, requirements gathering, project status, stakeholder updates, engineering review, change requests, scope management, effort estimation, cost analysis, implementation plan review, UI comparison, project kickoff, or heartbeat queue check. This skill handles all PM communication protocols, templates, and decision frameworks. It does NOT make Asana API calls directly (requires a separately installed Asana skill), does NOT send email directly (requires a separately installed Email skill), and does NOT interact with code repositories."
---

# Dev Project Manager Skill

## Credential Trust Model

**This skill does not access, store, request, or transmit any credentials or secrets.**

All external API calls — Asana task management, email delivery — are performed exclusively by separately installed dependency skills (an Asana skill and an Email skill). Those skills hold and use their own credentials, supplied by the agent operator through the agent runtime environment. This skill provides workflow instructions only. It never reads environment variables, never receives token values, and never calls external endpoints itself.

The env var names referenced in this skill (such as Asana PAT and Dev Manager email vars) are labels that identify which credential the dependency skills should use — this skill never sees the values behind those names.

## Agent Workspace Files

This skill references two operator-provisioned agent workspace files:

- **USER.md** — contains the agent's active project list, Asana project GIDs, repo URLs, and team agent IDs. This file is created and maintained by the agent operator (or by the build-development-team skill during setup). This skill reads guidance from it at runtime but does not create or modify it.
- **TOOLS.md** — contains the agent's available tools and which credential labels each dependency skill uses. Created and maintained by the operator. This skill does not create or modify it.

Both files live in the agent's workspace directory, managed by the OpenClaw operator. They contain no secret values — only project identifiers, GIDs, repo URLs, and env var name references.

## Heartbeat Scheduling

The 30-minute heartbeat is scheduled and triggered by the OpenClaw platform, not by this skill. This skill defines what the agent should do when a heartbeat session starts — it does not self-invoke, does not set timers, and does not persist between sessions. The operator configures heartbeat frequency in the OpenClaw agent configuration. Each heartbeat run is an isolated session.

## Dependency Skills Required

This skill requires the following separately installed skills to function. Install these before using this skill:

| Dependency | Purpose | Credential it uses |
|---|---|---|
| Asana skill | All Asana board and task operations | Asana PAT — held by the Asana skill, supplied by operator |
| Email skill | Dev Manager completion alerts only | Email credentials — held by the Email skill, supplied by operator |

GitHub access is not required for this agent — it does not interact with code repositories.

---

## Role Definition

You are the Project Manager (PM) agent. You bridge the client and the engineering agent. You translate client needs into structured requirements, coordinate technical assessments, produce client-facing documents, and maintain project visibility through Asana. You do not write code, design architecture, or interact directly with dev/QA agents — the engineer handles all technical planning and agent coordination.

You are dedicated to a specific project (defined in your USER.md). You communicate with the shared technical agents (engineer, dev-fe, dev-be, qa, n8n_engineer) about that project only. Every `sessions_send` message you send must include your project's Asana GID so recipients know which project they're acting on.

**Your communication style adapts by audience:**
- **To clients:** Plain language, no jargon, focus on what changes mean for their product and users.
- **To the engineer:** Semi-technical, precise, structured. Reference specific features, screens, data flows, and integration points.

---

## Asana Heartbeat Protocol

When a heartbeat session starts (triggered by the OpenClaw platform on the operator-configured schedule), perform the following checks for your project using the installed Asana skill:

### Heartbeat Steps

1. **Check PM Queue** — look for tasks moved here by the engineer or devs awaiting PM action.
2. **Check all columns** — scan for tasks stuck in any column for more than 2 hours without movement, and tasks marked Blocked.
3. Process any tasks found in PM Queue per the workflows below.
4. For stuck or blocked tasks: `sessions_send` nudge to the task owner with project GID + task name + task URL.

### Queue Check — Nothing Found

If PM Queue is empty and no tasks are stuck or blocked, the heartbeat session ends. No action taken.

---

## sessions_send Protocol

Every `sessions_send` message must include:
- Your project's Asana GID
- The task name
- The task URL

**Never reference work from another project in a message.**

sessions_send is an intra-instance OpenClaw communication tool. Messages are routed only to named agents within the same OpenClaw instance. No external network calls are made by sessions_send.

**Allowed send targets:** engineer, dev-fe, dev-be, qa, n8n_engineer

---

## Asana Board Structure

Every project board you manage must have these columns:

| Column | Purpose | Who Moves Tasks Here |
|---|---|---|
| Backlog | Work not yet started | PM |
| PM Queue | Work awaiting PM review or action | Engineer, Devs |
| Engineer Queue | Tasks for the engineer | PM |
| Frontend Dev Queue | Tasks for dev-fe | PM |
| Backend Dev Queue | Tasks for dev-be | PM |
| QA Queue | Completed dev work awaiting QA | Devs |
| N8N Engineer Queue | Automation tasks (if project uses it) | PM |
| In Progress | Actively being worked | Devs (self-move) |
| QA Review | Under active QA review | QA |
| Complete | Done and QA-approved | QA |
| Blocked | Cannot proceed | Anyone |

---

## Core Workflow

### Phase 1 — Requirements Elicitation

When a client comes with a request, understand what they actually need before scoping anything.

**For existing software changes:** Issue a Software Audit Request to the engineer first. Using the Asana skill, create a task "Software Audit: [feature area]" in the Engineer Queue. Send a `sessions_send` nudge to engineer: project GID + task name + task URL. Wait for engineer to move it to PM Queue before continuing.

**For 0-to-1 builds:** Skip the audit and go straight to discovery.

**Elicitation protocol** (see `references/requirements_elicitation.md`):
1. Problem-first discovery — "What problem are you trying to solve?"
2. Current-state walkthrough for existing software
3. Gap identification
4. Implicit requirements probe
5. Priority classification (MoSCoW)
6. Conflict detection

After elicitation, produce a Requirements Summary (see `references/templates.md`) and present to client for confirmation.

### Phase 2 — Technical Assessment Coordination

Once requirements are confirmed:
1. Using the Asana skill, create task "Technical Assessment: [feature name]" in Engineer Queue. Attach requirements summary MD.
2. `sessions_send` nudge to engineer: project GID + "requirements locked — assessment requested" + task URL.
3. Wait for engineer to return assessment to PM Queue.
4. Review for completeness — does it address every requirement?
5. Translate effort estimates into client-friendly language (see `references/estimation.md`).
6. Present to client using Client Assessment Summary template (`references/templates.md`).
7. Iterate with client until scope is agreed.

### Phase 3 — SRS Authoring

Once client agrees on scope, produce the Software Requirements Specification (SRS). Read `references/srs_standard.md` for the complete template.

Key SRS principles:
- Every requirement gets a unique, never-reused ID (FR-XXX, NFR-XXX)
- Every functional requirement has testable acceptance criteria
- Include cost/effort analysis with human vs. AI comparison
- Version-controlled with a change log
- Client sign-off section

**SRS Review Cycle:**
1. Produce draft SRS → present to client
2. Client reviews and requests changes
3. Cosmetic/document-level changes → PM makes directly
4. New scope or technical concerns → loop engineer via new Technical Assessment task
5. Update SRS, increment version, log changes
6. Repeat until signed off

### Phase 4 — Engineering Plan Review

After SRS sign-off:
1. Using the Asana skill, create task "Implementation Plan: [project/feature]" in Engineer Queue. Attach signed SRS MD.
2. `sessions_send` nudge to engineer: project GID + "SRS signed off — plan requested" + task URL.
3. Wait for engineer to deliver plan to PM Queue.
4. Review every SRS requirement ID against the plan — is each addressed?
5. If gaps found: create task "Plan Review — Gaps: [description]" in Engineer Queue, attach gap notice MD. `sessions_send` nudge to engineer.
6. Iterate until plan fully covers SRS.

### Phase 5 — Asana Task Setup

Once PM and engineer agree the plan covers the SRS:
1. Review the Task Manifest from the engineer.
2. Using the Asana skill, create one task per manifest entry.
3. Assign each task to the correct agent queue column.
4. Set branch names in task descriptions (format: `feature/[project]-[description]` or `fix/[project]-[bug-id]`).
5. Set dependencies between tasks per the manifest.
6. `sessions_send` nudge to each assigned agent: project GID + task name + task URL.

**Standard Asana task description format:**
```
[SRS Requirement: FR-XXX]
[Complexity: Low/Medium/High/Very High]
[AI Success Probability: XX%]
Branch: feature/[project]-[slug]
Spec Section: [FE-XXX / BE-XXX]

DESCRIPTION:
[What this task delivers]

ACCEPTANCE CRITERIA:
- [ ] Given X, when Y, then Z

EFFORT ESTIMATES:
- Human estimate: [X] hours
- AI estimate: [X] hours

DEPENDENCIES:
- Blocked by: [Task name/ID]
- Blocks: [Task name/ID]
```

### Phase 6 — Ongoing Monitoring

At every heartbeat (OpenClaw platform-triggered):
- Check for tasks stuck in any column for more than 2 hours
- Check for tasks marked Blocked
- For each: `sessions_send` nudge to the task owner with project GID + task name + task URL

When client asks for status:
1. Using the Asana skill, pull current task states.
2. Produce Status Update using template in `references/templates.md`.
3. Report: total tasks, tasks by column, blocked tasks, tasks with no movement.

### Phase 7 — Completion

When all tasks in an implementation plan are QA-approved and Complete:
1. Verify QA has marked all tasks Complete via the Asana skill.
2. Using the Email skill, send completion alert to the Dev Manager email address (from TOOLS.md — the Email skill holds the actual credential).
   Format: "Project: [project name] | Phase: [impl plan name] | Status: COMPLETE | Tasks: X/X"
3. If the Email skill is not installed or not configured, log completion in the Asana project description instead.

### Change Request Protocol

Once an SRS is signed off and work has begun, any new client requests are formal change requests. Read `references/change_management.md` for full protocol.

---

## Escalation Protocol

When monitoring the board, escalate to engineer when:
- A task has been In Progress significantly longer than its estimate with no comment update
- A task moves back from QA to In Progress more than twice
- Multiple tasks are blocked by a single dependency that isn't progressing

Escalation:
1. `sessions_send` to engineer: project GID + stuck task name + task URL + description of concern.
2. If engineer identifies a technical problem, assess timeline impact.
3. If significant, proactively update the client.

---

## Reference Files

| File | When to Read |
|------|-------------|
| `references/srs_standard.md` | When authoring or updating an SRS |
| `references/templates.md` | When producing any client-facing or engineer-facing document |
| `references/engineer_protocols.md` | When requesting work from the engineer |
| `references/requirements_elicitation.md` | During Phase 1 discovery |
| `references/estimation.md` | When translating estimates for clients |
| `references/change_management.md` | When handling post-SRS client requests |

FILE:_meta.json
{"ownerId":"kn77nfg6wv2expv6qs7k17dfqs83zp59","slug":"dev-project-manager","version":"1.2.0","publishedAt":1745648400000}

FILE:references/templates.md
# PM Communication Templates

## Requirements Summary Template

Use this after Phase 1 elicitation to confirm understanding with the client before proceeding to technical assessment.

```
REQUIREMENTS SUMMARY
[Project Name]
Date: [YYYY-MM-DD]
Prepared for: [Client Name]

═══════════════════════════════════════

BACKGROUND
[2-3 sentences: Why is the client requesting this? What business
problem or opportunity is driving the request?]

CURRENT STATE (for existing software)
[Brief summary of how the relevant areas of the software work
today, based on the engineer's software audit. Written so the
client can confirm accuracy.]

REQUESTED CHANGES / NEW FEATURES

1. [Feature/Change Name]
   - What: [Plain-language description of what the client wants]
   - Why: [Business reason / problem it solves]
   - Priority: [Must-Have / Should-Have / Nice-to-Have]
   - Current behavior (if applicable): [How it works now]
   - Desired behavior: [How it should work after]
   - Notes: [Any constraints, preferences, or specifics mentioned]

2. [Feature/Change Name]
   ...

IDENTIFIED CONCERNS
[Anything you spotted during elicitation that needs attention:
conflicts between requirements, implicit dependencies, areas
where the client's request may affect other parts of the system]

ITEMS EXPLICITLY OUT OF SCOPE
[Things discussed and agreed to NOT include]

OPEN QUESTIONS
[Anything you still need clarity on]

NEXT STEPS
- Client confirms this summary is accurate
- PM sends to engineering for technical assessment
- PM returns to client with effort estimates, timeline,
  and any technical concerns

═══════════════════════════════════════
```

## Client Assessment Summary Template

Use this to present the engineer's technical assessment back to the client in business-friendly language.

```
ASSESSMENT SUMMARY
[Project Name]
Date: [YYYY-MM-DD]

═══════════════════════════════════════

OVERVIEW
[1-2 paragraphs: Overall assessment of the requested changes.
Are they feasible? What's the general level of effort? Any
major considerations the client should know upfront?]

FEATURE-BY-FEATURE ASSESSMENT

1. [Feature Name]
   Status: [Feasible / Feasible with considerations / Needs discussion]
   Estimated Effort: [Low / Medium / High — with plain-language
     explanation like "roughly 1-2 weeks of development work"]
   Impact: [What parts of the application are affected]
   What you'll see: [Description of the end result from the
     user's perspective — what the interface/workflow will look
     like after implementation]
   Compared to now: [How this differs from current behavior]
   Considerations: [Any risks, trade-offs, or things the client
     should weigh in on]
   AI Delivery Outlook:
   - Estimated AI delivery time: [X hours/days]
   - Complexity: [Trivial/Low/Medium/High/Very High]
   - Confidence of successful AI delivery: [XX%]
   - Estimated AI cost: $[X.XX]
   - Comparable human cost: $[X.XX] ([X] hours at standard rates)
   - Estimated savings: $[X.XX] ([X]%)

2. [Feature Name]
   ...

OVERALL PROJECT ESTIMATES

   | Metric | Human (Traditional) | AI-Assisted |
   |--------|-------------------|-------------|
   | Total effort | [X] hours | [X] hours |
   | Estimated duration | [X] weeks | [X] days |
   | Total cost | $[X] | $[X] |
   | Savings | — | $[X] ([X]%) |

RECOMMENDATIONS
[Your PM-level recommendations: suggested phasing, things to
prioritize, items to defer, risks to mitigate]

OPEN ITEMS REQUIRING YOUR INPUT
[Decisions the client needs to make before we proceed]

NEXT STEPS
- Client reviews and provides feedback
- PM and client iterate until scope is agreed
- PM produces formal SRS for final review and sign-off

═══════════════════════════════════════
```

## Status Update Template

Use this when the client requests a project status update. Pull data from Asana task states.

```
PROJECT STATUS UPDATE
[Project Name]
Date: [YYYY-MM-DD]
Reporting Period: [Date range]
Overall Status: [On Track / At Risk / Blocked / Ahead of Schedule]

═══════════════════════════════════════

SUMMARY
[2-3 sentences: Where the project stands at a high level.
Lead with the most important information.]

PROGRESS OVERVIEW

   Completed:    [X] of [Y] tasks ([Z]%)
   In Progress:  [X] tasks
   In QA:        [X] tasks
   Not Started:  [X] tasks (Features: [X], Bugs: [X])
   Blocked:      [X] tasks

COMPLETED SINCE LAST UPDATE
- [Task/Feature name]: [Brief description of what was delivered]
- [Task/Feature name]: [Brief description]

CURRENTLY IN PROGRESS
- [Task/Feature name]: [What's happening, expected completion]
- [Task/Feature name]: [What's happening, expected completion]

IN QA / TESTING
- [Task/Feature name]: [Testing status]

RISKS & BLOCKERS
[If none, say "No current risks or blockers identified."]
- [Risk/Blocker]: [Description, impact, what's being done]

UPCOMING
[What's planned for the next period]

TIMELINE ASSESSMENT
[Are we on track relative to original estimates? If not, what
changed and what's the revised expectation?]

DECISIONS NEEDED FROM YOU
[Any items requiring client input or approval]

═══════════════════════════════════════
```

## Change Request Log Template

Use this to track change requests that come in after SRS sign-off.

```
CHANGE REQUEST
CR-[XXX]
Date Submitted: [YYYY-MM-DD]
Requested By: [Client name/contact]
Project: [Project Name]
SRS Version at Time of Request: [X.X]

═══════════════════════════════════════

DESCRIPTION OF REQUESTED CHANGE
[What the client is asking for, in their words]

PM CLASSIFICATION
- Impact Level: [1-5 per the Change Impact Classification]
- Scope Assessment: [Within existing scope / New scope]
- Affected SRS Requirements: [List any existing FR/NFR IDs affected]

PRELIMINARY IMPACT ASSESSMENT
(For Level 1-2, PM completes this directly. For Level 3+,
this section is completed after engineer consultation.)
- Affected areas: [What parts of the system are impacted]
- Effort estimate: [Hours / complexity]
- Timeline impact: [None / Delays completion by X days / etc.]
- Cost impact: [Additional cost estimate if applicable]
- Risk: [Any risks introduced by this change]

RECOMMENDATION
[PM's recommendation: approve, defer, modify, or decline — with
reasoning]

DECISION
- [ ] Approved — proceed with SRS amendment
- [ ] Approved with modifications — [describe modifications]
- [ ] Deferred to future phase
- [ ] Declined — [reason]

Decision Date: [YYYY-MM-DD]
Decided By: [Client name]

IF APPROVED:
- New SRS Version: [X.X]
- New/Modified Asana Tasks: [List task IDs or names]
- Revised Timeline Impact: [Description]
- Additional Cost: $[X.XX]

═══════════════════════════════════════
```

## Project Kickoff Checklist

Use this when starting a new project to ensure nothing is missed.

```
PROJECT KICKOFF CHECKLIST
[Project Name]
Date: [YYYY-MM-DD]

PRE-ENGAGEMENT
- [ ] Client contact information confirmed
- [ ] Project type identified: [New Build / Enhancement / Bug Fix]
- [ ] Existing software? If yes, request Software Audit from engineer
- [ ] Client provided any mockups, documents, or reference materials?

REQUIREMENTS PHASE
- [ ] Discovery conversation completed
- [ ] Requirements Summary produced and sent to client
- [ ] Client confirmed Requirements Summary
- [ ] Technical Assessment requested from engineer
- [ ] Technical Assessment received and reviewed for completeness
- [ ] Client Assessment Summary produced and sent to client
- [ ] Client feedback incorporated
- [ ] UI comparisons requested/reviewed (if applicable)
- [ ] Scope agreed upon by client

SRS PHASE
- [ ] SRS draft produced
- [ ] SRS sent to client as PDF
- [ ] Client review feedback received
- [ ] All revisions completed
- [ ] SRS formally signed off (version and date recorded)

ENGINEERING HANDOFF
- [ ] Engineer produced Implementation Plan
- [ ] PM verified plan covers all SRS requirements
- [ ] Gaps addressed and plan finalized

ASANA SETUP
- [ ] Checked for existing project board
- [ ] Board created/verified with correct columns
- [ ] All tasks created with descriptions, estimates, assignments
- [ ] Dependencies set between tasks
- [ ] Board is ready for agents to begin work

ONGOING
- [ ] Status update cadence agreed with client
- [ ] Change request protocol communicated to client
```

FILE:references/estimation.md
# Estimation & Effort Communication Guide

## Purpose

This reference covers how to translate engineering effort estimates into client-friendly language, how to calculate AI vs. human cost comparisons, and how to assess AI agent success probability for tasks.

## Translating Engineer Estimates for Clients

Engineers think in story points, components, and technical complexity. Clients think in time, money, and impact. Your job is to translate without losing accuracy.

### Effort-to-Timeline Translation

Never give the client a single number. Always give a range, because estimates are inherently uncertain.

| Engineer Says | You Tell the Client |
|--------------|-------------------|
| "Trivial, maybe 1 story point" | "This is a quick change — we're looking at a few hours of work, likely completed within a day." |
| "Small, 2-3 story points" | "A straightforward change that should take 1-3 days to implement and verify." |
| "Medium, 5-8 story points" | "This is a moderate piece of work — roughly 1-2 weeks including development and testing." |
| "Large, 13+ story points" | "This is a significant effort — we're looking at 2-4 weeks of dedicated development, possibly more depending on what we find during implementation." |
| "Epic, needs to be broken down" | "This is a major initiative that we'd recommend breaking into phases. Each phase would have its own timeline once we've mapped out the approach." |

### Language Rules for Client Communication

**Do say:**
- "Roughly," "approximately," "in the range of"
- "Based on our assessment, we estimate..."
- "This could take longer if [specific condition]"

**Do not say:**
- Story points, sprints, velocity (these are internal terms)
- Exact hours with false precision ("37.5 hours")
- "Easy" or "simple" (reduces perceived value and creates accountability risk)
- "Guaranteed by [date]" (estimates are estimates, not commitments)

### Surfacing Risk in Estimates

When an estimate has high uncertainty, communicate it clearly:

"We estimate this at roughly 2 weeks of development work. However, the [specific area] involves [specific risk — e.g., integration with a third-party API we haven't tested], which could add up to an additional week if we encounter complications. We'll have a clearer picture after the first few days of implementation."

## AI vs. Human Cost Model

Every SRS includes a cost comparison section. Here's how to build it.

### Standard Human Bill Rates

Use these default rates unless the client has established different rates. Rates should reflect typical market rates for qualified professionals. Adjust per region/context as needed.

| Role | Default Hourly Rate |
|------|-------------------|
| Senior Full-Stack Developer | $150/hr |
| Frontend Developer | $125/hr |
| Backend Developer | $135/hr |
| QA Engineer | $100/hr |
| DevOps / Infrastructure | $140/hr |
| Database Administrator | $130/hr |
| UI/UX Designer | $120/hr |
| Project Manager (human) | $130/hr |

### Human Effort Estimation by Complexity

These are baseline estimates for a single requirement or task. Actual hours come from the engineer's assessment — these are sanity-check ranges.

| Complexity | Typical Human Hours (per requirement) | Roles Typically Involved |
|-----------|---------------------------------------|------------------------|
| Trivial | 1-4 hours | 1 developer |
| Low | 4-16 hours | 1-2 developers |
| Medium | 16-40 hours | 2-3 developers + QA |
| High | 40-120 hours | 3-4 developers + QA + DevOps |
| Very High | 120-320 hours | Full team, multi-sprint |

### AI Agent Effort Estimation

AI agents work faster but their reliability varies by complexity. Use this model:

| Complexity | AI Estimated Time | Time Reduction vs Human | Notes |
|-----------|------------------|----------------------|-------|
| Trivial | 5-15 minutes | ~95-98% faster | Automated changes, config updates |
| Low | 15 min - 2 hours | ~90-95% faster | Straightforward component work |
| Medium | 2-8 hours | ~80-90% faster | Multi-component, moderate logic |
| High | 8-40 hours | ~60-80% faster | Complex logic, needs iteration |
| Very High | 40-100+ hours | ~40-70% faster | May need human oversight/intervention |

### AI Cost Calculation Formula

AI costs are estimated based on token consumption, which correlates with task complexity.

**Base assumptions:**
- Input token cost: $5.00 per million tokens
- Output token cost: $15.00 per million tokens
- Average input-to-output ratio: 3:1 (AI reads more than it writes)

**Token consumption estimates by complexity:**

| Complexity | Estimated Input Tokens | Estimated Output Tokens | Estimated Iterations | Total Cost |
|-----------|----------------------|----------------------|--------------------|-----------| 
| Trivial | 10,000 | 3,000 | 1-2 | $0.10 - $0.20 |
| Low | 50,000 | 15,000 | 2-3 | $0.50 - $1.00 |
| Medium | 200,000 | 60,000 | 3-5 | $2.00 - $5.00 |
| High | 800,000 | 250,000 | 5-10 | $8.00 - $25.00 |
| Very High | 3,000,000 | 1,000,000 | 10-20 | $30.00 - $100.00 |

**Formula per task:**
```
AI Cost = (input_tokens × $5.00 / 1,000,000) + (output_tokens × $15.00 / 1,000,000)
Total AI Cost = AI Cost × estimated_iterations
```

**Note:** These are estimates. Actual token consumption varies based on codebase size, context needed, and number of revision cycles. When presenting to clients, use the range and note that actual costs may vary.

### Savings Calculation

```
Human Cost = sum of (hours_per_role × hourly_rate) for each role involved
AI Cost = calculated per formula above
Savings = Human Cost - AI Cost
Savings Percentage = (Savings / Human Cost) × 100
```

### Risk-Adjusted Cost

For tasks where AI success probability is below 85%, calculate a risk-adjusted cost that accounts for potential fallback to human work:

```
Risk-Adjusted AI Cost = (AI Cost × Success%) + (Human Cost × (1 - Success%))
Risk-Adjusted Savings = Human Cost - Risk-Adjusted AI Cost
```

Present both the optimistic (straight AI) and risk-adjusted numbers in the SRS.

## AI Success Probability Framework

Every task in the SRS gets an AI success probability rating. This represents the PM's and engineer's joint assessment of how likely the AI agents are to successfully implement the task without significant human intervention.

### Probability Guidelines

| Probability | Criteria | Examples |
|------------|---------|---------|
| 95-99% | Trivial or highly repetitive tasks with clear patterns. Never say 100%. | Copy changes, config updates, simple CRUD, CSS adjustments, adding a standard form field |
| 85-94% | Straightforward tasks with clear requirements and established patterns | New component following existing patterns, standard API endpoint, basic validation logic |
| 70-84% | Moderate complexity with some ambiguity or multi-step coordination | Complex form with conditional logic, multi-table query changes, integration with familiar APIs |
| 50-69% | High complexity, novel patterns, or significant uncertainty | New architectural patterns, complex business rules, unfamiliar third-party integrations |
| Below 50% | Very high complexity, cutting-edge requirements, or high ambiguity | Novel algorithms, complex migrations with data transformation, real-time system changes, tasks requiring extensive domain knowledge not in codebase |

### Factors That Lower Probability

- Ambiguous requirements (even after elicitation)
- Complex state management across multiple components
- Integration with poorly documented third-party services
- Database migration with data transformation
- Requirements that cross multiple system boundaries
- Novel UI interactions without existing patterns to follow
- Performance optimization without clear benchmarks

### Factors That Raise Probability

- Clear, testable acceptance criteria
- Existing patterns in the codebase to follow
- Well-documented APIs and services
- Isolated changes that don't affect other components
- Standard CRUD operations
- Changes matching common software patterns

### Presenting Probability to Clients

Frame it positively and practically:

"For this feature, we rate our AI development confidence at 85%. That means we expect the automated development process to handle this smoothly, with a small chance we'll need some additional iteration. The cost estimate already accounts for this — we've included a buffer."

For lower-confidence items:

"This feature has some complexity that makes AI-only delivery less certain — we rate it at 65%. We've included a risk-adjusted cost estimate that accounts for the possibility that some portions may need additional work cycles. Even at this confidence level, the cost savings compared to traditional development are significant."

**Never present probability as a quality indicator.** A 65% success probability doesn't mean the end result will be 65% as good — it means there's a 35% chance the agents need more iterations or human guidance to get it right. The delivered quality standard is the same regardless.

## Presenting Estimates in Context

When you assemble the full cost/effort section of the SRS or Client Assessment Summary, organize the information in layers:

1. **Summary first** — Total project cost comparison, total savings, overall timeline comparison
2. **Feature-by-feature** — Individual breakdowns so the client can see where the value is
3. **Risk-adjusted view** — Show what happens if the lower-confidence items need extra work
4. **Methodology note** — Brief explanation of how estimates were derived (human rates based on market standards, AI costs based on computational resources consumed)

This lets the client see the forest before the trees, and gives them the detail they need to make informed decisions.

FILE:references/srs_standard.md
# SRS Standard — Software Requirements Specification

## Purpose

The SRS is the formal agreement between the client and the development team on what will be built. It is a client-facing document — written in accessible, non-technical language while being precise enough for engineering to implement against. Think of it as "what the software will do and how the user will experience it" — not "how the software will be built internally."

## Document Conventions

- **Version numbering**: Start at 1.0 for first client-ready draft. Minor revisions (typo fixes, clarifications) increment the decimal: 1.1, 1.2. Major scope changes increment the whole number: 2.0, 3.0.
- **Requirement IDs**: Every requirement gets a unique, never-reused ID. Functional requirements use `FR-XXX`, non-functional use `NFR-XXX`, interface requirements use `IR-XXX`.
- **Priority tags**: Every requirement is tagged `[Must-Have]`, `[Should-Have]`, `[Nice-to-Have]`, or `[Out-of-Scope]`.
- **Language**: Use "The system shall..." for mandatory requirements, "The system should..." for recommended, and "The system may..." for optional.

## SRS Template

```
SOFTWARE REQUIREMENTS SPECIFICATION
[Project Name]

Version: [X.X]
Date: [YYYY-MM-DD]
Prepared by: [PM Agent Name/ID]
Client: [Client Name/Organization]
Status: [Draft | In Review | Approved | Amended]

═══════════════════════════════════════════════════

TABLE OF CONTENTS
1. Introduction
2. Overall Description
3. System Features & Functional Requirements
4. Interface Requirements
5. Non-Functional Requirements
6. Cost & Effort Analysis
7. Assumptions & Dependencies
8. Out of Scope
9. Open Questions
10. Acceptance Criteria Summary
11. Change Log
12. Sign-Off

═══════════════════════════════════════════════════

1. INTRODUCTION

1.1 Purpose
   Describe what this document covers and its intended audience.
   Example: "This document defines the software requirements for
   the redesign of the Acme Corp customer portal's reporting
   module. It is intended for review and approval by the Acme
   Corp product team and will serve as the basis for engineering
   implementation."

1.2 Scope
   High-level description of what the project will deliver and
   what business goals it supports. Keep it to 2-3 paragraphs.

1.3 Definitions & Acronyms
   Define any terms that may be ambiguous. Include business terms
   the client uses that have specific meaning in this project.

1.4 References
   Link to related documents: existing documentation, mockups,
   previous SRS versions, relevant third-party specs.

═══════════════════════════════════════════════════

2. OVERALL DESCRIPTION

2.1 Product Perspective
   How does this software/feature fit into the larger ecosystem?
   Is it a standalone system, an update to an existing one, or a
   new module within a larger platform? For existing software,
   describe the current state and what is changing.

2.2 User Classes and Characteristics
   Who uses this software? Describe each user type:
   - Role/title
   - What they use the software for
   - Technical comfort level
   - Frequency of use

2.3 Operating Environment
   Where does the software run? (Web browser, mobile, desktop,
   specific OS requirements, integrations with other systems)

2.4 Constraints
   Business rules, regulatory requirements, technology
   constraints, timeline constraints that shape the solution.

═══════════════════════════════════════════════════

3. SYSTEM FEATURES & FUNCTIONAL REQUIREMENTS

   Organize by feature area. Each feature gets a subsection.

   3.X [Feature Name]

   3.X.1 Description
         What this feature does in plain language. Include the
         business value — why does this feature matter?

   3.X.2 Current State (for existing software changes)
         How this area works today. What the user currently sees
         and can do. Be specific enough that the client can
         confirm "yes, that's how it works now."

   3.X.3 Proposed Changes
         What will be different. Describe from the user's
         perspective: what they'll see, what they'll be able to
         do, how the workflow changes.

   3.X.4 Requirements

         FR-XXX: [Requirement title] [Must-Have]
         The system shall [specific behavior].
         Acceptance Criteria:
         - Given [context], when [action], then [expected result]
         - Given [context], when [action], then [expected result]

         FR-XXX: [Requirement title] [Should-Have]
         The system should [specific behavior].
         Acceptance Criteria:
         - Given [context], when [action], then [expected result]

   3.X.5 Interface Changes (if applicable)
         Description of visual/UI changes. Reference mockups
         if provided. Describe in terms of what the user sees
         and interacts with, not implementation details.

═══════════════════════════════════════════════════

4. INTERFACE REQUIREMENTS

4.1 User Interface Requirements
   General UI standards, accessibility requirements, responsive
   design expectations, branding guidelines.

4.2 External Interface Requirements
   Integrations with other systems, APIs, data feeds, third-
   party services that the software must interact with.

4.3 Data Migration Requirements (if applicable)
   What existing data needs to be preserved, transformed, or
   migrated as part of this project.

═══════════════════════════════════════════════════

5. NON-FUNCTIONAL REQUIREMENTS

   NFR-XXX: [Performance] [Must-Have]
   Example: "Page load times shall not exceed 2 seconds under
   normal load conditions."

   NFR-XXX: [Security] [Must-Have]
   Example: "All user data shall be encrypted in transit and
   at rest."

   NFR-XXX: [Availability] [Should-Have]
   Example: "The system should maintain 99.9% uptime."

   Common categories to address:
   - Performance / response times
   - Security & data protection
   - Scalability
   - Availability / uptime
   - Browser / device compatibility
   - Accessibility (WCAG compliance level)
   - Data retention / backup

═══════════════════════════════════════════════════

6. COST & EFFORT ANALYSIS

6.1 Effort Summary Table

   | Req ID | Feature | Complexity | Human Hours | Human Cost | AI Hours | AI Cost | AI Success % | Savings |
   |--------|---------|------------|-------------|------------|----------|---------|-------------|---------|
   | FR-001 | ...     | Low        | 8h          | $XXX       | 0.5h     | $X.XX   | 97%         | $XXX    |
   | FR-002 | ...     | Medium     | 24h         | $XXX       | 2h       | $X.XX   | 90%         | $XXX    |
   | FR-003 | ...     | High       | 80h         | $XXX       | 8h       | $X.XX   | 75%         | $XXX    |

6.2 Complexity Definitions
   - Trivial: Cosmetic changes, copy updates, config changes
   - Low: Single-component changes, straightforward logic
   - Medium: Multi-component changes, moderate logic, some
     integration work
   - High: Cross-cutting changes, complex business logic, data
     model changes, new integrations
   - Very High: Architectural changes, new subsystems, complex
     migration, novel technical challenges

6.3 Human Cost Basis
   Provide the standard bill rates used for comparison:
   - Senior Developer: $[X]/hr
   - Frontend Developer: $[X]/hr
   - Backend Developer: $[X]/hr
   - QA Engineer: $[X]/hr
   - DevOps/Infrastructure: $[X]/hr

6.4 AI Cost Basis
   Calculated using estimated token consumption per task
   complexity. See the estimation reference for the formula.

6.5 Total Project Summary
   - Total human effort estimate: [X] hours / $[X]
   - Total AI effort estimate: [X] hours / $[X]
   - Estimated client savings: $[X] ([X]%)
   - Weighted average AI success probability: [X]%

6.6 Risk-Adjusted Estimates
   For tasks with AI success probability below 85%, include
   a fallback estimate that accounts for potential rework or
   human intervention.

═══════════════════════════════════════════════════

7. ASSUMPTIONS & DEPENDENCIES

   List anything the project relies on that isn't guaranteed:
   - Client will provide [X] by [date]
   - Existing API [X] will remain stable
   - Third-party service [X] will be available
   - Client has [X] environment/access available for testing

═══════════════════════════════════════════════════

8. OUT OF SCOPE

   Explicitly list things that were discussed but are NOT
   included in this project. This prevents scope disputes later.
   For each item, note why it's excluded and whether it's
   planned for a future phase.

═══════════════════════════════════════════════════

9. OPEN QUESTIONS

   Any unresolved items that need client or stakeholder input
   before implementation can proceed. Each question should note
   who needs to answer it and what the impact of not answering
   it would be.

   | # | Question | Owner | Impact if Unresolved | Status |
   |---|----------|-------|---------------------|--------|

═══════════════════════════════════════════════════

10. ACCEPTANCE CRITERIA SUMMARY

   A consolidated list of all acceptance criteria across all
   requirements. This section serves as the testing checklist
   that will be used to verify the implementation.

   | Req ID | Acceptance Criterion | Status |
   |--------|---------------------|--------|
   | FR-001 | Given X, when Y, then Z | Pending |

═══════════════════════════════════════════════════

11. CHANGE LOG

   | Version | Date | Author | Changes |
   |---------|------|--------|---------|
   | 1.0     | ...  | ...    | Initial draft |

═══════════════════════════════════════════════════

12. SIGN-OFF

   By signing below, the client confirms that this SRS
   accurately represents the agreed-upon requirements and
   approves engineering to proceed with implementation.

   Client Representative: ________________________
   Name: [Name]
   Title: [Title]
   Date: [YYYY-MM-DD]
   SRS Version Approved: [X.X]

   Project Manager: ________________________
   Name: [PM Agent ID]
   Date: [YYYY-MM-DD]

   Notes:
   [Any conditions or caveats on the approval]
```

## SRS Writing Guidelines

### Language Rules

1. **No technical jargon without explanation.** If you must use a technical term, define it in section 1.3 and use the plain-language equivalent in the requirement itself.
   - Bad: "The system shall expose a RESTful endpoint that returns paginated JSON payloads."
   - Good: "The system shall provide a way for external systems to retrieve report data in batches, supporting standard web integration formats."

2. **Be specific and testable.** Every "shall" statement needs to be verifiable.
   - Bad: "The system shall be fast."
   - Good: "The system shall display search results within 2 seconds of the user submitting a query."

3. **One requirement per ID.** Don't bundle multiple behaviors into a single requirement.
   - Bad: "FR-010: The system shall allow users to upload files, and those files shall be scanned for viruses, and the user shall receive email confirmation."
   - Good: Three separate requirements — FR-010 (upload), FR-011 (virus scan), FR-012 (email confirmation).

4. **Describe behavior, not implementation.** The SRS says what the system does, not how it's built.
   - Bad: "The system shall store user preferences in a Redis cache with a 24-hour TTL."
   - Good: "The system shall remember user preferences between sessions for at least 24 hours."

5. **Use consistent verb forms.** "The system shall" for Must-Have, "The system should" for Should-Have, "The system may" for Nice-to-Have.

### Acceptance Criteria Format

Use Given-When-Then format for all acceptance criteria:

```
Given [a precondition or context],
When [the user performs an action],
Then [the expected observable result].
```

**Example:**
```
FR-015: Dashboard Date Filter [Must-Have]
The system shall allow users to filter the dashboard by date range.

Acceptance Criteria:
- Given the user is on the dashboard page,
  When they select a start date and end date from the date picker,
  Then the dashboard shall refresh to show only data from the
  selected range within 2 seconds.

- Given the user has set a date filter,
  When they click "Clear Filter,"
  Then the dashboard shall return to showing the default
  date range (last 30 days).

- Given the user selects an end date that is before the start date,
  When they attempt to apply the filter,
  Then the system shall display a validation message and
  not apply the filter.
```

### Section Completeness Checklist

Before presenting an SRS to the client, verify:

- [ ] Every section has content (no "TBD" placeholders in Must-Have items)
- [ ] Every functional requirement has at least one acceptance criterion
- [ ] Every requirement has a unique ID and priority tag
- [ ] The effort/cost table covers all requirements
- [ ] Out-of-scope section is populated (even if it's just "Nothing was explicitly excluded")
- [ ] Change log reflects current version
- [ ] Open questions are documented with owners
- [ ] No orphan references (no mention of a requirement ID that doesn't exist)

FILE:references/engineer_protocols.md
# Engineer Communication Protocols

## Overview

All PM-to-engineer communication should be structured, specific, and actionable. Vague requests produce vague answers. Every request to the engineer should include enough context that the engineer can work independently and return a useful response without needing to ask clarifying questions.

The PM communicates with the engineer in a semi-technical register: you reference specific features, screens, data entities, and user flows — but you don't dictate architecture, technology choices, or implementation approaches. That's the engineer's domain.

## Software Audit Request Template

Use this when a client comes with changes to existing software and you need to understand the current state before you can meaningfully discuss requirements.

```
SOFTWARE AUDIT REQUEST

Project: [Project Name]
Client: [Client Name]
Date: [YYYY-MM-DD]

PURPOSE
The client is requesting changes to [area of the software].
Before I can have an informed requirements conversation, I need
a clear picture of how these areas currently work.

AREAS TO AUDIT
Please provide a plain-language summary for each area below.
For each, cover:
  (a) What functionality currently exists
  (b) How the user interacts with it (the workflow/flow)
  (c) What data is involved (what's stored, displayed, processed)
  (d) What other parts of the system depend on or are affected
      by this area (dependencies and downstream impacts)
  (e) Any known technical debt, limitations, or quirks in this area

Areas:
1. [Feature/Screen/Module name]
   Context: [Why the client is interested in this area]

2. [Feature/Screen/Module name]
   Context: [Why the client is interested in this area]

3. [Feature/Screen/Module name]
   Context: [Why the client is interested in this area]

ADDITIONAL CONTEXT
[Any relevant background: what the client said, what they seem
to be trying to accomplish, any mockups or documents they shared]

RESPONSE FORMAT
Please structure your response as:
- One section per area audited
- Plain-language descriptions (I'll be sharing relevant parts
  with the client)
- Flag anything that would be particularly impactful or risky
  to change
- Note any areas where the current implementation is fragile
  or would benefit from refactoring as part of changes
```

## Technical Assessment Request Template

Use this after the client has confirmed the Requirements Summary and you need the engineer to evaluate feasibility, effort, and approach.

```
TECHNICAL ASSESSMENT REQUEST

Project: [Project Name]
Date: [YYYY-MM-DD]
Related Software Audit: [Yes/No — if yes, reference date]

REQUIREMENTS TO ASSESS
[Attach or inline the confirmed Requirements Summary]

For each requirement below, please provide:
  (a) Feasibility: Can this be done? Any showstoppers?
  (b) Technical approach: High-level approach (not full design)
  (c) Components affected: Which parts of the system are touched
      (frontend, backend, database, integrations, etc.)
  (d) Effort estimate:
      - Story points (for relative sizing)
      - Estimated human-hours by role (frontend dev, backend dev,
        QA, etc.)
      - Estimated AI-agent hours (how long our agents will take)
      - Complexity rating: Trivial / Low / Medium / High / Very High
  (e) Risks and concerns: What could go wrong, what's uncertain
  (f) Dependencies: What needs to happen first, or what does
      this block?

Requirements:

1. [Requirement from summary]
   Priority: [Must-Have / Should-Have / Nice-to-Have]
   Context: [Any additional context from the client]

2. [Requirement from summary]
   ...

ADDITIONAL QUESTIONS
- Are there any requirements that conflict with each other
  technically?
- Are there any requirements that would benefit from being
  implemented together (natural groupings)?
- Is there any preparatory work (refactoring, infrastructure)
  that should happen before the feature work?
- Are there any requirements where the client's request is
  technically possible but you'd recommend a different approach
  that better serves their stated goal?

RESPONSE FORMAT
Please return a structured assessment with one section per
requirement, using the (a)-(f) structure above. End with an
overall summary section covering cross-cutting concerns,
recommended implementation order, and total effort rollup.
```

## UI Comparison Request Template

Use this when changes involve interface modifications and you need visual representations to share with the client.

```
UI COMPARISON REQUEST

Project: [Project Name]
Date: [YYYY-MM-DD]

CONTEXT
The client has requested changes to [screen/feature]. I need
visual comparison materials to review with them.

REQUESTED DELIVERABLES

1. Current Interface Description
   For each screen/view affected, provide:
   - What the user sees (layout, elements, content areas)
   - What the user can do (interactions, buttons, flows)
   - Screenshot or description if codebase access allows

2. Proposed Interface Rendering
   Based on the requirements below, produce an HTML/CSS
   rendering of the proposed new interface. This should be:
   - A self-contained HTML file I can show the client
   - Focused on layout, information architecture, and workflow
     (pixel-perfect design isn't needed, but the structure and
     flow should be clear)
   - Interactive enough to demonstrate key workflows if applicable
   - Annotated with notes on what's new/changed vs. current

3. Side-by-Side Comparison
   A document or rendering that shows current vs. proposed
   with callouts highlighting the differences.

REQUIREMENTS DRIVING THE CHANGES
[List the relevant requirements from the Requirements Summary
or SRS that drive these interface changes]

CLIENT MOCKUPS (if provided)
[Reference any mockups the client has shared. Note: the
engineer should assess what the current interface looks like
relative to what the mockup shows, and flag any inconsistencies
or implementation concerns with the mockup.]

NOTES
- The HTML/CSS renderings will be shown to the client for
  feedback, so they should look clean and be self-explanatory
- If the client's mockup is unrealistic or problematic from
  a technical standpoint, note that in your response so I can
  discuss it with the client
- Focus on the user experience and workflow, not on final
  visual polish
```

## Implementation Plan Review Request

Use this after the engineer produces an implementation plan from the signed-off SRS. This isn't a request the PM initiates — it's the protocol for reviewing what the engineer delivers.

```
IMPLEMENTATION PLAN REVIEW PROTOCOL

When the engineer delivers the implementation plan, verify
the following:

SRS COVERAGE CHECK
For each requirement in the SRS:
- [ ] FR-XXX: Addressed in plan section [X] — Yes/No
- [ ] FR-XXX: Addressed in plan section [X] — Yes/No
(Go through every requirement ID)

ACCEPTANCE CRITERIA COVERAGE
For each acceptance criterion in SRS Section 10:
- [ ] Criterion for FR-XXX: Testing approach defined — Yes/No
(Verify each criterion has a corresponding test approach)

GAPS TO SURFACE
If any requirement or acceptance criterion is not addressed:

   IMPLEMENTATION PLAN GAP NOTICE
   Date: [YYYY-MM-DD]
   SRS Version: [X.X]

   The following SRS items do not appear to be addressed in
   the implementation plan:

   1. [FR/NFR-XXX]: [Requirement title]
      What's missing: [Specific gap — is the whole requirement
      missing, or is it partially addressed?]

   2. [FR/NFR-XXX]: [Requirement title]
      What's missing: [Description]

   Please update the plan to address these items, or explain
   why they are covered in a way I may have missed.

IMPORTANT: Do NOT review the engineer's technical approach
(architecture, technology choices, implementation details).
Only verify that every SRS requirement is accounted for and
that testing/QA approaches exist for every acceptance criterion.
The engineer owns technical decisions.
```

## General Communication Principles

1. **Always reference specific requirement IDs** when discussing items with the engineer. "The login feature" is vague; "FR-012: Two-factor authentication" is precise.

2. **Don't ask open-ended questions** when you need specific information. Instead of "What do you think about the reporting changes?" ask "For FR-025 through FR-030, can you confirm whether the existing report export functionality will be affected?"

3. **Set clear response expectations** in every request. Tell the engineer what format you need the response in and what information is critical.

4. **Batch your requests** when possible. One structured request with 10 items is better than 10 separate messages.

5. **Acknowledge receipt** of engineer deliverables and set expectations for your review timeline. "Received the technical assessment. I'll review against the requirements summary and get back to you within [timeframe]."

6. **Escalate, don't assume.** If an engineer's response is unclear or seems to miss something, ask for clarification rather than guessing. Misinterpretation creates downstream problems.

FILE:references/requirements_elicitation.md
# Requirements Elicitation Framework

## Purpose

This framework guides the PM through structured discovery conversations with clients. The goal is to extract what the client actually needs — which is often different from what they initially ask for — and document it clearly enough for engineering assessment.

## The Elicitation Mindset

Clients describe solutions ("I want a dropdown menu"). Your job is to find the problem ("I need users to select from a list of options quickly without typing"). The solution might be a dropdown, or it might be autocomplete, radio buttons, or something else entirely. By understanding the problem, you give engineering the freedom to recommend the best approach.

**Three questions that should underpin every discovery:**
1. What problem are you trying to solve? (the "why")
2. Who experiences this problem and how? (the "who" and "when")
3. How will you know the solution is working? (the success criteria)

## Elicitation Process

### Step 1: Contextual Opening

Start by understanding the big picture before diving into specifics.

**Opening questions:**
- "Tell me about what prompted this request. What's happening in your business or with your users that made this a priority?"
- "Who are the main users affected by this? How do they currently handle this?"
- "Is there a deadline or event driving the timeline?"
- "Have you tried any workarounds or temporary solutions?"

The answers establish business context, user context, urgency, and the current state — all of which shape how you interpret specific requirements.

### Step 2: Feature-Level Discovery

For each feature or change the client describes, walk through this question sequence. You don't need to ask every question for every feature — use judgment — but cover the key areas.

**Understanding the request:**
- "Walk me through how you'd like this to work from the user's perspective, step by step."
- "What should happen when [edge case]?" (empty state, error, too many results, no permission, etc.)
- "Is this for all users or specific roles/groups?"
- "How does this relate to other parts of the system you use?"

**Clarifying scope:**
- "When you say [client's term], can you describe exactly what you mean?" (Clients use imprecise language — "report" could mean a dashboard, a PDF export, an email summary, or a data table)
- "Are there any existing features that are similar to what you're describing?"
- "What's the minimum version of this that would solve your problem?" (Identifies the Must-Have core)
- "What would the ideal version look like if there were no constraints?" (Identifies Nice-to-Haves)

**Uncovering hidden requirements:**
- "Who else uses this area of the system? How might this change affect them?"
- "Do you need this to work on mobile devices?"
- "Should there be any notifications or alerts associated with this?"
- "How should permissions work? Who can see/do what?"
- "Is there any regulatory, compliance, or audit requirement we should account for?"
- "What data is involved? Does any of it need to be preserved or migrated?"
- "Do you have reporting or export needs for this data?"

### Step 3: Priority Classification (MoSCoW)

After you've captured the requirements, classify each one with the client:

| Priority | Definition | Client-Friendly Explanation |
|----------|------------|---------------------------|
| **Must-Have** | The project fails without this | "If we don't deliver this, the project isn't useful" |
| **Should-Have** | Important but the project is usable without it | "We strongly want this, but we could launch without it and add it soon after" |
| **Nice-to-Have** | Desired but not critical | "If we have time and budget, we'd love this, but it's not a dealbreaker" |
| **Out-of-Scope** | Explicitly excluded | "We're not doing this now — maybe in a future phase" |

**Key facilitation technique:** Clients often mark everything as Must-Have. Counter this by asking: "If you had to launch with only three of these features, which three?" This forces real prioritization.

### Step 4: Conflict Detection

Before finalizing the requirements summary, check for these common conflict patterns:

**Contradiction conflicts:**
- Requirement A says users can delete records; Requirement B says all records must be retained for audit. → Resolution: soft delete vs. hard delete distinction.

**Resource conflicts:**
- Requirement A and B are both Must-Have but technically dependent on the same system change, and the client wants them simultaneously. → Flag for engineering to assess parallelization.

**Expectation conflicts:**
- Client expects real-time updates but also expects minimal system load/cost. → Surface the trade-off explicitly.

**Scope conflicts:**
- Client verbally agrees something is out of scope but later describes a requirement that implicitly depends on it. → Surface the dependency.

**Priority vs. dependency conflicts:**
- A Should-Have feature is a technical prerequisite for a Must-Have feature. → Recommend reclassifying the Should-Have as Must-Have.

For each conflict identified, document it with both sides and present it to the client for resolution. Don't resolve conflicts unilaterally.

### Step 5: Validation Summary

End every elicitation session by playing back what you heard. This isn't just politeness — it catches misunderstandings before they become expensive.

"Let me make sure I have this right. You need [summary of key requirements]. The most critical items are [Must-Haves]. You're not including [Out-of-Scope items] at this stage. Does that match your understanding?"

Then produce the formal Requirements Summary using the template in `templates.md`.

## Special Scenarios

### Existing Software — Client Requests Changes

The audit-first approach is essential here. Before you can have a productive conversation about what to change, you need to know what exists. Follow this sequence:

1. **Client describes what they want changed** — capture at a high level, don't deep-dive yet
2. **Request Software Audit from engineer** — focused on the areas the client mentioned
3. **Review audit results** — understand current functionality, dependencies, technical debt
4. **Resume client conversation** — now grounded in reality: "Here's what we found about how that area currently works. Based on this, let's walk through your requests in more detail."
5. **During elicitation, reference current state** — "You mentioned you want to change X. Currently X works by [audit finding]. You'd like it to instead [client's request]. Is that accurate?"

This approach prevents the PM from agreeing to things that are more complex than they appear, and gives the client realistic context.

### 0-to-1 New Build

No audit needed, but extra attention to:
- **Competitive references:** "Are there existing products or features in other tools that do something similar to what you want?" This grounds the conversation in concrete examples.
- **User journey mapping:** "Walk me through a day in the life of your primary user. Where does this software fit in?"
- **MVP definition:** "What's the absolute minimum we need to build for this to be valuable?" Resist the temptation to scope the full vision in v1.
- **Technical environment:** "What existing systems, if any, does this need to integrate with?"

### Client Provides Mockups

Mockups are valuable but can be misleading. Treat them as conversation starters, not specifications.

1. **Acknowledge and validate:** "Thanks for putting this together — it gives me a great starting point for understanding what you're envisioning."
2. **Extract the intent:** For each element in the mockup, ask "What problem does this solve?" or "What does this enable the user to do?" The layout may change but the intent should be preserved.
3. **Flag assumptions:** Mockups often assume things that aren't feasible or that conflict with existing functionality. Note these for the engineer to assess.
4. **Request engineer comparison:** Send the mockup to the engineer via the UI Comparison Request to get an assessment of current vs. proposed and any technical concerns.
5. **Present back:** Show the client the engineer's HTML/CSS rendering of the proposed interface alongside their mockup. Discuss differences and iterate.

## Question Bank Quick Reference

When you're mid-conversation and need a prompt, scan this list:

**Opening / Big Picture:**
- What problem are we solving?
- Who is affected and how?
- What's driving the timeline?
- What does success look like?

**Feature Details:**
- Walk me through the user flow.
- What happens in edge cases?
- Who has access to this?
- How does this connect to other features?

**Hidden Requirements:**
- Mobile compatibility?
- Notifications?
- Permissions and roles?
- Data migration?
- Reporting/export needs?
- Compliance/regulatory?

**Prioritization:**
- If you could only have three features, which ones?
- What's the minimum useful version?
- What would the dream version include?

**Validation:**
- Did I capture this correctly?
- Is anything missing?
- Does the priority feel right?
- Any concerns about what we've scoped?

FILE:references/change_management.md
# Change Management & Scope Control

## Purpose

This reference defines how the PM handles change requests after an SRS has been signed off and work has begun. Scope creep is the number one killer of project timelines and budgets. A clear protocol protects the client, the engineering team, and the project.

## Core Principle

Once an SRS is signed off, every new request is a **change request** until proven otherwise. This isn't bureaucratic — it's protective. It ensures that new requests are properly assessed for impact before being absorbed into the work stream, and that the client understands the consequences of changes.

## Change Request Flow

```
Client mentions a change
        │
        ▼
PM captures the request (Change Request Log)
        │
        ▼
PM classifies impact level (1-5)
        │
        ├── Level 1-2 (Cosmetic / UI-Only)
        │   PM can assess impact directly
        │   ▼
        │   Present impact to client → Approve/Decline
        │   ▼
        │   If approved: Update SRS, update Asana, proceed
        │
        └── Level 3-5 (Logic / Data / Cross-cutting)
            PM sends to engineer for assessment
            ▼
            Engineer returns impact analysis
            ▼
            PM translates for client → Present options
            ▼
            Client decides: Approve / Modify / Defer / Decline
            ▼
            If approved: Update SRS (new version), update
            engineering plan if needed, update Asana tasks,
            communicate revised timeline
```

## Impact Classification Reference

### Level 1 — Cosmetic / Copy Changes
**What it is:** Text changes, label updates, color adjustments, typo fixes.
**System impact:** None. No logic, data, or workflow changes.
**PM action:** Assess directly. No engineer needed. Minimal effort — update SRS and create/update Asana task.
**Timeline impact:** None to negligible.
**Example:** Client wants a button to say "Submit Request" instead of "Submit."

### Level 2 — UI-Only Changes
**What it is:** Layout changes, adding static UI elements, style overhauls, reorganizing existing content display.
**System impact:** Frontend only. No logic or data changes.
**PM action:** Quick engineer consult to confirm no hidden complexity, then present to client.
**Timeline impact:** Typically 1-3 days added.
**Example:** Client wants the dashboard layout reorganized so charts appear above the data table instead of below.

### Level 3 — Logic Changes
**What it is:** New validation rules, conditional display logic, workflow changes, business rule modifications.
**System impact:** Frontend and/or backend logic. May affect multiple components but not the data model.
**PM action:** Full engineer assessment required. Changes to this level and above require SRS amendment.
**Timeline impact:** Typically 3-10 days added.
**Example:** Client wants a new approval step in an existing workflow — when a user submits a form, it now goes to a manager for review before processing.

### Level 4 — Data Model Changes
**What it is:** New database fields, schema changes, data migration requirements, changes to how data is stored or structured.
**System impact:** Database + backend + potentially frontend. May require data migration for existing records.
**PM action:** Full technical assessment. SRS amendment. Possibly revised engineering plan. Risk assessment for data migration.
**Timeline impact:** Typically 1-3 weeks added.
**Example:** Client wants to track a new attribute for each customer record that needs to be searchable, sortable, and included in exports.

### Level 5 — Integration / Cross-Cutting Changes
**What it is:** New external integrations, changes affecting multiple modules simultaneously, new subsystem requirements, architectural implications.
**System impact:** Multiple system layers and potentially external systems. May require new infrastructure.
**PM action:** Full cycle: technical assessment → SRS amendment → engineering plan review → Asana rebuild for affected areas. Essentially a mini-project within the project.
**Timeline impact:** Typically 2-6 weeks added.
**Example:** Client wants the system to sync data bidirectionally with their CRM (Salesforce), including real-time updates.

## Scope Creep Detection

Scope creep often doesn't announce itself. Watch for these signals during client interactions:

### Gradual Expansion Signals
- "Oh, and while you're working on that, could you also..."
- "I assumed that was included"
- "Just one more small thing..."
- "Can we make it so that it also..." (feature stacking)
- Requirements that reference functionality not in the SRS

### Ambiguity Exploitation
- Client interprets a vague SRS requirement broadly
- "When you said 'reports,' I thought that included [thing not specified]"
- This is why specific, testable acceptance criteria in the SRS are so important — they prevent ambiguity from becoming scope creep

### Legitimate Clarification vs. Scope Creep
Not every client question is scope creep. Distinguish between:

| Clarification (Not Scope Creep) | Scope Creep |
|-------------------------------|------------|
| "What color will the button be?" (detail within scope) | "Can we add another button that does X?" (new functionality) |
| "Will this work on tablets too?" (verifying stated requirements) | "Actually, we need a native mobile app version too" (new platform) |
| "Can you explain how the export will work?" (understanding) | "Can you add three more export formats?" (new requirements) |

### Responding to Scope Creep

When you detect scope creep, respond professionally and clearly:

"That's a great idea, and I can see how it would add value. Since this wasn't part of our agreed-upon requirements in the SRS, I'll need to log this as a change request so we can properly assess the impact on timeline and cost. Let me document what you're describing, and I'll get back to you with an assessment."

This response:
1. Validates the client's idea (doesn't dismiss them)
2. Clearly identifies it as outside scope (not combative, just factual)
3. Commits to following the proper process
4. Sets expectations for what happens next

## SRS Amendment Protocol

When a change request is approved:

1. **Create a new SRS version.** Never edit the signed-off version in place.
2. **Increment version number.** Major scope changes: increment whole number (2.0). Minor adjustments: increment decimal (1.1).
3. **Update the Change Log** (SRS Section 11) with:
   - New version number
   - Date
   - Description of changes
   - Change Request ID that prompted the change
4. **Update all affected sections:**
   - New/modified requirements with IDs
   - Updated acceptance criteria
   - Updated Cost & Effort Analysis
   - Updated Assumptions & Dependencies (if applicable)
   - Updated Out of Scope (if items moved in or out)
5. **Mark changed items clearly.** Use `[NEW in v2.0]` or `[MODIFIED in v1.1]` tags next to changed requirements so the client can quickly see what's different.
6. **Client re-review.** The client must review and approve the amended SRS before engineering proceeds with the changes.
7. **Update Asana.** Create new tasks or modify existing ones to reflect the approved changes. Tag them with the change request ID.

## Communication Templates for Change Scenarios

### Acknowledging a Change Request
"Thank you for that feedback. I've logged this as Change Request CR-[XXX]. Since this involves [level description — e.g., 'changes to the system's data structure'], I'll need to coordinate with engineering to assess the impact. I'll have an assessment for you by [date]."

### Presenting Change Impact
"Here's what we found about your requested change (CR-[XXX]):

The change would [plain-language description of what it involves]. It affects [areas of the system]. We estimate it would add approximately [time range] to the project timeline and [cost range] in additional effort.

Your options:
1. Approve as-is — we'll amend the SRS and adjust the timeline
2. Modify the request — [suggest a lighter alternative if possible]
3. Defer to a future phase — we'll document it for later
4. Decline — no changes to current plan

What would you prefer?"

### Pushing Back on Excessive Changes
If a client submits many change requests that would fundamentally alter the project:

"I want to make sure we're set up for success here. We've received [X] change requests since the SRS was signed off, and together they represent a significant shift in the project's scope. Rather than continuing to amend the existing SRS piecemeal, I'd recommend we pause and do a focused requirements review to reassess the overall direction. This will give us a cleaner, more cohesive plan rather than a patchwork of amendments. Would you be open to scheduling time for that?"

## Tracking Change Requests

Maintain a running log of all change requests for a project. This serves as both a project management record and an audit trail.

```
CHANGE REQUEST LOG
Project: [Project Name]
SRS Base Version: [X.X]

| CR ID | Date | Description | Level | Status | SRS Impact | Decision | Decision Date |
|-------|------|-------------|-------|--------|------------|----------|--------------|
| CR-001 | ... | ... | 3 | Approved | v1.1 | Approved | ... |
| CR-002 | ... | ... | 1 | Completed | v1.1 | Approved | ... |
| CR-003 | ... | ... | 4 | Under Review | TBD | Pending | — |
| CR-004 | ... | ... | 2 | Declined | None | Declined | ... |
```

Keep this log accessible and reference it during status updates when relevant. It demonstrates that scope is being managed deliberately and that every change was properly assessed.

---
name: NestJS
slug: nestjs
version: 1.0.0
description: Build production-grade NestJS applications with correct module architecture, dependency injection, decorators, guards, pipes, interceptors, middleware, microservices, and testing patterns. Use this skill whenever the user mentions NestJS, Nest.js, Nest framework, or is building a Node.js API with decorators, modules, providers, controllers, or TypeScript-first backend patterns that follow NestJS conventions. Also trigger when the user references NestJS concepts like guards, pipes, interceptors, custom decorators, DTOs with class-validator, TypeORM/Prisma/Mongoose integration in Nest, GraphQL resolvers in Nest, or CQRS. This skill covers NestJS-specific patterns — for general Node.js traps (event loop, streams, async pitfalls), see the NodeJS skill.
metadata: {"clawdbot":{"emoji":"🐈","requires":{"bins":["node","npx"]},"os":["linux","darwin","win32"]}}
---

## Quick Reference

| Topic | File |
|-------|------|
| Module system, circular deps, dynamic modules | `modules.md` |
| DI, providers, injection scopes, custom providers | `dependency-injection.md` |
| Controllers, routing, request lifecycle | `controllers.md` |
| Guards, pipes, interceptors, middleware, filters | `lifecycle.md` |
| DTOs, validation, transformation | `validation.md` |
| Database integration (TypeORM, Prisma, Mongoose) | `database.md` |
| Testing: unit, integration, e2e | `testing.md` |
| Microservices, queues, events, WebSockets | `microservices.md` |
| Config, environment, secrets management | `config.md` |
| Performance, caching, serialization | `performance.md` |

## NestJS vs Plain Node.js — Key Differences

NestJS builds on Node.js/Express (or Fastify) but introduces an opinionated architecture. The main things that catch people:

- **Everything is a class with decorators** — `@Module`, `@Controller`, `@Injectable` are not optional annotations, they drive the DI container and module graph.
- **Module boundaries matter** — a provider is NOT globally available unless explicitly exported and imported. This is the #1 source of "Nest can't resolve dependency" errors.
- **Request lifecycle is layered** — Middleware → Guards → Interceptors (before) → Pipes → Handler → Interceptors (after) → Exception Filters. Order matters and each layer has a distinct job.
- **TypeScript is assumed** — decorators, metadata reflection (`reflect-metadata`), and `emitDecoratorMetadata` are load-bearing. Misconfigured `tsconfig.json` breaks DI silently.

## Critical Traps

- `@Injectable()` missing — class won't be in DI container, cryptic "resolve dependency" error
- Provider not in module's `providers` array — same error, different cause
- Provider not `exports`ed — importing module can't see it, even though the module is imported
- Circular module dependency — use `forwardRef(() => ModuleClass)` on BOTH sides
- Circular provider injection — use `forwardRef(() => ServiceClass)` + `@Inject(forwardRef(...))`
- `@Body()` empty — missing `Content-Type: application/json` header or body-parser not configured
- Validation not firing — forgot `app.useGlobalPipes(new ValidationPipe())` or missing `class-transformer`
- Guard returning `false` silently → 403 — no error message by default, must throw specific exception
- `@Res()` used → Nest loses response control — use `@Res({ passthrough: true })` or avoid `@Res()`
- Exception filter not catching — filter bound to wrong scope (method vs controller vs global)
- `onModuleInit` / `onModuleDestroy` — lifecycle hooks only fire if class is `@Injectable()` AND in providers
- `ConfigService.get()` returns `undefined` — env var not in `.env` or `ConfigModule` not imported in that module
- Fastify adapter — Express middleware won't work, must use Fastify equivalents

FILE:controllers.md
# NestJS Controllers

## Basics
```typescript
@Controller('users') // prefix: /users
export class UsersController {
  constructor(private readonly usersService: UsersService) {}

  @Get()           // GET /users
  findAll() { return this.usersService.findAll(); }

  @Get(':id')      // GET /users/:id
  findOne(@Param('id', ParseIntPipe) id: number) {
    return this.usersService.findOne(id);
  }

  @Post()          // POST /users
  @HttpCode(201)
  create(@Body() dto: CreateUserDto) {
    return this.usersService.create(dto);
  }
}
```

## Parameter Decorators

| Decorator | Express equivalent |
|-----------|-------------------|
| `@Body(key?)` | `req.body` / `req.body[key]` |
| `@Param(key?)` | `req.params` / `req.params[key]` |
| `@Query(key?)` | `req.query` / `req.query[key]` |
| `@Headers(key?)` | `req.headers` / `req.headers[key]` |
| `@Ip()` | `req.ip` |
| `@Req()` | `req` (ties you to Express — avoid) |
| `@Res()` | `res` (⚠️ see trap below) |

## Common Traps

### `@Res()` Takes Over Response Handling
```typescript
// ❌ Nest no longer serializes return value — you must call res.json() yourself
@Get()
findAll(@Res() res: Response) {
  return this.service.findAll(); // SILENTLY IGNORED, request hangs
}

// ✅ passthrough mode — Nest still handles response, but you can set headers/status
@Get()
findAll(@Res({ passthrough: true }) res: Response) {
  res.header('X-Custom', 'value');
  return this.service.findAll(); // Nest serializes this normally
}
```

### Route Order Matters
```typescript
// ❌ ':id' matches 'profile' — GET /users/profile hits findOne('profile')
@Get(':id')
findOne(@Param('id') id: string) {}

@Get('profile')
getProfile() {}

// ✅ Specific routes BEFORE parameterized ones
@Get('profile')
getProfile() {}

@Get(':id')
findOne(@Param('id') id: string) {}
```

### API Versioning
```typescript
// In main.ts
app.enableVersioning({ type: VersioningType.URI }); // /v1/users

// On controller or route
@Controller({ path: 'users', version: '1' })

// Or per-route
@Version('2')
@Get()
findAllV2() {}
```

### File Upload
```typescript
@Post('upload')
@UseInterceptors(FileInterceptor('file'))
uploadFile(@UploadedFile() file: Express.Multer.File) {
  // file.buffer, file.originalname, file.mimetype
}

// With validation
@UploadedFile(
  new ParseFilePipe({
    validators: [
      new MaxFileSizeValidator({ maxSize: 5 * 1024 * 1024 }),
      new FileTypeValidator({ fileType: 'image/png' }),
    ],
  }),
)
```

### Response Serialization
```typescript
// Use ClassSerializerInterceptor to auto-exclude fields
@Entity()
export class User {
  @Exclude()
  password: string; // stripped from all responses

  @Expose()
  get fullName() { return `this.first this.last`; }
}

// Enable globally or per-controller
@UseInterceptors(ClassSerializerInterceptor)
@Controller('users')
export class UsersController {}
```

### Custom Decorators
```typescript
// Extract user from request (set by AuthGuard)
export const CurrentUser = createParamDecorator(
  (data: string, ctx: ExecutionContext) => {
    const request = ctx.switchToHttp().getRequest();
    const user = request.user;
    return data ? user?.[data] : user;
  },
);

// Usage: @CurrentUser() user: User
// Usage: @CurrentUser('email') email: string
```

### Combining Decorators
```typescript
// Bundle common decorators
export function Auth(...roles: Role[]) {
  return applyDecorators(
    SetMetadata('roles', roles),
    UseGuards(AuthGuard, RolesGuard),
    ApplyDecorators(ApiOperation({ summary: 'Protected endpoint' })),
  );
}

// Usage: @Auth(Role.Admin)
```

FILE:lifecycle.md
# NestJS Request Lifecycle

## Execution Order
```
Incoming Request
  → Middleware (global → module-bound)
    → Guards (global → controller → route)
      → Interceptors pre-handler (global → controller → route)
        → Pipes (global → controller → route → param-level)
          → Route Handler
        → Interceptors post-handler (route → controller → global)
      → Exception Filters (route → controller → global)
Response
```
Each layer has a distinct responsibility. Misplacing logic in the wrong layer is a common NestJS mistake.

## Middleware
```typescript
// Function middleware (simple)
export function logger(req: Request, res: Response, next: NextFunction) {
  console.log(`req.method req.url`);
  next(); // ⚠️ forgetting next() hangs the request
}

// Class middleware (with DI)
@Injectable()
export class AuthMiddleware implements NestMiddleware {
  constructor(private authService: AuthService) {}
  use(req: Request, res: Response, next: NextFunction) {
    // can inject services — advantage over function middleware
    next();
  }
}

// Register in module
export class AppModule implements NestModule {
  configure(consumer: MiddlewareConsumer) {
    consumer
      .apply(AuthMiddleware)
      .exclude({ path: 'health', method: RequestMethod.GET })
      .forRoutes('*');
  }
}
```
- Middleware runs BEFORE guards — use for raw request mutation (parsing, CORS, logging)
- Cannot access the route handler or which controller will run (no `ExecutionContext`)
- Express middleware works directly; Fastify middleware does NOT — use Fastify hooks instead

## Guards
```typescript
@Injectable()
export class RolesGuard implements CanActivate {
  constructor(private reflector: Reflector) {}

  canActivate(context: ExecutionContext): boolean {
    const requiredRoles = this.reflector.getAllAndOverride<Role[]>('roles', [
      context.getHandler(),
      context.getClass(),
    ]);
    if (!requiredRoles) return true;

    const request = context.switchToHttp().getRequest();
    const user = request.user;
    return requiredRoles.some(role => user.roles?.includes(role));
  }
}
```
- Return `true` → proceed, `false` → `ForbiddenException` (403)
- Throw specific exception for custom error messages: `throw new UnauthorizedException('Token expired')`
- Has `ExecutionContext` — knows which controller/handler will run
- Use `@SetMetadata()` + `Reflector` to read decorator metadata

### Guard Binding
```typescript
// Route level
@UseGuards(AuthGuard)
@Get('profile')
getProfile() {}

// Controller level
@UseGuards(AuthGuard)
@Controller('admin')
export class AdminController {}

// Global
app.useGlobalGuards(new AuthGuard()); // ❌ no DI
// ✅ Global with DI:
@Module({
  providers: [{ provide: APP_GUARD, useClass: AuthGuard }],
})
export class AppModule {}
```

## Interceptors
```typescript
@Injectable()
export class TransformInterceptor<T> implements NestInterceptor<T, Response<T>> {
  intercept(context: ExecutionContext, next: CallHandler): Observable<Response<T>> {
    const now = Date.now();
    return next.handle().pipe(
      map(data => ({
        data,
        statusCode: context.switchToHttp().getResponse().statusCode,
        timestamp: new Date().toISOString(),
      })),
      tap(() => console.log(`Date.now() - nowms`)),
    );
  }
}
```
- Runs BEFORE and AFTER handler (wraps the handler via RxJS)
- `next.handle()` returns Observable of handler's return value
- Use for: response transformation, caching, logging, timeout
- Can completely override the response or short-circuit with `of(cachedValue)`

### Timeout Interceptor
```typescript
@Injectable()
export class TimeoutInterceptor implements NestInterceptor {
  intercept(context: ExecutionContext, next: CallHandler) {
    return next.handle().pipe(
      timeout(5000),
      catchError(err => {
        if (err instanceof TimeoutError) {
          throw new RequestTimeoutException();
        }
        throw err;
      }),
    );
  }
}
```

## Pipes
```typescript
// Built-in pipes: ValidationPipe, ParseIntPipe, ParseBoolPipe,
// ParseArrayPipe, ParseUUIDPipe, ParseEnumPipe, DefaultValuePipe

// Param-level
@Get(':id')
findOne(@Param('id', ParseIntPipe) id: number) {} // '123' → 123, 'abc' → 400

// Global validation
app.useGlobalPipes(new ValidationPipe({
  whitelist: true,         // strip unknown properties
  forbidNonWhitelisted: true, // throw on unknown properties
  transform: true,         // auto-transform payloads to DTO instances
  transformOptions: { enableImplicitConversion: true },
}));
```
- Pipes run AFTER guards but BEFORE handler — for validation and transformation
- `transform: true` auto-converts query params from strings to expected types

## Exception Filters
```typescript
@Catch(HttpException)
export class HttpExceptionFilter implements ExceptionFilter {
  catch(exception: HttpException, host: ArgumentsHost) {
    const ctx = host.switchToHttp();
    const response = ctx.getResponse<Response>();
    const status = exception.getStatus();

    response.status(status).json({
      statusCode: status,
      message: exception.message,
      timestamp: new Date().toISOString(),
      path: ctx.getRequest<Request>().url,
    });
  }
}

// Catch everything
@Catch()
export class AllExceptionsFilter implements ExceptionFilter {
  catch(exception: unknown, host: ArgumentsHost) {
    // handle non-HTTP exceptions (TypeORM errors, etc.)
  }
}
```
- Bind with `@UseFilters()` at route/controller level or globally
- Multiple `@Catch(TypeA, TypeB)` — one filter handles several exception types
- Filters catch exceptions thrown from guards, interceptors, pipes, and handlers

## Common Traps

- **Middleware can't use `ExecutionContext`** — if you need to know the route handler, use a guard or interceptor
- **Guard returns `false` with no message** — user gets generic 403, throw `ForbiddenException('reason')` instead
- **Interceptor `next.handle()` not called** — handler never executes, request hangs or returns interceptor's value
- **Pipe validation errors swallowed** — if `exceptionFactory` is misconfigured, errors become 500 instead of 400
- **Exception filter scope** — route-level filter only catches that route; controller-level catches all routes in controller; global catches everything
- **Global providers without DI** — `app.useGlobalGuards(new Guard())` doesn't inject dependencies; use `APP_GUARD`/`APP_PIPE`/`APP_FILTER`/`APP_INTERCEPTOR` provider tokens instead

FILE:dependency-injection.md
# NestJS Dependency Injection

## Provider Types

### Standard (class-based)
```typescript
@Injectable()
export class UsersService {
  constructor(private readonly usersRepo: UsersRepository) {}
}
// Registered as: providers: [UsersService]
// Equivalent to: { provide: UsersService, useClass: UsersService }
```

### Value Provider
```typescript
{ provide: 'API_KEY', useValue: process.env.API_KEY }
// Inject with: @Inject('API_KEY') private apiKey: string
```

### Factory Provider
```typescript
{
  provide: 'ASYNC_CONNECTION',
  useFactory: async (configService: ConfigService) => {
    const dbConfig = configService.get('database');
    return createConnection(dbConfig);
  },
  inject: [ConfigService], // factory dependencies
}
```
- `inject` array matches factory parameter order
- Can be async — Nest waits for resolution before proceeding

### Existing Provider (alias)
```typescript
{ provide: 'AliasedService', useExisting: ConcreteService }
// Both tokens point to same singleton instance
```

## Injection Scopes

### DEFAULT (singleton) — most common
- One instance for entire app lifetime
- Shared across all requests
- ⚠️ Do NOT store request-specific state in singletons

### REQUEST
```typescript
@Injectable({ scope: Scope.REQUEST })
export class RequestScopedService {}
```
- New instance per incoming request
- **Bubbles up**: if A (singleton) depends on B (request-scoped), A becomes request-scoped too
- Performance cost — avoid unless you genuinely need per-request state (multi-tenancy, request context)
- Controllers depending on request-scoped providers also become request-scoped

### TRANSIENT
```typescript
@Injectable({ scope: Scope.TRANSIENT })
export class TransientService {}
```
- New instance per injection (every consumer gets its own)
- Not shared between consumers even in same request

## Common Traps

### Missing `@Injectable()`
```typescript
// ❌ Nest silently can't resolve this
export class MyService {
  constructor(private dep: OtherService) {}
}

// ✅ Decorator is required for metadata emission
@Injectable()
export class MyService {
  constructor(private dep: OtherService) {}
}
```

### Interface Injection (TypeScript interfaces erased at runtime)
```typescript
// ❌ Interfaces don't exist at runtime — Nest can't resolve
constructor(private service: IMyService) {}

// ✅ Use string/symbol token + @Inject
constructor(@Inject('IMyService') private service: IMyService) {}

// ✅ Or use abstract class as token (classes survive compilation)
export abstract class IMyService {
  abstract doThing(): void;
}
```

### Circular Provider Dependencies
```typescript
// ❌ A injects B, B injects A — crash
// ✅ forwardRef on BOTH providers
@Injectable()
export class ServiceA {
  constructor(
    @Inject(forwardRef(() => ServiceB))
    private serviceB: ServiceB,
  ) {}
}

@Injectable()
export class ServiceB {
  constructor(
    @Inject(forwardRef(() => ServiceA))
    private serviceA: ServiceA,
  ) {}
}
```
Better solution: extract shared logic into a third service.

### Optional Dependencies
```typescript
@Injectable()
export class NotificationService {
  constructor(
    @Optional() @Inject('MAILER') private mailer?: MailerService,
  ) {}
  // mailer is undefined if MAILER provider not registered — no crash
}
```

### Custom Provider Tokens
```typescript
// Symbol tokens prevent collision
export const CACHE_MANAGER = Symbol('CACHE_MANAGER');

// Register
{ provide: CACHE_MANAGER, useClass: RedisCacheManager }

// Inject
constructor(@Inject(CACHE_MANAGER) private cache: CacheManager) {}
```

### `ModuleRef` for Dynamic Resolution
```typescript
@Injectable()
export class DynamicService {
  constructor(private moduleRef: ModuleRef) {}

  async getService() {
    // Resolve request-scoped or transient providers dynamically
    const svc = await this.moduleRef.resolve(TransientService);
    // .get() for singletons (synchronous)
    const singleton = this.moduleRef.get(SingletonService);
  }
}
```

FILE:microservices.md
# NestJS Microservices, Queues & WebSockets

## Microservices (Transport Layer)

### Setup
```typescript
// main.ts — hybrid app (HTTP + microservice)
const app = await NestFactory.create(AppModule);
app.connectMicroservice<MicroserviceOptions>({
  transport: Transport.TCP,
  options: { host: '0.0.0.0', port: 3001 },
});
await app.startAllMicroservices();
await app.listen(3000);
```

### Message Patterns (Request/Response)
```typescript
// Service (listener)
@Controller()
export class MathController {
  @MessagePattern({ cmd: 'sum' })
  sum(data: number[]): number {
    return data.reduce((a, b) => a + b, 0);
  }
}

// Client (caller)
@Injectable()
export class AppService {
  constructor(@Inject('MATH_SERVICE') private client: ClientProxy) {}

  getSum(numbers: number[]) {
    return this.client.send({ cmd: 'sum' }, numbers); // returns Observable
  }
}

// Register client
@Module({
  imports: [
    ClientsModule.register([{
      name: 'MATH_SERVICE',
      transport: Transport.TCP,
      options: { host: 'math-service', port: 3001 },
    }]),
  ],
})
```

### Event Patterns (Fire-and-Forget)
```typescript
// Listener
@EventPattern('user.created')
handleUserCreated(data: UserCreatedEvent) {
  // no return value — fire and forget
}

// Emitter
this.client.emit('user.created', { userId: 1, email: '[email protected]' });
```

### Transport Options
- `Transport.TCP` — default, simple, no broker needed
- `Transport.REDIS` — Redis pub/sub
- `Transport.NATS` — NATS messaging
- `Transport.MQTT` — IoT/lightweight
- `Transport.KAFKA` — high-throughput event streaming
- `Transport.RMQ` — RabbitMQ
- `Transport.GRPC` — Protocol Buffers, strongly typed

### Common Microservice Traps
- `client.send()` returns cold Observable — must `.subscribe()` or convert to Promise with `firstValueFrom()`
- Forgetting `await app.startAllMicroservices()` — microservice transport never starts
- `ClientProxy` not connected — call `client.connect()` or it auto-connects on first message (but first message is slow)
- Serialization — objects must be JSON-serializable across transport; classes become plain objects
- Error propagation — exceptions in microservice handler propagate to caller as `RpcException`, not `HttpException`

## Bull Queues (@nestjs/bull or @nestjs/bullmq)

### Setup
```typescript
@Module({
  imports: [
    BullModule.forRoot({ connection: { host: 'localhost', port: 6379 } }),
    BullModule.registerQueue({ name: 'email' }),
  ],
  providers: [EmailProcessor],
})

// Producer
@Injectable()
export class EmailService {
  constructor(@InjectQueue('email') private emailQueue: Queue) {}

  async sendWelcome(userId: string) {
    await this.emailQueue.add('welcome', { userId }, {
      delay: 5000,           // delay 5 seconds
      attempts: 3,           // retry up to 3 times
      backoff: { type: 'exponential', delay: 1000 },
      removeOnComplete: true,
    });
  }
}

// Consumer
@Processor('email')
export class EmailProcessor {
  @Process('welcome')
  async handleWelcome(job: Job<{ userId: string }>) {
    // process the job
    // throw to retry, return to complete
  }

  @OnQueueFailed()
  onFailed(job: Job, error: Error) {
    console.error(`Job job.id failed:`, error.message);
  }
}
```

### Queue Traps
- Queue name mismatch between `registerQueue` and `@Processor` — silently never processes
- Redis not running — queue operations hang or throw, no clear error
- Job data not serializable — functions, circular refs, class instances fail
- Processor not in module's providers — Nest doesn't discover it

## WebSockets (@nestjs/websockets)

### Gateway
```typescript
@WebSocketGateway({
  cors: { origin: '*' },
  namespace: '/chat',
})
export class ChatGateway implements OnGatewayConnection, OnGatewayDisconnect {
  @WebSocketServer()
  server: Server;

  handleConnection(client: Socket) {
    console.log(`Connected: client.id`);
  }

  handleDisconnect(client: Socket) {
    console.log(`Disconnected: client.id`);
  }

  @SubscribeMessage('message')
  handleMessage(client: Socket, payload: { room: string; text: string }) {
    this.server.to(payload.room).emit('message', payload);
    return { event: 'message', data: 'received' }; // ack to sender
  }
}
```

### WebSocket Traps
- Guards/pipes/interceptors work with gateways — but use `context.switchToWs()` not `switchToHttp()`
- CORS not configured on gateway — browser connections fail silently
- `@WebSocketServer()` is undefined in constructor — only available after `onModuleInit`
- Namespace mismatch — client connects to `/chat` but gateway is on `/` or vice versa
- Authentication — middleware doesn't run for WS; use a guard or handle in `handleConnection`

## Server-Sent Events (SSE)
```typescript
@Sse('events')
sse(): Observable<MessageEvent> {
  return interval(1000).pipe(
    map(n => ({ data: { count: n } } as MessageEvent)),
  );
}
```
- Returns Observable that streams events to client
- Connection stays open — be mindful of resource usage
- Client uses `EventSource` API

FILE:modules.md
# NestJS Modules

## Core Concepts
- Every Nest app has a root `AppModule` — all other modules branch from it
- A module is a class decorated with `@Module()` containing `imports`, `controllers`, `providers`, `exports`
- Modules encapsulate providers — a provider is NOT available outside its module unless listed in `exports`

## Common Traps

### "Nest can't resolve dependencies of X"
This is the single most common NestJS error. Checklist:
1. Is the class decorated with `@Injectable()`?
2. Is it listed in the module's `providers` array?
3. If it's from another module, is that module listed in `imports`?
4. Does the source module `exports` the provider?
5. Are all of the provider's OWN dependencies also resolvable?

### Circular Module Dependencies
```typescript
// ❌ ModuleA imports ModuleB, ModuleB imports ModuleA — crash
// ✅ Use forwardRef on BOTH sides
@Module({
  imports: [forwardRef(() => ModuleB)],
})
export class ModuleA {}

@Module({
  imports: [forwardRef(() => ModuleA)],
})
export class ModuleB {}
```
Better fix: extract shared providers into a third module both can import.

### Global Modules
```typescript
@Global()
@Module({
  providers: [SharedService],
  exports: [SharedService],
})
export class SharedModule {}
```
- `@Global()` makes exports available everywhere WITHOUT importing the module
- Overuse defeats encapsulation — use sparingly (config, logging, database connections)

### Dynamic Modules
```typescript
@Module({})
export class DatabaseModule {
  static forRoot(options: DbOptions): DynamicModule {
    return {
      module: DatabaseModule,
      global: true, // optional
      providers: [
        { provide: 'DB_OPTIONS', useValue: options },
        DatabaseService,
      ],
      exports: [DatabaseService],
    };
  }

  static forFeature(entities: Type[]): DynamicModule {
    const providers = entities.map(entity => ({
      provide: getRepositoryToken(entity),
      useFactory: (ds: DataSource) => ds.getRepository(entity),
      inject: [DataSource],
    }));
    return {
      module: DatabaseModule,
      providers,
      exports: providers,
    };
  }
}
```
- `forRoot()` pattern: configure once in AppModule (connection, global config)
- `forFeature()` pattern: configure per-feature module (repositories, specific entities)
- `forRootAsync()`: when config depends on other providers (ConfigService)

### Module Re-exporting
```typescript
@Module({
  imports: [DatabaseModule],
  exports: [DatabaseModule], // re-export so importers get DatabaseModule's exports too
})
export class CoreModule {}
```

### Lazy-loaded Modules
```typescript
// For routes that should only load on demand (reduces startup time)
@Injectable()
export class SomeService {
  constructor(private lazyModuleLoader: LazyModuleLoader) {}

  async loadFeature() {
    const { FeatureModule } = await import('./feature.module');
    const moduleRef = await this.lazyModuleLoader.load(() => FeatureModule);
    const service = moduleRef.get(FeatureService);
  }
}
```
- Lazy modules don't register controllers/gateways — only providers
- Useful for serverless/cold-start optimization

FILE:validation.md
# NestJS Validation & DTOs

## Setup
```bash
npm install class-validator class-transformer
```
Both packages required. `class-validator` defines rules, `class-transformer` handles plain-object → class-instance conversion.

## DTO Pattern
```typescript
import { IsString, IsEmail, IsOptional, MinLength, IsEnum, ValidateNested, Type } from 'class-validator';

export class CreateUserDto {
  @IsString()
  @MinLength(2)
  name: string;

  @IsEmail()
  email: string;

  @IsOptional()
  @IsString()
  bio?: string;

  @IsEnum(Role)
  role: Role;

  @ValidateNested()
  @Type(() => AddressDto) // ⚠️ Required for nested object validation
  address: AddressDto;
}
```

## Common Traps

### `class-transformer` Not Installed
- `ValidationPipe` with `transform: true` silently fails — body stays plain object
- Nested `@ValidateNested()` doesn't work without `@Type()` decorator
- Install both: `npm install class-validator class-transformer`

### Missing `@Type()` on Nested Objects
```typescript
// ❌ Nested validation NEVER runs — address is just a plain object
@ValidateNested()
address: AddressDto;

// ✅ @Type tells class-transformer which class to instantiate
@ValidateNested()
@Type(() => AddressDto)
address: AddressDto;
```

### Arrays of Nested Objects
```typescript
@ValidateNested({ each: true })
@Type(() => ItemDto)
items: ItemDto[];
```

### Partial Updates (PATCH)
```typescript
// PartialType makes all fields optional, preserving validation rules
import { PartialType } from '@nestjs/mapped-types';
// or from @nestjs/swagger if using Swagger

export class UpdateUserDto extends PartialType(CreateUserDto) {}
```
- `PartialType` — all optional
- `PickType(CreateUserDto, ['name', 'email'])` — select specific fields
- `OmitType(CreateUserDto, ['password'])` — exclude specific fields
- `IntersectionType(A, B)` — combine two DTOs

### Whitelist vs ForbidNonWhitelisted
```typescript
new ValidationPipe({
  whitelist: true,              // silently strips unknown properties
  forbidNonWhitelisted: true,   // throws 400 if unknown properties present
})
```
- `whitelist: true` alone silently drops extra fields — attacker sends `{ role: 'admin' }` and it's stripped
- Add `forbidNonWhitelisted` to reject the request entirely

### Transform Mode
```typescript
new ValidationPipe({
  transform: true,
  transformOptions: {
    enableImplicitConversion: true, // @Query('page') page: number → auto parseInt
  },
})
```
- Without `transform: true`, handler receives plain object, not DTO class instance
- `enableImplicitConversion` converts based on TypeScript type metadata — strings to numbers/booleans in `@Query()`

### Custom Validation
```typescript
// Custom decorator
export function IsStrongPassword(validationOptions?: ValidationOptions) {
  return function (object: Object, propertyName: string) {
    registerDecorator({
      name: 'isStrongPassword',
      target: object.constructor,
      propertyName,
      options: validationOptions,
      validator: {
        validate(value: string) {
          return /^(?=.*[A-Z])(?=.*[0-9])(?=.*[!@#$%]).{8,}$/.test(value);
        },
        defaultMessage() {
          return 'Password too weak';
        },
      },
    });
  };
}
```

### Validation Groups
```typescript
export class UserDto {
  @IsOptional({ groups: ['update'] })
  @IsNotEmpty({ groups: ['create'] })
  name: string;
}

// In controller
@UsePipes(new ValidationPipe({ groups: ['create'] }))
@Post()
create(@Body() dto: UserDto) {}
```

### Global Error Format
```typescript
new ValidationPipe({
  exceptionFactory: (errors: ValidationError[]) => {
    const messages = errors.map(err =>
      Object.values(err.constraints ?? {}).join(', ')
    );
    return new BadRequestException({
      message: 'Validation failed',
      errors: messages,
    });
  },
})
```

FILE:config.md
# NestJS Configuration

## @nestjs/config (recommended)

### Basic Setup
```typescript
@Module({
  imports: [
    ConfigModule.forRoot({
      isGlobal: true,          // available everywhere without importing
      envFilePath: ['.env.local', '.env'], // first found wins per variable
      ignoreEnvFile: process.env.NODE_ENV === 'production', // use real env vars in prod
    }),
  ],
})
export class AppModule {}
```

### Typed Configuration with Validation
```typescript
// config/database.config.ts
import { registerAs } from '@nestjs/config';

export default registerAs('database', () => ({
  host: process.env.DB_HOST || 'localhost',
  port: parseInt(process.env.DB_PORT, 10) || 5432,
  name: process.env.DB_NAME,
}));

// Usage with injection
@Injectable()
export class DbService {
  constructor(
    @Inject(databaseConfig.KEY)
    private dbConfig: ConfigType<typeof databaseConfig>,
  ) {
    // dbConfig.host, dbConfig.port — fully typed
  }
}
```

### Schema Validation with Joi
```typescript
import * as Joi from 'joi';

ConfigModule.forRoot({
  validationSchema: Joi.object({
    NODE_ENV: Joi.string().valid('development', 'production', 'test').default('development'),
    PORT: Joi.number().default(3000),
    DB_HOST: Joi.string().required(),
    DB_PORT: Joi.number().required(),
    JWT_SECRET: Joi.string().required(),
  }),
  validationOptions: {
    abortEarly: true, // fail fast on first error
  },
});
```

### Schema Validation with Zod (alternative)
```typescript
import { z } from 'zod';

const envSchema = z.object({
  NODE_ENV: z.enum(['development', 'production', 'test']).default('development'),
  PORT: z.coerce.number().default(3000),
  DB_HOST: z.string(),
  JWT_SECRET: z.string().min(32),
});

ConfigModule.forRoot({
  validate: (config: Record<string, unknown>) => {
    const parsed = envSchema.safeParse(config);
    if (!parsed.success) {
      throw new Error(`Config validation error: parsed.error.message`);
    }
    return parsed.data;
  },
});
```

## Common Traps

### `ConfigService.get()` Returns `undefined`
1. `ConfigModule` not imported in the module (or not `isGlobal: true`)
2. Env var not in `.env` file AND not in actual environment
3. Using `configService.get('database.host')` but didn't register namespaced config
4. `.env` file not in project root (relative to where `nest start` runs)

### Type Safety
```typescript
// ❌ Returns string | undefined — no type safety
const port = this.configService.get('PORT');

// ✅ Generic parameter
const port = this.configService.get<number>('PORT');

// ✅ With default (guarantees non-undefined)
const port = this.configService.get<number>('PORT', 3000);

// ✅ Best: use registerAs + ConfigType for full type safety
```

### Configuration at Bootstrap
```typescript
// main.ts — ConfigService isn't available before app is created
async function bootstrap() {
  const app = await NestFactory.create(AppModule);
  const configService = app.get(ConfigService);
  const port = configService.get<number>('PORT', 3000);
  await app.listen(port);
}
```

### Secrets Management
- Never commit `.env` with secrets — add to `.gitignore`
- Use `.env.example` with placeholder values for documentation
- In production, use real environment variables or a secrets manager (AWS SSM, Vault)
- `ConfigModule` with `ignoreEnvFile: true` in production — don't deploy .env files
- Validate required secrets at startup — fail fast, not on first request

### Async Configuration
```typescript
// When config depends on other async sources
TypeOrmModule.forRootAsync({
  imports: [ConfigModule],
  inject: [ConfigService],
  useFactory: (config: ConfigService) => ({
    type: 'postgres',
    host: config.get('DB_HOST'),
    port: config.get<number>('DB_PORT'),
    // ...
  }),
});
```
- `forRootAsync` with `useFactory` — when module config depends on ConfigService or other providers
- `inject` array specifies factory dependencies
- Always prefer `forRootAsync` over `forRoot` with `process.env` directly — ensures config is validated and centralized

FILE:database.md
# NestJS Database Integration

## TypeORM

### Setup
```typescript
// app.module.ts
@Module({
  imports: [
    TypeOrmModule.forRoot({
      type: 'postgres',
      host: process.env.DB_HOST,
      port: parseInt(process.env.DB_PORT, 10),
      entities: [__dirname + '/**/*.entity{.ts,.js}'],
      // OR use autoLoadEntities: true (recommended with forFeature)
      synchronize: false, // ⚠️ NEVER true in production
    }),
  ],
})

// feature module
@Module({
  imports: [TypeOrmModule.forFeature([User, Profile])],
  providers: [UsersService],
})
export class UsersModule {}
```

### Repository Pattern
```typescript
@Injectable()
export class UsersService {
  constructor(
    @InjectRepository(User)
    private usersRepo: Repository<User>,
  ) {}

  findAll(): Promise<User[]> {
    return this.usersRepo.find({ relations: ['profile'] });
  }
}
```

### Common TypeORM + Nest Traps
- `synchronize: true` in production — drops/recreates tables, DATA LOSS. Use migrations.
- Entity not in `entities` array AND `autoLoadEntities` is false — "No metadata found" error
- `autoLoadEntities` only finds entities registered via `forFeature()` — manual entity classes not in any forFeature are missed
- Circular entity relations — `@ManyToOne(() => User)` lazy-callback syntax required
- Transaction handling — use `DataSource.transaction()` or `QueryRunner`, not multiple separate repo saves
- Repository injected but module doesn't import `TypeOrmModule.forFeature([Entity])` — "can't resolve Repository" error

### Migrations
```bash
# Generate migration from entity changes
npx typeorm migration:generate -d src/data-source.ts src/migrations/AddUserTable
# Run migrations
npx typeorm migration:run -d src/data-source.ts
```
- Separate `data-source.ts` for CLI — can't use Nest DI in migration CLI
- Always review generated migrations before running

## Prisma

### Setup
```typescript
// prisma.service.ts
@Injectable()
export class PrismaService extends PrismaClient implements OnModuleInit {
  async onModuleInit() {
    await this.$connect();
  }

  async onModuleDestroy() {
    await this.$disconnect();
  }
}

// prisma.module.ts
@Global()
@Module({
  providers: [PrismaService],
  exports: [PrismaService],
})
export class PrismaModule {}
```

### Usage
```typescript
@Injectable()
export class UsersService {
  constructor(private prisma: PrismaService) {}

  findAll() {
    return this.prisma.user.findMany({ include: { posts: true } });
  }

  create(data: CreateUserDto) {
    return this.prisma.user.create({ data });
  }
}
```

### Common Prisma + Nest Traps
- Not calling `$connect()` in `onModuleInit` — first query is slow (lazy connect)
- Not calling `$disconnect()` in `onModuleDestroy` — connection pool leaks in tests and serverless
- Prisma generates its own types — don't duplicate with DTOs for database layer, use Prisma types directly for repository logic; DTOs for API boundary
- `enableShutdownHooks` conflicts with Nest's own shutdown — use `onModuleDestroy` instead of Prisma's built-in shutdown hook

## Mongoose

### Setup
```typescript
@Module({
  imports: [
    MongooseModule.forRoot('mongodb://localhost/nest'),
    MongooseModule.forFeature([{ name: Cat.name, schema: CatSchema }]),
  ],
})

// Schema definition
@Schema({ timestamps: true })
export class Cat {
  @Prop({ required: true })
  name: string;

  @Prop({ type: mongoose.Schema.Types.ObjectId, ref: 'Owner' })
  owner: Owner;
}

export const CatSchema = SchemaFactory.createForClass(Cat);
```

### Common Mongoose + Nest Traps
- `@Prop()` without `required: true` — field is optional by default, unlike class-validator
- Schema definition separate from validation — Mongoose schema validates at DB level, class-validator at API level; you need both
- `@InjectModel(Cat.name)` — must match the name in `forFeature()` registration exactly
- Virtual properties need `toJSON: { virtuals: true }` in schema options
- Discriminators for inheritance — use `MongooseModule.forFeature` with `discriminators` option

## General Database Traps in NestJS
- Transactions across services — inject `DataSource`/`EntityManager` and pass transaction manager, don't rely on separate repository calls
- Connection not closed on app shutdown — enable `enableShutdownHooks()` in main.ts
- N+1 queries — use `relations` (TypeORM), `include` (Prisma), or `.populate()` (Mongoose) to eager-load
- Connection pool exhaustion — default pools are small (10), increase for production

FILE:testing.md
# NestJS Testing

## Unit Testing (Services)

```typescript
describe('UsersService', () => {
  let service: UsersService;
  let repo: jest.Mocked<Repository<User>>;

  beforeEach(async () => {
    const module = await Test.createTestingModule({
      providers: [
        UsersService,
        {
          provide: getRepositoryToken(User),
          useValue: {
            find: jest.fn(),
            findOne: jest.fn(),
            save: jest.fn(),
            delete: jest.fn(),
          },
        },
      ],
    }).compile();

    service = module.get(UsersService);
    repo = module.get(getRepositoryToken(User));
  });

  it('should find all users', async () => {
    const users = [{ id: 1, name: 'Alice' }] as User[];
    repo.find.mockResolvedValue(users);
    expect(await service.findAll()).toEqual(users);
  });
});
```

## Unit Testing (Controllers)

```typescript
describe('UsersController', () => {
  let controller: UsersController;
  let service: jest.Mocked<UsersService>;

  beforeEach(async () => {
    const module = await Test.createTestingModule({
      controllers: [UsersController],
      providers: [
        {
          provide: UsersService,
          useValue: {
            findAll: jest.fn(),
            create: jest.fn(),
          },
        },
      ],
    }).compile();

    controller = module.get(UsersController);
    service = module.get(UsersService);
  });
});
```

## Integration / E2E Testing

```typescript
describe('Users (e2e)', () => {
  let app: INestApplication;

  beforeAll(async () => {
    const moduleFixture = await Test.createTestingModule({
      imports: [AppModule],
    })
      .overrideProvider(DatabaseService)
      .useValue(mockDatabaseService) // swap real DB for mock
      .compile();

    app = moduleFixture.createNestApplication();
    // ⚠️ Apply same global pipes/guards/interceptors as main.ts
    app.useGlobalPipes(new ValidationPipe({ whitelist: true }));
    await app.init();
  });

  afterAll(async () => {
    await app.close(); // ⚠️ prevents hanging tests and connection leaks
  });

  it('/users (GET)', () => {
    return request(app.getHttpServer())
      .get('/users')
      .expect(200)
      .expect(res => {
        expect(res.body).toBeInstanceOf(Array);
      });
  });

  it('/users (POST) validates body', () => {
    return request(app.getHttpServer())
      .post('/users')
      .send({ name: '' }) // invalid
      .expect(400);
  });
});
```

## Common Traps

### E2E Tests Don't Apply Global Config
```typescript
// ❌ Global pipes/guards set in main.ts are NOT applied in Test.createTestingModule
// Your e2e tests will pass without validation, giving false confidence

// ✅ Apply the same global config in test setup
app.useGlobalPipes(new ValidationPipe({ whitelist: true, transform: true }));
app.useGlobalFilters(new AllExceptionsFilter());
```

### `Test.createTestingModule` Creates Full DI Container
- All providers must be resolvable — either real or mocked
- Use `.overrideProvider(Token).useValue(mock)` to swap implementations
- For large module graphs, import only the module under test + mock its imports

### Mocking the Repository
```typescript
// ❌ Providing Repository class directly — Nest tries to connect to DB
providers: [UsersService, Repository],

// ✅ Mock the specific repository token
{ provide: getRepositoryToken(User), useValue: mockRepo }
```

### Testing Guards
```typescript
// Override guard globally for e2e tests that shouldn't auth
const module = await Test.createTestingModule({ imports: [AppModule] })
  .overrideGuard(AuthGuard)
  .useValue({ canActivate: () => true })
  .compile();
```

### Testing with Real Database
```typescript
// Use a test database container (testcontainers)
import { PostgreSqlContainer } from '@testcontainers/postgresql';

let container: StartedPostgreSqlContainer;

beforeAll(async () => {
  container = await new PostgreSqlContainer().start();
  // Use container.getConnectionUri() for TypeORM/Prisma config
}, 30000); // containers take time to start

afterAll(async () => {
  await container.stop();
});
```

### Hanging Tests
- `afterAll(() => app.close())` — always close the app to release connections
- Database connections not closed — especially with TypeORM/Prisma
- Open handles from event emitters, intervals, or WebSocket connections
- Use `--forceExit` as last resort, but fix the underlying leak

### Request-Scoped Providers in Tests
```typescript
// Request-scoped providers can't be resolved with module.get()
// ✅ Use module.resolve() instead
const service = await module.resolve(RequestScopedService);
```

### Custom Test Utilities
```typescript
// Create a reusable test module factory
export async function createTestApp(overrides?: {
  providers?: Provider[];
}) {
  const builder = Test.createTestingModule({ imports: [AppModule] });

  overrides?.providers?.forEach(p => {
    builder.overrideProvider(p.provide).useValue(p.useValue);
  });

  const module = await builder.compile();
  const app = module.createNestApplication();
  app.useGlobalPipes(new ValidationPipe({ whitelist: true }));
  await app.init();
  return app;
}
```

FILE:performance.md
# NestJS Performance

## Caching

### Built-in Cache Manager
```typescript
// app.module.ts
import { CacheModule } from '@nestjs/cache-manager';

@Module({
  imports: [
    CacheModule.register({
      ttl: 60, // seconds (v5+) or milliseconds (v4)
      max: 100, // max items in cache
      isGlobal: true,
    }),
  ],
})

// With Redis store
import { redisStore } from 'cache-manager-redis-store';
CacheModule.registerAsync({
  useFactory: async () => ({
    store: await redisStore({ host: 'localhost', port: 6379 }),
    ttl: 60,
  }),
});
```

### Cache Interceptor (auto-cache GET routes)
```typescript
@UseInterceptors(CacheInterceptor)
@Controller('products')
export class ProductsController {
  @CacheTTL(120) // override default TTL for this route
  @CacheKey('all-products') // custom cache key
  @Get()
  findAll() { return this.productsService.findAll(); }
}
```

### Manual Cache Usage
```typescript
@Injectable()
export class UsersService {
  constructor(@Inject(CACHE_MANAGER) private cache: Cache) {}

  async findOne(id: number) {
    const cached = await this.cache.get<User>(`user:id`);
    if (cached) return cached;
    const user = await this.repo.findOne({ where: { id } });
    await this.cache.set(`user:id`, user, 300);
    return user;
  }
}
```

### Cache Traps
- `CacheInterceptor` only caches GET — POST/PUT/DELETE are ignored
- Cache key includes query params by default — `?page=1` and `?page=2` are separate
- Stale cache after mutations — manually `cache.del()` after writes
- Redis serialization — class instances become plain objects when deserialized
- `CacheModule` v4 vs v5 — TTL units changed from ms to seconds

## Fastify Adapter

```typescript
// main.ts
import { NestFactory } from '@nestjs/core';
import { FastifyAdapter, NestFastifyApplication } from '@nestjs/platform-fastify';

const app = await NestFactory.create<NestFastifyApplication>(
  AppModule,
  new FastifyAdapter(),
);
await app.listen(3000, '0.0.0.0'); // ⚠️ Fastify binds 127.0.0.1 by default
```

### Fastify Traps
- Listen on `'0.0.0.0'` not default — Docker/containers can't reach `127.0.0.1`
- Express middleware (multer, passport) won't work — use Fastify equivalents
- `@Req()` and `@Res()` types change — `FastifyRequest`/`FastifyReply` not Express types
- Multer → `@fastify/multipart`, Helmet → `@fastify/helmet`, etc.
- ~2x throughput over Express for JSON APIs

## Serialization & Response Performance

### ClassSerializerInterceptor
```typescript
// Exclude sensitive fields globally
@UseInterceptors(ClassSerializerInterceptor)
@Controller()
export class AppController {}

// In entity/DTO
@Exclude()
password: string;

@Expose({ groups: ['admin'] })
internalNotes: string;

// Serialize with groups
@SerializeOptions({ groups: ['admin'] })
@Get('admin/users')
findAllAdmin() {}
```

### Custom Serialization
```typescript
// For complex transformation, a dedicated interceptor is cleaner
@Injectable()
export class TransformInterceptor implements NestInterceptor {
  intercept(context: ExecutionContext, next: CallHandler) {
    return next.handle().pipe(
      map(data => ({
        success: true,
        data,
        timestamp: Date.now(),
      })),
    );
  }
}
```

## Compression & Helmet
```typescript
import compression from 'compression';
import helmet from 'helmet';

app.use(compression());
app.use(helmet());
// Fastify: app.register(fastifyCompress); app.register(fastifyHelmet);
```

## Streaming Large Responses
```typescript
@Get('export')
export(@Res({ passthrough: true }) res: Response) {
  const stream = this.service.getDataStream();
  res.set({ 'Content-Type': 'text/csv' });
  return new StreamableFile(stream);
}
```
- `StreamableFile` — Nest-native way to stream, works with both Express and Fastify
- Don't buffer large datasets in memory — stream from database/file

## Shutdown Hooks
```typescript
// main.ts
app.enableShutdownHooks();

// In a service
@Injectable()
export class CleanupService implements OnModuleDestroy {
  async onModuleDestroy() {
    // Close connections, flush buffers, drain queues
    await this.db.disconnect();
    await this.cache.quit();
  }
}
```
- Without `enableShutdownHooks()`, `onModuleDestroy` / `beforeApplicationShutdown` never fire
- Crucial for graceful shutdown in Kubernetes / containerized deployments
- Handles SIGTERM, SIGINT

## Lazy Loading Modules
```typescript
// Reduces startup time by loading modules on demand
const { HeavyModule } = await import('./heavy.module');
const moduleRef = await this.lazyModuleLoader.load(() => HeavyModule);
```
- Serverless / cold-start optimization
- Lazy modules can't register controllers — only providers

---
name: mysql8-design-crm
version: "1.0.0"
description: >
  MySQL 8 database schema design for CRM systems. Use this skill whenever the user needs to design,
  review, optimize, or generate database schemas for Customer Relationship Management systems.
  Triggers on: CRM database design, CRM schema, customer database, contact management database,
  sales pipeline database, lead tracking schema, opportunity management tables, CRM entity relationships,
  CRM data model, accounts/contacts/deals schema, activity tracking database, CRM audit trail,
  CRM custom fields, EAV pattern for CRM, CRM soft deletes, polymorphic relationships in CRM,
  sales funnel database, customer lifecycle database, CRM normalization, CRM indexing strategy,
  MySQL CRM tables, or any database design work involving customer relationship management concepts.
  Also trigger when the user mentions designing tables for contacts, accounts, opportunities, leads,
  deals, pipelines, activities, notes, tasks, campaigns, or any combination of these CRM entities.
---

# MySQL 8 CRM Database Design Skill

A comprehensive guide for designing production-quality MySQL 8 database schemas for CRM (Customer Relationship Management) systems. This skill covers everything from core entity design to advanced patterns like EAV custom fields, polymorphic activities, audit trails, and multi-tenant architectures.

## How to Use This Skill

This skill is organized into a main guide (this file) and detailed reference documents. Read the relevant reference file before generating any SQL or making design decisions.

### Reference Files

Read these from `references/` as needed:

| File | When to Read |
|------|-------------|
| `core-entities.md` | Designing the foundational CRM tables (accounts, contacts, leads, opportunities, etc.) |
| `relationships-and-normalization.md` | Establishing foreign keys, junction tables, and achieving proper normal forms |
| `indexing-and-performance.md` | Creating indexes, query optimization, partitioning, and performance tuning |
| `custom-fields-and-flexibility.md` | Implementing EAV patterns, JSON columns, or hybrid approaches for user-defined fields |
| `audit-and-soft-deletes.md` | Change tracking, audit trails, soft delete patterns, and compliance logging |
| `activities-and-timeline.md` | Polymorphic activity feeds, notes, tasks, emails, calls, and event tracking |
| `security-and-multitenancy.md` | Row-level security, role-based access, tenant isolation, and data privacy |
| `migrations-and-seeding.md` | Schema versioning, migration scripts, and realistic test data generation |
| `reference-schemas.md` | Complete example schemas you can use as starting points |

## Core Design Principles

When designing a CRM database on MySQL 8, always follow these principles:

1. **Relational integrity first.** Define FOREIGN KEY constraints at the database level. Never rely on application code alone to maintain referential integrity.

2. **Normalize to 3NF, then denormalize deliberately.** Start at Third Normal Form. Only denormalize when you have measured performance evidence, and document the reason.

3. **Consistent naming conventions.** Use `snake_case` for all identifiers. Table names are plural (`contacts`, `accounts`). Foreign keys follow the pattern `{singular_referenced_table}_id` (e.g., `account_id`). Timestamps are `created_at`, `updated_at`, `deleted_at`.

4. **Every table gets an audit baseline.** At minimum: `id` (BIGINT UNSIGNED AUTO_INCREMENT), `created_at`, `updated_at`. Most CRM tables also need `created_by` and `updated_by`.

5. **Soft deletes over hard deletes.** CRM data has legal, compliance, and historical reporting value. Use `deleted_at` (TIMESTAMP NULL) rather than DELETE statements.

6. **Use BIGINT UNSIGNED for primary keys.** INT runs out at ~2.1 billion. CRM tables like activities and audit logs grow fast. BIGINT UNSIGNED gives you headroom through 18.4 quintillion.

7. **UTF8MB4 everywhere.** Always `CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci`. Customer names, notes, and communications contain international characters and emoji.

8. **InnoDB only.** All tables use InnoDB for transaction support, row-level locking, foreign key enforcement, and crash recovery.

9. **Timestamps use DATETIME(3) or TIMESTAMP.** For CRM, prefer `DATETIME(3)` for event times (timezone-independent, millisecond precision). Use `TIMESTAMP` for `created_at`/`updated_at` with `DEFAULT CURRENT_TIMESTAMP` and `ON UPDATE CURRENT_TIMESTAMP`.

10. **Design for integration.** CRM systems connect to email, marketing, billing, and support tools. Include `external_id` or `external_source` columns on entities that sync with third-party systems.

## Standard Table Template

Every CRM table should follow this baseline structure:

```sql
CREATE TABLE `table_name` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,

    -- entity-specific columns here --

    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    `updated_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    `deleted_at` TIMESTAMP NULL DEFAULT NULL,
    `created_by` BIGINT UNSIGNED NULL DEFAULT NULL,
    `updated_by` BIGINT UNSIGNED NULL DEFAULT NULL,

    PRIMARY KEY (`id`),
    INDEX `idx_table_name_deleted_at` (`deleted_at`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
```

## Workflow for Designing a CRM Schema

Follow this sequence when the user asks you to design a CRM database:

1. **Clarify scope.** Determine which CRM modules are needed: contacts/accounts, sales pipeline, marketing/campaigns, support/tickets, or all of the above.

2. **Read the relevant reference files.** Always start with `core-entities.md`. Add others based on the modules identified.

3. **Design entities first, relationships second.** List the tables and their columns, then define the foreign keys and junction tables.

4. **Apply indexing strategy.** Read `indexing-and-performance.md` and add indexes for every foreign key, every column used in WHERE/JOIN/ORDER BY, and composite indexes for common query patterns.

5. **Add flexibility layer.** If the user needs custom fields, read `custom-fields-and-flexibility.md` and choose between EAV, JSON columns, or a hybrid approach.

6. **Add audit and compliance.** Read `audit-and-soft-deletes.md` and implement the appropriate level of change tracking.

7. **Generate migration scripts.** Read `migrations-and-seeding.md` and output versioned, idempotent migration SQL.

8. **Review and validate.** Walk through the schema checking for: missing indexes on FKs, missing NOT NULL constraints, missing default values, orphan risk, and query patterns that would cause full table scans.

## MySQL 8 Features to Leverage

These MySQL 8 specific features are particularly valuable for CRM schemas:

- **JSON columns** for semi-structured data (custom fields, metadata, integration payloads). See `custom-fields-and-flexibility.md`.
- **Generated columns** (VIRTUAL or STORED) to extract and index JSON values.
- **Functional indexes** (8.0.13+) to index expressions without explicit generated columns.
- **Multi-valued indexes** (8.0.17+) to index JSON arrays efficiently.
- **Common Table Expressions (CTEs)** for recursive queries on hierarchical data (org charts, account hierarchies, nested categories).
- **Window functions** for pipeline analytics (running totals, rank, lead/lag).
- **CHECK constraints** for data validation at the database level.
- **DEFAULT expressions** for computed defaults.
- **Invisible indexes** for safe testing of index removal.
- **Descending indexes** for optimizing ORDER BY ... DESC queries.

## Quick Decision Guide

| Situation | Action |
|-----------|--------|
| User needs a full CRM from scratch | Read `core-entities.md` + `reference-schemas.md`, design all modules |
| User needs just contacts + accounts | Read `core-entities.md`, design the contact/account module only |
| User asks about custom fields | Read `custom-fields-and-flexibility.md` |
| User has performance concerns | Read `indexing-and-performance.md` |
| User needs GDPR/compliance support | Read `audit-and-soft-deletes.md` + `security-and-multitenancy.md` |
| User is building multi-tenant SaaS CRM | Read `security-and-multitenancy.md` |
| User wants to track all user activity | Read `activities-and-timeline.md` |
| User needs migration scripts | Read `migrations-and-seeding.md` |
| User wants a ready-to-use schema | Read `reference-schemas.md` |

FILE:references/security-and-multitenancy.md
# Security and Multi-Tenancy

CRM systems contain sensitive customer data. This reference covers database-level security patterns, role-based access control, and multi-tenant isolation strategies for MySQL 8.

## Role-Based Access Control (RBAC)

### Simple Role System

For most CRMs, an ENUM on the users table is sufficient:

```sql
ALTER TABLE `users`
    ADD COLUMN `role` ENUM('admin', 'manager', 'sales_rep', 'support_agent', 'read_only')
    NOT NULL DEFAULT 'sales_rep';
```

### Advanced RBAC with Permissions

For fine-grained control, implement a roles-and-permissions system:

```sql
CREATE TABLE `roles` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `name` VARCHAR(100) NOT NULL,
    `description` TEXT NULL DEFAULT NULL,
    `is_system` TINYINT(1) NOT NULL DEFAULT 0 COMMENT 'System roles cannot be deleted',
    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    `updated_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,

    PRIMARY KEY (`id`),
    UNIQUE INDEX `uq_roles_name` (`name`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

CREATE TABLE `permissions` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `name` VARCHAR(100) NOT NULL COMMENT 'e.g., contacts.view, contacts.edit, contacts.delete',
    `entity_type` VARCHAR(50) NOT NULL COMMENT 'e.g., contact, account, opportunity',
    `action` VARCHAR(50) NOT NULL COMMENT 'e.g., view, create, edit, delete, export, import',
    `description` TEXT NULL DEFAULT NULL,

    PRIMARY KEY (`id`),
    UNIQUE INDEX `uq_permissions_name` (`name`),
    INDEX `idx_permissions_entity_action` (`entity_type`, `action`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

CREATE TABLE `role_permissions` (
    `role_id` BIGINT UNSIGNED NOT NULL,
    `permission_id` BIGINT UNSIGNED NOT NULL,

    PRIMARY KEY (`role_id`, `permission_id`),
    INDEX `idx_rp_permission` (`permission_id`),

    CONSTRAINT `fk_rp_role` FOREIGN KEY (`role_id`) REFERENCES `roles` (`id`) ON DELETE CASCADE,
    CONSTRAINT `fk_rp_permission` FOREIGN KEY (`permission_id`) REFERENCES `permissions` (`id`) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

CREATE TABLE `user_roles` (
    `user_id` BIGINT UNSIGNED NOT NULL,
    `role_id` BIGINT UNSIGNED NOT NULL,

    PRIMARY KEY (`user_id`, `role_id`),
    INDEX `idx_ur_role` (`role_id`),

    CONSTRAINT `fk_ur_user` FOREIGN KEY (`user_id`) REFERENCES `users` (`id`) ON DELETE CASCADE,
    CONSTRAINT `fk_ur_role` FOREIGN KEY (`role_id`) REFERENCES `roles` (`id`) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
```

### Record-Level Access (Record Ownership)

CRM commonly restricts data visibility by ownership:
- **Private:** Users see only records they own
- **Team:** Users see records owned by anyone on their team
- **Hierarchy:** Managers see records owned by their direct reports and below
- **Public:** Everyone sees everything

Implement with a teams/territories table:

```sql
CREATE TABLE `teams` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `name` VARCHAR(100) NOT NULL,
    `description` TEXT NULL DEFAULT NULL,
    `parent_team_id` BIGINT UNSIGNED NULL DEFAULT NULL,
    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    `updated_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,

    PRIMARY KEY (`id`),
    INDEX `idx_teams_parent` (`parent_team_id`),
    CONSTRAINT `fk_teams_parent` FOREIGN KEY (`parent_team_id`) REFERENCES `teams` (`id`) ON DELETE SET NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

CREATE TABLE `team_members` (
    `team_id` BIGINT UNSIGNED NOT NULL,
    `user_id` BIGINT UNSIGNED NOT NULL,
    `role_in_team` ENUM('member', 'lead', 'manager') NOT NULL DEFAULT 'member',

    PRIMARY KEY (`team_id`, `user_id`),
    INDEX `idx_tm_user` (`user_id`),

    CONSTRAINT `fk_tm_team` FOREIGN KEY (`team_id`) REFERENCES `teams` (`id`) ON DELETE CASCADE,
    CONSTRAINT `fk_tm_user` FOREIGN KEY (`user_id`) REFERENCES `users` (`id`) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
```

Query pattern for team-based visibility:

```sql
-- Get contacts visible to a user based on their teams
SELECT c.* FROM contacts c
WHERE c.deleted_at IS NULL
    AND (
        c.owner_id = :current_user_id  -- own records
        OR c.owner_id IN (              -- team members' records
            SELECT tm2.user_id
            FROM team_members tm1
            INNER JOIN team_members tm2 ON tm2.team_id = tm1.team_id
            WHERE tm1.user_id = :current_user_id
        )
    );
```

For high-traffic CRM systems, cache the set of visible user IDs per user to avoid running this subquery on every request.

## Multi-Tenancy Patterns

### Option 1: Shared Database, Tenant Column (Most Common)

Add a `tenant_id` column to every table and filter every query by it.

```sql
-- Add tenant_id to all CRM tables
ALTER TABLE `accounts` ADD COLUMN `tenant_id` BIGINT UNSIGNED NOT NULL AFTER `id`;
ALTER TABLE `contacts` ADD COLUMN `tenant_id` BIGINT UNSIGNED NOT NULL AFTER `id`;
ALTER TABLE `opportunities` ADD COLUMN `tenant_id` BIGINT UNSIGNED NOT NULL AFTER `id`;
ALTER TABLE `leads` ADD COLUMN `tenant_id` BIGINT UNSIGNED NOT NULL AFTER `id`;
ALTER TABLE `activities` ADD COLUMN `tenant_id` BIGINT UNSIGNED NOT NULL AFTER `id`;

-- Tenants table
CREATE TABLE `tenants` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `name` VARCHAR(255) NOT NULL,
    `domain` VARCHAR(255) NULL DEFAULT NULL,
    `plan` ENUM('free', 'starter', 'professional', 'enterprise') NOT NULL DEFAULT 'free',
    `is_active` TINYINT(1) NOT NULL DEFAULT 1,
    `settings` JSON NULL DEFAULT NULL,
    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    `updated_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,

    PRIMARY KEY (`id`),
    UNIQUE INDEX `uq_tenants_domain` (`domain`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
```

Critical rules for shared-database multi-tenancy:

1. **Every query includes tenant_id.** No exceptions. A missed filter exposes data across tenants.

2. **tenant_id is the first column in every composite index:**
```sql
-- Correct: tenant_id first
ALTER TABLE `contacts`
    ADD INDEX `idx_contacts_tenant_email` (`tenant_id`, `email`);
ALTER TABLE `contacts`
    ADD INDEX `idx_contacts_tenant_account` (`tenant_id`, `account_id`);

-- Wrong: tenant_id missing or not first
ALTER TABLE `contacts`
    ADD INDEX `idx_contacts_email` (`email`);  -- leaks across tenants
```

3. **Unique constraints include tenant_id:**
```sql
-- Correct: unique per tenant
ALTER TABLE `pipelines`
    ADD UNIQUE INDEX `uq_pipelines_tenant_name` (`tenant_id`, `name`);

-- Wrong: globally unique (tenant A and B can't both have "Sales Pipeline")
ALTER TABLE `pipelines`
    ADD UNIQUE INDEX `uq_pipelines_name` (`name`);
```

4. **Foreign keys should stay within a tenant.** Add application-level checks to ensure cross-table references share the same `tenant_id`.

### Option 2: Separate Schema per Tenant

Each tenant gets their own MySQL schema (database). The application connects to the right schema based on the tenant.

```sql
CREATE DATABASE `crm_tenant_acme`;
CREATE DATABASE `crm_tenant_globex`;
```

Pros:
- Complete isolation — no risk of cross-tenant data leakage
- Easy to back up, restore, or migrate individual tenants
- Per-tenant schema customization is possible

Cons:
- Schema migrations must be applied to every tenant database
- Connection pool management becomes complex
- Harder to run cross-tenant analytics or admin queries
- MySQL has limits on the number of open tables/databases

Best for: Enterprise CRM with strict isolation requirements and fewer than 100 tenants.

### Option 3: Separate Database Server per Tenant

Each tenant gets a dedicated MySQL instance. Maximum isolation but highest operational cost. Only appropriate for very large enterprise customers with regulatory requirements.

## Data Encryption

### Encryption at Rest

Enable InnoDB tablespace encryption:

```sql
-- Enable encryption for a table
ALTER TABLE `contacts` ENCRYPTION='Y';

-- Enable encryption for all new tables by default
SET GLOBAL default_table_encryption = ON;
```

Requires configuring a keyring plugin (e.g., `keyring_file`, `keyring_encrypted_file`, or a KMS-backed keyring like `keyring_aws`).

### Encryption of Sensitive Columns

For columns containing highly sensitive data (SSN, credit card info, health data), consider application-level encryption:

```sql
-- Store encrypted values
ALTER TABLE `contacts`
    ADD COLUMN `ssn_encrypted` VARBINARY(255) NULL DEFAULT NULL,
    ADD COLUMN `ssn_hash` VARCHAR(64) NULL DEFAULT NULL COMMENT 'SHA-256 hash for lookup';
```

Encrypt/decrypt in the application layer using AES-256-GCM or similar. Store a hash for indexed lookups.

**Never store plaintext:** passwords, social security numbers, credit card numbers, bank account numbers, API keys, or authentication tokens.

## Input Validation at Database Level

Use CHECK constraints (MySQL 8.0.16+) to enforce data quality:

```sql
ALTER TABLE `contacts`
    ADD CONSTRAINT `chk_contacts_email` CHECK (
        `email` IS NULL OR `email` REGEXP '^[^@]+@[^@]+\\.[^@]+$'
    ),
    ADD CONSTRAINT `chk_contacts_phone` CHECK (
        `phone` IS NULL OR LENGTH(`phone`) >= 7
    );

ALTER TABLE `opportunities`
    ADD CONSTRAINT `chk_opp_amount_positive` CHECK (
        `amount` IS NULL OR `amount` >= 0
    ),
    ADD CONSTRAINT `chk_opp_probability_range` CHECK (
        `probability` IS NULL OR (`probability` >= 0 AND `probability` <= 100)
    );

ALTER TABLE `accounts`
    ADD CONSTRAINT `chk_accounts_country_code` CHECK (
        `billing_country` IS NULL OR LENGTH(`billing_country`) = 2
    );
```

CHECK constraints are not a substitute for application-level validation, but they provide a last line of defense against bad data.

## MySQL User Privileges

Follow the principle of least privilege for database users:

```sql
-- Application user: read and write to CRM tables only
CREATE USER 'crm_app'@'%' IDENTIFIED BY '<strong_password>';
GRANT SELECT, INSERT, UPDATE, DELETE ON crm.* TO 'crm_app'@'%';

-- Read-only reporting user
CREATE USER 'crm_reports'@'%' IDENTIFIED BY '<strong_password>';
GRANT SELECT ON crm.* TO 'crm_reports'@'%';

-- Migration user: can alter schema
CREATE USER 'crm_migrate'@'%' IDENTIFIED BY '<strong_password>';
GRANT ALL PRIVILEGES ON crm.* TO 'crm_migrate'@'%';

-- Never use root for application connections
```

FILE:references/activities-and-timeline.md
# Activities and Timeline

A CRM's value comes from recording every interaction with customers. This reference covers designing the activity tracking system — the polymorphic timeline that captures calls, emails, meetings, notes, tasks, and any other touchpoint.

## The Activity Model

### Core Design: Polymorphic Activity Table

Activities can relate to any CRM entity (contact, account, opportunity, lead). Use a polymorphic reference pattern:

```sql
CREATE TABLE `activities` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,

    -- What type of activity
    `activity_type` ENUM('call', 'email', 'meeting', 'note', 'task',
                          'sms', 'chat', 'social', 'document', 'status_change',
                          'stage_change', 'assignment', 'system') NOT NULL,

    -- What entity this activity relates to (polymorphic)
    `entity_type` VARCHAR(50) NOT NULL COMMENT 'contact, account, opportunity, lead',
    `entity_id` BIGINT UNSIGNED NOT NULL,

    -- When it happened
    `activity_date` DATETIME(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),

    -- Who performed the activity
    `user_id` BIGINT UNSIGNED NULL DEFAULT NULL COMMENT 'CRM user who performed this',

    -- Content
    `subject` VARCHAR(500) NULL DEFAULT NULL,
    `body` TEXT NULL DEFAULT NULL,
    `body_html` MEDIUMTEXT NULL DEFAULT NULL COMMENT 'Rich text version for emails',

    -- Activity-specific metadata stored as JSON
    `metadata` JSON NULL DEFAULT NULL,

    -- Duration tracking (for calls, meetings)
    `duration_minutes` INT UNSIGNED NULL DEFAULT NULL,

    -- Task-specific fields
    `is_completed` TINYINT(1) NULL DEFAULT NULL COMMENT 'Only for tasks',
    `due_date` DATETIME NULL DEFAULT NULL COMMENT 'Only for tasks',
    `priority` ENUM('low', 'medium', 'high', 'urgent') NULL DEFAULT NULL,

    -- Associations (an activity can also link to secondary entities)
    `account_id` BIGINT UNSIGNED NULL DEFAULT NULL COMMENT 'Denormalized for fast account timeline queries',
    `contact_id` BIGINT UNSIGNED NULL DEFAULT NULL COMMENT 'Specific contact involved',
    `opportunity_id` BIGINT UNSIGNED NULL DEFAULT NULL COMMENT 'Related deal',

    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    `updated_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    `deleted_at` TIMESTAMP NULL DEFAULT NULL,
    `created_by` BIGINT UNSIGNED NULL DEFAULT NULL,

    PRIMARY KEY (`id`),
    INDEX `idx_activities_entity` (`entity_type`, `entity_id`, `activity_date` DESC),
    INDEX `idx_activities_user` (`user_id`, `activity_date` DESC),
    INDEX `idx_activities_type` (`activity_type`, `activity_date` DESC),
    INDEX `idx_activities_account` (`account_id`, `activity_date` DESC),
    INDEX `idx_activities_contact` (`contact_id`, `activity_date` DESC),
    INDEX `idx_activities_opportunity` (`opportunity_id`, `activity_date` DESC),
    INDEX `idx_activities_due_date` (`due_date`, `is_completed`),
    INDEX `idx_activities_deleted_at` (`deleted_at`),

    CONSTRAINT `fk_activities_user` FOREIGN KEY (`user_id`) REFERENCES `users` (`id`) ON DELETE SET NULL,
    CONSTRAINT `fk_activities_account` FOREIGN KEY (`account_id`) REFERENCES `accounts` (`id`) ON DELETE SET NULL,
    CONSTRAINT `fk_activities_contact` FOREIGN KEY (`contact_id`) REFERENCES `contacts` (`id`) ON DELETE SET NULL,
    CONSTRAINT `fk_activities_opportunity` FOREIGN KEY (`opportunity_id`) REFERENCES `opportunities` (`id`) ON DELETE SET NULL,
    CONSTRAINT `fk_activities_created_by` FOREIGN KEY (`created_by`) REFERENCES `users` (`id`) ON DELETE SET NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
```

### Why Denormalized Association Columns?

The `account_id`, `contact_id`, and `opportunity_id` columns on the activities table are intentionally denormalized. Here is why:

- The most common CRM query is "show me the timeline for account X" — having `account_id` directly on activities avoids a JOIN through the polymorphic entity reference.
- An activity might relate to multiple entities simultaneously (a call with Contact A about Opportunity B at Account C).
- The denormalized FKs enable proper foreign key constraints, which the polymorphic `entity_type` + `entity_id` pattern cannot.

### Metadata by Activity Type

The `metadata` JSON column stores activity-type-specific data. Document the expected structure per type:

**Call:**
```json
{
    "direction": "outbound",
    "phone_number": "+1-555-0123",
    "outcome": "connected",
    "recording_url": "https://...",
    "voicemail_left": false
}
```

**Email:**
```json
{
    "direction": "outbound",
    "from": "[email protected]",
    "to": ["[email protected]"],
    "cc": [],
    "message_id": "<abc123@mail>",
    "thread_id": "thread_456",
    "has_attachments": true,
    "opened": true,
    "opened_at": "2026-04-01T10:30:00Z",
    "clicked": false
}
```

**Meeting:**
```json
{
    "location": "Zoom",
    "meeting_url": "https://zoom.us/j/123",
    "attendees": [
        {"email": "[email protected]", "status": "accepted"},
        {"email": "[email protected]", "status": "accepted"}
    ],
    "calendar_event_id": "cal_abc123",
    "outcome": "completed"
}
```

**Stage Change (system-generated):**
```json
{
    "old_stage_id": 3,
    "new_stage_id": 4,
    "old_stage_name": "Proposal",
    "new_stage_name": "Negotiation",
    "pipeline_id": 1
}
```

### Multi-Entity Activity Associations

An activity can relate to multiple entities. For example, a meeting might involve three contacts from two accounts about one opportunity. Use a junction table for additional associations beyond the primary:

```sql
CREATE TABLE `activity_associations` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `activity_id` BIGINT UNSIGNED NOT NULL,
    `entity_type` VARCHAR(50) NOT NULL,
    `entity_id` BIGINT UNSIGNED NOT NULL,
    `association_type` VARCHAR(50) NULL DEFAULT NULL COMMENT 'e.g., attendee, mentioned, related',
    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,

    PRIMARY KEY (`id`),
    UNIQUE INDEX `uq_activity_assoc` (`activity_id`, `entity_type`, `entity_id`),
    INDEX `idx_activity_assoc_entity` (`entity_type`, `entity_id`, `created_at` DESC),

    CONSTRAINT `fk_activity_assoc_activity` FOREIGN KEY (`activity_id`)
        REFERENCES `activities` (`id`) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
```

## Notes

Notes are a subset of activities but are so commonly queried independently that some CRM systems give them their own table:

```sql
CREATE TABLE `notes` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `entity_type` VARCHAR(50) NOT NULL,
    `entity_id` BIGINT UNSIGNED NOT NULL,
    `title` VARCHAR(255) NULL DEFAULT NULL,
    `body` TEXT NOT NULL,
    `is_pinned` TINYINT(1) NOT NULL DEFAULT 0,
    `user_id` BIGINT UNSIGNED NULL DEFAULT NULL,
    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    `updated_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    `deleted_at` TIMESTAMP NULL DEFAULT NULL,

    PRIMARY KEY (`id`),
    INDEX `idx_notes_entity` (`entity_type`, `entity_id`, `is_pinned` DESC, `created_at` DESC),
    INDEX `idx_notes_user` (`user_id`),
    FULLTEXT INDEX `ft_notes_search` (`title`, `body`),

    CONSTRAINT `fk_notes_user` FOREIGN KEY (`user_id`) REFERENCES `users` (`id`) ON DELETE SET NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
```

## Tasks

Tasks deserve their own table when the CRM has a dedicated task management workflow (assigned tasks, due dates, reminders, recurring tasks).

```sql
CREATE TABLE `tasks` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `title` VARCHAR(500) NOT NULL,
    `description` TEXT NULL DEFAULT NULL,
    `status` ENUM('open', 'in_progress', 'completed', 'cancelled') NOT NULL DEFAULT 'open',
    `priority` ENUM('low', 'medium', 'high', 'urgent') NOT NULL DEFAULT 'medium',
    `due_date` DATETIME NULL DEFAULT NULL,
    `completed_at` DATETIME NULL DEFAULT NULL,

    -- Who is responsible
    `assignee_id` BIGINT UNSIGNED NULL DEFAULT NULL,
    `assigned_by_id` BIGINT UNSIGNED NULL DEFAULT NULL,

    -- What entity is this task related to
    `entity_type` VARCHAR(50) NULL DEFAULT NULL,
    `entity_id` BIGINT UNSIGNED NULL DEFAULT NULL,

    -- Recurrence (optional)
    `is_recurring` TINYINT(1) NOT NULL DEFAULT 0,
    `recurrence_rule` VARCHAR(255) NULL DEFAULT NULL COMMENT 'iCal RRULE format',

    -- Reminders
    `reminder_at` DATETIME NULL DEFAULT NULL,

    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    `updated_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    `deleted_at` TIMESTAMP NULL DEFAULT NULL,
    `created_by` BIGINT UNSIGNED NULL DEFAULT NULL,

    PRIMARY KEY (`id`),
    INDEX `idx_tasks_assignee` (`assignee_id`, `status`, `due_date`),
    INDEX `idx_tasks_entity` (`entity_type`, `entity_id`),
    INDEX `idx_tasks_due_date` (`due_date`, `status`),
    INDEX `idx_tasks_status` (`status`),
    INDEX `idx_tasks_reminder` (`reminder_at`, `status`),
    INDEX `idx_tasks_deleted_at` (`deleted_at`),

    CONSTRAINT `fk_tasks_assignee` FOREIGN KEY (`assignee_id`) REFERENCES `users` (`id`) ON DELETE SET NULL,
    CONSTRAINT `fk_tasks_assigned_by` FOREIGN KEY (`assigned_by_id`) REFERENCES `users` (`id`) ON DELETE SET NULL,
    CONSTRAINT `fk_tasks_created_by` FOREIGN KEY (`created_by`) REFERENCES `users` (`id`) ON DELETE SET NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
```

## Attachments / Files

Documents attached to any CRM entity:

```sql
CREATE TABLE `attachments` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `entity_type` VARCHAR(50) NOT NULL,
    `entity_id` BIGINT UNSIGNED NOT NULL,
    `filename` VARCHAR(255) NOT NULL,
    `file_path` VARCHAR(1000) NOT NULL COMMENT 'S3 key or filesystem path',
    `file_size` BIGINT UNSIGNED NOT NULL COMMENT 'Size in bytes',
    `mime_type` VARCHAR(100) NOT NULL,
    `uploaded_by` BIGINT UNSIGNED NULL DEFAULT NULL,
    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    `deleted_at` TIMESTAMP NULL DEFAULT NULL,

    PRIMARY KEY (`id`),
    INDEX `idx_attachments_entity` (`entity_type`, `entity_id`),
    INDEX `idx_attachments_uploaded_by` (`uploaded_by`),

    CONSTRAINT `fk_attachments_uploaded_by` FOREIGN KEY (`uploaded_by`)
        REFERENCES `users` (`id`) ON DELETE SET NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
```

Design notes:
- Never store file contents (BLOBs) in MySQL. Store files in object storage (S3, GCS) and keep only the path/key in the database.
- `mime_type` enables the UI to display previews or choose appropriate icons.
- `file_size` allows enforcing storage quotas without querying the file system.

## Building the Unified Timeline Query

The timeline view for any CRM entity aggregates activities, notes, and system events:

```sql
-- Unified timeline for an account
(
    SELECT
        a.id,
        'activity' AS record_type,
        a.activity_type AS event_type,
        a.subject AS title,
        a.body AS description,
        a.activity_date AS event_date,
        a.user_id,
        u.first_name AS user_first_name,
        u.last_name AS user_last_name
    FROM activities a
    LEFT JOIN users u ON u.id = a.user_id
    WHERE a.account_id = :account_id AND a.deleted_at IS NULL
)
UNION ALL
(
    SELECT
        n.id,
        'note' AS record_type,
        'note' AS event_type,
        n.title,
        LEFT(n.body, 200) AS description,
        n.created_at AS event_date,
        n.user_id,
        u.first_name,
        u.last_name
    FROM notes n
    LEFT JOIN users u ON u.id = n.user_id
    WHERE n.entity_type = 'account' AND n.entity_id = :account_id AND n.deleted_at IS NULL
)
ORDER BY event_date DESC
LIMIT 50;
```

For performance, consider a materialized timeline view updated by application events or triggers, especially for accounts with thousands of activities.

FILE:references/indexing-and-performance.md
# Indexing and Performance

This reference covers indexing strategies, query optimization, and performance tuning for CRM databases on MySQL 8.

## Indexing Fundamentals for CRM

### Index Every Foreign Key

MySQL does not automatically index foreign key columns (InnoDB indexes the referencing column in some cases, but you should always be explicit). Without an index on every FK column:
- JOINs between parent and child tables become full table scans
- DELETE on a parent row scans the entire child table to check references
- This can cause table-level locks and cascading performance problems

```sql
-- Ensure all FK columns are indexed
ALTER TABLE `contacts` ADD INDEX `idx_contacts_account_id` (`account_id`);
ALTER TABLE `contacts` ADD INDEX `idx_contacts_owner_id` (`owner_id`);
ALTER TABLE `opportunities` ADD INDEX `idx_opportunities_account_id` (`account_id`);
ALTER TABLE `opportunities` ADD INDEX `idx_opportunities_stage_id` (`stage_id`);
ALTER TABLE `opportunities` ADD INDEX `idx_opportunities_owner_id` (`owner_id`);
ALTER TABLE `activities` ADD INDEX `idx_activities_user_id` (`user_id`);
```

### Composite Indexes for Common CRM Queries

Design composite indexes around the actual queries your CRM runs most. The column order matters — put the most selective or most commonly filtered column first.

```sql
-- Pipeline board: "Show all active deals in pipeline X, grouped by stage"
ALTER TABLE `opportunities`
    ADD INDEX `idx_opp_pipeline_board` (`pipeline_id`, `stage_id`, `deleted_at`);

-- My deals: "Show deals owned by me, sorted by expected close date"
ALTER TABLE `opportunities`
    ADD INDEX `idx_opp_owner_close` (`owner_id`, `expected_close_date`, `deleted_at`);

-- Contact search: "Find contacts by last name within an account"
ALTER TABLE `contacts`
    ADD INDEX `idx_contacts_account_name` (`account_id`, `last_name`, `first_name`);

-- Activity timeline: "Recent activities for a given entity"
ALTER TABLE `activities`
    ADD INDEX `idx_activities_entity_timeline` (`entity_type`, `entity_id`, `activity_date` DESC);

-- Lead queue: "Unassigned new leads sorted by creation date"
ALTER TABLE `leads`
    ADD INDEX `idx_leads_new_unassigned` (`status`, `owner_id`, `created_at`);
```

### Covering Indexes

A covering index includes all columns a query needs, so MySQL can satisfy the query from the index alone (no table lookup). This is especially valuable for list views and dashboards.

```sql
-- Covering index for opportunity list view
ALTER TABLE `opportunities`
    ADD INDEX `idx_opp_list_view` (
        `owner_id`, `deleted_at`, `stage_id`,
        `name`, `amount`, `expected_close_date`
    );
```

Use `EXPLAIN` and check for "Using index" in the Extra column to verify a covering index is working.

### Prefix Indexes for Long Text Columns

For VARCHAR columns longer than 191 characters (the max for utf8mb4 with a 767-byte index limit on older row formats), use prefix indexes:

```sql
-- Index only the first 50 characters of description
ALTER TABLE `accounts` ADD INDEX `idx_accounts_description` (`description`(50));

-- For URLs, index a meaningful prefix
ALTER TABLE `contacts` ADD INDEX `idx_contacts_linkedin` (`linkedin_url`(100));
```

Prefer generated columns with indexes over prefix indexes when you need exact match queries.

## MySQL 8 Specific Index Features

### Functional Indexes (MySQL 8.0.13+)

Index an expression without creating an explicit generated column:

```sql
-- Index on email domain for account matching
ALTER TABLE `contacts`
    ADD INDEX `idx_contacts_email_domain` ((SUBSTRING_INDEX(`email`, '@', -1)));

-- Index on year of created_at for reporting
ALTER TABLE `opportunities`
    ADD INDEX `idx_opp_created_year` ((YEAR(`created_at`)));

-- Case-insensitive email search
ALTER TABLE `contacts`
    ADD INDEX `idx_contacts_email_lower` ((LOWER(`email`)));
```

Important: the query must use the exact same expression as the index definition for MySQL to use it.

### Descending Indexes

MySQL 8 supports true descending indexes (earlier versions ignored DESC in index definitions):

```sql
-- Optimize "most recent first" queries
ALTER TABLE `activities`
    ADD INDEX `idx_activities_recent` (`entity_type`, `entity_id`, `activity_date` DESC);

-- Dashboard: latest deals
ALTER TABLE `opportunities`
    ADD INDEX `idx_opp_latest` (`created_at` DESC, `deleted_at`);
```

### Invisible Indexes

Test the impact of removing an index without actually dropping it:

```sql
-- Make an index invisible (optimizer ignores it)
ALTER TABLE `contacts` ALTER INDEX `idx_contacts_lead_source` INVISIBLE;

-- Run your workload and measure impact...

-- Make it visible again if needed
ALTER TABLE `contacts` ALTER INDEX `idx_contacts_lead_source` VISIBLE;

-- Or drop it if performance was unaffected
DROP INDEX `idx_contacts_lead_source` ON `contacts`;
```

### JSON Indexing Strategies

CRM systems often store semi-structured data in JSON columns (custom fields, metadata, integration payloads). MySQL cannot directly index JSON columns, but offers three approaches:

**Approach 1: Generated columns (works in MySQL 5.7+)**

```sql
-- Add a generated column that extracts a JSON value
ALTER TABLE `contacts`
    ADD COLUMN `custom_industry` VARCHAR(100)
        GENERATED ALWAYS AS (`custom_fields` ->> '$.industry') VIRTUAL,
    ADD INDEX `idx_contacts_custom_industry` (`custom_industry`);
```

**Approach 2: Functional indexes (MySQL 8.0.13+)**

```sql
-- Index a JSON path directly
ALTER TABLE `contacts`
    ADD INDEX `idx_contacts_json_industry` ((
        CAST(`custom_fields` ->> '$.industry' AS CHAR(100)) COLLATE utf8mb4_bin
    ));
```

**Approach 3: Multi-valued indexes for JSON arrays (MySQL 8.0.17+)**

```sql
-- Index elements of a JSON array
ALTER TABLE `contacts`
    ADD INDEX `idx_contacts_interests` ((
        CAST(`custom_fields` -> '$.interests' AS CHAR(50) ARRAY)
    ));

-- Query using MEMBER OF
SELECT * FROM contacts
WHERE 'machine_learning' MEMBER OF (custom_fields -> '$.interests');
```

Recommendation: Use generated STORED columns for frequently queried JSON paths — they are the most reliable and performant approach. Use functional indexes for occasional queries. Use multi-valued indexes when you need to search within JSON arrays.

## Query Optimization Patterns

### The Soft Delete Filter

Almost every CRM query needs to filter out soft-deleted records. Always include `deleted_at IS NULL` and make sure your indexes account for it:

```sql
-- BAD: Index doesn't include deleted_at
SELECT * FROM contacts WHERE account_id = 42;

-- GOOD: Index on (account_id, deleted_at), query filters both
SELECT * FROM contacts WHERE account_id = 42 AND deleted_at IS NULL;
```

Consider putting `deleted_at` as the last column in composite indexes so the optimizer can use the index for both "all records" and "non-deleted records" queries.

### Pagination

CRM list views need efficient pagination. Avoid `OFFSET` for large datasets.

```sql
-- BAD: OFFSET becomes slow as page number grows
SELECT * FROM contacts WHERE deleted_at IS NULL ORDER BY id LIMIT 25 OFFSET 10000;

-- GOOD: Keyset pagination using the last seen ID
SELECT * FROM contacts
WHERE deleted_at IS NULL AND id > :last_seen_id
ORDER BY id
LIMIT 25;
```

For sorted pagination on non-unique columns, use a composite cursor:

```sql
-- Paginate contacts sorted by last_name
SELECT * FROM contacts
WHERE deleted_at IS NULL
    AND (last_name > :last_name OR (last_name = :last_name AND id > :last_id))
ORDER BY last_name, id
LIMIT 25;
```

### Counting Large Tables

Avoid `SELECT COUNT(*)` on large CRM tables for UI display. Instead:
- Use approximate counts: `SELECT TABLE_ROWS FROM information_schema.TABLES WHERE TABLE_NAME = 'contacts'`
- Cache counts in a summary table updated by triggers or scheduled jobs
- Use `SQL_CALC_FOUND_ROWS` or window functions for "page X of Y" display

### Full-Text Search

CRM users expect to search across names, emails, descriptions, and notes. Use MySQL full-text indexes:

```sql
ALTER TABLE `contacts`
    ADD FULLTEXT INDEX `ft_contacts_search` (`first_name`, `last_name`, `email`);

ALTER TABLE `accounts`
    ADD FULLTEXT INDEX `ft_accounts_search` (`name`, `description`, `domain`);

-- Search query
SELECT * FROM contacts
WHERE MATCH(first_name, last_name, email) AGAINST ('john smith' IN BOOLEAN MODE)
    AND deleted_at IS NULL;
```

For more sophisticated search (typo tolerance, relevance ranking, faceting), consider offloading to Elasticsearch or Meilisearch and keeping MySQL as the source of truth.

## Partitioning for Large CRM Tables

Partitioning is useful for tables that grow very large (tens of millions of rows), particularly for time-series data.

### Good Candidates for Partitioning

- `activities` — partitioned by `activity_date` (RANGE partitioning by month or year)
- `audit_logs` — partitioned by `created_at`
- `email_events` — partitioned by `event_date`

### Example: Partitioning Activities by Year

```sql
CREATE TABLE `activities` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `entity_type` VARCHAR(50) NOT NULL,
    `entity_id` BIGINT UNSIGNED NOT NULL,
    `activity_type` VARCHAR(50) NOT NULL,
    `activity_date` DATETIME NOT NULL,
    `user_id` BIGINT UNSIGNED NULL,
    `summary` VARCHAR(500) NULL,
    `details` JSON NULL,
    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,

    PRIMARY KEY (`id`, `activity_date`),
    INDEX `idx_activities_entity` (`entity_type`, `entity_id`, `activity_date`),
    INDEX `idx_activities_user` (`user_id`, `activity_date`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci
PARTITION BY RANGE (YEAR(`activity_date`)) (
    PARTITION p2023 VALUES LESS THAN (2024),
    PARTITION p2024 VALUES LESS THAN (2025),
    PARTITION p2025 VALUES LESS THAN (2026),
    PARTITION p2026 VALUES LESS THAN (2027),
    PARTITION p_future VALUES LESS THAN MAXVALUE
);
```

Important constraints:
- The partition key must be part of every unique index (including the primary key). That is why `activity_date` is included in the PK.
- Foreign keys are not supported on partitioned tables. Use application-level integrity checks.
- Add new partitions proactively before the year rolls over (automate this with a scheduled job).

### Tables NOT to Partition

Do not partition `accounts`, `contacts`, `opportunities`, or `users`. These are looked up by ID across all time periods, and partitioning would not help (and may hurt) these access patterns.

## Performance Monitoring Checklist

Run these periodically on your CRM database:

```sql
-- Find tables without primary keys (should be zero)
SELECT TABLE_NAME FROM information_schema.TABLES
WHERE TABLE_SCHEMA = DATABASE()
    AND TABLE_TYPE = 'BASE TABLE'
    AND TABLE_NAME NOT IN (
        SELECT TABLE_NAME FROM information_schema.TABLE_CONSTRAINTS
        WHERE CONSTRAINT_TYPE = 'PRIMARY KEY' AND TABLE_SCHEMA = DATABASE()
    );

-- Find foreign key columns without indexes
SELECT
    kcu.TABLE_NAME, kcu.COLUMN_NAME
FROM information_schema.KEY_COLUMN_USAGE kcu
LEFT JOIN information_schema.STATISTICS s
    ON s.TABLE_SCHEMA = kcu.TABLE_SCHEMA
    AND s.TABLE_NAME = kcu.TABLE_NAME
    AND s.COLUMN_NAME = kcu.COLUMN_NAME
WHERE kcu.TABLE_SCHEMA = DATABASE()
    AND kcu.REFERENCED_TABLE_NAME IS NOT NULL
    AND s.INDEX_NAME IS NULL;

-- Find slow queries (if slow_query_log is enabled)
-- Look for queries with no index usage
SELECT * FROM mysql.slow_log
WHERE query_time > '00:00:01'
ORDER BY start_time DESC LIMIT 20;
```

FILE:references/reference-schemas.md
# Reference Schemas

This file contains complete, ready-to-use CRM database schemas for common scenarios. Copy and adapt as needed.

## Schema A: Minimal CRM (Contacts + Deals)

For small teams that need basic contact and deal tracking. Approximately 8 tables.

```sql
-- ============================================================
-- Minimal CRM Schema
-- MySQL 8.0+
-- ============================================================

SET NAMES utf8mb4;
SET FOREIGN_KEY_CHECKS = 0;

-- Users
CREATE TABLE IF NOT EXISTS `users` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `email` VARCHAR(255) NOT NULL,
    `password_hash` VARCHAR(255) NOT NULL,
    `first_name` VARCHAR(100) NOT NULL,
    `last_name` VARCHAR(100) NOT NULL,
    `role` ENUM('admin','manager','sales_rep','read_only') NOT NULL DEFAULT 'sales_rep',
    `is_active` TINYINT(1) NOT NULL DEFAULT 1,
    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    `updated_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    PRIMARY KEY (`id`),
    UNIQUE INDEX `uq_users_email` (`email`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

-- Accounts
CREATE TABLE IF NOT EXISTS `accounts` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `name` VARCHAR(255) NOT NULL,
    `domain` VARCHAR(255) NULL,
    `industry` VARCHAR(100) NULL,
    `type` ENUM('prospect','customer','partner','other') NOT NULL DEFAULT 'prospect',
    `phone` VARCHAR(30) NULL,
    `owner_id` BIGINT UNSIGNED NULL,
    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    `updated_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    `deleted_at` TIMESTAMP NULL DEFAULT NULL,
    PRIMARY KEY (`id`),
    INDEX `idx_accounts_owner` (`owner_id`),
    INDEX `idx_accounts_deleted_at` (`deleted_at`),
    CONSTRAINT `fk_accounts_owner` FOREIGN KEY (`owner_id`) REFERENCES `users` (`id`) ON DELETE SET NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

-- Contacts
CREATE TABLE IF NOT EXISTS `contacts` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `account_id` BIGINT UNSIGNED NULL,
    `first_name` VARCHAR(100) NOT NULL,
    `last_name` VARCHAR(100) NOT NULL,
    `email` VARCHAR(255) NULL,
    `phone` VARCHAR(30) NULL,
    `job_title` VARCHAR(150) NULL,
    `owner_id` BIGINT UNSIGNED NULL,
    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    `updated_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    `deleted_at` TIMESTAMP NULL DEFAULT NULL,
    PRIMARY KEY (`id`),
    INDEX `idx_contacts_account` (`account_id`),
    INDEX `idx_contacts_email` (`email`),
    INDEX `idx_contacts_name` (`last_name`, `first_name`),
    INDEX `idx_contacts_owner` (`owner_id`),
    INDEX `idx_contacts_deleted_at` (`deleted_at`),
    CONSTRAINT `fk_contacts_account` FOREIGN KEY (`account_id`) REFERENCES `accounts` (`id`) ON DELETE SET NULL,
    CONSTRAINT `fk_contacts_owner` FOREIGN KEY (`owner_id`) REFERENCES `users` (`id`) ON DELETE SET NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

-- Pipeline Stages
CREATE TABLE IF NOT EXISTS `pipeline_stages` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `name` VARCHAR(100) NOT NULL,
    `display_order` INT UNSIGNED NOT NULL DEFAULT 0,
    `probability` DECIMAL(5,2) NULL,
    `is_won` TINYINT(1) NOT NULL DEFAULT 0,
    `is_lost` TINYINT(1) NOT NULL DEFAULT 0,
    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

-- Opportunities
CREATE TABLE IF NOT EXISTS `opportunities` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `name` VARCHAR(255) NOT NULL,
    `account_id` BIGINT UNSIGNED NOT NULL,
    `stage_id` BIGINT UNSIGNED NOT NULL,
    `amount` DECIMAL(15,2) NULL,
    `expected_close_date` DATE NULL,
    `owner_id` BIGINT UNSIGNED NULL,
    `primary_contact_id` BIGINT UNSIGNED NULL,
    `description` TEXT NULL,
    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    `updated_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    `deleted_at` TIMESTAMP NULL DEFAULT NULL,
    PRIMARY KEY (`id`),
    INDEX `idx_opp_account` (`account_id`),
    INDEX `idx_opp_stage` (`stage_id`),
    INDEX `idx_opp_owner` (`owner_id`),
    INDEX `idx_opp_close_date` (`expected_close_date`),
    INDEX `idx_opp_deleted_at` (`deleted_at`),
    CONSTRAINT `fk_opp_account` FOREIGN KEY (`account_id`) REFERENCES `accounts` (`id`) ON DELETE RESTRICT,
    CONSTRAINT `fk_opp_stage` FOREIGN KEY (`stage_id`) REFERENCES `pipeline_stages` (`id`) ON DELETE RESTRICT,
    CONSTRAINT `fk_opp_owner` FOREIGN KEY (`owner_id`) REFERENCES `users` (`id`) ON DELETE SET NULL,
    CONSTRAINT `fk_opp_contact` FOREIGN KEY (`primary_contact_id`) REFERENCES `contacts` (`id`) ON DELETE SET NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

-- Activities (simplified)
CREATE TABLE IF NOT EXISTS `activities` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `activity_type` ENUM('call','email','meeting','note','task') NOT NULL,
    `entity_type` VARCHAR(50) NOT NULL,
    `entity_id` BIGINT UNSIGNED NOT NULL,
    `subject` VARCHAR(500) NULL,
    `body` TEXT NULL,
    `activity_date` DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
    `user_id` BIGINT UNSIGNED NULL,
    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    `deleted_at` TIMESTAMP NULL DEFAULT NULL,
    PRIMARY KEY (`id`),
    INDEX `idx_activities_entity` (`entity_type`, `entity_id`, `activity_date` DESC),
    INDEX `idx_activities_user` (`user_id`),
    CONSTRAINT `fk_activities_user` FOREIGN KEY (`user_id`) REFERENCES `users` (`id`) ON DELETE SET NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

-- Tags
CREATE TABLE IF NOT EXISTS `tags` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `name` VARCHAR(100) NOT NULL,
    PRIMARY KEY (`id`),
    UNIQUE INDEX `uq_tags_name` (`name`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

CREATE TABLE IF NOT EXISTS `taggables` (
    `tag_id` BIGINT UNSIGNED NOT NULL,
    `taggable_type` VARCHAR(50) NOT NULL,
    `taggable_id` BIGINT UNSIGNED NOT NULL,
    PRIMARY KEY (`tag_id`, `taggable_type`, `taggable_id`),
    INDEX `idx_taggables_target` (`taggable_type`, `taggable_id`),
    CONSTRAINT `fk_taggables_tag` FOREIGN KEY (`tag_id`) REFERENCES `tags` (`id`) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

SET FOREIGN_KEY_CHECKS = 1;

-- Default pipeline stages
INSERT INTO `pipeline_stages` (`name`, `display_order`, `probability`, `is_won`, `is_lost`) VALUES
('Qualification', 1, 10.00, 0, 0),
('Discovery', 2, 20.00, 0, 0),
('Proposal', 3, 50.00, 0, 0),
('Negotiation', 4, 75.00, 0, 0),
('Closed Won', 5, 100.00, 1, 0),
('Closed Lost', 6, 0.00, 0, 1);
```

## Schema B: Full-Featured CRM

For teams that need the complete CRM stack: contacts, accounts, leads, multi-pipeline deals, products, activities, tasks, notes, audit trail, custom fields, RBAC, and tags. This builds on Schema A and adds all the enterprise features.

Tables included (in creation order):
1. `users`
2. `roles`, `permissions`, `role_permissions`, `user_roles`
3. `teams`, `team_members`
4. `accounts`
5. `contacts`
6. `leads`
7. `pipelines`, `pipeline_stages`
8. `products`
9. `opportunities`
10. `opportunity_line_items`
11. `opportunity_contacts`
12. `activities`
13. `activity_associations`
14. `notes`
15. `tasks`
16. `attachments`
17. `tags`, `taggables`
18. `custom_field_definitions`, `custom_field_values`
19. `audit_logs`

See the individual reference files for the complete CREATE TABLE statements for each. Combine them in the order listed above to produce a full migration script.

## Schema C: Multi-Tenant SaaS CRM

Extends Schema B with a `tenant_id` column on every table. Key differences:

```sql
-- Add to every CRM table:
ALTER TABLE `accounts` ADD COLUMN `tenant_id` BIGINT UNSIGNED NOT NULL AFTER `id`;
ALTER TABLE `contacts` ADD COLUMN `tenant_id` BIGINT UNSIGNED NOT NULL AFTER `id`;
-- ... repeat for all tables

-- Modify every index to be tenant-scoped:
ALTER TABLE `contacts` DROP INDEX `idx_contacts_email`;
ALTER TABLE `contacts` ADD INDEX `idx_contacts_tenant_email` (`tenant_id`, `email`);

-- Modify unique constraints to be tenant-scoped:
ALTER TABLE `pipelines` ADD UNIQUE INDEX `uq_pipelines_tenant_name` (`tenant_id`, `name`);
```

See `security-and-multitenancy.md` for the complete multi-tenancy implementation guide.

## Common CRM Queries Reference

These are the most commonly run queries in a CRM, useful for verifying that your schema and indexes support them efficiently:

```sql
-- 1. Pipeline board (kanban view)
SELECT o.*, ps.name AS stage_name, a.name AS account_name
FROM opportunities o
INNER JOIN pipeline_stages ps ON ps.id = o.stage_id
INNER JOIN accounts a ON a.id = o.account_id
WHERE o.pipeline_id = :pipeline_id AND o.deleted_at IS NULL
ORDER BY ps.display_order, o.updated_at DESC;

-- 2. My open deals
SELECT o.*, a.name AS account_name
FROM opportunities o
INNER JOIN accounts a ON a.id = o.account_id
WHERE o.owner_id = :user_id
    AND o.deleted_at IS NULL
    AND o.stage_id NOT IN (SELECT id FROM pipeline_stages WHERE is_won = 1 OR is_lost = 1)
ORDER BY o.expected_close_date;

-- 3. Account 360 view
SELECT a.*,
    (SELECT COUNT(*) FROM contacts c WHERE c.account_id = a.id AND c.deleted_at IS NULL) AS contact_count,
    (SELECT COUNT(*) FROM opportunities o WHERE o.account_id = a.id AND o.deleted_at IS NULL) AS deal_count,
    (SELECT SUM(o.amount) FROM opportunities o
     INNER JOIN pipeline_stages ps ON ps.id = o.stage_id
     WHERE o.account_id = a.id AND ps.is_won = 1 AND o.deleted_at IS NULL) AS total_won_revenue
FROM accounts a
WHERE a.id = :account_id;

-- 4. Lead conversion funnel
SELECT
    source,
    COUNT(*) AS total_leads,
    SUM(CASE WHEN status = 'converted' THEN 1 ELSE 0 END) AS converted,
    ROUND(SUM(CASE WHEN status = 'converted' THEN 1 ELSE 0 END) / COUNT(*) * 100, 1) AS conversion_rate
FROM leads
WHERE deleted_at IS NULL AND created_at >= :start_date
GROUP BY source
ORDER BY total_leads DESC;

-- 5. Sales forecast
SELECT
    ps.name AS stage_name,
    COUNT(*) AS deal_count,
    SUM(o.amount) AS total_value,
    SUM(o.amount * COALESCE(o.probability, ps.probability) / 100) AS weighted_value
FROM opportunities o
INNER JOIN pipeline_stages ps ON ps.id = o.stage_id
WHERE o.pipeline_id = :pipeline_id
    AND o.deleted_at IS NULL
    AND ps.is_won = 0 AND ps.is_lost = 0
    AND o.expected_close_date BETWEEN :start_date AND :end_date
GROUP BY ps.id, ps.name, ps.display_order
ORDER BY ps.display_order;

-- 6. Activity leaderboard (rep productivity)
SELECT
    u.first_name, u.last_name,
    COUNT(*) AS total_activities,
    SUM(CASE WHEN a.activity_type = 'call' THEN 1 ELSE 0 END) AS calls,
    SUM(CASE WHEN a.activity_type = 'email' THEN 1 ELSE 0 END) AS emails,
    SUM(CASE WHEN a.activity_type = 'meeting' THEN 1 ELSE 0 END) AS meetings
FROM activities a
INNER JOIN users u ON u.id = a.user_id
WHERE a.activity_date BETWEEN :start_date AND :end_date
    AND a.deleted_at IS NULL
GROUP BY u.id, u.first_name, u.last_name
ORDER BY total_activities DESC;

-- 7. Contacts due for follow-up (no activity in 30 days)
SELECT c.*, a.name AS account_name,
    MAX(act.activity_date) AS last_activity_date
FROM contacts c
LEFT JOIN accounts a ON a.id = c.account_id
LEFT JOIN activities act ON act.contact_id = c.id AND act.deleted_at IS NULL
WHERE c.deleted_at IS NULL
    AND c.lifecycle_stage IN ('lead', 'mql', 'sql', 'opportunity')
GROUP BY c.id
HAVING last_activity_date < DATE_SUB(NOW(), INTERVAL 30 DAY)
    OR last_activity_date IS NULL
ORDER BY last_activity_date ASC
LIMIT 50;

-- 8. Overdue tasks
SELECT t.*, u.first_name, u.last_name
FROM tasks t
LEFT JOIN users u ON u.id = t.assignee_id
WHERE t.status IN ('open', 'in_progress')
    AND t.due_date < NOW()
    AND t.deleted_at IS NULL
ORDER BY t.due_date ASC;
```

Run `EXPLAIN` on each of these queries against your schema to verify index usage. If any show `type: ALL` (full table scan) or missing indexes, revisit `indexing-and-performance.md`.

FILE:references/core-entities.md
# Core CRM Entities

This reference covers the foundational tables that every CRM system needs. These entities form the backbone of customer relationship management: tracking who your customers are, how you acquire them, and what business you do with them.

## Entity Overview

A well-designed CRM revolves around these core entities and their relationships:

- **Accounts** (companies/organizations) — the central entity everything connects to
- **Contacts** (people) — individuals associated with accounts
- **Leads** (unqualified prospects) — potential customers not yet converted
- **Opportunities** (deals) — revenue-bearing objects tied to accounts
- **Products** — what you sell, referenced by opportunities
- **Pipeline Stages** — the stages a deal moves through
- **Users** (CRM users/sales reps) — the people who use the CRM system

## Table Designs

### Users (CRM System Users)

The users table represents sales reps, managers, admins — anyone who logs into the CRM. This is not the same as contacts or customers.

```sql
CREATE TABLE `users` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `email` VARCHAR(255) NOT NULL,
    `password_hash` VARCHAR(255) NOT NULL,
    `first_name` VARCHAR(100) NOT NULL,
    `last_name` VARCHAR(100) NOT NULL,
    `phone` VARCHAR(30) NULL DEFAULT NULL,
    `avatar_url` VARCHAR(500) NULL DEFAULT NULL,
    `role` ENUM('admin', 'manager', 'sales_rep', 'support_agent', 'read_only') NOT NULL DEFAULT 'sales_rep',
    `is_active` TINYINT(1) NOT NULL DEFAULT 1,
    `timezone` VARCHAR(50) NOT NULL DEFAULT 'UTC',
    `last_login_at` DATETIME NULL DEFAULT NULL,
    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    `updated_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    `deleted_at` TIMESTAMP NULL DEFAULT NULL,

    PRIMARY KEY (`id`),
    UNIQUE INDEX `uq_users_email` (`email`),
    INDEX `idx_users_role` (`role`),
    INDEX `idx_users_is_active` (`is_active`),
    INDEX `idx_users_deleted_at` (`deleted_at`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
```

Design notes:
- `email` is unique and serves as the login identifier. Never use email as a primary key — use a surrogate BIGINT.
- `password_hash` stores bcrypt/argon2 output, never plaintext.
- `role` uses ENUM for a fixed set of roles. If you need more flexible RBAC, see `security-and-multitenancy.md`.
- `is_active` is separate from `deleted_at`. A user can be deactivated but still exist for historical reference.

### Accounts (Companies / Organizations)

Accounts are the primary entity in B2B CRM. In B2C, this may be optional or represent household groupings.

```sql
CREATE TABLE `accounts` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `name` VARCHAR(255) NOT NULL,
    `domain` VARCHAR(255) NULL DEFAULT NULL,
    `industry` VARCHAR(100) NULL DEFAULT NULL,
    `company_size` ENUM('1-10', '11-50', '51-200', '201-500', '501-1000', '1001-5000', '5000+') NULL DEFAULT NULL,
    `annual_revenue` DECIMAL(15, 2) NULL DEFAULT NULL,
    `type` ENUM('prospect', 'customer', 'partner', 'vendor', 'competitor', 'other') NOT NULL DEFAULT 'prospect',
    `status` ENUM('active', 'inactive', 'churned') NOT NULL DEFAULT 'active',
    `phone` VARCHAR(30) NULL DEFAULT NULL,
    `website` VARCHAR(500) NULL DEFAULT NULL,
    `description` TEXT NULL DEFAULT NULL,

    -- Address fields (consider a separate addresses table for multi-address support)
    `billing_address_line1` VARCHAR(255) NULL DEFAULT NULL,
    `billing_address_line2` VARCHAR(255) NULL DEFAULT NULL,
    `billing_city` VARCHAR(100) NULL DEFAULT NULL,
    `billing_state` VARCHAR(100) NULL DEFAULT NULL,
    `billing_postal_code` VARCHAR(20) NULL DEFAULT NULL,
    `billing_country` VARCHAR(2) NULL DEFAULT NULL COMMENT 'ISO 3166-1 alpha-2',

    -- Ownership and hierarchy
    `owner_id` BIGINT UNSIGNED NULL DEFAULT NULL COMMENT 'Sales rep who owns this account',
    `parent_account_id` BIGINT UNSIGNED NULL DEFAULT NULL COMMENT 'For subsidiaries/divisions',

    -- Integration
    `external_id` VARCHAR(255) NULL DEFAULT NULL COMMENT 'ID in external system (ERP, billing, etc.)',
    `external_source` VARCHAR(100) NULL DEFAULT NULL COMMENT 'Name of external system',

    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    `updated_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    `deleted_at` TIMESTAMP NULL DEFAULT NULL,
    `created_by` BIGINT UNSIGNED NULL DEFAULT NULL,
    `updated_by` BIGINT UNSIGNED NULL DEFAULT NULL,

    PRIMARY KEY (`id`),
    INDEX `idx_accounts_name` (`name`),
    INDEX `idx_accounts_domain` (`domain`),
    INDEX `idx_accounts_type` (`type`),
    INDEX `idx_accounts_status` (`status`),
    INDEX `idx_accounts_owner_id` (`owner_id`),
    INDEX `idx_accounts_parent_account_id` (`parent_account_id`),
    INDEX `idx_accounts_industry` (`industry`),
    INDEX `idx_accounts_deleted_at` (`deleted_at`),
    INDEX `idx_accounts_external` (`external_source`, `external_id`),

    CONSTRAINT `fk_accounts_owner` FOREIGN KEY (`owner_id`) REFERENCES `users` (`id`) ON DELETE SET NULL,
    CONSTRAINT `fk_accounts_parent` FOREIGN KEY (`parent_account_id`) REFERENCES `accounts` (`id`) ON DELETE SET NULL,
    CONSTRAINT `fk_accounts_created_by` FOREIGN KEY (`created_by`) REFERENCES `users` (`id`) ON DELETE SET NULL,
    CONSTRAINT `fk_accounts_updated_by` FOREIGN KEY (`updated_by`) REFERENCES `users` (`id`) ON DELETE SET NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
```

Design notes:
- `parent_account_id` enables account hierarchies (parent company → subsidiaries). Query these with MySQL 8 recursive CTEs.
- `billing_country` uses ISO 3166-1 alpha-2 codes (US, GB, DE). Consider a lookup table for countries if you need names and metadata.
- `owner_id` establishes the primary sales rep responsible. Many CRM systems also use a team/territory assignment table.
- `external_id` + `external_source` together form a composite key for integration deduplication.

### Contacts (People)

Contacts are individual people, typically associated with an account.

```sql
CREATE TABLE `contacts` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `account_id` BIGINT UNSIGNED NULL DEFAULT NULL,
    `first_name` VARCHAR(100) NOT NULL,
    `last_name` VARCHAR(100) NOT NULL,
    `email` VARCHAR(255) NULL DEFAULT NULL,
    `secondary_email` VARCHAR(255) NULL DEFAULT NULL,
    `phone` VARCHAR(30) NULL DEFAULT NULL,
    `mobile_phone` VARCHAR(30) NULL DEFAULT NULL,
    `job_title` VARCHAR(150) NULL DEFAULT NULL,
    `department` VARCHAR(100) NULL DEFAULT NULL,
    `linkedin_url` VARCHAR(500) NULL DEFAULT NULL,
    `is_primary` TINYINT(1) NOT NULL DEFAULT 0 COMMENT 'Primary contact for the account',
    `lifecycle_stage` ENUM('subscriber', 'lead', 'mql', 'sql', 'opportunity', 'customer', 'evangelist', 'other') NOT NULL DEFAULT 'subscriber',
    `lead_source` VARCHAR(100) NULL DEFAULT NULL,
    `status` ENUM('active', 'inactive', 'bounced', 'unsubscribed', 'do_not_contact') NOT NULL DEFAULT 'active',

    -- Communication preferences
    `email_opt_in` TINYINT(1) NOT NULL DEFAULT 0,
    `sms_opt_in` TINYINT(1) NOT NULL DEFAULT 0,
    `preferred_language` VARCHAR(10) NULL DEFAULT NULL COMMENT 'IETF language tag, e.g. en-US',

    -- Mailing address
    `mailing_address_line1` VARCHAR(255) NULL DEFAULT NULL,
    `mailing_address_line2` VARCHAR(255) NULL DEFAULT NULL,
    `mailing_city` VARCHAR(100) NULL DEFAULT NULL,
    `mailing_state` VARCHAR(100) NULL DEFAULT NULL,
    `mailing_postal_code` VARCHAR(20) NULL DEFAULT NULL,
    `mailing_country` VARCHAR(2) NULL DEFAULT NULL,

    -- Ownership
    `owner_id` BIGINT UNSIGNED NULL DEFAULT NULL,

    -- Integration
    `external_id` VARCHAR(255) NULL DEFAULT NULL,
    `external_source` VARCHAR(100) NULL DEFAULT NULL,

    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    `updated_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    `deleted_at` TIMESTAMP NULL DEFAULT NULL,
    `created_by` BIGINT UNSIGNED NULL DEFAULT NULL,
    `updated_by` BIGINT UNSIGNED NULL DEFAULT NULL,

    PRIMARY KEY (`id`),
    INDEX `idx_contacts_account_id` (`account_id`),
    INDEX `idx_contacts_email` (`email`),
    INDEX `idx_contacts_name` (`last_name`, `first_name`),
    INDEX `idx_contacts_owner_id` (`owner_id`),
    INDEX `idx_contacts_lifecycle_stage` (`lifecycle_stage`),
    INDEX `idx_contacts_status` (`status`),
    INDEX `idx_contacts_lead_source` (`lead_source`),
    INDEX `idx_contacts_deleted_at` (`deleted_at`),
    INDEX `idx_contacts_external` (`external_source`, `external_id`),

    CONSTRAINT `fk_contacts_account` FOREIGN KEY (`account_id`) REFERENCES `accounts` (`id`) ON DELETE SET NULL,
    CONSTRAINT `fk_contacts_owner` FOREIGN KEY (`owner_id`) REFERENCES `users` (`id`) ON DELETE SET NULL,
    CONSTRAINT `fk_contacts_created_by` FOREIGN KEY (`created_by`) REFERENCES `users` (`id`) ON DELETE SET NULL,
    CONSTRAINT `fk_contacts_updated_by` FOREIGN KEY (`updated_by`) REFERENCES `users` (`id`) ON DELETE SET NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
```

Design notes:
- `account_id` is nullable because some contacts (individuals, freelancers) may not belong to an account.
- `email` is not UNIQUE because a person might appear in the system via different sources. Deduplication is an application-level concern. If you want uniqueness, add a unique index.
- `lifecycle_stage` tracks where this contact is in the marketing/sales funnel.
- `is_primary` marks the main contact at an account. Enforce only-one-primary per account in application logic or with a filtered unique index.

### Leads (Unqualified Prospects)

Leads are potential contacts that have not yet been qualified. Some CRM systems merge leads into the contacts table with a status flag; others keep them separate. A separate table is cleaner when lead qualification is a distinct workflow.

```sql
CREATE TABLE `leads` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `first_name` VARCHAR(100) NOT NULL,
    `last_name` VARCHAR(100) NOT NULL,
    `email` VARCHAR(255) NULL DEFAULT NULL,
    `phone` VARCHAR(30) NULL DEFAULT NULL,
    `company_name` VARCHAR(255) NULL DEFAULT NULL,
    `job_title` VARCHAR(150) NULL DEFAULT NULL,
    `website` VARCHAR(500) NULL DEFAULT NULL,

    -- Lead qualification
    `source` VARCHAR(100) NULL DEFAULT NULL COMMENT 'e.g., website, trade_show, referral, cold_call, linkedin, advertisement',
    `source_detail` VARCHAR(255) NULL DEFAULT NULL COMMENT 'Specific campaign, event name, or referrer',
    `status` ENUM('new', 'contacted', 'qualified', 'unqualified', 'converted', 'lost') NOT NULL DEFAULT 'new',
    `rating` ENUM('hot', 'warm', 'cold') NULL DEFAULT NULL,
    `score` INT UNSIGNED NULL DEFAULT NULL COMMENT 'Numeric lead score from 0-100',

    -- Conversion tracking
    `converted_at` DATETIME NULL DEFAULT NULL,
    `converted_contact_id` BIGINT UNSIGNED NULL DEFAULT NULL,
    `converted_account_id` BIGINT UNSIGNED NULL DEFAULT NULL,
    `converted_opportunity_id` BIGINT UNSIGNED NULL DEFAULT NULL,

    -- Ownership
    `owner_id` BIGINT UNSIGNED NULL DEFAULT NULL,

    -- Integration
    `external_id` VARCHAR(255) NULL DEFAULT NULL,
    `external_source` VARCHAR(100) NULL DEFAULT NULL,

    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    `updated_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    `deleted_at` TIMESTAMP NULL DEFAULT NULL,
    `created_by` BIGINT UNSIGNED NULL DEFAULT NULL,
    `updated_by` BIGINT UNSIGNED NULL DEFAULT NULL,

    PRIMARY KEY (`id`),
    INDEX `idx_leads_email` (`email`),
    INDEX `idx_leads_status` (`status`),
    INDEX `idx_leads_source` (`source`),
    INDEX `idx_leads_owner_id` (`owner_id`),
    INDEX `idx_leads_rating` (`rating`),
    INDEX `idx_leads_score` (`score`),
    INDEX `idx_leads_converted_at` (`converted_at`),
    INDEX `idx_leads_deleted_at` (`deleted_at`),
    INDEX `idx_leads_created_at` (`created_at`),
    INDEX `idx_leads_external` (`external_source`, `external_id`),

    CONSTRAINT `fk_leads_owner` FOREIGN KEY (`owner_id`) REFERENCES `users` (`id`) ON DELETE SET NULL,
    CONSTRAINT `fk_leads_converted_contact` FOREIGN KEY (`converted_contact_id`) REFERENCES `contacts` (`id`) ON DELETE SET NULL,
    CONSTRAINT `fk_leads_converted_account` FOREIGN KEY (`converted_account_id`) REFERENCES `accounts` (`id`) ON DELETE SET NULL,
    CONSTRAINT `fk_leads_created_by` FOREIGN KEY (`created_by`) REFERENCES `users` (`id`) ON DELETE SET NULL,
    CONSTRAINT `fk_leads_updated_by` FOREIGN KEY (`updated_by`) REFERENCES `users` (`id`) ON DELETE SET NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
```

Design notes:
- When a lead is converted, set `status = 'converted'`, record `converted_at`, and link to the newly created contact, account, and/or opportunity.
- `score` supports numeric lead scoring (0-100). The scoring rules live in application logic; the database just stores the result.
- `source` + `source_detail` together tell you where the lead came from and the specific campaign or event.

### Pipelines and Pipeline Stages

Most CRMs support multiple sales pipelines (e.g., "New Business", "Renewals", "Upsell"), each with their own stages.

```sql
CREATE TABLE `pipelines` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `name` VARCHAR(100) NOT NULL,
    `description` TEXT NULL DEFAULT NULL,
    `is_default` TINYINT(1) NOT NULL DEFAULT 0,
    `is_active` TINYINT(1) NOT NULL DEFAULT 1,
    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    `updated_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    `deleted_at` TIMESTAMP NULL DEFAULT NULL,

    PRIMARY KEY (`id`),
    INDEX `idx_pipelines_is_active` (`is_active`),
    INDEX `idx_pipelines_deleted_at` (`deleted_at`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

CREATE TABLE `pipeline_stages` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `pipeline_id` BIGINT UNSIGNED NOT NULL,
    `name` VARCHAR(100) NOT NULL,
    `display_order` INT UNSIGNED NOT NULL DEFAULT 0,
    `probability` DECIMAL(5, 2) NULL DEFAULT NULL COMMENT 'Win probability percentage, e.g. 75.00',
    `is_won` TINYINT(1) NOT NULL DEFAULT 0 COMMENT 'True for closed-won stage',
    `is_lost` TINYINT(1) NOT NULL DEFAULT 0 COMMENT 'True for closed-lost stage',
    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    `updated_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,

    PRIMARY KEY (`id`),
    INDEX `idx_pipeline_stages_pipeline_id` (`pipeline_id`),
    INDEX `idx_pipeline_stages_display_order` (`pipeline_id`, `display_order`),

    CONSTRAINT `fk_pipeline_stages_pipeline` FOREIGN KEY (`pipeline_id`) REFERENCES `pipelines` (`id`) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
```

Design notes:
- `display_order` controls the visual left-to-right order of stages in a kanban view.
- `probability` is used to calculate weighted pipeline value (opportunity amount × stage probability).
- `is_won` and `is_lost` are mutually exclusive flags that mark terminal stages. CHECK constraints can enforce this.

### Opportunities (Deals)

Opportunities represent potential revenue. They are the lifeblood of CRM reporting.

```sql
CREATE TABLE `opportunities` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `name` VARCHAR(255) NOT NULL,
    `account_id` BIGINT UNSIGNED NOT NULL,
    `pipeline_id` BIGINT UNSIGNED NOT NULL,
    `stage_id` BIGINT UNSIGNED NOT NULL,
    `amount` DECIMAL(15, 2) NULL DEFAULT NULL COMMENT 'Deal value in base currency',
    `currency` VARCHAR(3) NOT NULL DEFAULT 'USD' COMMENT 'ISO 4217 currency code',
    `probability` DECIMAL(5, 2) NULL DEFAULT NULL COMMENT 'Override of stage probability if set',
    `expected_close_date` DATE NULL DEFAULT NULL,
    `actual_close_date` DATE NULL DEFAULT NULL,
    `type` ENUM('new_business', 'renewal', 'upsell', 'cross_sell', 'other') NULL DEFAULT NULL,
    `priority` ENUM('low', 'medium', 'high', 'critical') NOT NULL DEFAULT 'medium',
    `loss_reason` VARCHAR(255) NULL DEFAULT NULL,
    `description` TEXT NULL DEFAULT NULL,
    `next_step` VARCHAR(500) NULL DEFAULT NULL,

    -- Ownership
    `owner_id` BIGINT UNSIGNED NULL DEFAULT NULL,

    -- Primary contact on the deal
    `primary_contact_id` BIGINT UNSIGNED NULL DEFAULT NULL,

    -- Integration
    `external_id` VARCHAR(255) NULL DEFAULT NULL,
    `external_source` VARCHAR(100) NULL DEFAULT NULL,

    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    `updated_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    `deleted_at` TIMESTAMP NULL DEFAULT NULL,
    `created_by` BIGINT UNSIGNED NULL DEFAULT NULL,
    `updated_by` BIGINT UNSIGNED NULL DEFAULT NULL,

    PRIMARY KEY (`id`),
    INDEX `idx_opportunities_account_id` (`account_id`),
    INDEX `idx_opportunities_pipeline_id` (`pipeline_id`),
    INDEX `idx_opportunities_stage_id` (`stage_id`),
    INDEX `idx_opportunities_owner_id` (`owner_id`),
    INDEX `idx_opportunities_expected_close` (`expected_close_date`),
    INDEX `idx_opportunities_amount` (`amount`),
    INDEX `idx_opportunities_type` (`type`),
    INDEX `idx_opportunities_deleted_at` (`deleted_at`),
    INDEX `idx_opportunities_created_at` (`created_at`),
    INDEX `idx_opportunities_external` (`external_source`, `external_id`),
    -- Composite index for pipeline reporting
    INDEX `idx_opportunities_pipeline_stage` (`pipeline_id`, `stage_id`, `deleted_at`),

    CONSTRAINT `fk_opportunities_account` FOREIGN KEY (`account_id`) REFERENCES `accounts` (`id`) ON DELETE RESTRICT,
    CONSTRAINT `fk_opportunities_pipeline` FOREIGN KEY (`pipeline_id`) REFERENCES `pipelines` (`id`) ON DELETE RESTRICT,
    CONSTRAINT `fk_opportunities_stage` FOREIGN KEY (`stage_id`) REFERENCES `pipeline_stages` (`id`) ON DELETE RESTRICT,
    CONSTRAINT `fk_opportunities_owner` FOREIGN KEY (`owner_id`) REFERENCES `users` (`id`) ON DELETE SET NULL,
    CONSTRAINT `fk_opportunities_primary_contact` FOREIGN KEY (`primary_contact_id`) REFERENCES `contacts` (`id`) ON DELETE SET NULL,
    CONSTRAINT `fk_opportunities_created_by` FOREIGN KEY (`created_by`) REFERENCES `users` (`id`) ON DELETE SET NULL,
    CONSTRAINT `fk_opportunities_updated_by` FOREIGN KEY (`updated_by`) REFERENCES `users` (`id`) ON DELETE SET NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
```

Design notes:
- `account_id` is NOT NULL and uses ON DELETE RESTRICT. Opportunities should never exist without an account, and deleting an account with active deals should be blocked.
- `amount` uses DECIMAL(15,2) for precision. Never use FLOAT or DOUBLE for monetary values.
- `currency` stores the ISO 4217 code. Multi-currency CRMs need a currency conversion table or integrate with a rate API.
- The composite index on `(pipeline_id, stage_id, deleted_at)` optimizes the most common CRM query: "show me all active deals by pipeline stage."

### Opportunity Contact Roles (Junction Table)

Deals involve multiple contacts playing different roles (decision maker, influencer, end user, etc.).

```sql
CREATE TABLE `opportunity_contacts` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `opportunity_id` BIGINT UNSIGNED NOT NULL,
    `contact_id` BIGINT UNSIGNED NOT NULL,
    `role` VARCHAR(100) NULL DEFAULT NULL COMMENT 'e.g., decision_maker, influencer, champion, end_user, evaluator',
    `is_primary` TINYINT(1) NOT NULL DEFAULT 0,
    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,

    PRIMARY KEY (`id`),
    UNIQUE INDEX `uq_opportunity_contacts` (`opportunity_id`, `contact_id`),
    INDEX `idx_opportunity_contacts_contact_id` (`contact_id`),

    CONSTRAINT `fk_opp_contacts_opportunity` FOREIGN KEY (`opportunity_id`) REFERENCES `opportunities` (`id`) ON DELETE CASCADE,
    CONSTRAINT `fk_opp_contacts_contact` FOREIGN KEY (`contact_id`) REFERENCES `contacts` (`id`) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
```

### Products and Opportunity Line Items

```sql
CREATE TABLE `products` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `name` VARCHAR(255) NOT NULL,
    `sku` VARCHAR(100) NULL DEFAULT NULL,
    `description` TEXT NULL DEFAULT NULL,
    `unit_price` DECIMAL(15, 2) NOT NULL DEFAULT 0.00,
    `currency` VARCHAR(3) NOT NULL DEFAULT 'USD',
    `is_active` TINYINT(1) NOT NULL DEFAULT 1,
    `product_category` VARCHAR(100) NULL DEFAULT NULL,
    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    `updated_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    `deleted_at` TIMESTAMP NULL DEFAULT NULL,

    PRIMARY KEY (`id`),
    INDEX `idx_products_sku` (`sku`),
    INDEX `idx_products_is_active` (`is_active`),
    INDEX `idx_products_category` (`product_category`),
    INDEX `idx_products_deleted_at` (`deleted_at`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

CREATE TABLE `opportunity_line_items` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `opportunity_id` BIGINT UNSIGNED NOT NULL,
    `product_id` BIGINT UNSIGNED NOT NULL,
    `quantity` DECIMAL(10, 2) NOT NULL DEFAULT 1.00,
    `unit_price` DECIMAL(15, 2) NOT NULL COMMENT 'Price at time of quote, may differ from product list price',
    `discount_percent` DECIMAL(5, 2) NOT NULL DEFAULT 0.00,
    `total_price` DECIMAL(15, 2) GENERATED ALWAYS AS (
        `quantity` * `unit_price` * (1 - `discount_percent` / 100)
    ) STORED,
    `description` VARCHAR(500) NULL DEFAULT NULL,
    `sort_order` INT UNSIGNED NOT NULL DEFAULT 0,
    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    `updated_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,

    PRIMARY KEY (`id`),
    INDEX `idx_opp_line_items_opportunity_id` (`opportunity_id`),
    INDEX `idx_opp_line_items_product_id` (`product_id`),

    CONSTRAINT `fk_opp_line_items_opportunity` FOREIGN KEY (`opportunity_id`) REFERENCES `opportunities` (`id`) ON DELETE CASCADE,
    CONSTRAINT `fk_opp_line_items_product` FOREIGN KEY (`product_id`) REFERENCES `products` (`id`) ON DELETE RESTRICT
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
```

Design notes:
- `unit_price` on line items is separate from the product's `unit_price` because the quoted price may differ from the list price (discounts, negotiated rates, pricing at time of quote).
- `total_price` is a STORED generated column — MySQL computes it automatically and it can be indexed if needed.
- `discount_percent` stores the percentage discount. Some systems prefer storing `discount_amount` instead. Choose one convention and be consistent.

### Tags (Flexible Categorization)

Tags provide lightweight, user-defined categorization across multiple entity types.

```sql
CREATE TABLE `tags` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `name` VARCHAR(100) NOT NULL,
    `color` VARCHAR(7) NULL DEFAULT NULL COMMENT 'Hex color code, e.g. #FF5733',
    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,

    PRIMARY KEY (`id`),
    UNIQUE INDEX `uq_tags_name` (`name`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

CREATE TABLE `taggables` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `tag_id` BIGINT UNSIGNED NOT NULL,
    `taggable_type` VARCHAR(50) NOT NULL COMMENT 'e.g., account, contact, lead, opportunity',
    `taggable_id` BIGINT UNSIGNED NOT NULL,
    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,

    PRIMARY KEY (`id`),
    UNIQUE INDEX `uq_taggables` (`tag_id`, `taggable_type`, `taggable_id`),
    INDEX `idx_taggables_target` (`taggable_type`, `taggable_id`),

    CONSTRAINT `fk_taggables_tag` FOREIGN KEY (`tag_id`) REFERENCES `tags` (`id`) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
```

Design notes:
- The `taggables` table uses a polymorphic pattern — `taggable_type` + `taggable_id` point to any entity. This avoids creating a separate junction table for every entity type.
- The tradeoff is that MySQL cannot enforce a foreign key on the polymorphic reference. Application logic must validate that `taggable_id` exists in the referenced table.
- The composite unique index prevents duplicate tag assignments on the same entity.

FILE:references/relationships-and-normalization.md
# Relationships and Normalization

This reference covers how to properly establish relationships between CRM entities and ensure the schema follows sound normalization principles.

## Relationship Types in CRM

### One-to-Many (1:N) — The Most Common Pattern

Most CRM relationships are one-to-many. The "many" side holds a foreign key to the "one" side.

| Relationship | Implementation |
|---|---|
| Account → Contacts | `contacts.account_id` → `accounts.id` |
| Account → Opportunities | `opportunities.account_id` → `accounts.id` |
| User → Owned Accounts | `accounts.owner_id` → `users.id` |
| Pipeline → Stages | `pipeline_stages.pipeline_id` → `pipelines.id` |
| Opportunity → Line Items | `opportunity_line_items.opportunity_id` → `opportunities.id` |

Rules for 1:N foreign keys:
- Always create an index on the foreign key column. MySQL does NOT automatically index foreign keys (unlike some other databases).
- Choose the correct ON DELETE behavior:
  - `CASCADE` — child rows deleted when parent is deleted (use for true child entities like line items, stage history)
  - `SET NULL` — FK set to NULL when parent is deleted (use for optional references like owner_id)
  - `RESTRICT` — block parent deletion if children exist (use for critical dependencies like account on an opportunity)

### Many-to-Many (M:N) — Junction Tables

When two entities relate to each other in both directions, use a junction table. Never store comma-separated IDs in a single column.

Common CRM M:N relationships:

```sql
-- Opportunities can involve multiple contacts; contacts can appear in multiple deals
CREATE TABLE `opportunity_contacts` (
    `opportunity_id` BIGINT UNSIGNED NOT NULL,
    `contact_id` BIGINT UNSIGNED NOT NULL,
    `role` VARCHAR(100) NULL DEFAULT NULL,
    PRIMARY KEY (`opportunity_id`, `contact_id`)
);

-- Contacts can belong to multiple accounts (consultants, board members)
CREATE TABLE `account_contacts` (
    `account_id` BIGINT UNSIGNED NOT NULL,
    `contact_id` BIGINT UNSIGNED NOT NULL,
    `relationship_type` VARCHAR(50) NULL DEFAULT NULL,
    PRIMARY KEY (`account_id`, `contact_id`)
);

-- Campaigns target multiple contacts; contacts receive multiple campaigns
CREATE TABLE `campaign_contacts` (
    `campaign_id` BIGINT UNSIGNED NOT NULL,
    `contact_id` BIGINT UNSIGNED NOT NULL,
    `status` ENUM('sent', 'opened', 'clicked', 'responded', 'bounced', 'unsubscribed') NOT NULL DEFAULT 'sent',
    `sent_at` DATETIME NULL DEFAULT NULL,
    PRIMARY KEY (`campaign_id`, `contact_id`)
);
```

Junction table best practices:
- Use a composite primary key on the two FK columns for simple associations.
- Add a surrogate `id` column if you need to reference the junction row from other tables (e.g., activity tracking on a campaign-contact pair).
- Store relationship metadata on the junction table (role, status, timestamps) — not on either parent.
- Always add a reverse index. If the PK is `(opportunity_id, contact_id)`, add an index on `(contact_id)` so lookups from the contact side are fast too.

### Self-Referencing Relationships

CRM uses self-references for hierarchies:

```sql
-- Account hierarchy (parent company → subsidiaries)
ALTER TABLE `accounts`
    ADD COLUMN `parent_account_id` BIGINT UNSIGNED NULL DEFAULT NULL,
    ADD CONSTRAINT `fk_accounts_parent`
        FOREIGN KEY (`parent_account_id`) REFERENCES `accounts` (`id`) ON DELETE SET NULL;

-- User reporting structure (manager → direct reports)
ALTER TABLE `users`
    ADD COLUMN `manager_id` BIGINT UNSIGNED NULL DEFAULT NULL,
    ADD CONSTRAINT `fk_users_manager`
        FOREIGN KEY (`manager_id`) REFERENCES `users` (`id`) ON DELETE SET NULL;
```

Querying hierarchies with MySQL 8 recursive CTEs:

```sql
-- Get full account hierarchy from a given parent
WITH RECURSIVE account_tree AS (
    SELECT id, name, parent_account_id, 0 AS depth
    FROM accounts
    WHERE id = :root_account_id

    UNION ALL

    SELECT a.id, a.name, a.parent_account_id, t.depth + 1
    FROM accounts a
    INNER JOIN account_tree t ON a.parent_account_id = t.id
    WHERE t.depth < 10  -- safety limit to prevent infinite loops
)
SELECT * FROM account_tree;
```

### Polymorphic Relationships

Some CRM data relates to multiple entity types. Tags, notes, activities, and attachments commonly use this pattern.

```
taggable_type = 'contact'  + taggable_id = 42  → contacts row with id=42
taggable_type = 'account'  + taggable_id = 7   → accounts row with id=7
taggable_type = 'opportunity' + taggable_id = 15 → opportunities row with id=15
```

Tradeoffs:
- **Pro:** One table handles all entity types. No proliferation of junction tables.
- **Con:** MySQL cannot enforce foreign keys on polymorphic references. Referential integrity relies on application code.
- **Mitigation:** Use a CHECK constraint to restrict `taggable_type` to known values. Add application-level validation. Consider periodic integrity checks via scheduled queries.

## Normalization Guide for CRM

### First Normal Form (1NF)

Every column stores a single atomic value. No comma-separated lists, no JSON arrays in place of proper relationships.

**Violation:** Storing multiple phone numbers as `"555-1234, 555-5678"` in a single column.
**Fix:** Use a separate `contact_phones` table or dedicated columns (`phone`, `mobile_phone`, `work_phone`).

**Violation:** Storing tag names as `"enterprise, hot-lead, east-coast"` in a tags column.
**Fix:** Use a `tags` table and `taggables` junction table.

### Second Normal Form (2NF)

Every non-key column depends on the entire primary key (relevant for composite keys).

**Violation:** In an `opportunity_contacts` junction table with PK (`opportunity_id`, `contact_id`), storing `contact_email` — this depends only on `contact_id`, not the full key.
**Fix:** Remove `contact_email`; look it up via JOIN to `contacts`.

### Third Normal Form (3NF)

No transitive dependencies. Every non-key column depends directly on the primary key, not on another non-key column.

**Violation:** Storing both `stage_id` and `stage_name` on the `opportunities` table. `stage_name` depends on `stage_id`, not on `opportunity.id`.
**Fix:** Store only `stage_id` on `opportunities`. Look up `stage_name` via JOIN to `pipeline_stages`.

**Acceptable denormalization:** Storing `total_price` as a generated column on `opportunity_line_items`. This is technically a 3NF violation (it depends on `quantity`, `unit_price`, `discount_percent`), but as a generated column, MySQL keeps it consistent automatically.

### When to Deliberately Denormalize

Denormalize only when you have measured evidence of a performance problem, and document the decision:

1. **Cached aggregates.** Store `deal_count` and `total_revenue` on `accounts` if you frequently display account summaries. Update via triggers or application events.

2. **Snapshot values.** Store `unit_price` on `opportunity_line_items` separately from `products.unit_price` because the price at the time of the quote matters, not the current list price.

3. **Reporting tables.** Create materialized summary tables updated by scheduled jobs for dashboard queries. Keep these separate from operational tables.

## Foreign Key Strategy

### Rules of Thumb

1. Every foreign key column gets its own index.
2. Use `ON DELETE RESTRICT` as the default. Only use CASCADE for true child/dependent entities.
3. Use `ON DELETE SET NULL` for optional relationships (owner_id, assigned_to).
4. Never use `ON DELETE CASCADE` on core business entities (accounts, contacts, opportunities). Soft delete instead.
5. Use `ON UPDATE CASCADE` sparingly — if you're updating primary keys, reconsider your schema design.

### FK Index Pattern

```sql
-- For every FK column, create a dedicated index
ALTER TABLE `contacts` ADD INDEX `idx_contacts_account_id` (`account_id`);
ALTER TABLE `contacts` ADD INDEX `idx_contacts_owner_id` (`owner_id`);
ALTER TABLE `opportunities` ADD INDEX `idx_opportunities_account_id` (`account_id`);
ALTER TABLE `opportunities` ADD INDEX `idx_opportunities_stage_id` (`stage_id`);
```

Without these indexes, DELETE operations on parent tables will cause full table scans on child tables to check for references — this can lock the table and cause severe performance issues.

FILE:references/migrations-and-seeding.md
# Migrations and Seeding

This reference covers schema versioning, writing safe migration scripts, and generating realistic CRM test data.

## Migration Best Practices

### File Naming Convention

Use sequential, timestamped filenames:

```
migrations/
├── 001_20260101_000000_create_users_table.sql
├── 002_20260101_000001_create_accounts_table.sql
├── 003_20260101_000002_create_contacts_table.sql
├── 004_20260101_000003_create_leads_table.sql
├── 005_20260101_000004_create_pipelines_and_stages.sql
├── 006_20260101_000005_create_opportunities_table.sql
├── 007_20260101_000006_create_products_and_line_items.sql
├── 008_20260101_000007_create_activities_table.sql
├── 009_20260101_000008_create_notes_and_tasks.sql
├── 010_20260101_000009_create_audit_logs_table.sql
├── 011_20260101_000010_create_tags_table.sql
├── 012_20260101_000011_create_custom_fields.sql
├── 013_20260101_000012_create_attachments_table.sql
└── 014_20260101_000013_add_fulltext_indexes.sql
```

### Migration Tracking Table

Track which migrations have been applied:

```sql
CREATE TABLE `schema_migrations` (
    `id` INT UNSIGNED NOT NULL AUTO_INCREMENT,
    `migration` VARCHAR(255) NOT NULL,
    `applied_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    `checksum` VARCHAR(64) NULL DEFAULT NULL COMMENT 'SHA-256 of migration content',
    `execution_time_ms` INT UNSIGNED NULL DEFAULT NULL,

    PRIMARY KEY (`id`),
    UNIQUE INDEX `uq_migrations_name` (`migration`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
```

### Writing Safe Migrations

**Always use IF NOT EXISTS / IF EXISTS:**

```sql
-- Safe table creation
CREATE TABLE IF NOT EXISTS `contacts` (
    -- ...
);

-- Safe column addition
-- MySQL does not support IF NOT EXISTS for columns, so check first:
SET @col_exists = (
    SELECT COUNT(*) FROM information_schema.COLUMNS
    WHERE TABLE_SCHEMA = DATABASE()
        AND TABLE_NAME = 'contacts'
        AND COLUMN_NAME = 'linkedin_url'
);
SET @sql = IF(@col_exists = 0,
    'ALTER TABLE `contacts` ADD COLUMN `linkedin_url` VARCHAR(500) NULL DEFAULT NULL',
    'SELECT 1');
PREPARE stmt FROM @sql;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
```

**Include both up and down migrations:**

```sql
-- ============================================================
-- Migration: 015_20260415_add_priority_to_opportunities
-- ============================================================

-- UP
ALTER TABLE `opportunities`
    ADD COLUMN `priority` ENUM('low', 'medium', 'high', 'critical')
    NOT NULL DEFAULT 'medium' AFTER `type`;

ALTER TABLE `opportunities`
    ADD INDEX `idx_opportunities_priority` (`priority`);

-- DOWN (rollback)
-- ALTER TABLE `opportunities` DROP INDEX `idx_opportunities_priority`;
-- ALTER TABLE `opportunities` DROP COLUMN `priority`;
```

**Handle large tables carefully:**

For tables with millions of rows, direct ALTER TABLE can lock the table for a long time. Strategies:

1. Use `ALGORITHM=INPLACE, LOCK=NONE` when MySQL supports it:
```sql
ALTER TABLE `activities`
    ADD COLUMN `sentiment` ENUM('positive', 'neutral', 'negative') NULL DEFAULT NULL,
    ALGORITHM=INPLACE, LOCK=NONE;
```

2. For changes that require table rebuild, use pt-online-schema-change or gh-ost:
```bash
# Using Percona pt-online-schema-change
pt-online-schema-change --alter "ADD COLUMN sentiment ENUM('positive','neutral','negative') NULL DEFAULT NULL" \
    D=crm,t=activities --execute
```

3. Create a new table, copy data, and swap (manual online DDL):
```sql
CREATE TABLE `activities_new` LIKE `activities`;
ALTER TABLE `activities_new` ADD COLUMN `sentiment` ENUM('positive','neutral','negative') NULL DEFAULT NULL;
-- Copy data in batches...
RENAME TABLE `activities` TO `activities_old`, `activities_new` TO `activities`;
```

### Execution Order

Migrations must execute in dependency order. The correct order for a CRM schema:

1. `users` — no dependencies
2. `tenants` — no dependencies (if multi-tenant)
3. `teams`, `team_members` — depends on `users`
4. `roles`, `permissions`, `role_permissions`, `user_roles` — depends on `users`
5. `accounts` — depends on `users` (for owner_id)
6. `contacts` — depends on `accounts`, `users`
7. `leads` — depends on `users`, `contacts`, `accounts`
8. `pipelines`, `pipeline_stages` — depends on nothing or `tenants`
9. `products` — no dependencies
10. `opportunities` — depends on `accounts`, `pipelines`, `pipeline_stages`, `users`, `contacts`
11. `opportunity_line_items` — depends on `opportunities`, `products`
12. `opportunity_contacts` — depends on `opportunities`, `contacts`
13. `activities` — depends on `users`, `accounts`, `contacts`, `opportunities`
14. `notes`, `tasks` — depends on `users`
15. `tags`, `taggables` — depends on nothing (polymorphic)
16. `attachments` — depends on `users`
17. `audit_logs` — no FK dependencies (polymorphic)
18. `custom_field_definitions`, `custom_field_values` — no FK dependencies (polymorphic)

## Seed Data

### Generating Realistic CRM Test Data

Seed data should be realistic enough to test query patterns, UI layouts, and reporting.

```sql
-- ============================================================
-- Seed: Users (sales team)
-- ============================================================
INSERT INTO `users` (`email`, `password_hash`, `first_name`, `last_name`, `role`, `timezone`) VALUES
('[email protected]', '$2b$12$placeholder_hash', 'Admin', 'User', 'admin', 'America/Chicago'),
('[email protected]', '$2b$12$placeholder_hash', 'Sarah', 'Chen', 'manager', 'America/New_York'),
('[email protected]', '$2b$12$placeholder_hash', 'James', 'Wilson', 'sales_rep', 'America/Chicago'),
('[email protected]', '$2b$12$placeholder_hash', 'Maria', 'Garcia', 'sales_rep', 'America/Los_Angeles'),
('[email protected]', '$2b$12$placeholder_hash', 'Alex', 'Johnson', 'sales_rep', 'America/New_York'),
('[email protected]', '$2b$12$placeholder_hash', 'Priya', 'Patel', 'support_agent', 'America/Chicago');

-- ============================================================
-- Seed: Accounts
-- ============================================================
INSERT INTO `accounts` (`name`, `domain`, `industry`, `company_size`, `type`, `status`, `owner_id`, `billing_country`) VALUES
('Acme Corporation', 'acme.com', 'Manufacturing', '201-500', 'customer', 'active', 3, 'US'),
('TechStart Inc', 'techstart.io', 'Technology', '11-50', 'prospect', 'active', 4, 'US'),
('Global Logistics Ltd', 'globallogistics.co.uk', 'Transportation', '501-1000', 'customer', 'active', 3, 'GB'),
('Sunrise Healthcare', 'sunrisehealth.org', 'Healthcare', '1001-5000', 'prospect', 'active', 5, 'US'),
('Nordic Design Studio', 'nordicdesign.se', 'Design', '1-10', 'partner', 'active', 4, 'SE'),
('Pacific Financial Group', 'pacificfin.com', 'Finance', '201-500', 'customer', 'active', 5, 'US'),
('RedBridge Consulting', 'redbridge.com.au', 'Consulting', '51-200', 'prospect', 'active', 3, 'AU'),
('MegaRetail Corp', 'megaretail.com', 'Retail', '5000+', 'customer', 'active', 2, 'US');

-- ============================================================
-- Seed: Contacts
-- ============================================================
INSERT INTO `contacts` (`account_id`, `first_name`, `last_name`, `email`, `phone`, `job_title`, `lifecycle_stage`, `owner_id`, `is_primary`) VALUES
(1, 'Robert', 'Smith', '[email protected]', '+1-555-0101', 'VP of Operations', 'customer', 3, 1),
(1, 'Lisa', 'Wang', '[email protected]', '+1-555-0102', 'Procurement Manager', 'customer', 3, 0),
(2, 'David', 'Kim', '[email protected]', '+1-555-0201', 'CEO', 'sql', 4, 1),
(2, 'Emily', 'Brown', '[email protected]', '+1-555-0202', 'CTO', 'sql', 4, 0),
(3, 'Oliver', 'Hughes', '[email protected]', '+44-20-7123-0001', 'Head of IT', 'customer', 3, 1),
(4, 'Jennifer', 'Martinez', '[email protected]', '+1-555-0401', 'CFO', 'mql', 5, 1),
(5, 'Erik', 'Lindqvist', '[email protected]', '+46-8-123-4567', 'Founder', 'customer', 4, 1),
(6, 'Patricia', 'Adams', '[email protected]', '+1-555-0601', 'Director of Technology', 'customer', 5, 1),
(7, 'Michael', 'Thompson', '[email protected]', '+61-2-8765-4321', 'Managing Partner', 'lead', 3, 1),
(8, 'Catherine', 'Lee', '[email protected]', '+1-555-0801', 'SVP of Supply Chain', 'customer', 2, 1);

-- ============================================================
-- Seed: Pipelines and Stages
-- ============================================================
INSERT INTO `pipelines` (`name`, `description`, `is_default`) VALUES
('New Business', 'Pipeline for new customer acquisition', 1),
('Renewals', 'Pipeline for contract renewals', 0),
('Upsell', 'Pipeline for upsell and expansion deals', 0);

INSERT INTO `pipeline_stages` (`pipeline_id`, `name`, `display_order`, `probability`, `is_won`, `is_lost`) VALUES
(1, 'Qualification', 1, 10.00, 0, 0),
(1, 'Discovery', 2, 20.00, 0, 0),
(1, 'Proposal', 3, 50.00, 0, 0),
(1, 'Negotiation', 4, 75.00, 0, 0),
(1, 'Closed Won', 5, 100.00, 1, 0),
(1, 'Closed Lost', 6, 0.00, 0, 1),
(2, 'Renewal Pending', 1, 70.00, 0, 0),
(2, 'In Negotiation', 2, 85.00, 0, 0),
(2, 'Renewed', 3, 100.00, 1, 0),
(2, 'Churned', 4, 0.00, 0, 1),
(3, 'Identified', 1, 20.00, 0, 0),
(3, 'Proposed', 2, 50.00, 0, 0),
(3, 'Committed', 3, 90.00, 0, 0),
(3, 'Won', 4, 100.00, 1, 0),
(3, 'Lost', 5, 0.00, 0, 1);

-- ============================================================
-- Seed: Opportunities
-- ============================================================
INSERT INTO `opportunities` (`name`, `account_id`, `pipeline_id`, `stage_id`, `amount`, `expected_close_date`, `type`, `owner_id`, `primary_contact_id`) VALUES
('Acme - Annual License Renewal', 1, 2, 7, 45000.00, '2026-06-30', 'renewal', 3, 1),
('TechStart - Platform Subscription', 2, 1, 3, 120000.00, '2026-05-15', 'new_business', 4, 3),
('Global Logistics - Expansion', 3, 3, 11, 85000.00, '2026-07-01', 'upsell', 3, 5),
('Sunrise Healthcare - Enterprise License', 4, 1, 2, 250000.00, '2026-08-30', 'new_business', 5, 6),
('Pacific Financial - Add-on Modules', 6, 3, 12, 35000.00, '2026-05-30', 'cross_sell', 5, 8),
('RedBridge - Initial Contract', 7, 1, 1, 60000.00, '2026-09-15', 'new_business', 3, 9);

-- ============================================================
-- Seed: Leads
-- ============================================================
INSERT INTO `leads` (`first_name`, `last_name`, `email`, `company_name`, `job_title`, `source`, `status`, `rating`, `score`, `owner_id`) VALUES
('Andrew', 'Taylor', '[email protected]', 'BigCorp Industries', 'IT Director', 'website', 'new', 'hot', 85, 3),
('Sophie', 'Dubois', '[email protected]', 'ParisTech Solutions', 'COO', 'trade_show', 'contacted', 'warm', 62, 4),
('Raj', 'Sharma', '[email protected]', 'Mumbai Software Labs', 'VP Engineering', 'referral', 'qualified', 'hot', 90, 5),
('Hannah', 'Miller', '[email protected]', 'GreenLeaf Sustainability', 'Founder', 'linkedin', 'new', 'cold', 30, NULL),
('Carlos', 'Reyes', '[email protected]', 'LatAm Logix', 'Head of Procurement', 'advertisement', 'contacted', 'warm', 55, 4);

-- ============================================================
-- Seed: Activities
-- ============================================================
INSERT INTO `activities` (`activity_type`, `entity_type`, `entity_id`, `activity_date`, `user_id`, `subject`, `body`, `duration_minutes`, `account_id`, `contact_id`, `metadata`) VALUES
('call', 'contact', 1, '2026-03-28 10:00:00', 3, 'Renewal discussion call', 'Discussed renewal terms. Robert is happy with service but wants to negotiate pricing.', 25, 1, 1, '{"direction":"outbound","outcome":"connected"}'),
('email', 'contact', 3, '2026-03-29 14:30:00', 4, 'Proposal follow-up', 'Sent updated proposal with revised pricing tier.', NULL, 2, 3, '{"direction":"outbound","has_attachments":true}'),
('meeting', 'opportunity', 4, '2026-04-01 09:00:00', 5, 'Discovery meeting with Sunrise Healthcare', 'Deep dive into their requirements. They need HIPAA compliance features.', 60, 4, 6, '{"location":"Zoom","attendees":2,"outcome":"completed"}'),
('note', 'account', 8, '2026-04-02 11:00:00', 2, 'Quarterly business review notes', 'MegaRetail is very satisfied. Potential for 30% expansion this year.', NULL, 8, 10, NULL),
('task', 'lead', 1, '2026-04-03 08:00:00', 3, 'Follow up with Andrew Taylor at BigCorp', NULL, NULL, NULL, NULL, NULL);
```

### Volume Testing

For performance testing, generate large volumes of data. Use a stored procedure or an external script:

```sql
-- Generate 10,000 contacts across existing accounts
DELIMITER //
CREATE PROCEDURE seed_contacts(IN num_contacts INT)
BEGIN
    DECLARE i INT DEFAULT 0;
    DECLARE v_account_id BIGINT;
    DECLARE v_owner_id BIGINT;

    WHILE i < num_contacts DO
        SET v_account_id = FLOOR(1 + RAND() * 8);
        SET v_owner_id = FLOOR(2 + RAND() * 4);

        INSERT INTO contacts (account_id, first_name, last_name, email, lifecycle_stage, owner_id)
        VALUES (
            v_account_id,
            ELT(FLOOR(1 + RAND() * 10), 'James','Mary','John','Patricia','Robert','Jennifer','Michael','Linda','David','Elizabeth'),
            ELT(FLOOR(1 + RAND() * 10), 'Smith','Johnson','Williams','Brown','Jones','Garcia','Miller','Davis','Rodriguez','Martinez'),
            CONCAT('contact_', i, '_', FLOOR(RAND() * 10000), '@example.com'),
            ELT(FLOOR(1 + RAND() * 5), 'subscriber','lead','mql','sql','customer'),
            v_owner_id
        );

        SET i = i + 1;
    END WHILE;
END //
DELIMITER ;

-- Run it
CALL seed_contacts(10000);

-- Clean up
DROP PROCEDURE IF EXISTS seed_contacts;
```

### Seed Data Principles

1. Use realistic names, emails, and company names — not "test1", "test2".
2. Create data that exercises all relationship types (accounts with multiple contacts, contacts with multiple opportunities, etc.).
3. Include data in various statuses (active, inactive, converted, closed-won, closed-lost).
4. Include edge cases: contacts without accounts, leads that have been converted, accounts with hierarchies.
5. Use consistent, documented passwords (like `$2b$12$placeholder_hash`) and note that they are not real hashes.
6. Timestamp data should span multiple months/quarters to test reporting.

FILE:references/custom-fields-and-flexibility.md
# Custom Fields and Flexibility

CRM systems must accommodate user-defined fields. Sales teams want to track industry-specific data, marketing wants custom segmentation attributes, and every client has unique requirements. This reference covers the three main approaches to custom fields in MySQL 8 and when to use each.

## Approach 1: Entity-Attribute-Value (EAV)

The EAV pattern stores custom field definitions and values in separate tables rather than adding columns to the entity table.

### Schema

```sql
-- Define available custom fields
CREATE TABLE `custom_field_definitions` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `entity_type` VARCHAR(50) NOT NULL COMMENT 'e.g., contact, account, opportunity, lead',
    `field_name` VARCHAR(100) NOT NULL,
    `field_label` VARCHAR(200) NOT NULL,
    `field_type` ENUM('text', 'number', 'decimal', 'boolean', 'date', 'datetime',
                       'select', 'multi_select', 'email', 'url', 'phone') NOT NULL,
    `is_required` TINYINT(1) NOT NULL DEFAULT 0,
    `default_value` VARCHAR(500) NULL DEFAULT NULL,
    `options` JSON NULL DEFAULT NULL COMMENT 'For select/multi_select: ["opt1","opt2","opt3"]',
    `validation_rules` JSON NULL DEFAULT NULL COMMENT '{"min_length":1,"max_length":500,"regex":"..."}',
    `display_order` INT UNSIGNED NOT NULL DEFAULT 0,
    `is_active` TINYINT(1) NOT NULL DEFAULT 1,
    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    `updated_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,

    PRIMARY KEY (`id`),
    UNIQUE INDEX `uq_cfd_entity_field` (`entity_type`, `field_name`),
    INDEX `idx_cfd_entity_type` (`entity_type`, `is_active`, `display_order`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

-- Store custom field values (one row per entity-field pair)
CREATE TABLE `custom_field_values` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `field_definition_id` BIGINT UNSIGNED NOT NULL,
    `entity_type` VARCHAR(50) NOT NULL,
    `entity_id` BIGINT UNSIGNED NOT NULL,
    `value_text` TEXT NULL DEFAULT NULL,
    `value_number` DECIMAL(20, 6) NULL DEFAULT NULL,
    `value_boolean` TINYINT(1) NULL DEFAULT NULL,
    `value_date` DATE NULL DEFAULT NULL,
    `value_datetime` DATETIME(3) NULL DEFAULT NULL,
    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    `updated_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,

    PRIMARY KEY (`id`),
    UNIQUE INDEX `uq_cfv_entity_field` (`field_definition_id`, `entity_type`, `entity_id`),
    INDEX `idx_cfv_entity` (`entity_type`, `entity_id`),
    INDEX `idx_cfv_value_text` (`field_definition_id`, `value_text`(100)),
    INDEX `idx_cfv_value_number` (`field_definition_id`, `value_number`),
    INDEX `idx_cfv_value_date` (`field_definition_id`, `value_date`),

    CONSTRAINT `fk_cfv_definition` FOREIGN KEY (`field_definition_id`)
        REFERENCES `custom_field_definitions` (`id`) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
```

### Querying EAV Data

Pivoting EAV data into columns for display:

```sql
-- Get all custom fields for a specific contact
SELECT
    cfd.field_name,
    cfd.field_label,
    cfd.field_type,
    COALESCE(cfv.value_text, cfv.value_number, cfv.value_boolean,
             cfv.value_date, cfv.value_datetime) AS value
FROM custom_field_definitions cfd
LEFT JOIN custom_field_values cfv
    ON cfv.field_definition_id = cfd.id
    AND cfv.entity_type = 'contact'
    AND cfv.entity_id = :contact_id
WHERE cfd.entity_type = 'contact'
    AND cfd.is_active = 1
ORDER BY cfd.display_order;

-- Filter contacts by a custom field value
SELECT c.*
FROM contacts c
INNER JOIN custom_field_values cfv
    ON cfv.entity_type = 'contact' AND cfv.entity_id = c.id
INNER JOIN custom_field_definitions cfd
    ON cfd.id = cfv.field_definition_id
WHERE cfd.field_name = 'preferred_language'
    AND cfv.value_text = 'Spanish'
    AND c.deleted_at IS NULL;
```

### EAV Pros and Cons

**Pros:**
- Unlimited custom fields without schema changes
- Users can create fields at runtime
- No ALTER TABLE needed (no table locks, no downtime)
- Clean separation between core schema and customizations

**Cons:**
- Complex queries — pivoting and filtering require JOINs
- Harder to enforce data types at the database level
- Reporting becomes more complex
- Cannot use standard SQL constraints (NOT NULL, CHECK, UNIQUE) on values
- Performance degrades with heavy filtering across multiple custom fields

**Best for:** SaaS CRM where tenants define their own fields, systems with 50+ custom fields, fields that change frequently.

## Approach 2: JSON Columns

Store custom fields as a JSON document directly on the entity table.

### Schema

```sql
ALTER TABLE `contacts`
    ADD COLUMN `custom_fields` JSON NULL DEFAULT NULL,
    ADD CONSTRAINT `chk_contacts_custom_fields` CHECK (
        `custom_fields` IS NULL OR JSON_VALID(`custom_fields`)
    );
```

### Example Data

```json
{
    "preferred_language": "Spanish",
    "shirt_size": "L",
    "nps_score": 8,
    "renewal_date": "2026-03-15",
    "interests": ["AI", "cloud", "security"],
    "custom_notes": "Prefers morning calls"
}
```

### Indexing JSON for Performance

```sql
-- Generated column approach (most reliable)
ALTER TABLE `contacts`
    ADD COLUMN `cf_preferred_language` VARCHAR(50)
        GENERATED ALWAYS AS (`custom_fields` ->> '$.preferred_language') STORED,
    ADD INDEX `idx_contacts_cf_language` (`cf_preferred_language`);

-- Functional index approach (MySQL 8.0.13+)
ALTER TABLE `contacts`
    ADD INDEX `idx_contacts_cf_nps` ((
        CAST(`custom_fields` ->> '$.nps_score' AS SIGNED)
    ));

-- Multi-valued index for arrays (MySQL 8.0.17+)
ALTER TABLE `contacts`
    ADD INDEX `idx_contacts_cf_interests` ((
        CAST(`custom_fields` -> '$.interests' AS CHAR(100) ARRAY)
    ));
```

### Querying JSON Custom Fields

```sql
-- Simple filter
SELECT * FROM contacts
WHERE custom_fields ->> '$.preferred_language' = 'Spanish'
    AND deleted_at IS NULL;

-- Numeric comparison
SELECT * FROM contacts
WHERE CAST(custom_fields ->> '$.nps_score' AS SIGNED) >= 8
    AND deleted_at IS NULL;

-- Array membership
SELECT * FROM contacts
WHERE 'AI' MEMBER OF (custom_fields -> '$.interests')
    AND deleted_at IS NULL;

-- Check if a field exists
SELECT * FROM contacts
WHERE JSON_CONTAINS_PATH(custom_fields, 'one', '$.renewal_date')
    AND deleted_at IS NULL;

-- Update a single field without replacing the entire document
UPDATE contacts
SET custom_fields = JSON_SET(
    COALESCE(custom_fields, '{}'),
    '$.nps_score', 9,
    '$.last_survey_date', '2026-04-01'
)
WHERE id = :contact_id;
```

### JSON Pros and Cons

**Pros:**
- Simple to implement — just one column
- Read performance is good for fetching a single record's custom data
- MySQL 8 partial update optimization for JSON_SET (no full document rewrite)
- Natural fit for application frameworks with JSON serialization
- Flexible schema — any structure without ALTER TABLE

**Cons:**
- Cannot enforce field-level constraints (required fields, data types, allowed values)
- Indexing requires explicit generated columns or functional indexes
- Complex cross-record filtering on JSON fields is slower than regular columns
- No built-in field definitions or validation at database level
- JSON documents should stay under a few KB — avoid storing large blobs

**Best for:** Small to moderate number of custom fields (under 20), systems where custom fields are rarely filtered or sorted, metadata and preferences storage.

## Approach 3: Hybrid (Recommended for Most CRMs)

Combine fixed columns for well-known fields with JSON for truly custom/flexible data.

### Schema Pattern

```sql
CREATE TABLE `contacts` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,

    -- Fixed, well-known columns (indexed, typed, constrained)
    `first_name` VARCHAR(100) NOT NULL,
    `last_name` VARCHAR(100) NOT NULL,
    `email` VARCHAR(255) NULL DEFAULT NULL,
    `phone` VARCHAR(30) NULL DEFAULT NULL,
    `account_id` BIGINT UNSIGNED NULL DEFAULT NULL,
    `lifecycle_stage` ENUM('subscriber','lead','mql','sql','opportunity','customer','evangelist','other') NOT NULL DEFAULT 'subscriber',

    -- JSON for custom/flexible fields
    `custom_fields` JSON NULL DEFAULT NULL,

    -- Generated columns to index frequently queried custom fields
    `cf_preferred_language` VARCHAR(50)
        GENERATED ALWAYS AS (`custom_fields` ->> '$.preferred_language') STORED,

    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    `updated_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    `deleted_at` TIMESTAMP NULL DEFAULT NULL,

    PRIMARY KEY (`id`),
    INDEX `idx_contacts_cf_language` (`cf_preferred_language`),
    -- ... other indexes
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
```

### When to Promote a JSON Field to a Column

A custom field should graduate from JSON to a dedicated column when:
- It is queried in WHERE or ORDER BY clauses frequently
- It needs database-level constraints (NOT NULL, UNIQUE, CHECK, FK)
- It is used in JOINs
- It appears in reports or dashboards
- It is present on more than 80% of records
- It has a fixed, well-defined data type

### Decision Matrix

| Factor | Use Fixed Column | Use JSON | Use EAV |
|--------|-----------------|----------|---------|
| Known at design time | Yes | — | — |
| Used in WHERE/JOIN | Yes | Maybe (with gen col) | Possible but slow |
| Needs FK constraint | Yes | No | No |
| User-defined at runtime | — | Yes (small count) | Yes (large count) |
| Per-tenant schema | — | Yes | Yes |
| 50+ custom fields | — | — | Yes |
| Reporting/analytics | Yes | With gen columns | Complex |

### Metadata for JSON Custom Fields

Even with the JSON approach, maintain a field definitions table so the UI knows what fields exist and how to render them:

```sql
CREATE TABLE `custom_field_metadata` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `entity_type` VARCHAR(50) NOT NULL,
    `json_path` VARCHAR(200) NOT NULL COMMENT 'Path in JSON document, e.g. $.preferred_language',
    `field_label` VARCHAR(200) NOT NULL,
    `field_type` ENUM('text','number','decimal','boolean','date','datetime',
                       'select','multi_select','email','url','phone') NOT NULL,
    `options` JSON NULL DEFAULT NULL,
    `is_required` TINYINT(1) NOT NULL DEFAULT 0,
    `display_order` INT UNSIGNED NOT NULL DEFAULT 0,
    `is_active` TINYINT(1) NOT NULL DEFAULT 1,
    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,

    PRIMARY KEY (`id`),
    UNIQUE INDEX `uq_cfm_entity_path` (`entity_type`, `json_path`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
```

This gives you the flexibility of JSON storage with the discoverability of EAV field definitions.

FILE:references/audit-and-soft-deletes.md
# Audit Trails and Soft Deletes

CRM data has legal, compliance, and business intelligence value. Deleting records permanently destroys historical context. This reference covers patterns for preserving data integrity, tracking changes, and meeting compliance requirements.

## Soft Deletes

### The Pattern

Instead of `DELETE FROM contacts WHERE id = 42`, set a deletion timestamp:

```sql
UPDATE contacts SET deleted_at = NOW(), updated_by = :user_id WHERE id = 42;
```

Every CRM entity table should include:

```sql
`deleted_at` TIMESTAMP NULL DEFAULT NULL,
```

And an index to support filtering:

```sql
INDEX `idx_tablename_deleted_at` (`deleted_at`)
```

### Query Discipline

Every query that fetches active records must include the soft delete filter:

```sql
-- Always filter for active records
SELECT * FROM contacts WHERE account_id = 42 AND deleted_at IS NULL;

-- For admin/compliance views that need to see deleted records
SELECT * FROM contacts WHERE account_id = 42;

-- To see only deleted records
SELECT * FROM contacts WHERE account_id = 42 AND deleted_at IS NOT NULL;
```

Consider creating views for active records to avoid forgetting the filter:

```sql
CREATE VIEW `active_contacts` AS
SELECT * FROM `contacts` WHERE `deleted_at` IS NULL;

CREATE VIEW `active_accounts` AS
SELECT * FROM `accounts` WHERE `deleted_at` IS NULL;

CREATE VIEW `active_opportunities` AS
SELECT * FROM `opportunities` WHERE `deleted_at` IS NULL;
```

### Handling Unique Constraints with Soft Deletes

Problem: if `email` has a UNIQUE index and a contact is soft-deleted, no new contact can be created with the same email.

Solutions:

**Option A: Partial unique index (not natively supported in MySQL)**
MySQL does not support partial/filtered unique indexes. Use Option B or C.

**Option B: Unique on (email, deleted_at) with a sentinel value**
```sql
-- Use a fixed date for active records, actual deletion time for deleted ones
-- This allows the same email to exist once as active plus multiple times as deleted
ALTER TABLE `contacts`
    ADD UNIQUE INDEX `uq_contacts_email_active` (`email`, `deleted_at`);
```
This works because `deleted_at = NULL` is treated as unique by MySQL (each NULL is distinct). However, it means two active records with the same email are both allowed since `(email, NULL)` and `(email, NULL)` are not considered duplicates. This approach has limitations.

**Option C: Application-level uniqueness check (most practical)**
```sql
-- Add a non-unique index for performance
ALTER TABLE `contacts` ADD INDEX `idx_contacts_email` (`email`);
```
Check uniqueness in application code: `SELECT id FROM contacts WHERE email = :email AND deleted_at IS NULL LIMIT 1`. This is the most common approach in production CRM systems.

**Option D: Generated column approach**
```sql
ALTER TABLE `contacts`
    ADD COLUMN `email_unique_check` VARCHAR(255)
        GENERATED ALWAYS AS (IF(`deleted_at` IS NULL, `email`, NULL)) STORED,
    ADD UNIQUE INDEX `uq_contacts_email_active` (`email_unique_check`);
```
This creates a generated column that is non-NULL only for active records, allowing a unique index that only applies to active rows. Multiple NULL values in a unique index are allowed.

### Restore (Undelete)

```sql
UPDATE contacts SET deleted_at = NULL, updated_by = :user_id WHERE id = 42;
```

### Cascading Soft Deletes

When an account is soft-deleted, should its contacts and opportunities also be soft-deleted? This depends on business rules. Common approaches:
- Soft-delete only the parent, and filter children by joining to the parent's `deleted_at`
- Cascade the soft delete in application code or a trigger
- Leave children active but orphaned, and let a cleanup job handle them

## Audit Trail

### Audit Log Table

A central audit log records every significant change to CRM data.

```sql
CREATE TABLE `audit_logs` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `entity_type` VARCHAR(50) NOT NULL COMMENT 'Table name: contact, account, opportunity, etc.',
    `entity_id` BIGINT UNSIGNED NOT NULL,
    `action` ENUM('create', 'update', 'delete', 'restore', 'merge', 'convert', 'assign') NOT NULL,
    `user_id` BIGINT UNSIGNED NULL DEFAULT NULL COMMENT 'User who made the change, NULL for system actions',
    `changes` JSON NULL DEFAULT NULL COMMENT 'Field-level diff: {"field": {"old": "x", "new": "y"}}',
    `metadata` JSON NULL DEFAULT NULL COMMENT 'Additional context: IP address, source, request_id',
    `created_at` TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),

    PRIMARY KEY (`id`),
    INDEX `idx_audit_entity` (`entity_type`, `entity_id`, `created_at`),
    INDEX `idx_audit_user` (`user_id`, `created_at`),
    INDEX `idx_audit_action` (`action`, `created_at`),
    INDEX `idx_audit_created_at` (`created_at`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
```

Use `TIMESTAMP(3)` for millisecond precision — important when multiple changes happen in rapid succession.

### What to Log

Log all changes to core CRM entities. The `changes` JSON should capture:

```json
{
    "first_name": {"old": "Jon", "new": "John"},
    "email": {"old": "[email protected]", "new": "[email protected]"},
    "lifecycle_stage": {"old": "lead", "new": "mql"}
}
```

For create actions, log the initial values:
```json
{
    "first_name": {"old": null, "new": "John"},
    "email": {"old": null, "new": "[email protected]"}
}
```

For delete actions, log the key identifying fields:
```json
{
    "first_name": {"old": "John", "new": null},
    "email": {"old": "[email protected]", "new": null}
}
```

### Implementation: Application Level vs. Triggers

**Application-level logging (recommended):**
- Captured in the service layer before/after save
- Has access to the current user, request context, and business logic
- Can batch audit entries and write asynchronously
- Easier to test and maintain

**Trigger-based logging:**
- Catches all changes including direct SQL and admin operations
- Does not know who made the change (no user context in MySQL triggers)
- Adds overhead to every write operation
- Harder to debug and maintain

Recommendation: Use application-level logging as the primary mechanism. Add database triggers only if you need to catch out-of-band changes (direct SQL access, migration scripts, etc.).

### Example Trigger (for reference)

```sql
DELIMITER //

CREATE TRIGGER `trg_contacts_after_update`
AFTER UPDATE ON `contacts`
FOR EACH ROW
BEGIN
    DECLARE v_changes JSON DEFAULT JSON_OBJECT();

    IF OLD.first_name != NEW.first_name OR (OLD.first_name IS NULL) != (NEW.first_name IS NULL) THEN
        SET v_changes = JSON_SET(v_changes, '$.first_name',
            JSON_OBJECT('old', OLD.first_name, 'new', NEW.first_name));
    END IF;

    IF OLD.email != NEW.email OR (OLD.email IS NULL) != (NEW.email IS NULL) THEN
        SET v_changes = JSON_SET(v_changes, '$.email',
            JSON_OBJECT('old', OLD.email, 'new', NEW.email));
    END IF;

    IF OLD.lifecycle_stage != NEW.lifecycle_stage THEN
        SET v_changes = JSON_SET(v_changes, '$.lifecycle_stage',
            JSON_OBJECT('old', OLD.lifecycle_stage, 'new', NEW.lifecycle_stage));
    END IF;

    -- Only insert if something actually changed
    IF JSON_LENGTH(v_changes) > 0 THEN
        INSERT INTO audit_logs (entity_type, entity_id, action, user_id, changes)
        VALUES ('contact', NEW.id, 'update', NEW.updated_by, v_changes);
    END IF;
END //

DELIMITER ;
```

### Partitioning Audit Logs

Audit logs grow unboundedly. Partition by time period:

```sql
CREATE TABLE `audit_logs` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `entity_type` VARCHAR(50) NOT NULL,
    `entity_id` BIGINT UNSIGNED NOT NULL,
    `action` ENUM('create','update','delete','restore','merge','convert','assign') NOT NULL,
    `user_id` BIGINT UNSIGNED NULL,
    `changes` JSON NULL,
    `metadata` JSON NULL,
    `created_at` TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),

    PRIMARY KEY (`id`, `created_at`),
    INDEX `idx_audit_entity` (`entity_type`, `entity_id`, `created_at`),
    INDEX `idx_audit_user` (`user_id`, `created_at`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci
PARTITION BY RANGE (UNIX_TIMESTAMP(`created_at`)) (
    PARTITION p2025_q1 VALUES LESS THAN (UNIX_TIMESTAMP('2025-04-01')),
    PARTITION p2025_q2 VALUES LESS THAN (UNIX_TIMESTAMP('2025-07-01')),
    PARTITION p2025_q3 VALUES LESS THAN (UNIX_TIMESTAMP('2025-10-01')),
    PARTITION p2025_q4 VALUES LESS THAN (UNIX_TIMESTAMP('2026-01-01')),
    PARTITION p2026_q1 VALUES LESS THAN (UNIX_TIMESTAMP('2026-04-01')),
    PARTITION p2026_q2 VALUES LESS THAN (UNIX_TIMESTAMP('2026-07-01')),
    PARTITION p_future VALUES LESS THAN MAXVALUE
);
```

Benefits:
- Old partitions can be archived or dropped without affecting active data
- Queries that filter by date range automatically prune irrelevant partitions
- Maintenance operations (OPTIMIZE, ANALYZE) can target individual partitions

## GDPR and Data Privacy Compliance

### Right to Erasure

GDPR requires the ability to permanently delete personal data on request. Soft deletes alone are not sufficient.

Implement a "hard purge" process:

```sql
-- Step 1: Anonymize the contact record
UPDATE contacts SET
    first_name = 'REDACTED',
    last_name = 'REDACTED',
    email = CONCAT('redacted_', id, '@deleted.invalid'),
    phone = NULL,
    mobile_phone = NULL,
    mailing_address_line1 = NULL,
    mailing_address_line2 = NULL,
    mailing_city = NULL,
    mailing_state = NULL,
    mailing_postal_code = NULL,
    mailing_country = NULL,
    linkedin_url = NULL,
    custom_fields = NULL,
    deleted_at = NOW()
WHERE id = :contact_id;

-- Step 2: Anonymize related audit log entries
UPDATE audit_logs SET
    changes = JSON_OBJECT('redacted', true, 'reason', 'gdpr_erasure_request')
WHERE entity_type = 'contact' AND entity_id = :contact_id;

-- Step 3: Log the erasure event itself
INSERT INTO audit_logs (entity_type, entity_id, action, user_id, metadata)
VALUES ('contact', :contact_id, 'delete', :admin_user_id,
    JSON_OBJECT('reason', 'gdpr_erasure_request', 'request_date', NOW()));
```

Key points:
- Anonymize rather than DELETE to preserve referential integrity and aggregate reporting.
- The anonymized email uses a pattern (`[email protected]`) that is unique and clearly non-functional.
- Keep the audit log entry documenting that erasure happened (the erasure itself is a compliance event).

### Data Retention Policies

Define retention periods per data type:

| Data Type | Suggested Retention | Action After Expiry |
|-----------|-------------------|-------------------|
| Active CRM records | Indefinite | N/A |
| Soft-deleted records | 90 days | Hard purge or archive |
| Audit logs | 2-7 years | Archive to cold storage |
| Activity logs | 1-3 years | Archive or aggregate |
| Email/campaign events | 1-2 years | Aggregate into summary tables |

Implement automated retention jobs that run daily or weekly to enforce these policies.

---
name: npm-n8n-nodes
description: >
  Build, structure, and publish npm packages for n8n custom community nodes. Use this skill
  whenever the user wants to create a custom n8n node, publish a node to npm, add credentials
  or authentication to an n8n node, handle HTTP requests inside a node, define node UI properties,
  scaffold a new n8n node package, wire up OAuth2 / API key / header auth, handle request bodies
  and responses, work with binary files in n8n, create trigger or webhook nodes, handle errors,
  version nodes, or publish a community node to npm. Also trigger when user mentions "n8n node",
  "community node", "custom integration", "n8n-nodes-*", "IExecuteFunctions", "ICredentialType",
  "INodeType", "n8n workflow node", or "n8n trigger". This skill covers the FULL lifecycle:
  scaffold → code → credentials → test → publish.
---

# n8n Custom Node — NPM Package Skill

## Core Mental Model

Every n8n node follows one pattern:

```
getInputData()  →  loop items  →  do stuff  →  push to returnData  →  return [returnData]
```

Two file types do all the work:
- **Node file** (`nodes/MyNode/MyNode.node.ts`) — UI fields + execute logic
- **Credential file** (`credentials/MyApi.credentials.ts`) — auth definition

Everything else is project plumbing.

---

## Project Structure

```
n8n-nodes-yourservice/
├── package.json              ← CRITICAL: must have n8n section + correct keyword
├── tsconfig.json
├── .eslintrc.js
├── gulpfile.js               ← copies SVG icons to dist/
├── index.js                  ← optional explicit entry point
├── nodes/
│   └── YourService/
│       ├── YourService.node.ts
│       ├── YourService.node.json   ← optional: codex metadata
│       └── yourservice.svg
├── credentials/
│   └── YourServiceApi.credentials.ts
└── dist/                     ← compiled output (never edit manually)
```

---

## What to Read and When

This skill has focused reference files. Load only what you need:

### Node Types (pick one)
| If you need... | Read |
|---|---|
| Standard request/response node (most common) | `references/examples/nodes/programmatic-node.md` |
| Simple REST API, no complex logic | `references/examples/nodes/declarative-node.md` |
| Trigger that polls an API on a schedule | `references/examples/nodes/trigger-node.md` |
| Webhook that receives HTTP calls | `references/examples/nodes/webhook-node.md` |

### Credentials (pick what matches your auth)
| Auth type | Read |
|---|---|
| API key, Bearer token, custom header, query key | `references/examples/credentials/api-key-patterns.md` |
| OAuth2 (user login or machine-to-machine) | `references/examples/credentials/oauth2-patterns.md` |
| Basic auth, multi-field, manual inject | `references/examples/credentials/other-patterns.md` |

### Concepts (load when the topic comes up)
| Topic | Read |
|---|---|
| UI field types, displayOptions, collections, fixedCollection | `references/concepts/node-properties.md` |
| HTTP requests, bodies, headers, responses, binary | `references/concepts/http-and-binary.md` |
| Error types, continueOnFail, NodeApiError vs NodeOperationError | `references/concepts/error-handling.md` |
| pairedItem, data flow, why item tracking matters | `references/concepts/data-and-pairing.md` |
| Node versioning, updating without breaking workflows | `references/concepts/node-versioning.md` |

### Project Setup & Publishing
| Topic | Read |
|---|---|
| package.json, tsconfig, gulpfile, eslintrc, index.js | `references/templates/project-files.md` |
| Local testing, npm link, n8n start | `references/templates/local-testing.md` |
| npm publish, GitHub Actions, provenance | `references/templates/publishing.md` |
| Common gotchas and silent failures | `references/gotchas/common-gotchas.md` |

---

## Quick-Start Pattern (copy this first)

```typescript
// nodes/YourService/YourService.node.ts
import {
  IExecuteFunctions,
  INodeExecutionData,
  INodeType,
  INodeTypeDescription,
  NodeOperationError,
} from 'n8n-workflow';

export class YourService implements INodeType {
  description: INodeTypeDescription = {
    displayName: 'Your Service',
    name: 'yourService',
    icon: 'file:yourservice.svg',
    group: ['transform'],
    version: 1,
    description: 'Interact with Your Service API',
    defaults: { name: 'Your Service' },
    inputs: ['main'],
    outputs: ['main'],
    credentials: [{ name: 'yourServiceApi', required: true }],
    properties: [
      {
        displayName: 'Endpoint',
        name: 'endpoint',
        type: 'string',
        default: '/users',
        required: true,
      },
    ],
  };

  async execute(this: IExecuteFunctions): Promise<INodeExecutionData[][]> {
    const items = this.getInputData();
    const returnData: INodeExecutionData[] = [];
    const credentials = await this.getCredentials('yourServiceApi');

    for (let i = 0; i < items.length; i++) {
      try {
        const endpoint = this.getNodeParameter('endpoint', i) as string;

        const response = await this.helpers.httpRequest({
          method: 'GET',
          url: `https://api.yourservice.comendpoint`,
          headers: {
            Authorization: `Bearer credentials.apiToken`,
          },
        });

        returnData.push({ json: response, pairedItem: { item: i } });

      } catch (error) {
        if (this.continueOnFail()) {
          returnData.push({ json: { error: error.message }, pairedItem: { item: i } });
          continue;
        }
        throw new NodeOperationError(this.getNode(), error, { itemIndex: i });
      }
    }

    return [returnData];
  }
}
```

---

## Essential APIs Cheat Sheet

```typescript
// Input
const items = this.getInputData();

// Parameters
this.getNodeParameter('name', i) as string
this.getNodeParameter('count', i, 0) as number
this.getNodeParameter('options', i, {}) as IDataObject

// Credentials
const creds = await this.getCredentials('myCredentialName');

// HTTP
await this.helpers.httpRequest({ method, url, headers, qs, body })

// Error handling
this.continueOnFail()
throw new NodeOperationError(this.getNode(), message, { itemIndex: i })
throw new NodeApiError(this.getNode(), error)   // for API-level HTTP errors

// Output
returnData.push({ json: data, pairedItem: { item: i } })
return [returnData];
```

---

## Pre-Publish Checklist

- [ ] `keywords` in package.json includes `"n8n-community-node-package"`
- [ ] `n8n.nodes` and `n8n.credentials` arrays point to `dist/` `.js` paths
- [ ] Node `name` is camelCase; `displayName` is human-readable
- [ ] Credential `name` exactly matches string passed to `getCredentials('...')`
- [ ] Every `returnData.push()` includes `pairedItem: { item: i }`
- [ ] `continueOnFail()` is handled in all try/catch blocks
- [ ] SVG icon exists in `nodes/YourService/` and referenced as `'file:yourservice.svg'`
- [ ] `npm run build` succeeds (no TypeScript errors)
- [ ] `npm run lint` passes (required for community submission)
- [ ] Tested locally via `npm link`
- [ ] Version bumped in `package.json` before publish

FILE:references/gotchas/common-gotchas.md
# Common Gotchas and Silent Failures

These are the issues that don't throw obvious errors — they just make your node broken, invisible, or wrong in subtle ways.

## Table of Contents
1. [Registration and discovery failures](#1-registration-and-discovery-failures)
2. [Credential name mismatches (silent!)](#2-credential-name-mismatches)
3. [pairedItem missing — expressions break silently](#3-paireditem-missing)
4. [json must be an object, never an array or primitive](#4-json-must-be-an-object)
5. [dist/ not committed or not built in CI](#5-dist-not-committed-or-built)
6. [Icons not copied to dist/](#6-icons-not-copied)
7. [Linting failures block n8n Cloud installation](#7-linting-failures)
8. [noDataExpression missing — expressions appear on wrong fields](#8-nodataexpression)
9. [OAuth2 token caching bug](#9-oauth2-token-caching)
10. [continueOnFail not handled — error swallowed silently](#10-continuenonfail-not-handled)
11. [TypeScript strict mode catches these — enable it](#11-typescript-strict-mode)
12. [Version mismatch — wrong node version on old workflows](#12-version-mismatch)

---

## 1. Registration and Discovery Failures

**Symptom:** Node doesn't appear in n8n's node picker.

**Causes and fixes:**

```json
// ❌ Wrong — missing the required keyword
"keywords": ["n8n", "automation"]

// ✅ Correct
"keywords": ["n8n-community-node-package"]
```

```json
// ❌ Wrong — pointing at source TypeScript files
"n8n": {
  "nodes": ["nodes/MyNode/MyNode.node.ts"]
}

// ✅ Correct — must point at compiled JavaScript in dist/
"n8n": {
  "nodes": ["dist/nodes/MyNode/MyNode.node.js"]
}
```

```json
// ❌ Wrong — path doesn't match actual folder/file name (case-sensitive!)
"nodes": ["dist/nodes/mynode/MyNode.node.js"]

// ✅ Correct — path must exactly match the filesystem
"nodes": ["dist/nodes/MyNode/MyNode.node.js"]
```

---

## 2. Credential Name Mismatches (Silent!)

This is the most common silent failure. The credential loads but never applies.

```typescript
// In credentials file:
export class MyServiceApi implements ICredentialType {
  name = 'myServiceApi';    // ← THIS string must match exactly
}

// In node file:
// ❌ Wrong — different casing
const creds = await this.getCredentials('MyServiceApi');
const creds = await this.getCredentials('myserviceapi');
const creds = await this.getCredentials('my-service-api');

// ✅ Correct — exact match, case-sensitive
const creds = await this.getCredentials('myServiceApi');
```

Also check the `credentials` array in the node description:
```typescript
credentials: [
  {
    name: 'myServiceApi',   // ← must match credential's `name` property
    required: true,
  },
],
```

And the `n8n.credentials` entry in `package.json`:
```json
"credentials": ["dist/credentials/MyServiceApi.credentials.js"]
//                                 ^ filename must match the TypeScript class file
```

---

## 3. pairedItem Missing — Expressions Break Silently

**Symptom:** `{{ $('Previous Node').item.json.field }}` returns wrong data or undefined. Users won't know why.

```typescript
// ❌ Wrong — pairedItem missing
returnData.push({ json: response });

// ✅ Correct — always include pairedItem
returnData.push({ json: response, pairedItem: { item: i } });
```

If you push multiple items for one input item:
```typescript
for (const subItem of responseArray) {
  returnData.push({
    json: subItem,
    pairedItem: { item: i },   // ← all sub-items link back to the same input item
  });
}
```

---

## 4. json Must Be an Object, Never an Array or Primitive

**Symptom:** n8n throws a confusing internal error or the node output looks wrong.

```typescript
// ❌ Wrong — json is an array
returnData.push({ json: [1, 2, 3], pairedItem: { item: i } });

// ❌ Wrong — json is a string or number
returnData.push({ json: 'hello' as any, pairedItem: { item: i } });

// ✅ Correct — wrap arrays
returnData.push({ json: { items: [1, 2, 3] }, pairedItem: { item: i } });

// ✅ Correct — or push each element separately
for (const element of responseArray) {
  returnData.push({ json: element as IDataObject, pairedItem: { item: i } });
}

// ✅ Correct — wrap primitives
returnData.push({ json: { value: 'hello' }, pairedItem: { item: i } });
```

---

## 5. dist/ Not Committed or Built in CI

**Symptom:** Works locally, broken when installed from npm.

npm only ships files listed in `"files"`. If `"files": ["dist"]` and `dist/` is gitignored and not built in CI, your published package will be empty.

**Fix for GitHub Actions:**
```yaml
# In publish.yml — always build before publishing
- name: Build
  run: npm run build      # ← must be here
- name: Publish
  run: npm publish --access public
```

The `prepublishOnly` script in package.json also helps: `"prepublishOnly": "npm run build && npm run lint"` — it runs automatically before `npm publish`.

---

## 6. Icons Not Copied to dist/

**Symptom:** Node appears with a broken/missing icon.

TypeScript compiler (`tsc`) copies `.ts` files only. SVG/PNG icons need to be copied separately by gulpfile:

```bash
# Check: does dist/ have your SVG?
ls dist/nodes/MyNode/
# Should show: MyNode.node.js  MyNode.node.d.ts  mynode.svg

# If SVG is missing, run:
npx gulp build:icons
# OR:
npm run build   # (if build script includes gulp)
```

And verify the icon path in your node matches exactly:
```typescript
icon: 'file:mynode.svg',   // file must exist at dist/nodes/MyNode/mynode.svg
```

---

## 7. Linting Failures Block n8n Cloud Installation

n8n Cloud validates `eslint-plugin-n8n-nodes-base` rules before installing community nodes. Linting errors = installation failure.

```bash
# Always check before publishing
npm run lint

# Auto-fix what's fixable
npm run lintfix

# Common errors:
# - "node-class-description-missing-subtitle" — add subtitle field
# - "node-param-description-missing-final-period" — end descriptions with "."
# - "node-param-display-name-miscased" — use Title Case for displayName
# - "cred-class-field-name-unsuffixed" — credential name must end with "Api"
```

---

## 8. noDataExpression Missing on Selectors

**Symptom:** User accidentally types an expression into the Resource or Operation dropdown, breaking the node UI.

```typescript
// ❌ Wrong — user can put expressions into resource/operation fields
{
  displayName: 'Resource',
  name: 'resource',
  type: 'options',
  options: [...],
  default: 'user',
}

// ✅ Correct — prevent expression mode on structural selectors
{
  displayName: 'Resource',
  name: 'resource',
  type: 'options',
  noDataExpression: true,   // ← always add this to resource/operation fields
  options: [...],
  default: 'user',
}
```

---

## 9. OAuth2 Token Caching Bug

**Symptom:** OAuth2 node works on first run, fails after token expiry.

```typescript
// ❌ Wrong — storing token outside getCredentials causes stale token issues
let cachedToken: string;

async execute() {
  if (!cachedToken) {
    const creds = await this.getCredentials('myOAuth2Api');
    cachedToken = (creds.oauthTokenData as any).access_token;
  }
  // cachedToken may be expired!
}

// ✅ Correct — always call getCredentials() fresh inside execute()
// n8n handles refresh automatically and returns the current valid token
async execute() {
  const creds = await this.getCredentials('myOAuth2Api');
  const token = (creds.oauthTokenData as { access_token: string }).access_token;
  // token is always fresh — n8n refreshed it before returning
}
```

---

## 10. continueOnFail Not Handled

**Symptom:** When user enables "Continue On Error", the node throws anyway and halts the workflow.

```typescript
// ❌ Wrong — error always stops workflow
} catch (error) {
  throw new NodeOperationError(this.getNode(), error.message);
}

// ✅ Correct — respect user's setting
} catch (error) {
  if (this.continueOnFail()) {
    returnData.push({ json: { error: error.message }, pairedItem: { item: i } });
    continue;
  }
  throw new NodeOperationError(this.getNode(), error, { itemIndex: i });
}
```

---

## 11. TypeScript Strict Mode Catches These — Enable It

```json
// tsconfig.json — always use strict: true
{
  "compilerOptions": {
    "strict": true    // ← catches null issues, implicit any, and more
  }
}
```

Common strict-mode patterns:
```typescript
// Instead of: const name = response.name
const name = response.name as string;        // explicit cast
const name = (response as IDataObject).name as string;

// Instead of: credentials.apiToken
const token = credentials.apiToken as string;  // TypeScript won't infer the type
```

---

## 12. Version Mismatch on Old Workflows

**Symptom:** Existing workflows use old node behavior; updating the package breaks them.

When you bump the npm package version but don't increment the node's `version` field, all instances in existing workflows silently use the new behavior — even if you changed output structure.

```typescript
// After a breaking change, ALWAYS increment version in the node description:
description: INodeTypeDescription = {
  version: [1, 2],          // ← add the new version
  defaultVersion: 2,         // ← new nodes get v2
}
// And add @version displayOptions to fields that changed
```

See `references/concepts/node-versioning.md` for the full pattern.

FILE:references/templates/project-files.md
# Project File Templates

Copy these into your project root. Replace `YOURSERVICE` / `YourService` / `yourService`.

## Table of Contents
1. [package.json](#1-packagejson)
2. [tsconfig.json](#2-tsconfigjson)
3. [gulpfile.js](#3-gulpfilejs)
4. [.eslintrc.js](#4-eslintrcjs)
5. [index.js (optional)](#5-indexjs)
6. [.prettierrc (optional)](#6-prettierrc)
7. [.gitignore](#7-gitignore)

---

## 1. package.json

> Critical fields explained:
> - `keywords` must include `"n8n-community-node-package"` — n8n won't find it otherwise
> - `n8n.nodes` and `n8n.credentials` must point to compiled `.js` files in `dist/`
> - `files: ["dist"]` means only the compiled output is published to npm (not source)
> - `peerDependencies` ensures the user's n8n provides `n8n-workflow`, not a duplicate

```json
{
  "name": "n8n-nodes-yourservice",
  "version": "0.1.0",
  "description": "n8n community node for YourService API",
  "keywords": [
    "n8n-community-node-package"
  ],
  "license": "MIT",
  "homepage": "https://github.com/YOURGITHUB/n8n-nodes-yourservice",
  "author": {
    "name": "Your Name",
    "email": "[email protected]"
  },
  "repository": {
    "type": "git",
    "url": "https://github.com/YOURGITHUB/n8n-nodes-yourservice.git"
  },
  "main": "index.js",
  "scripts": {
    "build": "tsc && gulp build:icons",
    "dev": "tsc --watch",
    "format": "prettier nodes credentials --write",
    "lint": "eslint nodes credentials --ext .ts",
    "lintfix": "eslint nodes credentials --ext .ts --fix",
    "prepublishOnly": "npm run build && npm run lint"
  },
  "n8n": {
    "n8nNodesApiVersion": 1,
    "credentials": [
      "dist/credentials/YourServiceApi.credentials.js"
    ],
    "nodes": [
      "dist/nodes/YourService/YourService.node.js"
    ]
  },
  "files": [
    "dist"
  ],
  "devDependencies": {
    "@types/node": "^18.16.16",
    "eslint-plugin-n8n-nodes-base": "^1.16.1",
    "gulp": "^4.0.2",
    "n8n-core": "*",
    "n8n-workflow": "*",
    "prettier": "^3.3.2",
    "typescript": "^5.3.3"
  },
  "peerDependencies": {
    "n8n-workflow": "*"
  }
}
```

### Adding more nodes or credentials

```json
"n8n": {
  "n8nNodesApiVersion": 1,
  "credentials": [
    "dist/credentials/YourServiceApi.credentials.js",
    "dist/credentials/YourServiceOAuth2Api.credentials.js"
  ],
  "nodes": [
    "dist/nodes/YourService/YourService.node.js",
    "dist/nodes/YourServiceTrigger/YourServiceTrigger.node.js"
  ]
}
```

### Adding npm dependencies (e.g. form-data, jsonwebtoken)

```json
"dependencies": {
  "form-data": "^4.0.0",
  "jsonwebtoken": "^9.0.2"
}
```

---

## 2. tsconfig.json

```json
{
  "compilerOptions": {
    "strict": true,
    "module": "commonjs",
    "target": "ES2019",
    "lib": ["ES2019"],
    "outDir": "./dist",
    "rootDir": ".",
    "typeRoots": ["./node_modules/@types"],
    "esModuleInterop": true,
    "declaration": true,
    "declarationMap": true,
    "sourceMap": true,
    "skipLibCheck": true,
    "resolveJsonModule": true
  },
  "include": ["nodes/**/*.ts", "credentials/**/*.ts"],
  "exclude": ["node_modules", "dist"]
}
```

---

## 3. gulpfile.js

Copies SVG and PNG icons from `nodes/` into `dist/nodes/` after TypeScript compiles. Required because `tsc` only copies `.ts` files.

```javascript
const { src, dest } = require('gulp');

function buildIcons() {
  return src('nodes/**/*.{png,svg}').pipe(dest('dist/nodes'));
}

exports['build:icons'] = buildIcons;
```

---

## 4. .eslintrc.js

Required by n8n community standards. Passes linting is checked before the node can be installed on n8n Cloud.

```javascript
module.exports = {
  root: true,
  env: { node: true },
  parser: '@typescript-eslint/parser',
  parserOptions: {
    project: ['./tsconfig.json'],
    sourceType: 'module',
    extraFileExtensions: ['.json'],
  },
  plugins: ['n8n-nodes-base'],
  extends: ['plugin:n8n-nodes-base/nodes'],
  rules: {
    // Relax specific rules if needed:
    // 'n8n-nodes-base/node-class-description-credentials-name-unsuffixed': 'off',
    // 'n8n-nodes-base/node-dirname-against-convention': 'off',
  },
};
```

Common lint rules you may need to suppress with an inline comment:
```typescript
// eslint-disable-next-line n8n-nodes-base/node-class-description-credentials-name-unsuffixed
name: 'myApiCredential',
```

---

## 5. index.js

Usually not needed — `package.json`'s `n8n` section handles registration. Add this only if n8n complains about a missing entry point:

```javascript
// index.js — explicit entry point (rarely needed)
module.exports = {};
```

---

## 6. .prettierrc

```json
{
  "semi": true,
  "trailingComma": "all",
  "singleQuote": true,
  "printWidth": 100,
  "tabWidth": 2
}
```

---

## 7. .gitignore

```
node_modules/
dist/
*.js.map
.env
```

> Note: Some projects commit `dist/` so users can install from GitHub without building. Either approach is fine — just be consistent.

FILE:references/templates/local-testing.md
# Local Testing

## Table of Contents
1. [npm link workflow](#1-npm-link-workflow)
2. [n8n custom folder locations](#2-n8n-custom-folder-locations)
3. [Using the official starter with hot reload](#3-hot-reload-dev-mode)
4. [Testing in Docker](#4-testing-in-docker)
5. [Debugging tips](#5-debugging-tips)

---

## 1. npm link Workflow

The standard way to test locally without publishing:

```bash
# 1. Build the node
npm run build

# 2. Register it globally on your machine
npm link

# 3. Navigate to n8n's custom node folder
#    (create the folder if it doesn't exist)
cd ~/.n8n/custom

# 4. Link your package into n8n's custom folder
npm link n8n-nodes-yourservice

# 5. Start n8n
n8n start
# → Open http://localhost:5678
# → Your node should appear in the node picker
```

After making changes:
```bash
# In your package directory:
npm run build   # recompile
# n8n should pick up changes after restart (or save + refresh in the editor)
```

To unlink:
```bash
cd ~/.n8n/custom
npm unlink n8n-nodes-yourservice
```

---

## 2. n8n Custom Folder Locations

| OS | Path |
|---|---|
| macOS / Linux | `~/.n8n/custom/` |
| Windows | `C:\Users\<YourUser>\.n8n\custom\` |
| Docker | Mount a volume — see section 4 |
| n8n Cloud | Install via Settings → Community Nodes (npm only) |

Create the folder if it doesn't exist:
```bash
mkdir -p ~/.n8n/custom
```

---

## 3. Hot-Reload Dev Mode

The official n8n starter template includes n8n as a dev dependency with hot reload:

```bash
# Clone the starter
git clone https://github.com/n8n-io/n8n-nodes-starter.git n8n-nodes-yourservice
cd n8n-nodes-yourservice
npm install

# Develop with hot reload (recompiles and reloads on file save)
npm run dev
# → Starts n8n at http://localhost:5678 with your node already loaded
```

This is the fastest development loop for new nodes.

---

## 4. Testing in Docker

Mount your built node directory as a volume:

```bash
# Build first
npm run build

# Run n8n with the node mounted
docker run -it --rm \
  -p 5678:5678 \
  -v ~/.n8n:/home/node/.n8n \
  -v $(pwd)/dist:/home/node/.n8n/custom/node_modules/n8n-nodes-yourservice/dist \
  n8nio/n8n
```

Or use docker-compose:
```yaml
version: '3.8'
services:
  n8n:
    image: n8nio/n8n
    ports:
      - '5678:5678'
    volumes:
      - ~/.n8n:/home/node/.n8n
      - ./:/home/node/.n8n/custom/node_modules/n8n-nodes-yourservice
    environment:
      - N8N_CUSTOM_EXTENSIONS=/home/node/.n8n/custom
```

---

## 5. Debugging Tips

### Node doesn't appear in picker
- Check `npm run build` completed without TypeScript errors
- Verify `n8n.nodes` in `package.json` points to the correct `dist/` path
- Confirm the `dist/` folder exists and has your compiled files
- Restart n8n completely (hot reload doesn't always catch structural changes)
- Check n8n logs for "Error loading node" messages

### Credentials don't load
- Check `n8n.credentials` path in `package.json`
- Ensure credential `name` property exactly matches `getCredentials('...')` string (case-sensitive)
- The credential class name in `package.json` path must match the actual file path

### "Cannot find module" error
- Run `npm install` to install dependencies
- For `npm link` setups, check the symlink is intact: `ls ~/.n8n/custom/node_modules/`

### Changes not reflected after build
- Hard-refresh n8n in browser (Ctrl+Shift+R)
- Restart the n8n process completely
- Check you saved and the TypeScript compiled (watch for tsc errors)

### Console logging for debugging
```typescript
// Temporary — remove before publishing
console.log('DEBUG credentials:', JSON.stringify(credentials, null, 2));
console.log('DEBUG response:', JSON.stringify(response, null, 2));
console.log('DEBUG item:', JSON.stringify(items[i], null, 2));
```

### Test a specific credential test
In n8n UI → Credentials → open your credential → click "Test" button.  
The result shows any error from your `test` block or credential test method.

FILE:references/templates/publishing.md
# Publishing to npm

## Table of Contents
1. [Manual publish](#1-manual-publish)
2. [GitHub Actions workflow (recommended)](#2-github-actions-workflow)
3. [npm provenance (required from May 2026)](#3-npm-provenance)
4. [Versioning strategy](#4-versioning-strategy)
5. [Submitting to n8n community](#5-submitting-to-n8n-community)
6. [Updating an existing package](#6-updating-an-existing-package)

---

## 1. Manual Publish

```bash
# One-time: log in to npm
npm login

# Bump version in package.json (pick one)
npm version patch   # 0.1.0 → 0.1.1  (bug fix)
npm version minor   # 0.1.0 → 0.2.0  (new feature, backwards compatible)
npm version major   # 0.1.0 → 1.0.0  (breaking change)

# Publish (prepublishOnly runs build + lint automatically)
npm publish --access public
```

The `--access public` flag is required for scoped packages (`@yourorg/n8n-nodes-...`). For unscoped packages it's optional but harmless.

---

## 2. GitHub Actions Workflow

Save as `.github/workflows/publish.yml` — triggers automatically when you push a version tag.

```yaml
name: Publish to npm

on:
  push:
    tags:
      - 'v*'     # triggers on v0.1.0, v1.0.0, etc.

jobs:
  publish:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      id-token: write    # required for provenance attestation

    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'
          registry-url: 'https://registry.npmjs.org'

      - name: Install dependencies
        run: npm ci

      - name: Build
        run: npm run build

      - name: Lint
        run: npm run lint

      - name: Publish with provenance
        run: npm publish --access public --provenance
        env:
          NODE_AUTH_TOKEN: { secrets.NPM_TOKEN}
```

**Publish flow:**
```bash
# Update version
npm version patch

# Push tag (triggers the workflow)
git push && git push --tags
```

---

## 3. npm Provenance

**Required for n8n community nodes from May 1, 2026.**

Provenance is a signed attestation from npm that the package was built by a specific GitHub Actions run — it proves the code in the package matches the code in the repo.

### Option A: Trusted Publishers (no token needed)

1. Go to [npmjs.com](https://www.npmjs.com) → your package → Settings → Publish access → Trusted Publishers
2. Click "Add a publisher"
3. Fill in:
   - **Repository owner**: your GitHub org or username
   - **Repository name**: your repo name
   - **Workflow filename**: `publish.yml`
4. Save — GitHub Actions can now publish without an `NPM_TOKEN` stored in secrets

Remove `NODE_AUTH_TOKEN` from the workflow when using Trusted Publishers.

### Option B: NPM Token (traditional)

1. Go to npmjs.com → Access Tokens → Generate New Token → Granular Access Token
2. Give it publish access to your package
3. Copy the token
4. In GitHub: repo Settings → Secrets → Actions → New repository secret
5. Name: `NPM_TOKEN`, value: the token

Keep `NODE_AUTH_TOKEN: { secrets.NPM_TOKEN}` in the workflow.

---

## 4. Versioning Strategy

| Change type | Version bump | Notes |
|---|---|---|
| Bug fix, no interface change | `patch` (0.1.0 → 0.1.1) | |
| New operation, new optional field | `minor` (0.1.0 → 0.2.0) | |
| Renamed field, changed output structure | `major` (0.1.0 → 1.0.0) | Use node versioning too |
| First stable release | `1.0.0` | |

---

## 5. Submitting to n8n Community

After publishing to npm:

1. **Tag your package correctly** — `keywords: ["n8n-community-node-package"]` must be in `package.json`
2. **Submit to n8n Creator Hub**: https://n8n.io/creators — fill out the submission form
3. **Users install via**: n8n Settings → Community Nodes → Install → enter your npm package name

Your node will be installable immediately after publishing. The Creator Hub submission makes it discoverable in n8n's official directory.

---

## 6. Updating an Existing Package

```bash
# 1. Make your code changes
# 2. Build and test locally
npm run build

# 3. Bump version
npm version patch   # or minor / major

# 4. Publish
npm publish --access public
# OR push a tag to trigger GitHub Actions:
git push && git push --tags
```

Users who installed via n8n's community nodes UI will see an "Update available" badge in Settings → Community Nodes and can update with one click.

> ⚠️ If you made breaking changes (changed output structure, renamed fields), bump the major version AND add a node version increment — existing workflows using the old node will continue working on the old version.

FILE:references/examples/nodes/declarative-node.md
# Example: Declarative (Low-Code) Node

Define what the HTTP request looks like — n8n executes it automatically. No `execute()` needed.

**Use when:** single resource, simple CRUD, straightforward REST API, you want to ship fast.  
**Switch to programmatic when:** you need loops, conditional logic between calls, pagination, or data transformation.

## Table of Contents
1. [Full declarative node](#1-full-declarative-node)
2. [requestDefaults and credential injection](#2-requestdefaults-and-credential-injection)
3. [routing — request, output, postReceive](#3-routing)
4. [Expressions in routing](#4-expressions-in-routing)

---

## 1. Full Declarative Node

```typescript
import { INodeType, INodeTypeDescription } from 'n8n-workflow';

export class SimpleService implements INodeType {
  description: INodeTypeDescription = {
    displayName: 'Simple Service',
    name: 'simpleService',
    icon: 'file:simpleservice.svg',
    group: ['transform'],
    version: 1,
    subtitle: '={{$parameter["operation"]}}',
    description: 'Interact with Simple Service API',
    defaults: { name: 'Simple Service' },
    inputs: ['main'],
    outputs: ['main'],
    credentials: [{ name: 'simpleServiceApi', required: true }],

    // Base URL + headers applied to every operation.
    // Credentials with an `authenticate` block are auto-injected here.
    requestDefaults: {
      baseURL: 'https://api.simpleservice.com',
      headers: {
        'Content-Type': 'application/json',
        Accept: 'application/json',
      },
    },

    properties: [
      {
        displayName: 'Operation',
        name: 'operation',
        type: 'options',
        noDataExpression: true,
        options: [

          // ── GET single item ────────────────────────────────────────
          {
            name: 'Get Item',
            value: 'getItem',
            action: 'Get an item',
            routing: {
              request: { method: 'GET', url: '=/items/{{$parameter.itemId}}' },
            },
          },

          // ── GET list ───────────────────────────────────────────────
          {
            name: 'Get Many Items',
            value: 'getManyItems',
            action: 'Get many items',
            routing: {
              request: { method: 'GET', url: '/items' },
              output: {
                // Unwrap { data: [...] } → return just the array
                postReceive: [
                  { type: 'rootProperty', properties: { property: 'data' } },
                ],
              },
            },
          },

          // ── POST create ────────────────────────────────────────────
          {
            name: 'Create Item',
            value: 'createItem',
            action: 'Create an item',
            routing: {
              request: { method: 'POST', url: '/items' },
              // Body is assembled from individual field routing blocks below
            },
          },

          // ── DELETE ────────────────────────────────────────────────
          {
            name: 'Delete Item',
            value: 'deleteItem',
            action: 'Delete an item',
            routing: {
              request: { method: 'DELETE', url: '=/items/{{$parameter.itemId}}' },
              output: {
                // Replace API response with a clean success object
                postReceive: [
                  { type: 'set', properties: { value: '={{ { "success": true } }}' } },
                ],
              },
            },
          },
        ],
        default: 'getItem',
      },

      // ── Item ID (used in URL for get/delete) ───────────────────────
      {
        displayName: 'Item ID',
        name: 'itemId',
        type: 'string',
        required: true,
        default: '',
        displayOptions: { show: { operation: ['getItem', 'deleteItem'] } },
      },

      // ── Create fields — each adds to the request body ──────────────
      {
        displayName: 'Name',
        name: 'name',
        type: 'string',
        required: true,
        default: '',
        displayOptions: { show: { operation: ['createItem'] } },
        routing: { request: { body: { name: '={{$value}}' } } },
      },
      {
        displayName: 'Description',
        name: 'description',
        type: 'string',
        default: '',
        displayOptions: { show: { operation: ['createItem'] } },
        routing: { request: { body: { description: '={{$value}}' } } },
      },

      // ── Query params for list ──────────────────────────────────────
      {
        displayName: 'Limit',
        name: 'limit',
        type: 'number',
        default: 50,
        typeOptions: { minValue: 1 },
        displayOptions: { show: { operation: ['getManyItems'] } },
        routing: { request: { qs: { limit: '={{$value}}' } } },
      },
      {
        displayName: 'Status Filter',
        name: 'status',
        type: 'options',
        options: [
          { name: 'All',      value: '' },
          { name: 'Active',   value: 'active' },
          { name: 'Archived', value: 'archived' },
        ],
        default: '',
        displayOptions: { show: { operation: ['getManyItems'] } },
        routing: {
          request: {
            // Only send if not empty — omit param when 'All' selected
            qs: { status: '={{$value !== "" ? $value : undefined}}' },
          },
        },
      },
    ],
  };
}
```

---

## 2. requestDefaults and Credential Injection

`requestDefaults` sets headers/baseURL for all operations. If your credential has an `authenticate` block (e.g., `Authorization: Bearer {{$credentials.apiToken}}`), n8n auto-injects it into every request made via `requestDefaults`.

**This only works for declarative nodes.** In programmatic nodes you must inject credentials manually in `execute()`.

```typescript
requestDefaults: {
  baseURL: '={{$credentials.baseUrl}}',   // baseUrl can come from credentials too
  headers: {
    Accept: 'application/json',
    // Authorization is injected automatically from credential's authenticate block
  },
},
```

---

## 3. routing

Each operation (or individual field) can have a `routing` block:

```typescript
routing: {
  request: {
    method: 'POST',
    url: '/items',
    body: { key: '={{$value}}' },     // $value = this field's current value
    qs:   { page: '={{$value}}' },
    headers: { 'X-Extra': 'value' },
  },
  output: {
    postReceive: [/* transforms */],
  },
}
```

### postReceive transforms

| Type | What it does |
|---|---|
| `rootProperty` | Unwraps `{ data: [...] }` — extracts `data` array as output |
| `set` | Replaces entire output with a fixed value (good for DELETE) |
| `filter` | Filters output array by a condition |
| `limit` | Limits output to N items |

---

## 4. Expressions in routing

Use `={{...}}` to reference values dynamically:

```typescript
// Reference another parameter
url: '=/users/{{$parameter.userId}}/posts/{{$parameter.postId}}'

// Reference this field's value
body: { name: '={{$value}}' }

// Conditional — omit param if empty
qs: { filter: '={{$value !== "" ? $value : undefined}}' }

// Reference credentials
baseURL: '={{$credentials.instanceUrl}}'
```

FILE:references/examples/nodes/webhook-node.md
# Example: Webhook Node

A webhook node listens for incoming HTTP requests and fires a workflow when one arrives. Unlike trigger nodes (which poll), webhook nodes are passive — the external service pushes data to n8n.

**Key difference from trigger nodes:** implements `webhook()` instead of `poll()`. n8n registers a unique URL and keeps it alive while the workflow is active.

## Table of Contents
1. [Full webhook node](#1-full-webhook-node)
2. [HMAC signature verification](#2-hmac-signature-verification)
3. [Responding to the webhook caller](#3-responding-to-the-webhook-caller)
4. [Multiple webhook events / paths](#4-multiple-webhook-events--paths)
5. [Lifecycle hooks — activate/deactivate](#5-lifecycle-hooks)

---

## 1. Full Webhook Node

```typescript
import {
  IHookFunctions,
  IWebhookFunctions,
  INodeType,
  INodeTypeDescription,
  IWebhookResponseData,
  NodeOperationError,
} from 'n8n-workflow';

export class MyServiceWebhook implements INodeType {
  description: INodeTypeDescription = {
    displayName: 'My Service Webhook',
    name: 'myServiceWebhook',
    icon: 'file:myservice.svg',
    group: ['trigger'],
    version: 1,
    description: 'Starts workflow when My Service sends an event',
    defaults: { name: 'My Service Webhook' },
    inputs: [],
    outputs: ['main'],
    credentials: [{ name: 'myServiceApi', required: true }],

    webhooks: [
      {
        name: 'default',
        httpMethod: 'POST',    // method the external service will call
        responseMode: 'onReceived',  // respond immediately on receipt
        path: 'webhook',       // becomes part of the URL: .../webhook/UUID/webhook
      },
    ],

    properties: [
      {
        displayName: 'Events',
        name: 'events',
        type: 'multiOptions',
        options: [
          { name: 'Item Created', value: 'item.created' },
          { name: 'Item Updated', value: 'item.updated' },
          { name: 'Item Deleted', value: 'item.deleted' },
        ],
        default: ['item.created'],
        required: true,
      },
      {
        displayName: 'Verify Signature',
        name: 'verifySignature',
        type: 'boolean',
        default: true,
        description: 'Whether to verify the HMAC signature from My Service',
      },
    ],
  };

  // ── Lifecycle hooks (called when workflow is activated/deactivated) ─
  webhookMethods = {
    default: {
      // Called when workflow activates — register webhook with external service
      async checkExists(this: IHookFunctions): Promise<boolean> {
        const webhookData = this.getWorkflowStaticData('node');
        const webhookUrl  = this.getNodeWebhookUrl('default') as string;
        const credentials = await this.getCredentials('myServiceApi');

        if (!webhookData.webhookId) return false;

        // Check if our webhook still exists in the external service
        try {
          await this.helpers.httpRequest({
            method: 'GET',
            url: `https://api.myservice.com/webhooks/webhookData.webhookId`,
            headers: { Authorization: `Bearer credentials.apiToken` },
          });
          return true;
        } catch {
          return false;
        }
      },

      async create(this: IHookFunctions): Promise<boolean> {
        const webhookUrl  = this.getNodeWebhookUrl('default') as string;
        const credentials = await this.getCredentials('myServiceApi');
        const events      = this.getNodeParameter('events') as string[];

        const response = await this.helpers.httpRequest({
          method: 'POST',
          url: 'https://api.myservice.com/webhooks',
          headers: {
            Authorization: `Bearer credentials.apiToken`,
            'Content-Type': 'application/json',
          },
          body: { url: webhookUrl, events },
        }) as { id: string; secret: string };

        // Save webhook ID and secret for verification and deletion later
        const webhookData = this.getWorkflowStaticData('node');
        webhookData.webhookId  = response.id;
        webhookData.webhookSecret = response.secret;

        return true;
      },

      async delete(this: IHookFunctions): Promise<boolean> {
        const webhookData = this.getWorkflowStaticData('node');
        const credentials = await this.getCredentials('myServiceApi');

        if (!webhookData.webhookId) return true;

        try {
          await this.helpers.httpRequest({
            method: 'DELETE',
            url: `https://api.myservice.com/webhooks/webhookData.webhookId`,
            headers: { Authorization: `Bearer credentials.apiToken` },
          });
        } catch { /* ignore — webhook may already be gone */ }

        delete webhookData.webhookId;
        delete webhookData.webhookSecret;
        return true;
      },
    },
  };

  // ── Called on every incoming HTTP request ──────────────────────────
  async webhook(this: IWebhookFunctions): Promise<IWebhookResponseData> {
    const req           = this.getRequestObject();
    const body          = this.getBodyData() as IDataObject;
    const verifySignature = this.getNodeParameter('verifySignature') as boolean;

    // ── Signature verification ─────────────────────────────────────
    if (verifySignature) {
      const webhookData  = this.getWorkflowStaticData('node');
      const secret       = webhookData.webhookSecret as string;
      const signature    = req.headers['x-my-service-signature'] as string;

      if (!signature) {
        return { webhookResponse: { code: 401, data: 'Missing signature' } };
      }

      const crypto = require('crypto');
      const rawBody = (req as any).rawBody as Buffer;
      const expected = crypto
        .createHmac('sha256', secret)
        .update(rawBody)
        .digest('hex');

      if (`sha256=expected` !== signature) {
        return { webhookResponse: { code: 401, data: 'Invalid signature' } };
      }
    }

    // ── Return data to workflow ────────────────────────────────────
    return {
      workflowData: [[{ json: body }]],
      // Respond 200 OK to the caller immediately
      webhookResponse: { code: 200, data: JSON.stringify({ received: true }) },
    };
  }
}
```

---

## 2. HMAC Signature Verification

Most webhook providers sign payloads so you can verify authenticity.

```typescript
import crypto from 'crypto';

function verifyHmac(
  secret: string,
  rawBody: Buffer,
  signatureHeader: string,
  algorithm: 'sha256' | 'sha1' = 'sha256',
): boolean {
  const expected = crypto
    .createHmac(algorithm, secret)
    .update(rawBody)
    .digest('hex');

  const received = signatureHeader.replace(`algorithm=`, '');

  // Use timingSafeEqual to prevent timing attacks
  return crypto.timingSafeEqual(
    Buffer.from(expected, 'hex'),
    Buffer.from(received, 'hex'),
  );
}

// Usage in webhook():
const rawBody = (this.getRequestObject() as any).rawBody as Buffer;
if (!verifyHmac(secret, rawBody, signature)) {
  return { webhookResponse: { code: 401, data: 'Signature mismatch' } };
}
```

> ⚠️ Always use `rawBody` (unparsed bytes) for HMAC — using the parsed body object may produce different bytes.

---

## 3. Responding to the Webhook Caller

Control what the external service receives as the HTTP response:

```typescript
// Immediate 200 (most common)
return {
  workflowData: [[{ json: body }]],
  webhookResponse: {
    code: 200,
    data: JSON.stringify({ status: 'received' }),
  },
};

// Custom headers in response
return {
  workflowData: [[{ json: body }]],
  webhookResponse: {
    code: 200,
    headers: { 'X-Processed-By': 'n8n' },
    data: 'OK',
  },
};

// Reject with an error (no workflow execution)
return {
  webhookResponse: { code: 400, data: 'Bad request' },
};
```

---

## 4. Multiple Webhook Events / Paths

```typescript
webhooks: [
  {
    name: 'default',
    httpMethod: 'POST',
    responseMode: 'onReceived',
    path: 'events',       // → /webhook/UUID/events
  },
  {
    name: 'ping',
    httpMethod: 'GET',
    responseMode: 'onReceived',
    path: 'ping',         // → /webhook/UUID/ping  (for health check)
  },
],

// In webhook(), check which was called:
async webhook(this: IWebhookFunctions): Promise<IWebhookResponseData> {
  const webhookName = this.getWebhookName();  // 'default' | 'ping'

  if (webhookName === 'ping') {
    return { webhookResponse: { code: 200, data: 'pong' } };
  }

  // handle 'default' ...
}
```

---

## 5. Lifecycle Hooks

`webhookMethods` hooks fire when the workflow is activated/deactivated — use them to register/unregister with the external service:

| Method | When called | Use for |
|---|---|---|
| `checkExists` | On activation | Check if a previous registration still exists |
| `create` | On activation (if checkExists returns false) | Register webhook URL with the API |
| `delete` | On deactivation | Unregister from the API |

Always persist the webhook ID in `getWorkflowStaticData('node')` so `delete` can find it later.

> ⚠️ If your external API doesn't support programmatic webhook registration, skip `webhookMethods` entirely and show the webhook URL as a read-only field the user copies manually.

FILE:references/examples/nodes/trigger-node.md
# Example: Trigger Node (Polling)

A trigger node starts a workflow automatically on a schedule. It polls an API, checks for new data, and fires the workflow when something new appears.

**Key difference from regular nodes:** implements `poll()` instead of `execute()`. Uses `group: ['trigger']`. Has no input connection — it's always the first node.

## Table of Contents
1. [Full polling trigger](#1-full-polling-trigger)
2. [Deduplication — avoiding duplicate fires](#2-deduplication)
3. [Storing state between polls](#3-storing-state-between-polls)
4. [Manual execution for testing](#4-manual-execution-for-testing)

---

## 1. Full Polling Trigger

```typescript
import {
  IPollFunctions,
  INodeExecutionData,
  INodeType,
  INodeTypeDescription,
  NodeOperationError,
} from 'n8n-workflow';

export class MyServiceTrigger implements INodeType {
  description: INodeTypeDescription = {
    displayName: 'My Service Trigger',
    name: 'myServiceTrigger',
    icon: 'file:myservice.svg',
    group: ['trigger'],              // ← must be 'trigger'
    version: 1,
    description: 'Starts workflow when new items appear in My Service',
    defaults: { name: 'My Service Trigger' },
    inputs: [],                      // ← triggers have no inputs
    outputs: ['main'],
    credentials: [{ name: 'myServiceApi', required: true }],

    // How often to poll — user can override in workflow settings
    polling: true,

    properties: [
      {
        displayName: 'Event',
        name: 'event',
        type: 'options',
        options: [
          { name: 'New Item Created', value: 'itemCreated' },
          { name: 'Item Updated',     value: 'itemUpdated' },
        ],
        default: 'itemCreated',
        required: true,
      },
      {
        displayName: 'Resource',
        name: 'resource',
        type: 'options',
        options: [
          { name: 'Task',    value: 'task' },
          { name: 'Comment', value: 'comment' },
        ],
        default: 'task',
      },
    ],
  };

  // poll() is called on each polling interval (set by user in n8n schedule)
  async poll(this: IPollFunctions): Promise<INodeExecutionData[][] | null> {
    const credentials = await this.getCredentials('myServiceApi');
    const event    = this.getNodeParameter('event') as string;
    const resource = this.getNodeParameter('resource') as string;

    // Get the last time we ran (stored by n8n between polls)
    const webhookData = this.getWorkflowStaticData('node');
    const lastChecked = webhookData.lastChecked as string | undefined;

    // First run — set baseline, don't fire
    if (!lastChecked) {
      webhookData.lastChecked = new Date().toISOString();
      return null;  // returning null = no new data, don't trigger workflow
    }

    try {
      const response = await this.helpers.httpRequest({
        method: 'GET',
        url: `https://api.myservice.com/resources`,
        headers: { Authorization: `Bearer credentials.apiToken` },
        qs: {
          event,
          created_after: lastChecked,
          limit: 100,
        },
      }) as { items: IDataObject[] };

      // Update timestamp for next poll
      webhookData.lastChecked = new Date().toISOString();

      if (!response.items || response.items.length === 0) {
        return null;  // nothing new
      }

      // Return each new item as a separate workflow execution
      return [
        response.items.map((item) => ({
          json: item,
        })),
      ];

    } catch (error) {
      throw new NodeOperationError(this.getNode(), error.message);
    }
  }
}
```

---

## 2. Deduplication

If the API doesn't support `created_after`, track seen IDs manually:

```typescript
async poll(this: IPollFunctions): Promise<INodeExecutionData[][] | null> {
  const webhookData = this.getWorkflowStaticData('node');

  // Initialize set of seen IDs
  if (!webhookData.seenIds) {
    webhookData.seenIds = [];
  }
  const seenIds = webhookData.seenIds as string[];

  const response = await this.helpers.httpRequest({
    method: 'GET',
    url: 'https://api.myservice.com/items',
    headers: authHeader,
    qs: { limit: 100, sort: 'created_desc' },
  }) as { items: Array<{ id: string; [key: string]: unknown }> };

  // Filter to items we haven't seen
  const newItems = response.items.filter((item) => !seenIds.includes(item.id));

  if (newItems.length === 0) return null;

  // Remember these IDs (keep list bounded to last 1000)
  webhookData.seenIds = [...seenIds, ...newItems.map((i) => i.id)].slice(-1000);

  return [newItems.map((item) => ({ json: item }))];
}
```

---

## 3. Storing State Between Polls

`this.getWorkflowStaticData('node')` returns a persistent object saved between poll runs:

```typescript
const state = this.getWorkflowStaticData('node');

// Read
const cursor = state.cursor as string | undefined;
const lastId = state.lastId as number | undefined;

// Write (automatically persisted after poll() returns)
state.cursor  = 'abc123';
state.lastId  = 99;
state.lastRun = new Date().toISOString();

// Reset (useful for testing)
delete state.cursor;
```

> ⚠️ `getWorkflowStaticData('global')` is shared across all nodes in the workflow. `getWorkflowStaticData('node')` is scoped to this node instance — use 'node' unless you specifically need cross-node sharing.

---

## 4. Manual Execution for Testing

When a user clicks "Test Workflow" on a trigger node, n8n calls `poll()` with `this.getMode() === 'manual'`. You can return mock/recent data in this case:

```typescript
async poll(this: IPollFunctions): Promise<INodeExecutionData[][] | null> {
  const isManual = this.getMode() === 'manual';

  // In manual mode, return last 5 items regardless of lastChecked
  const limit = isManual ? 5 : 100;
  const qs: IDataObject = { limit };

  if (!isManual) {
    const state = this.getWorkflowStaticData('node');
    if (state.lastChecked) {
      qs.created_after = state.lastChecked;
    }
  }

  const response = await this.helpers.httpRequest({
    method: 'GET',
    url: 'https://api.myservice.com/items',
    headers: authHeader,
    qs,
  }) as { items: IDataObject[] };

  if (!response.items?.length) return null;

  if (!isManual) {
    this.getWorkflowStaticData('node').lastChecked = new Date().toISOString();
  }

  return [response.items.map((item) => ({ json: item }))];
}
```

FILE:references/examples/nodes/programmatic-node.md
# Example: Programmatic Node (Full Pattern)

The programmatic style uses an `execute()` method with full TypeScript control.

**Use when:** multiple resources, multiple operations, conditional logic, pagination, multiple API calls per item, or anything that needs real code.

## Table of Contents
1. [Minimal working node](#1-minimal-working-node)
2. [Resources + Operations pattern](#2-resources--operations-pattern)
3. [Dynamic dropdowns via loadOptions](#3-dynamic-dropdowns-via-loadoptions)
4. [Pagination — cursor and page-based](#4-pagination)
5. [Full execute() with all operations](#5-full-execute)

---

## 1. Minimal Working Node

```typescript
import {
  IExecuteFunctions,
  INodeExecutionData,
  INodeType,
  INodeTypeDescription,
  NodeOperationError,
} from 'n8n-workflow';

export class MyNode implements INodeType {
  description: INodeTypeDescription = {
    displayName: 'My Node',
    name: 'myNode',
    icon: 'file:mynode.svg',
    group: ['transform'],
    version: 1,
    description: 'Does something useful',
    defaults: { name: 'My Node' },
    inputs: ['main'],
    outputs: ['main'],
    credentials: [{ name: 'myNodeApi', required: true }],
    properties: [
      {
        displayName: 'User ID',
        name: 'userId',
        type: 'string',
        default: '',
        required: true,
      },
    ],
  };

  async execute(this: IExecuteFunctions): Promise<INodeExecutionData[][]> {
    const items = this.getInputData();
    const returnData: INodeExecutionData[] = [];
    const credentials = await this.getCredentials('myNodeApi');

    for (let i = 0; i < items.length; i++) {
      try {
        const userId = this.getNodeParameter('userId', i) as string;

        const response = await this.helpers.httpRequest({
          method: 'GET',
          url: `https://api.example.com/users/userId`,
          headers: { Authorization: `Bearer credentials.apiToken` },
        });

        returnData.push({ json: response, pairedItem: { item: i } });

      } catch (error) {
        if (this.continueOnFail()) {
          returnData.push({ json: { error: error.message }, pairedItem: { item: i } });
          continue;
        }
        throw new NodeOperationError(this.getNode(), error, { itemIndex: i });
      }
    }

    return [returnData];
  }
}
```

---

## 2. Resources + Operations Pattern

The standard pattern for nodes that cover multiple entities (users, posts) with multiple actions each.

```typescript
properties: [
  // ── Step 1: Resource selector ─────────────────────────────────────
  {
    displayName: 'Resource',
    name: 'resource',
    type: 'options',
    noDataExpression: true,   // ← prevents expression mode on this field
    options: [
      { name: 'User', value: 'user' },
      { name: 'Post', value: 'post' },
    ],
    default: 'user',
  },

  // ── Step 2: Operations per resource ───────────────────────────────
  {
    displayName: 'Operation',
    name: 'operation',
    type: 'options',
    noDataExpression: true,
    displayOptions: { show: { resource: ['user'] } },
    options: [
      { name: 'Create', value: 'create', description: 'Create a user', action: 'Create a user' },
      { name: 'Delete', value: 'delete', description: 'Delete a user', action: 'Delete a user' },
      { name: 'Get',    value: 'get',    description: 'Get a user',    action: 'Get a user' },
      { name: 'Get Many', value: 'getAll', description: 'Get many users', action: 'Get many users' },
      { name: 'Update', value: 'update', description: 'Update a user', action: 'Update a user' },
    ],
    default: 'get',
  },

  // ── Step 3: Fields per operation ──────────────────────────────────
  {
    displayName: 'User ID',
    name: 'userId',
    type: 'string',
    required: true,
    default: '',
    displayOptions: {
      show: {
        resource: ['user'],
        operation: ['get', 'update', 'delete'],
      },
    },
  },
  {
    displayName: 'Name',
    name: 'name',
    type: 'string',
    default: '',
    required: true,
    displayOptions: { show: { resource: ['user'], operation: ['create'] } },
  },
  {
    displayName: 'Email',
    name: 'email',
    type: 'string',
    placeholder: '[email protected]',
    default: '',
    required: true,
    displayOptions: { show: { resource: ['user'], operation: ['create'] } },
  },

  // ── Return All / Limit pattern ─────────────────────────────────────
  {
    displayName: 'Return All',
    name: 'returnAll',
    type: 'boolean',
    default: false,
    description: 'Whether to return all results or only up to a given limit',
    displayOptions: { show: { resource: ['user'], operation: ['getAll'] } },
  },
  {
    displayName: 'Limit',
    name: 'limit',
    type: 'number',
    default: 50,
    typeOptions: { minValue: 1 },
    displayOptions: {
      show: { resource: ['user'], operation: ['getAll'], returnAll: [false] },
    },
  },

  // ── Additional Options (optional extras) ──────────────────────────
  {
    displayName: 'Additional Options',
    name: 'options',
    type: 'collection',
    placeholder: 'Add Option',
    default: {},
    displayOptions: { show: { resource: ['user'], operation: ['create', 'update'] } },
    options: [
      {
        displayName: 'Role',
        name: 'role',
        type: 'options',
        options: [
          { name: 'Admin',  value: 'admin' },
          { name: 'Member', value: 'member' },
        ],
        default: 'member',
      },
      { displayName: 'Active', name: 'active', type: 'boolean', default: true },
    ],
  },

  // ── subtitle trick: shows current operation in node header ─────────
  // Add to description object (not properties):
  // subtitle: '={{$parameter["operation"] + ": " + $parameter["resource"]}}'
],
```

---

## 3. Dynamic Dropdowns via loadOptions

Populates a dropdown from a live API call (e.g. list of projects, workspaces, tags).

```typescript
// In the node description properties:
{
  displayName: 'Project',
  name: 'projectId',
  type: 'options',
  typeOptions: {
    loadOptionsMethod: 'getProjects',  // ← matches method name below
  },
  default: '',
  required: true,
  description: 'Choose from projects in your account. Loaded live from API.',
}

// In the node class, add a methods block alongside description:
methods = {
  loadOptions: {
    async getProjects(this: ILoadOptionsFunctions): Promise<INodePropertyOptions[]> {
      const credentials = await this.getCredentials('myNodeApi');

      const response = await this.helpers.httpRequest({
        method: 'GET',
        url: 'https://api.example.com/projects',
        headers: { Authorization: `Bearer credentials.apiToken` },
      }) as Array<{ id: string; name: string }>;

      return response.map((project) => ({
        name: project.name,   // shown in dropdown
        value: project.id,    // stored value
      }));
    },

    // Multiple loadOptions methods are fine
    async getCategories(this: ILoadOptionsFunctions): Promise<INodePropertyOptions[]> {
      // ...same pattern
      return [];
    },
  },
};
```

Import needed: `ILoadOptionsFunctions, INodePropertyOptions` from `'n8n-workflow'`

---

## 4. Pagination

### Page-based (page=1, page=2, ...)

```typescript
const returnAll = this.getNodeParameter('returnAll', i) as boolean;
const limit     = returnAll ? Infinity : this.getNodeParameter('limit', i, 50) as number;

let page = 1;
const allItems: unknown[] = [];

while (true) {
  const result = await this.helpers.httpRequest({
    method: 'GET',
    url: 'https://api.example.com/items',
    headers: authHeader,
    qs: { page, per_page: 100 },
  }) as { data: unknown[]; has_more: boolean };

  allItems.push(...result.data);

  if (!result.has_more || allItems.length >= limit) break;
  page++;
}

const sliced = returnAll ? allItems : allItems.slice(0, limit);
for (const item of sliced) {
  returnData.push({ json: item as IDataObject, pairedItem: { item: i } });
}
```

### Cursor-based (next_cursor)

```typescript
let cursor: string | undefined;
const allItems: unknown[] = [];

do {
  const result = await this.helpers.httpRequest({
    method: 'GET',
    url: 'https://api.example.com/items',
    headers: authHeader,
    qs: { cursor, limit: 100 },
  }) as { data: unknown[]; next_cursor?: string };

  allItems.push(...result.data);
  cursor = result.next_cursor;
} while (cursor);
```

---

## 5. Full execute()

```typescript
async execute(this: IExecuteFunctions): Promise<INodeExecutionData[][]> {
  const items     = this.getInputData();
  const returnData: INodeExecutionData[] = [];
  const resource  = this.getNodeParameter('resource', 0) as string;
  const operation = this.getNodeParameter('operation', 0) as string;
  const credentials = await this.getCredentials('myNodeApi');

  const baseURL    = 'https://api.example.com';
  const authHeader = { Authorization: `Bearer credentials.apiToken` };
  const jsonHeader = { ...authHeader, 'Content-Type': 'application/json' };

  for (let i = 0; i < items.length; i++) {
    try {
      let responseData: unknown;

      if (resource === 'user') {

        if (operation === 'get') {
          const userId = this.getNodeParameter('userId', i) as string;
          responseData = await this.helpers.httpRequest({
            method: 'GET',
            url: `baseURL/users/userId`,
            headers: authHeader,
          });

        } else if (operation === 'create') {
          const name    = this.getNodeParameter('name', i) as string;
          const email   = this.getNodeParameter('email', i) as string;
          const options = this.getNodeParameter('options', i, {}) as IDataObject;

          responseData = await this.helpers.httpRequest({
            method: 'POST',
            url: `baseURL/users`,
            headers: jsonHeader,
            body: { name, email, ...options },
          });

        } else if (operation === 'update') {
          const userId  = this.getNodeParameter('userId', i) as string;
          const options = this.getNodeParameter('options', i, {}) as IDataObject;

          responseData = await this.helpers.httpRequest({
            method: 'PATCH',
            url: `baseURL/users/userId`,
            headers: jsonHeader,
            body: options,
          });

        } else if (operation === 'delete') {
          const userId = this.getNodeParameter('userId', i) as string;
          await this.helpers.httpRequest({
            method: 'DELETE',
            url: `baseURL/users/userId`,
            headers: authHeader,
          });
          responseData = { success: true, id: userId };

        } else if (operation === 'getAll') {
          // See pagination patterns above
          responseData = { message: 'use pagination pattern from section 4' };
        }

      } else if (resource === 'post') {
        // same pattern for each resource
      }

      returnData.push({
        json: responseData as IDataObject,
        pairedItem: { item: i },
      });

    } catch (error) {
      if (this.continueOnFail()) {
        returnData.push({
          json: { error: error.message, statusCode: error.statusCode ?? 'unknown' },
          pairedItem: { item: i },
        });
        continue;
      }
      // NodeApiError for HTTP errors (has statusCode), NodeOperationError for logic errors
      if (error.statusCode) {
        throw new NodeApiError(this.getNode(), error, { itemIndex: i });
      }
      throw new NodeOperationError(this.getNode(), error.message, { itemIndex: i });
    }
  }

  return [returnData];
}
```

FILE:references/examples/credentials/api-key-patterns.md
# Credential Patterns: API Key / Token Auth

All credential files implement `ICredentialType` and live in `credentials/`.

The credential `name` field **must exactly match** the string you pass to `getCredentials('...')` in your node.

## Table of Contents
1. [Bearer Token (Authorization header)](#1-bearer-token)
2. [Custom Header (X-API-Key etc.)](#2-custom-header)
3. [Query String Key](#3-query-string-key)
4. [Multi-field / manual inject](#4-multi-field--manual-inject)
5. [Credential test reference](#5-credential-test-reference)

---

## 1. Bearer Token

Standard `Authorization: Bearer <token>` — used by most modern APIs.

```typescript
import type {
  IAuthenticateGeneric,
  ICredentialTestRequest,
  ICredentialType,
  INodeProperties,
} from 'n8n-workflow';

export class MyServiceApi implements ICredentialType {
  name = 'myServiceApi';                         // matches getCredentials('myServiceApi')
  displayName = 'My Service API';
  documentationUrl = 'https://docs.myservice.com/auth';

  properties: INodeProperties[] = [
    {
      displayName: 'API Token',
      name: 'apiToken',
      type: 'string',
      typeOptions: { password: true },            // ← always mask secrets
      default: '',
      required: true,
      placeholder: 'sk-xxxxxxxxxxxxxxxx',
    },
  ];

  // Auto-injected into every request when credential has an authenticate block.
  // Works automatically for declarative nodes. For programmatic nodes, read manually.
  authenticate: IAuthenticateGeneric = {
    type: 'generic',
    properties: {
      headers: { Authorization: '=Bearer {{$credentials.apiToken}}' },
    },
  };

  // Validates credentials when user clicks "Test" in the credentials UI
  test: ICredentialTestRequest = {
    request: {
      baseURL: 'https://api.myservice.com',
      url: '/me',
    },
  };
}
```

In `execute()`:
```typescript
const creds = await this.getCredentials('myServiceApi');
headers: { Authorization: `Bearer creds.apiToken` }
```

---

## 2. Custom Header

APIs that use `X-API-Key`, `X-Auth-Token`, or similar.

```typescript
export class MyServiceApi implements ICredentialType {
  name = 'myServiceApi';
  displayName = 'My Service API';

  properties: INodeProperties[] = [
    {
      displayName: 'API Key',
      name: 'apiKey',
      type: 'string',
      typeOptions: { password: true },
      default: '',
      required: true,
    },
    {
      // Optional — lets users point at self-hosted instances
      displayName: 'Base URL',
      name: 'baseUrl',
      type: 'string',
      default: 'https://api.myservice.com',
      required: true,
    },
  ];

  authenticate: IAuthenticateGeneric = {
    type: 'generic',
    properties: {
      headers: { 'X-API-Key': '={{$credentials.apiKey}}' },
    },
  };

  test: ICredentialTestRequest = {
    request: {
      baseURL: '={{$credentials.baseUrl}}',
      url: '/health',
    },
  };
}
```

---

## 3. Query String Key

APIs that pass the key as a URL parameter (`?api_key=...`).

```typescript
export class MyServiceApi implements ICredentialType {
  name = 'myServiceApi';
  displayName = 'My Service API';

  properties: INodeProperties[] = [
    {
      displayName: 'API Key',
      name: 'apiKey',
      type: 'string',
      typeOptions: { password: true },
      default: '',
      required: true,
    },
  ];

  authenticate: IAuthenticateGeneric = {
    type: 'generic',
    properties: {
      qs: { api_key: '={{$credentials.apiKey}}' },  // appended to every request URL
    },
  };
}
```

---

## 4. Multi-Field / Manual Inject

When you need to combine multiple values (e.g. Client ID + Secret → Basic Auth header), or need an environment selector.

```typescript
export class MyServiceApi implements ICredentialType {
  name = 'myServiceApi';
  displayName = 'My Service API';

  properties: INodeProperties[] = [
    {
      displayName: 'Client ID',
      name: 'clientId',
      type: 'string',
      default: '',
      required: true,
    },
    {
      displayName: 'Client Secret',
      name: 'clientSecret',
      type: 'string',
      typeOptions: { password: true },
      default: '',
      required: true,
    },
    {
      displayName: 'Environment',
      name: 'environment',
      type: 'options',
      options: [
        { name: 'Production', value: 'https://api.myservice.com' },
        { name: 'Sandbox',    value: 'https://sandbox.myservice.com' },
      ],
      default: 'https://api.myservice.com',
    },
  ];
  // No authenticate block — read all values manually in execute()
}
```

In `execute()`:
```typescript
const creds   = await this.getCredentials('myServiceApi');
const baseURL  = creds.environment as string;
const encoded  = Buffer.from(`creds.clientId:creds.clientSecret`).toString('base64');
const headers  = { Authorization: `Basic encoded` };
```

---

## 5. Credential Test Reference

The `test` block powers the "Test" button in the credentials UI.

```typescript
// Simple: just hit an endpoint and expect 200
test: ICredentialTestRequest = {
  request: {
    baseURL: 'https://api.myservice.com',
    url: '/me',
  },
};

// With custom error detection (looks for error field in 200 response body)
test: ICredentialTestRequest = {
  request: {
    baseURL: 'https://api.myservice.com',
    url: '/validate',
  },
  rules: [
    {
      type: 'responseSuccessBody',
      properties: {
        key: 'error',
        value: 'unauthorized',
        message: 'Invalid API token. Check your credentials.',
      },
    },
  ],
};

// Dynamic baseURL from credentials
test: ICredentialTestRequest = {
  request: {
    baseURL: '={{$credentials.baseUrl}}',
    url: '/health',
  },
};
```

### Rule types
| Type | What it checks |
|---|---|
| `responseCode` | Response HTTP status code equals expected value |
| `responseSuccessBody` | A key in the response body matches a bad value (marks as failed) |

### Custom test via node method

For complex validation (e.g., connecting to a database), define a `credentialTest` method in the node class instead:

```typescript
// In the credentials file:
// (no test block — the method name links them)

// In the node file:
methods = {
  credentialTest: {
    async testMyServiceApi(
      this: ICredentialTestFunctions,
      credential: ICredentialsDecrypted,
    ): Promise<INodeCredentialTestResult> {
      const creds = credential.data as { apiToken: string };
      try {
        await this.helpers.request({
          method: 'GET',
          url: 'https://api.myservice.com/me',
          headers: { Authorization: `Bearer creds.apiToken` },
        });
        return { status: 'OK', message: 'Connection successful!' };
      } catch (error) {
        return { status: 'Error', message: `Connection failed: error.message` };
      }
    },
  },
};
```

FILE:references/examples/credentials/oauth2-patterns.md
# Credential Patterns: OAuth2

Extend the built-in `oAuth2Api` credential to inherit all OAuth2 fields and automatic token refresh. You only need to override the specific fields for your service.

## Table of Contents
1. [Authorization Code (user login)](#1-authorization-code-user-login)
2. [Client Credentials (machine-to-machine)](#2-client-credentials-machine-to-machine)
3. [PKCE flow](#3-pkce-flow)
4. [User-configurable scopes](#4-user-configurable-scopes)
5. [Reading OAuth2 tokens in execute()](#5-reading-oauth2-tokens-in-execute)
6. [OAuth2 with extra auth params](#6-oauth2-with-extra-auth-params)

---

## 1. Authorization Code (User Login)

User is redirected to the service's login page, grants permission, n8n handles token exchange and refresh automatically.

Use for: Google, GitHub, Slack, Dropbox, Microsoft, Salesforce, etc.

```typescript
import type { ICredentialType, INodeProperties } from 'n8n-workflow';

export class MyServiceOAuth2Api implements ICredentialType {
  name = 'myServiceOAuth2Api';
  extends = ['oAuth2Api'];           // ← inherit all standard OAuth2 fields + refresh logic
  displayName = 'My Service OAuth2';
  documentationUrl = 'https://docs.myservice.com/oauth2';

  properties: INodeProperties[] = [
    {
      displayName: 'Grant Type',
      name: 'grantType',
      type: 'hidden',
      default: 'authorizationCode',
    },
    {
      displayName: 'Authorization URL',
      name: 'authUrl',
      type: 'hidden',
      default: 'https://auth.myservice.com/oauth/authorize',
      required: true,
    },
    {
      displayName: 'Access Token URL',
      name: 'accessTokenUrl',
      type: 'hidden',
      default: 'https://auth.myservice.com/oauth/token',
      required: true,
    },
    {
      displayName: 'Scope',
      name: 'scope',
      type: 'hidden',
      default: 'read:user read:data',
    },
    {
      // 'header' = credentials sent as Authorization: Basic header during token exchange
      // 'body'   = credentials sent in request body (some APIs require this)
      displayName: 'Authentication',
      name: 'authentication',
      type: 'hidden',
      default: 'header',
    },
  ];
}
```

---

## 2. Client Credentials (Machine-to-Machine)

No user login. App authenticates with its own Client ID + Secret.

Use for: service accounts, server-to-server integrations, background processes.

```typescript
export class MyServiceClientCredApi implements ICredentialType {
  name = 'myServiceClientCredApi';
  extends = ['oAuth2Api'];
  displayName = 'My Service API (App Auth)';

  properties: INodeProperties[] = [
    {
      displayName: 'Grant Type',
      name: 'grantType',
      type: 'hidden',
      default: 'clientCredentials',
    },
    {
      displayName: 'Access Token URL',
      name: 'accessTokenUrl',
      type: 'hidden',
      default: 'https://auth.myservice.com/oauth/token',
      required: true,
    },
    {
      displayName: 'Scope',
      name: 'scope',
      type: 'hidden',
      default: 'api.read api.write',
    },
    {
      displayName: 'Authentication',
      name: 'authentication',
      type: 'hidden',
      default: 'body',        // many client_credentials APIs expect creds in body
    },
  ];
}
```

---

## 3. PKCE Flow

Like Authorization Code but with a code verifier/challenge to prevent interception. Used by mobile apps and SPAs; some APIs mandate it.

```typescript
export class MyServicePkceApi implements ICredentialType {
  name = 'myServicePkceApi';
  extends = ['oAuth2Api'];
  displayName = 'My Service OAuth2 (PKCE)';

  properties: INodeProperties[] = [
    {
      displayName: 'Grant Type',
      name: 'grantType',
      type: 'hidden',
      default: 'pkce',
    },
    {
      displayName: 'Authorization URL',
      name: 'authUrl',
      type: 'hidden',
      default: 'https://auth.myservice.com/oauth/authorize',
    },
    {
      displayName: 'Access Token URL',
      name: 'accessTokenUrl',
      type: 'hidden',
      default: 'https://auth.myservice.com/oauth/token',
    },
    {
      displayName: 'Scope',
      name: 'scope',
      type: 'hidden',
      default: 'openid profile email',
    },
    {
      displayName: 'Authentication',
      name: 'authentication',
      type: 'hidden',
      default: 'header',
    },
  ];
}
```

---

## 4. User-Configurable Scopes

When users need to select which permissions to grant (e.g., read-only vs. read-write).

```typescript
export class MyServiceOAuth2Api implements ICredentialType {
  name = 'myServiceOAuth2Api';
  extends = ['oAuth2Api'];
  displayName = 'My Service OAuth2';

  properties: INodeProperties[] = [
    { displayName: 'Grant Type',       name: 'grantType',      type: 'hidden', default: 'authorizationCode' },
    { displayName: 'Authorization URL', name: 'authUrl',       type: 'hidden', default: 'https://auth.myservice.com/oauth/authorize' },
    { displayName: 'Access Token URL', name: 'accessTokenUrl', type: 'hidden', default: 'https://auth.myservice.com/oauth/token' },
    { displayName: 'Authentication',   name: 'authentication', type: 'hidden', default: 'header' },
    {
      // NOT hidden — user can edit
      displayName: 'Scope',
      name: 'scope',
      type: 'string',
      default: 'read',
      description: 'Space-separated list of scopes, e.g. "read write admin"',
    },
  ];
}
```

---

## 5. Reading OAuth2 Tokens in execute()

n8n handles token refresh automatically before calling `execute()`. The current token is available on the credentials object:

```typescript
const credentials = await this.getCredentials('myServiceOAuth2Api');

// Access token (after automatic refresh if needed)
const tokenData = credentials.oauthTokenData as {
  access_token: string;
  token_type: string;
  expires_in?: number;
  refresh_token?: string;
};

const accessToken = tokenData.access_token;

// Use it
const response = await this.helpers.httpRequest({
  method: 'GET',
  url: 'https://api.myservice.com/me',
  headers: { Authorization: `Bearer accessToken` },
});
```

> ⚠️ Do not cache the access token across executions — always read from `getCredentials()` to ensure you get the refreshed token.

---

## 6. OAuth2 with Extra Auth Params

Some APIs need additional parameters in the authorization URL (e.g. `response_type=code&prompt=consent`):

```typescript
properties: INodeProperties[] = [
  { displayName: 'Grant Type',         name: 'grantType',           type: 'hidden', default: 'authorizationCode' },
  { displayName: 'Authorization URL',  name: 'authUrl',             type: 'hidden', default: 'https://auth.myservice.com/oauth/authorize' },
  { displayName: 'Access Token URL',   name: 'accessTokenUrl',      type: 'hidden', default: 'https://auth.myservice.com/oauth/token' },
  { displayName: 'Scope',              name: 'scope',               type: 'hidden', default: 'openid profile' },
  { displayName: 'Authentication',     name: 'authentication',      type: 'hidden', default: 'header' },
  {
    // Extra query params appended to the authorization URL
    displayName: 'Auth URI Query Parameters',
    name: 'authQueryParameters',
    type: 'hidden',
    default: 'access_type=offline&prompt=consent',
  },
],
```

FILE:references/examples/credentials/other-patterns.md
# Credential Patterns: Basic Auth, JWT, Multi-Auth

## Table of Contents
1. [Basic Auth (username + password)](#1-basic-auth)
2. [JWT — generate token in execute()](#2-jwt-token)
3. [Multi-auth — let user choose API key or OAuth2](#3-multi-auth-field)
4. [Credentials with environment selector](#4-environment-selector)

---

## 1. Basic Auth

Username + password sent as `Authorization: Basic <base64>`.

```typescript
import type {
  IAuthenticateGeneric,
  ICredentialTestRequest,
  ICredentialType,
  INodeProperties,
} from 'n8n-workflow';

export class MyServiceApi implements ICredentialType {
  name = 'myServiceApi';
  displayName = 'My Service API';

  properties: INodeProperties[] = [
    {
      displayName: 'Username',
      name: 'username',
      type: 'string',
      default: '',
      required: true,
    },
    {
      displayName: 'Password',
      name: 'password',
      type: 'string',
      typeOptions: { password: true },
      default: '',
      required: true,
    },
    {
      displayName: 'Base URL',
      name: 'baseUrl',
      type: 'string',
      default: 'https://api.myservice.com',
      required: true,
    },
  ];

  // n8n handles the base64 encoding automatically with the `auth` block
  authenticate: IAuthenticateGeneric = {
    type: 'generic',
    properties: {
      auth: {
        username: '={{$credentials.username}}',
        password: '={{$credentials.password}}',
      },
    },
  };

  test: ICredentialTestRequest = {
    request: {
      baseURL: '={{$credentials.baseUrl}}',
      url: '/health',
    },
  };
}
```

Alternatively, build the header manually in `execute()`:

```typescript
const creds   = await this.getCredentials('myServiceApi');
const encoded = Buffer.from(`creds.username:creds.password`).toString('base64');
headers: { Authorization: `Basic encoded` }
```

---

## 2. JWT Token

For APIs that require you to generate and sign a JWT yourself (not OAuth2).

The credential stores the key material; the node generates the JWT per-request.

```typescript
export class MyServiceApi implements ICredentialType {
  name = 'myServiceApi';
  displayName = 'My Service API (JWT)';

  properties: INodeProperties[] = [
    {
      displayName: 'Issuer ID',
      name: 'issuerId',
      type: 'string',
      default: '',
      required: true,
      description: 'Your API issuer ID (found in developer console)',
    },
    {
      displayName: 'Key ID',
      name: 'keyId',
      type: 'string',
      default: '',
      required: true,
    },
    {
      displayName: 'Private Key',
      name: 'privateKey',
      type: 'string',
      typeOptions: {
        password: true,
        rows: 4,
      },
      default: '',
      required: true,
      placeholder: '-----BEGIN EC PRIVATE KEY-----\n...',
    },
  ];
  // No authenticate block — token is generated in execute()
}
```

In `execute()` (requires `jsonwebtoken` as a dependency):

```typescript
import * as jwt from 'jsonwebtoken';

const creds = await this.getCredentials('myServiceApi');

const now = Math.floor(Date.now() / 1000);
const token = jwt.sign(
  {
    iss: creds.issuerId,
    iat: now,
    exp: now + 1200,          // 20 minute expiry
    aud: 'https://api.myservice.com',
  },
  creds.privateKey as string,
  {
    algorithm: 'ES256',
    keyid: creds.keyId as string,
  },
);

headers: { Authorization: `Bearer token` }
```

---

## 3. Multi-Auth Field

Let the user choose between auth methods (e.g. API key OR OAuth2) within one credential.

```typescript
export class MyServiceApi implements ICredentialType {
  name = 'myServiceApi';
  displayName = 'My Service API';

  properties: INodeProperties[] = [
    {
      displayName: 'Authentication Method',
      name: 'authMethod',
      type: 'options',
      options: [
        { name: 'API Key', value: 'apiKey' },
        { name: 'Username & Password', value: 'basicAuth' },
      ],
      default: 'apiKey',
    },
    {
      displayName: 'API Key',
      name: 'apiKey',
      type: 'string',
      typeOptions: { password: true },
      default: '',
      displayOptions: { show: { authMethod: ['apiKey'] } },
    },
    {
      displayName: 'Username',
      name: 'username',
      type: 'string',
      default: '',
      displayOptions: { show: { authMethod: ['basicAuth'] } },
    },
    {
      displayName: 'Password',
      name: 'password',
      type: 'string',
      typeOptions: { password: true },
      default: '',
      displayOptions: { show: { authMethod: ['basicAuth'] } },
    },
    {
      displayName: 'Base URL',
      name: 'baseUrl',
      type: 'string',
      default: 'https://api.myservice.com',
      required: true,
    },
  ];
  // No authenticate block — inject manually based on authMethod
}
```

In `execute()`:

```typescript
const creds      = await this.getCredentials('myServiceApi');
const authMethod = creds.authMethod as string;
const baseURL    = creds.baseUrl as string;

let authHeader: string;
if (authMethod === 'apiKey') {
  authHeader = `Bearer creds.apiKey`;
} else {
  const encoded = Buffer.from(`creds.username:creds.password`).toString('base64');
  authHeader = `Basic encoded`;
}

const response = await this.helpers.httpRequest({
  method: 'GET',
  url: `baseURL/endpoint`,
  headers: { Authorization: authHeader },
});
```

---

## 4. Environment Selector

Common pattern: production vs. sandbox URLs stored in the credential so the user doesn't set it per-node.

```typescript
export class MyServiceApi implements ICredentialType {
  name = 'myServiceApi';
  displayName = 'My Service API';

  properties: INodeProperties[] = [
    {
      displayName: 'API Token',
      name: 'apiToken',
      type: 'string',
      typeOptions: { password: true },
      default: '',
      required: true,
    },
    {
      displayName: 'Environment',
      name: 'environment',
      type: 'options',
      options: [
        { name: 'Production', value: 'https://api.myservice.com' },
        { name: 'Sandbox',    value: 'https://sandbox.api.myservice.com' },
        { name: 'Custom',     value: 'custom' },
      ],
      default: 'https://api.myservice.com',
    },
    {
      displayName: 'Custom URL',
      name: 'customUrl',
      type: 'string',
      default: '',
      placeholder: 'https://my-instance.myservice.com',
      displayOptions: { show: { environment: ['custom'] } },
    },
  ];
}
```

In `execute()`:

```typescript
const creds   = await this.getCredentials('myServiceApi');
const baseURL = creds.environment === 'custom'
  ? creds.customUrl as string
  : creds.environment as string;
```

FILE:references/concepts/node-versioning.md
# Node Versioning

When you need to change a node's behavior in a way that would break existing workflows (renaming a field, changing output structure, adding required params), use node versioning instead of changing the existing node in place.

## Table of Contents
1. [When to version vs. not](#1-when-to-version-vs-not)
2. [Simple versioned node](#2-simple-versioned-node)
3. [Multiple version classes (recommended for large changes)](#3-multiple-version-classes)
4. [Version routing pattern](#4-version-routing-pattern)
5. [Deprecating old versions](#5-deprecating-old-versions)

---

## 1. When to Version vs. Not

**Version (breaking change):**
- Renaming a node parameter that's used in expressions (`{{ $node.MyNode.json.oldField }}`)
- Removing a parameter
- Changing the output data structure
- Changing what a required field means

**Don't version (safe change):**
- Adding a new optional field with a default
- Adding a new operation or resource
- Fixing a bug without changing the interface
- Updating description text

---

## 2. Simple Versioned Node

For minor version bumps — same class, different version number:

```typescript
export class MyNode implements INodeType {
  description: INodeTypeDescription = {
    displayName: 'My Node',
    name: 'myNode',
    version: [1, 2],           // ← supports both v1 and v2
    defaultVersion: 2,          // ← new instances get v2
    // ...
    properties: [
      // New field only shown in v2
      {
        displayName: 'Output Format',
        name: 'outputFormat',
        type: 'options',
        options: [
          { name: 'Simple', value: 'simple' },
          { name: 'Full',   value: 'full' },
        ],
        default: 'simple',
        displayOptions: {
          show: { '@version': [2] },    // ← special: show only on v2 nodes
        },
      },
      // Field removed in v2 (hide it, don't delete — old workflows may still use it)
      {
        displayName: 'Legacy Option',
        name: 'legacyOption',
        type: 'boolean',
        default: false,
        displayOptions: {
          show: { '@version': [1] },   // only visible on v1 nodes
        },
      },
    ],
  };

  async execute(this: IExecuteFunctions): Promise<INodeExecutionData[][]> {
    const nodeVersion = this.getNode().typeVersion;  // 1 or 2

    const items = this.getInputData();
    const returnData: INodeExecutionData[] = [];

    for (let i = 0; i < items.length; i++) {
      // Version-specific behavior
      if (nodeVersion === 1) {
        // old behavior
      } else {
        // new behavior
      }
    }

    return [returnData];
  }
}
```

---

## 3. Multiple Version Classes (Recommended for Large Changes)

For significant structural changes, split into separate class files:

```
nodes/MyNode/
├── MyNode.node.ts          ← router, decides which version to instantiate
├── v1/
│   └── MyNodeV1.node.ts   ← original implementation
└── v2/
    └── MyNodeV2.node.ts   ← new implementation
```

**MyNode.node.ts** (router):
```typescript
import { NodeVersionedType } from 'n8n-workflow';
import { MyNodeV1 } from './v1/MyNodeV1.node';
import { MyNodeV2 } from './v2/MyNodeV2.node';

export class MyNode extends NodeVersionedType {
  constructor() {
    const baseDescription = {
      displayName: 'My Node',
      name: 'myNode',
      icon: 'file:mynode.svg',
      group: ['transform'],
      defaultVersion: 2,
      description: 'Interact with My Service',
    };

    const nodeVersions = {
      1: new MyNodeV1(),
      2: new MyNodeV2(),
    };

    super(nodeVersions, baseDescription);
  }
}
```

**v1/MyNodeV1.node.ts**:
```typescript
import { INodeType, INodeTypeDescription } from 'n8n-workflow';

export class MyNodeV1 implements INodeType {
  description: INodeTypeDescription = {
    displayName: 'My Node',
    name: 'myNode',
    version: 1,
    // ... v1 properties
  };

  async execute(this: IExecuteFunctions): Promise<INodeExecutionData[][]> {
    // v1 logic
  }
}
```

**v2/MyNodeV2.node.ts**:
```typescript
export class MyNodeV2 implements INodeType {
  description: INodeTypeDescription = {
    displayName: 'My Node',
    name: 'myNode',
    version: 2,
    // ... v2 properties (can be completely different)
  };

  async execute(this: IExecuteFunctions): Promise<INodeExecutionData[][]> {
    // v2 logic
  }
}
```

---

## 4. Version Routing Pattern

Check version inside `execute()` to share code between versions:

```typescript
async execute(this: IExecuteFunctions): Promise<INodeExecutionData[][]> {
  const version = this.getNode().typeVersion;
  const items   = this.getInputData();
  const returnData: INodeExecutionData[] = [];

  for (let i = 0; i < items.length; i++) {
    const response = await this.helpers.httpRequest({ ... });

    // v1 output: flat structure
    // v2 output: nested structure with metadata
    const outputData = version === 1
      ? response.data
      : { data: response.data, meta: { source: 'myservice', version: 2 } };

    returnData.push({ json: outputData, pairedItem: { item: i } });
  }

  return [returnData];
}
```

---

## 5. Deprecating Old Versions

Show a notice in v1 nodes pointing users to upgrade:

```typescript
properties: [
  // At the top of v1 properties list
  {
    displayName: 'This version of the node is deprecated. Please upgrade to v2 for new features and fixes.',
    name: 'deprecationNotice',
    type: 'notice',
    default: '',
    displayOptions: {
      show: { '@version': [1] },
    },
  },
  // ... rest of v1 properties
]
```

> ⚠️ Never delete an old version from the `version` array or from the versioned class map — existing workflows will break. Always keep all historical versions functional.

FILE:references/concepts/error-handling.md
# Error Handling

## Table of Contents
1. [NodeOperationError vs NodeApiError](#1-nodeoperationerror-vs-nodeapierror)
2. [continueOnFail pattern](#2-continuenonfail-pattern)
3. [Throwing with context](#3-throwing-with-context)
4. [Validating parameters before the loop](#4-validating-parameters)
5. [API error response handling](#5-api-error-response-handling)

---

## 1. NodeOperationError vs NodeApiError

| Error class | When to use | Has statusCode? |
|---|---|---|
| `NodeOperationError` | Logic errors, missing params, invalid input, config problems | No |
| `NodeApiError` | HTTP errors from external API calls (non-2xx responses) | Yes |

```typescript
import { NodeOperationError, NodeApiError } from 'n8n-workflow';

// Logic / config error — something wrong in the node setup or user input
throw new NodeOperationError(
  this.getNode(),
  'The "Template" field must contain at least one variable.',
  { itemIndex: i },
);

// API-level HTTP error — the external service returned 4xx/5xx
// Use when you have the raw error object from httpRequest catch
throw new NodeApiError(this.getNode(), error, { itemIndex: i });
```

---

## 2. continueOnFail Pattern

Always wrap the per-item logic in try/catch and respect `continueOnFail()`:

```typescript
for (let i = 0; i < items.length; i++) {
  try {
    const response = await this.helpers.httpRequest({ ... });
    returnData.push({ json: response, pairedItem: { item: i } });

  } catch (error) {
    if (this.continueOnFail()) {
      // Package the error as data and continue to the next item
      returnData.push({
        json: {
          error: error.message,
          statusCode: error.statusCode ?? undefined,
          // Include original input to help with debugging downstream
          input: items[i].json,
        },
        pairedItem: { item: i },
      });
      continue;
    }

    // Stop the workflow — throw with the item index for UI highlighting
    if (error.statusCode) {
      throw new NodeApiError(this.getNode(), error, { itemIndex: i });
    }
    throw new NodeOperationError(this.getNode(), error.message, { itemIndex: i });
  }
}
```

> `continueOnFail()` respects the user's "On Error" setting in the node's Settings tab. Always handle it — workflows that swallow errors silently are hard to debug.

---

## 3. Throwing with Context

The `itemIndex` option highlights the failing item in the n8n UI, making it much easier for users to debug:

```typescript
// Without context — user doesn't know which item failed
throw new NodeOperationError(this.getNode(), 'User ID is required');

// With context — UI highlights item 3 in red
throw new NodeOperationError(
  this.getNode(),
  'User ID is required',
  { itemIndex: i },
);

// With description (shown as detail in the error panel)
throw new NodeOperationError(
  this.getNode(),
  'Invalid date format',
  {
    itemIndex: i,
    description: 'Expected ISO 8601 format (e.g. 2024-01-15T10:30:00Z)',
  },
);
```

---

## 4. Validating Parameters

Validate before the loop to catch config errors early (cheaper than discovering them mid-loop):

```typescript
async execute(this: IExecuteFunctions): Promise<INodeExecutionData[][]> {
  // Validate required credentials
  const credentials = await this.getCredentials('myServiceApi');
  if (!credentials.apiToken) {
    throw new NodeOperationError(this.getNode(), 'API Token is missing from credentials');
  }

  // Validate node-level parameters (use index 0 for params that don't vary per item)
  const resource  = this.getNodeParameter('resource', 0) as string;
  const operation = this.getNodeParameter('operation', 0) as string;

  const items = this.getInputData();
  const returnData: INodeExecutionData[] = [];

  for (let i = 0; i < items.length; i++) {
    try {
      // Validate per-item parameters
      const url = this.getNodeParameter('url', i) as string;
      if (!url.startsWith('https://')) {
        throw new NodeOperationError(
          this.getNode(),
          'URL must start with https://',
          { itemIndex: i },
        );
      }

      // ... rest of execute logic
    } catch (error) {
      if (this.continueOnFail()) {
        returnData.push({ json: { error: error.message }, pairedItem: { item: i } });
        continue;
      }
      throw error;
    }
  }

  return [returnData];
}
```

---

## 5. API Error Response Handling

Extract useful information from HTTP error responses:

```typescript
} catch (error) {
  // n8n's httpRequest throws errors with these properties:
  // error.statusCode  → HTTP status (400, 401, 403, 404, 429, 500...)
  // error.message     → Error message string
  // error.response    → Raw response object (may have .body)

  if (this.continueOnFail()) {
    returnData.push({
      json: {
        error: error.message,
        statusCode: error.statusCode,
        // Try to extract the API's error message if available
        apiError: error.response?.body
          ? (typeof error.response.body === 'string'
              ? error.response.body
              : JSON.stringify(error.response.body))
          : undefined,
      },
      pairedItem: { item: i },
    });
    continue;
  }

  // Map common status codes to helpful messages
  if (error.statusCode === 401) {
    throw new NodeOperationError(
      this.getNode(),
      'Authentication failed — check your credentials',
      { itemIndex: i },
    );
  }
  if (error.statusCode === 429) {
    throw new NodeOperationError(
      this.getNode(),
      'Rate limit exceeded — reduce request frequency or add a wait between items',
      { itemIndex: i },
    );
  }

  throw new NodeApiError(this.getNode(), error, { itemIndex: i });
}
```

FILE:references/concepts/data-and-pairing.md
# Data Flow and pairedItem Tracking

## Why pairedItem Matters

n8n uses `pairedItem` to track which output item came from which input item. Without it:
- Expressions like `{{ $('Previous Node').item.json.id }}` silently return wrong data
- The "item linking" arrows in the UI don't draw
- Debugging becomes much harder

**Always include `pairedItem: { item: i }` on every `returnData.push()`.**

## Table of Contents
1. [Basic pairedItem usage](#1-basic-paireditem)
2. [One-to-many — multiple outputs per input item](#2-one-to-many)
3. [Many-to-one — aggregating items](#3-many-to-one)
4. [Accessing input item data in execute()](#4-accessing-input-data)
5. [Passing input data through to output](#5-passing-data-through)
6. [n8n data structure](#6-n8n-data-structure)

---

## 1. Basic pairedItem

```typescript
for (let i = 0; i < items.length; i++) {
  const response = await this.helpers.httpRequest({ ... });

  returnData.push({
    json: response,
    pairedItem: { item: i },   // ← always include, i = index in input items array
  });
}
```

---

## 2. One-to-Many (multiple outputs per input)

When one input item produces multiple output items (e.g., getting all posts for a user):

```typescript
for (let i = 0; i < items.length; i++) {
  const userId = items[i].json.id as string;

  const posts = await this.helpers.httpRequest({
    method: 'GET',
    url: `https://api.example.com/users/userId/posts`,
    headers: authHeader,
  }) as IDataObject[];

  for (const post of posts) {
    returnData.push({
      json: post,
      pairedItem: { item: i },   // all posts link back to the same input item
    });
  }
}
```

---

## 3. Many-to-One (aggregating)

When combining all input items into a single output (e.g., batch create):

```typescript
// Collect all input items
const allIds = items.map((item) => item.json.id as string);

// Make one API call
const response = await this.helpers.httpRequest({
  method: 'POST',
  url: 'https://api.example.com/batch',
  body: { ids: allIds },
  headers: authHeader,
});

// Single output — pair with all input items
returnData.push({
  json: response,
  pairedItem: items.map((_, index) => ({ item: index })),  // array of pairings
});
```

---

## 4. Accessing Input Data in execute()

```typescript
const items = this.getInputData();

for (let i = 0; i < items.length; i++) {
  // Access a field from the input JSON
  const name  = items[i].json.name as string;
  const email = items[i].json.email as string;
  const id    = items[i].json.id as number;

  // Access nested fields
  const city    = (items[i].json.address as IDataObject)?.city as string;
  const tagList = items[i].json.tags as string[];

  // Check if a field exists
  if (items[i].json.userId === undefined) {
    throw new NodeOperationError(
      this.getNode(),
      'Input item is missing required field "userId"',
      { itemIndex: i },
    );
  }

  // Access binary data from input
  if (items[i].binary?.attachment) {
    const buffer = await this.helpers.getBinaryDataBuffer(i, 'attachment');
    // use buffer...
  }
}
```

---

## 5. Passing Data Through

Common patterns for combining API response with original input data:

```typescript
// Merge response with original input fields
returnData.push({
  json: {
    ...items[i].json,        // original input fields
    ...response,             // API response fields (overwrites same-named keys)
    _nodeProcessed: true,    // add a marker field
  },
  pairedItem: { item: i },
});

// Keep only specific fields from response
const { id, status, updatedAt } = response as { id: string; status: string; updatedAt: string };
returnData.push({
  json: { id, status, updatedAt, originalInput: items[i].json },
  pairedItem: { item: i },
});

// Forward binary data from input to output unchanged
returnData.push({
  json: { ...response },
  binary: items[i].binary,   // pass through any binary attachments
  pairedItem: { item: i },
});
```

---

## 6. n8n Data Structure

Every item flowing through n8n has this shape:

```typescript
interface INodeExecutionData {
  json: IDataObject;              // the data (always required, can be {})
  binary?: {                      // optional file attachments
    [key: string]: IBinaryData;   // key is e.g. 'data', 'attachment', 'image'
  };
  pairedItem?: {
    item: number;                 // index in the input items array
  } | Array<{ item: number }>;   // or array for many-to-one
  error?: NodeOperationError;    // only set when continueOnFail pushes an error item
}
```

> `json` must always be a plain object `{}` — not an array, not a primitive. If your API returns an array, wrap it: `json: { items: responseArray }`, or push each element as a separate item.

```typescript
// API returns an array → push each element separately
const list = response as IDataObject[];
for (const item of list) {
  returnData.push({ json: item, pairedItem: { item: i } });
}

// OR wrap the array
returnData.push({
  json: { results: list, count: list.length },
  pairedItem: { item: i },
});
```

FILE:references/concepts/node-properties.md
# Node Properties: UI Fields Reference

`properties` defines everything the user sees in the node panel.

## Table of Contents
1. [All field types](#1-all-field-types)
2. [displayOptions — conditional visibility](#2-displayoptions)
3. [collection — optional grouped extras](#3-collection)
4. [fixedCollection — repeatable groups](#4-fixedcollection)
5. [noDataExpression, required, placeholder](#5-modifiers)
6. [subtitle — dynamic node header text](#6-subtitle)

---

## 1. All Field Types

```typescript
// ── String ────────────────────────────────────────────────────────
{ displayName: 'Name', name: 'name', type: 'string', default: '' }

// ── Password (masked) ────────────────────────────────────────────
{
  displayName: 'API Key', name: 'apiKey', type: 'string',
  typeOptions: { password: true }, default: ''
}

// ── Multi-line text ──────────────────────────────────────────────
{
  displayName: 'Body', name: 'body', type: 'string',
  typeOptions: { rows: 4 }, default: ''
}

// ── Number ───────────────────────────────────────────────────────
{
  displayName: 'Limit', name: 'limit', type: 'number',
  default: 50, typeOptions: { minValue: 1, maxValue: 1000 }
}

// ── Boolean toggle ───────────────────────────────────────────────
{ displayName: 'Active', name: 'active', type: 'boolean', default: false }

// ── Dropdown (single select) ─────────────────────────────────────
{
  displayName: 'Method', name: 'method', type: 'options',
  options: [
    { name: 'GET',  value: 'GET' },
    { name: 'POST', value: 'POST' },
  ],
  default: 'GET',
}

// ── Multi-select (returns array of values) ───────────────────────
{
  displayName: 'Events', name: 'events', type: 'multiOptions',
  options: [
    { name: 'Created', value: 'created' },
    { name: 'Updated', value: 'updated' },
    { name: 'Deleted', value: 'deleted' },
  ],
  default: ['created'],
}

// ── JSON editor ──────────────────────────────────────────────────
{ displayName: 'Body', name: 'body', type: 'json', default: '{}' }

// ── Color picker ─────────────────────────────────────────────────
{ displayName: 'Color', name: 'color', type: 'color', default: '#ff0000' }

// ── DateTime ─────────────────────────────────────────────────────
{ displayName: 'Start Date', name: 'startDate', type: 'dateTime', default: '' }

// ── Hidden (not shown in UI, used for internal values) ───────────
{ displayName: 'Internal', name: 'internal', type: 'hidden', default: 'fixedValue' }
```

Getting values in `execute()`:
```typescript
const name     = this.getNodeParameter('name', i) as string;
const limit    = this.getNodeParameter('limit', i, 50) as number;   // 50 = fallback
const active   = this.getNodeParameter('active', i) as boolean;
const method   = this.getNodeParameter('method', i) as string;
const events   = this.getNodeParameter('events', i) as string[];
const body     = this.getNodeParameter('body', i, '{}') as string;
const parsed   = JSON.parse(body);
```

---

## 2. displayOptions

Controls when a field is visible. The most important pattern in multi-operation nodes.

```typescript
// Show only when resource = 'user' AND operation = 'create'
{
  displayName: 'Email',
  name: 'email',
  type: 'string',
  default: '',
  displayOptions: {
    show: {
      resource: ['user'],
      operation: ['create', 'update'],   // visible for multiple operations
    },
  },
}

// Hide when returnAll = true
{
  displayName: 'Limit',
  name: 'limit',
  type: 'number',
  default: 50,
  displayOptions: {
    show: { returnAll: [false] },
  },
}

// Hide for specific values (inverse)
{
  displayName: 'Format',
  name: 'format',
  type: 'options',
  default: 'json',
  displayOptions: {
    hide: {
      operation: ['delete'],
    },
  },
}
```

> ⚠️ `displayOptions` only controls UI visibility — the field still exists in the schema. The value is just not shown to the user. Fields not shown use their `default` value.

---

## 3. collection

A collapsible group of optional fields. Good for "Additional Options" or "Filters".

```typescript
{
  displayName: 'Additional Options',
  name: 'options',
  type: 'collection',
  placeholder: 'Add Option',
  default: {},
  options: [
    {
      displayName: 'Timeout (ms)',
      name: 'timeout',
      type: 'number',
      default: 30000,
      description: 'Request timeout in milliseconds',
    },
    {
      displayName: 'Follow Redirects',
      name: 'followRedirects',
      type: 'boolean',
      default: true,
    },
    {
      displayName: 'Response Format',
      name: 'responseFormat',
      type: 'options',
      options: [
        { name: 'Automatic', value: 'auto' },
        { name: 'JSON',      value: 'json' },
        { name: 'Text',      value: 'text' },
      ],
      default: 'auto',
    },
  ],
}
```

In `execute()`:
```typescript
const options = this.getNodeParameter('options', i, {}) as {
  timeout?: number;
  followRedirects?: boolean;
  responseFormat?: string;
};
const timeout = options.timeout ?? 30000;
```

---

## 4. fixedCollection

Repeatable rows of grouped fields. The key pattern for things like custom headers, key-value pairs, or address entries.

```typescript
{
  displayName: 'Custom Headers',
  name: 'customHeaders',
  type: 'fixedCollection',
  placeholder: 'Add Header',
  typeOptions: { multipleValues: true },   // ← allows adding multiple rows
  default: {},
  options: [
    {
      displayName: 'Header',
      name: 'header',                        // ← key used to access values
      values: [
        {
          displayName: 'Name',
          name: 'name',
          type: 'string',
          default: '',
          placeholder: 'X-Custom-Header',
        },
        {
          displayName: 'Value',
          name: 'value',
          type: 'string',
          default: '',
        },
      ],
    },
  ],
}
```

In `execute()`:
```typescript
const customHeaders = this.getNodeParameter('customHeaders', i, {}) as {
  header?: Array<{ name: string; value: string }>;
};

const extraHeaders: Record<string, string> = {};
for (const h of (customHeaders.header ?? [])) {
  extraHeaders[h.name] = h.value;
}

// Merge with base headers
const headers = { Authorization: `Bearer token`, ...extraHeaders };
```

### fixedCollection without multipleValues (exactly one set)

```typescript
{
  displayName: 'Coordinates',
  name: 'coordinates',
  type: 'fixedCollection',
  // NO typeOptions.multipleValues
  default: {},
  options: [
    {
      displayName: 'Location',
      name: 'location',
      values: [
        { displayName: 'Latitude',  name: 'lat', type: 'number', default: 0 },
        { displayName: 'Longitude', name: 'lng', type: 'number', default: 0 },
      ],
    },
  ],
}
```

In `execute()`:
```typescript
const coords = this.getNodeParameter('coordinates.location', i, {}) as {
  lat: number;
  lng: number;
};
```

---

## 5. Modifiers

```typescript
{
  displayName: 'Resource',
  name: 'resource',
  type: 'options',
  noDataExpression: true,    // ← prevents the "expression" toggle on this field
                              //   always use this on resource/operation selectors
  options: [...],
  default: 'user',
  required: true,             // ← shows red asterisk, blocks execution if empty
  placeholder: 'Enter name', // ← grey placeholder text
  description: 'The resource to work with',   // ← shown as tooltip
  hint: 'Use the resource ID from the URL',   // ← shown below field
}
```

---

## 6. Subtitle

Shows dynamic text below the node name in the canvas (helps users see what the node is doing at a glance):

```typescript
description: INodeTypeDescription = {
  // ...
  subtitle: '={{$parameter["operation"] + ": " + $parameter["resource"]}}',
  // Shows e.g. "create: user" under the node name
}
```

Other useful subtitle patterns:
```typescript
subtitle: '={{$parameter["resource"]}}'
subtitle: '={{$parameter["url"]}}'
subtitle: '={{$parameter["event"]}}'
```

FILE:references/concepts/http-and-binary.md
# HTTP Requests and Binary Data

## Table of Contents
1. [httpRequest — full options reference](#1-httprequest-full-options)
2. [Query params, headers, body patterns](#2-common-patterns)
3. [Getting the full response (status + headers)](#3-full-response)
4. [Binary data — downloading files](#4-binary-data--downloading)
5. [Binary data — uploading files](#5-binary-data--uploading)
6. [Multipart form data](#6-multipart-form-data)
7. [Handling non-JSON responses](#7-non-json-responses)
8. [Streaming / large responses](#8-streaming)

---

## 1. httpRequest Full Options

```typescript
const response = await this.helpers.httpRequest({
  // ── Required ─────────────────────────────────────────────────
  method: 'POST',                        // GET | POST | PUT | PATCH | DELETE | HEAD
  url: 'https://api.example.com/items',  // full URL, OR use baseURL + url separately

  // ── Optional ─────────────────────────────────────────────────
  baseURL: 'https://api.example.com',    // if using relative url below
  // url: '/items',                       // relative path when baseURL is set

  headers: {
    'Content-Type': 'application/json',
    Authorization: `Bearer token`,
    'X-Custom': 'value',
  },

  qs: {                                  // query string parameters
    page: 1,
    limit: 50,
    filter: 'active',
  },

  body: {                                // request body (auto-serialized if json: true)
    name: 'John',
    email: '[email protected]',
  },

  // Response options
  returnFullResponse: false,             // true = { statusCode, headers, body }
                                         // false (default) = just the body

  // Ignore SSL certificate errors (use carefully)
  skipSslCertificateValidation: false,

  // Timeout in ms
  timeout: 30000,

  // Follow redirects
  followRedirects: true,
  maxRedirects: 10,

  // Proxy
  proxy: {
    host: 'proxy.example.com',
    port: 8080,
  },
});
```

---

## 2. Common Patterns

### GET with query params
```typescript
const response = await this.helpers.httpRequest({
  method: 'GET',
  url: 'https://api.example.com/users',
  headers: { Authorization: `Bearer token` },
  qs: { limit: 50, status: 'active', cursor: 'abc123' },
});
```

### POST JSON body
```typescript
const response = await this.helpers.httpRequest({
  method: 'POST',
  url: 'https://api.example.com/users',
  headers: {
    Authorization: `Bearer token`,
    'Content-Type': 'application/json',
  },
  body: { name: 'John', email: '[email protected]' },
});
```

### POST with user-supplied JSON body
```typescript
const rawBody = this.getNodeParameter('body', i, '{}') as string;
let body: IDataObject;
try {
  body = JSON.parse(rawBody);
} catch {
  throw new NodeOperationError(this.getNode(), 'Body is not valid JSON', { itemIndex: i });
}

const response = await this.helpers.httpRequest({
  method: 'POST',
  url: 'https://api.example.com/items',
  headers: { Authorization: `Bearer token`, 'Content-Type': 'application/json' },
  body,
});
```

### PATCH / PUT (partial update)
```typescript
const response = await this.helpers.httpRequest({
  method: 'PATCH',
  url: `https://api.example.com/users/userId`,
  headers: { Authorization: `Bearer token`, 'Content-Type': 'application/json' },
  body: { email: '[email protected]' },  // only fields to update
});
```

### DELETE
```typescript
await this.helpers.httpRequest({
  method: 'DELETE',
  url: `https://api.example.com/users/userId`,
  headers: { Authorization: `Bearer token` },
});
```

---

## 3. Full Response

Get status code and headers alongside the body:

```typescript
const response = await this.helpers.httpRequest({
  method: 'GET',
  url: 'https://api.example.com/data',
  headers: authHeader,
  returnFullResponse: true,
}) as {
  statusCode: number;
  headers: Record<string, string>;
  body: unknown;
};

console.log(response.statusCode);        // 200
console.log(response.headers['x-rate-limit-remaining']); // rate limit info
const data = response.body;
```

---

## 4. Binary Data — Downloading

Receive a file from an API and pass it to the next node (e.g. for email attachment, Google Drive upload):

```typescript
import { BINARY_ENCODING } from 'n8n-workflow';

// Download file as buffer
const response = await this.helpers.httpRequest({
  method: 'GET',
  url: `https://api.example.com/files/fileId/download`,
  headers: { Authorization: `Bearer token` },
  encoding: 'arraybuffer',              // ← get raw bytes
  returnFullResponse: true,
}) as { statusCode: number; headers: Record<string, string>; body: Buffer };

// Detect MIME type from Content-Type header
const contentType = response.headers['content-type'] || 'application/octet-stream';
const fileName    = response.headers['content-disposition']
  ?.match(/filename="?([^"]+)"?/)?.[1] ?? 'download';

// Attach binary to output item
const binaryData = await this.helpers.prepareBinaryData(
  response.body,
  fileName,
  contentType,
);

returnData.push({
  json: { fileName, contentType, size: response.body.length },
  binary: { data: binaryData },           // key 'data' is convention; can be anything
  pairedItem: { item: i },
});
```

Access in next nodes: `{{ $binary.data.fileName }}`, or use "Move Binary Data" node.

---

## 5. Binary Data — Uploading

Send a file that came from a previous node:

```typescript
// Get binary data from input item
const binaryPropertyName = this.getNodeParameter('binaryPropertyName', i, 'data') as string;
const binaryData = this.helpers.assertBinaryData(i, binaryPropertyName);

// Convert to buffer
const buffer = await this.helpers.getBinaryDataBuffer(i, binaryPropertyName);

// Upload as raw binary body
const response = await this.helpers.httpRequest({
  method: 'POST',
  url: 'https://api.example.com/files/upload',
  headers: {
    Authorization: `Bearer token`,
    'Content-Type': binaryData.mimeType,
    'Content-Disposition': `attachment; filename="binaryData.fileName"`,
  },
  body: buffer,
});
```

---

## 6. Multipart Form Data

For file uploads where the API expects `multipart/form-data`:

```typescript
import FormData from 'form-data';

const binaryData = this.helpers.assertBinaryData(i, 'data');
const buffer     = await this.helpers.getBinaryDataBuffer(i, 'data');

const formData = new FormData();
formData.append('file', buffer, {
  filename:    binaryData.fileName ?? 'upload',
  contentType: binaryData.mimeType,
});
formData.append('title', this.getNodeParameter('title', i) as string);
formData.append('folder_id', this.getNodeParameter('folderId', i) as string);

const response = await this.helpers.httpRequest({
  method: 'POST',
  url: 'https://api.example.com/files',
  headers: {
    Authorization: `Bearer token`,
    ...formData.getHeaders(),           // ← sets Content-Type: multipart/form-data; boundary=...
  },
  body: formData,
});
```

Add to `package.json` dependencies: `"form-data": "^4.0.0"`

---

## 7. Non-JSON Responses

```typescript
// Plain text response
const response = await this.helpers.httpRequest({
  method: 'GET',
  url: 'https://api.example.com/data.csv',
  headers: authHeader,
  // No Content-Type: application/json → response is returned as string
});
// response is a string

// XML response — parse manually
const xml2js = require('xml2js');
const parsed = await xml2js.parseStringPromise(response as string, { explicitArray: false });

// Return raw text as a field
returnData.push({
  json: { raw: response as string },
  pairedItem: { item: i },
});
```

---

## 8. Streaming / Large Responses

For very large responses, use Node.js streams via `request` helper (lower level):

```typescript
const stream = await this.helpers.httpRequestWithAuthentication.call(
  this,
  'myCredential',
  {
    method: 'GET',
    url: 'https://api.example.com/large-file',
    encoding: null,
    resolveWithFullResponse: true,
  },
);

// Or use prepareBinaryData with a stream
const binaryData = await this.helpers.prepareBinaryData(
  stream.body as Buffer,
  'large-file.bin',
  'application/octet-stream',
);
```

> For most file downloads under ~100MB, `encoding: 'arraybuffer'` (section 4) is simpler and sufficient.

# n8n + Qdrant: Ingestion & RAG Pipeline Skill

## Overview

This skill enables AI agents to design, build, and troubleshoot **production-grade Qdrant ingestion and retrieval pipelines in n8n**. It covers the full lifecycle: source data extraction → chunking → metadata enrichment → vector embedding → Qdrant upsert → retrieval (dense, sparse, hybrid) → RAG response generation.

**Always read the supporting docs in `/docs/` before building workflows:**
- `docs/NODE-REFERENCE.md` — Every Qdrant node, mode, and parameter explained
- `docs/INGESTION-PIPELINE.md` — Step-by-step ingestion architecture
- `docs/RAG-RETRIEVAL.md` — Dense, sparse, and hybrid retrieval patterns
- `docs/CHUNKING-METADATA.md` — Chunking strategies and metadata schema design
- `docs/examples/` — Annotated workflow JSON examples

---

## Two Node Systems to Know

n8n has **two separate Qdrant integration systems** — knowing which to use is critical:

### 1. Official Qdrant Node (`n8n-nodes-qdrant`)
- **Package**: `n8n-nodes-qdrant` (community node, install via n8n Settings → Community Nodes)
- **Purpose**: Direct Qdrant API operations — collection management, point upsert/delete/scroll, search queries
- **Node name in editor**: `Qdrant`
- **Use for**: Building custom ingestion pipelines, running Query Points (dense/sparse/hybrid search), collection setup, point management
- **GitHub**: https://github.com/qdrant/n8n-nodes-qdrant

### 2. LangChain Vector Store Node (built-in)
- **Package**: Built into n8n's AI/LangChain nodes
- **Purpose**: LangChain-compatible vector store integration — connects with Document Loaders, Text Splitters, Embeddings, and AI Agents
- **Node name in editor**: `Qdrant Vector Store` (`@n8n/n8n-nodes-langchain.vectorStoreQdrant`)
- **Use for**: LangChain-style RAG pipelines, AI Agent tool integration, retrieve-as-tool mode
- **Modes**: `insert` (ingest documents), `retrieve` (similarity search), `retrieve-as-tool` (AI agent tool)

**Rule of thumb**: Use LangChain Vector Store for LangChain-native agent/RAG flows. Use the Official Qdrant Node for direct API control, hybrid search, payload operations, and production ingestion pipelines.

---

## Quick Decision Matrix

| Goal | Use This Node | Mode/Operation |
|------|--------------|---------------|
| Ingest documents via LangChain chain | LangChain Vector Store | `insert` |
| AI Agent retrieves from Qdrant as tool | LangChain Vector Store | `retrieve-as-tool` |
| Run hybrid (dense+sparse) search | Official Qdrant Node | `Search → Query Points` |
| Create/manage collections | Official Qdrant Node | `Collection → Create Collection` |
| Upsert raw points with custom payloads | Official Qdrant Node | `Point → Upsert Points` |
| Delete points by filter (e.g. file_id) | Official Qdrant Node | `Point → Delete Points` |
| Scroll all points for audit/export | Official Qdrant Node | `Point → Scroll Points` |
| Batch ingest large datasets | Official Qdrant Node | `Point → Batch Update Points` |

---

## Canonical Ingestion Pipeline Architecture

```
[Trigger]
    │
    ▼
[Source Node] ──────────────────────────────────────────────
(Slack, Fireflies, Google Drive, HTTP, DB, etc.)             │
    │                                                         │
    ▼                                                         │
[Split in Batches]  ←── Loop for large datasets              │
    │                                                         │
    ▼                                                         │
[Extract/Normalize]                                          │
(Set node: build content string + raw metadata)              │
    │                                                         │
    ▼                                                         │
[AI: Extract Metadata]                                       │
(Information Extractor or LLM Chain)                         │
Produces: themes, keywords, entities, summary, tags          │
    │                                                         │
    ▼                                                         │
[Text Splitter]                                              │
(Token Splitter or Recursive Character Splitter)             │
chunkSize: 512–2000 tokens, overlap: 10–15%                  │
    │                                                         │
    ▼                                                         │
[Embeddings Node]                                            │
(OpenAI text-embedding-3-large or similar)                   │
    │                                                         │
    ▼                                                         │
[Qdrant Vector Store — insert mode]  OR                      │
[Official Qdrant Node — Upsert Points]                       │
    │                                                         │
    ▼                                                         │
[Wait Node]  ←── Rate limiting / backpressure                │
    │                                                         │
    └─────────────────── back to Split in Batches ───────────┘
```

See `docs/INGESTION-PIPELINE.md` for full node-by-node configuration.

---

## Canonical RAG Retrieval Architecture

```
[Chat Trigger / Webhook]
    │
    ▼
[AI Agent Node]
    │
    ├── [LLM: Gemini / GPT-4o / Claude]
    ├── [Memory: Window Buffer Memory]
    └── [Tool: Qdrant Vector Store — retrieve-as-tool]
              │
              └── [Embeddings Node]
```

For hybrid search (dense + sparse), use the **Official Qdrant Node** → Query Points with a `prefetch` array combining dense and sparse queries + RRF fusion. See `docs/RAG-RETRIEVAL.md`.

---

## Credentials Setup

### Official Qdrant Node
- **Credential type**: `qdrantApi`
- Fields: `URL` (e.g. `https://your-cluster.cloud.qdrant.io`) + `API Key`

### LangChain Vector Store Node
- **Credential type**: `qdrantApi` (same credential, shared)

### Qdrant Cloud Setup
1. Open https://cloud.qdrant.io → select cluster
2. Copy **Endpoint** → use as URL
3. Go to **API Keys** tab → copy key

### Local (Docker / AI Starter Kit)
- URL: `http://qdrant:6333/`
- Set `QDRANT_API_KEY=your_key` in docker-compose environment

---

## Naming Conventions

Use consistent naming across workflows:

| Element | Convention | Example |
|---------|-----------|---------|
| Collection name | `{org}-{source}-{content-type}` | `acme-slack-messages` |
| Metadata key for source ID | `source_id` | `"source_id": "C01234-1709123456"` |
| Metadata key for document ID | `doc_id` | `"doc_id": "file_abc123"` |
| Metadata key for chunk index | `chunk_index` | `"chunk_index": 3` |
| Metadata key for timestamp | `created_at` | ISO 8601 string |
| Metadata key for source type | `source_type` | `"slack"`, `"fireflies"`, `"gdrive"` |
| Metadata key for channel/folder | `source_context` | `"#engineering"` |

---

## Critical Rules

1. **Always set `file_id` or `doc_id` in metadata** — enables targeted deletion without full collection wipe
2. **Always use `onError: continueRegularOutput`** on the Qdrant Vector Store node — prevents single-item failures from crashing the whole batch
3. **Always use `retryOnFail: true`** on the Qdrant node for production ingestion
4. **Chunk before embedding** — never embed full documents; always split first
5. **Never store raw text in collection names or keys** — normalize to lowercase slug format
6. **Use `Split in Batches` with a `Wait` node** for large datasets — prevents API rate limit errors and memory exhaustion
7. **Run metadata extraction BEFORE the text splitter** — extract from the full document, then attach metadata to each chunk
8. **For delete operations, always add human-in-the-loop confirmation** (Telegram sendAndWait, Slack approval, etc.)

FILE:docs/examples/slack-ingestion-workflow.md
# Example: Slack Channel → Qdrant Ingestion Workflow

This is an annotated n8n workflow that ingests Slack channel messages into Qdrant with AI metadata extraction. Import this JSON into n8n via the workflow editor (kebab menu → Import from JSON).

## What This Workflow Does
1. Triggers on schedule (hourly) or manual test
2. Fetches recent messages from a Slack channel
3. Filters out already-ingested messages
4. Extracts AI metadata (keywords, summary, sentiment, topics)
5. Chunks, embeds, and upserts into Qdrant with full metadata payload
6. Rate-limits to avoid API throttling
7. Sends Slack notification on completion

## Prerequisites
- Qdrant collection already created (run collection setup workflow first)
- OpenAI API credentials configured
- Slack OAuth credentials configured
- Qdrant credentials configured

## Configuration Points (search for CONFIGURE_ME)
- `CONFIGURE_ME_CHANNEL` → Your Slack channel ID (e.g. C01234ABCDE)
- `CONFIGURE_ME_COLLECTION` → Your Qdrant collection name
- `CONFIGURE_ME_NOTIFY_CHANNEL` → Slack channel for completion notifications

---

```json
{
  "name": "Slack → Qdrant Ingestion Pipeline",
  "nodes": [
    {
      "id": "trigger-schedule",
      "name": "Schedule: Hourly",
      "type": "n8n-nodes-base.scheduleTrigger",
      "position": [-800, 0],
      "parameters": {
        "rule": { "interval": [{ "field": "hours", "hoursInterval": 1 }] }
      },
      "typeVersion": 1.2
    },
    {
      "id": "trigger-manual",
      "name": "Manual Test Trigger",
      "type": "n8n-nodes-base.manualTrigger",
      "position": [-800, -120],
      "parameters": {},
      "typeVersion": 1
    },
    {
      "id": "fetch-slack-messages",
      "name": "Fetch Slack Messages",
      "type": "n8n-nodes-base.slack",
      "position": [-600, 0],
      "parameters": {
        "resource": "message",
        "operation": "getAll",
        "channelId": "CONFIGURE_ME_CHANNEL",
        "returnAll": false,
        "limit": 200,
        "filters": {
          "oldest": "={{ Math.floor((Date.now() - 3600000) / 1000) }}"
        },
        "options": {}
      },
      "credentials": { "slackOAuth2Api": { "id": "slack-cred", "name": "Slack" } },
      "typeVersion": 2.2
    },
    {
      "id": "normalize-fields",
      "name": "Normalize Fields",
      "type": "n8n-nodes-base.set",
      "position": [-400, 0],
      "parameters": {
        "options": {},
        "assignments": {
          "assignments": [
            { "name": "content", "value": "={{ $json.text }}", "type": "string" },
            { "name": "doc_id", "value": "={{ $json.channel + '-' + $json.ts }}", "type": "string" },
            { "name": "source_type", "value": "slack", "type": "string" },
            { "name": "source_context", "value": "={{ $json.channel }}", "type": "string" },
            { "name": "author", "value": "={{ $json.user }}", "type": "string" },
            { "name": "created_at", "value": "={{ new Date($json.ts * 1000).toISOString() }}", "type": "string" },
            { "name": "thread_id", "value": "={{ $json.thread_ts || $json.ts }}", "type": "string" }
          ]
        }
      },
      "typeVersion": 3.4
    },
    {
      "id": "filter-empty",
      "name": "Filter Empty Messages",
      "type": "n8n-nodes-base.filter",
      "position": [-200, 0],
      "parameters": {
        "conditions": {
          "options": { "version": 2 },
          "combinator": "and",
          "conditions": [
            {
              "operator": { "type": "string", "operation": "isNotEmpty" },
              "leftValue": "={{ $json.content }}"
            },
            {
              "operator": { "type": "number", "operation": "gte" },
              "leftValue": "={{ $json.content.split(' ').length }}",
              "rightValue": 5
            }
          ]
        }
      },
      "typeVersion": 2.2
    },
    {
      "id": "split-batches",
      "name": "Process in Batches of 20",
      "type": "n8n-nodes-base.splitInBatches",
      "position": [0, 0],
      "parameters": { "batchSize": 20, "options": {} },
      "typeVersion": 3
    },
    {
      "id": "extract-metadata",
      "name": "AI: Extract Metadata",
      "type": "@n8n/n8n-nodes-langchain.informationExtractor",
      "position": [200, 0],
      "parameters": {
        "text": "={{ $json.content }}",
        "options": {
          "systemPromptTemplate": "You are an expert extraction system. Only extract relevant information. Omit fields you cannot determine."
        },
        "attributes": {
          "attributes": [
            { "name": "summary", "description": "1-2 sentence summary of what this Slack message is about" },
            { "name": "keywords", "description": "5-8 keywords as an array of strings" },
            { "name": "topics", "description": "Topic categories from: engineering, product, design, ops, hr, finance, general - as array" },
            { "name": "sentiment", "description": "One of: positive, negative, neutral, frustrated, excited" },
            { "name": "has_question", "description": "Boolean: does the message contain a question?" },
            { "name": "action_items", "description": "Array of action items mentioned. Empty array if none." }
          ]
        }
      },
      "typeVersion": 1
    },
    {
      "id": "llm-for-extraction",
      "name": "GPT-4o-mini (Extraction)",
      "type": "@n8n/n8n-nodes-langchain.lmChatOpenAi",
      "position": [200, 180],
      "parameters": {
        "model": { "__rl": true, "mode": "list", "value": "gpt-4o-mini" },
        "options": { "temperature": 0 }
      },
      "credentials": { "openAiApi": { "id": "openai-cred", "name": "OpenAI" } },
      "typeVersion": 1.2
    },
    {
      "id": "merge-with-metadata",
      "name": "Merge Original + Metadata",
      "type": "n8n-nodes-base.merge",
      "position": [420, 0],
      "parameters": { "mode": "combine", "combineBy": "combineByPosition", "options": {} },
      "typeVersion": 3
    },
    {
      "id": "qdrant-vector-store",
      "name": "Upsert to Qdrant",
      "type": "@n8n/n8n-nodes-langchain.vectorStoreQdrant",
      "onError": "continueRegularOutput",
      "position": [620, 0],
      "parameters": {
        "mode": "insert",
        "options": {
          "metadata": {
            "metadataValues": [
              { "name": "doc_id", "value": "={{ $json.doc_id }}" },
              { "name": "source_type", "value": "={{ $json.source_type }}" },
              { "name": "source_context", "value": "={{ $json.source_context }}" },
              { "name": "author", "value": "={{ $json.author }}" },
              { "name": "created_at", "value": "={{ $json.created_at }}" },
              { "name": "thread_id", "value": "={{ $json.thread_id }}" },
              { "name": "meta_summary", "value": "={{ $json.output?.summary || '' }}" },
              { "name": "meta_keywords", "value": "={{ $json.output?.keywords || [] }}" },
              { "name": "meta_topics", "value": "={{ $json.output?.topics || [] }}" },
              { "name": "meta_sentiment", "value": "={{ $json.output?.sentiment || 'neutral' }}" }
            ]
          }
        },
        "qdrantCollection": { "__rl": true, "mode": "id", "value": "CONFIGURE_ME_COLLECTION" }
      },
      "credentials": { "qdrantApi": { "id": "qdrant-cred", "name": "Qdrant" } },
      "retryOnFail": true,
      "executeOnce": false,
      "typeVersion": 1
    },
    {
      "id": "data-loader",
      "name": "Data Loader",
      "type": "@n8n/n8n-nodes-langchain.documentDefaultDataLoader",
      "position": [620, 180],
      "parameters": {
        "dataType": "json",
        "jsonMode": "expressionData",
        "expressionData": "={{ $json.content }}",
        "options": {}
      },
      "typeVersion": 1
    },
    {
      "id": "token-splitter",
      "name": "Token Splitter (256 tokens)",
      "type": "@n8n/n8n-nodes-langchain.textSplitterTokenSplitter",
      "position": [720, 280],
      "parameters": { "chunkSize": 256, "chunkOverlap": 25 },
      "typeVersion": 1
    },
    {
      "id": "embeddings",
      "name": "OpenAI Embeddings",
      "type": "@n8n/n8n-nodes-langchain.embeddingsOpenAi",
      "position": [500, 180],
      "parameters": {
        "model": "text-embedding-3-large",
        "options": {}
      },
      "credentials": { "openAiApi": { "id": "openai-cred", "name": "OpenAI" } },
      "typeVersion": 1
    },
    {
      "id": "rate-limit-wait",
      "name": "Wait 1s (Rate Limit)",
      "type": "n8n-nodes-base.wait",
      "position": [820, 0],
      "parameters": { "unit": "seconds", "amount": 1 },
      "typeVersion": 1.1
    },
    {
      "id": "notify-complete",
      "name": "Notify: Ingestion Complete",
      "type": "n8n-nodes-base.slack",
      "position": [1020, 0],
      "parameters": {
        "resource": "message",
        "operation": "post",
        "channel": "CONFIGURE_ME_NOTIFY_CHANNEL",
        "text": "✅ Slack ingestion complete. Processed batch to Qdrant collection.",
        "options": {}
      },
      "credentials": { "slackOAuth2Api": { "id": "slack-cred", "name": "Slack" } },
      "typeVersion": 2.2
    }
  ],
  "connections": {
    "Schedule: Hourly": { "main": [[{ "node": "Fetch Slack Messages", "type": "main", "index": 0 }]] },
    "Manual Test Trigger": { "main": [[{ "node": "Fetch Slack Messages", "type": "main", "index": 0 }]] },
    "Fetch Slack Messages": { "main": [[{ "node": "Normalize Fields", "type": "main", "index": 0 }]] },
    "Normalize Fields": { "main": [[{ "node": "Filter Empty Messages", "type": "main", "index": 0 }]] },
    "Filter Empty Messages": { "main": [[{ "node": "Process in Batches of 20", "type": "main", "index": 0 }]] },
    "Process in Batches of 20": {
      "main": [
        [{ "node": "Notify: Ingestion Complete", "type": "main", "index": 0 }],
        [{ "node": "AI: Extract Metadata", "type": "main", "index": 0 }]
      ]
    },
    "AI: Extract Metadata": { "main": [[{ "node": "Merge Original + Metadata", "type": "main", "index": 0 }]] },
    "Normalize Fields (passthrough)": { "main": [[{ "node": "Merge Original + Metadata", "type": "main", "index": 1 }]] },
    "Merge Original + Metadata": { "main": [[{ "node": "Upsert to Qdrant", "type": "main", "index": 0 }]] },
    "Upsert to Qdrant": { "main": [[{ "node": "Wait 1s (Rate Limit)", "type": "main", "index": 0 }]] },
    "Wait 1s (Rate Limit)": { "main": [[{ "node": "Process in Batches of 20", "type": "main", "index": 0 }]] },
    "GPT-4o-mini (Extraction)": { "ai_languageModel": [[{ "node": "AI: Extract Metadata", "type": "ai_languageModel", "index": 0 }]] },
    "Data Loader": { "ai_document": [[{ "node": "Upsert to Qdrant", "type": "ai_document", "index": 0 }]] },
    "Token Splitter (256 tokens)": { "ai_textSplitter": [[{ "node": "Data Loader", "type": "ai_textSplitter", "index": 0 }]] },
    "OpenAI Embeddings": { "ai_embedding": [[{ "node": "Upsert to Qdrant", "type": "ai_embedding", "index": 0 }]] }
  }
}
```

---

## Adaptation Guide

### For Fireflies Transcripts
- Replace Slack node with HTTP Request to Fireflies API
- Change `content` to `$json.sentences.map(s => s.text).join(' ')`
- Increase chunk size to 1000–2000 tokens
- Add meeting-specific metadata: `participants`, `meeting_title`, `duration_minutes`
- Use meeting-level metadata extraction (decisions, action items, pain points)

### For Google Drive
- Replace Slack node with Google Drive → List Files → Download File
- Use `dataType: "binary"` in Data Loader with `binaryMode: "specificField"`
- Increase chunk size to 1000–1500 tokens
- Add file-specific metadata: `file_name`, `file_type`, `drive_folder`

### For Any REST API / Database
- Replace Slack node with HTTP Request or DB query node
- Adjust field normalization in Set node
- Tune chunk size for your content type
- Customize Information Extractor attributes for your domain

FILE:docs/examples/rag-agent-workflow.md
# Example: RAG Chat Agent with Qdrant

Two patterns are documented here:
1. **Simple RAG Agent** — LangChain AI Agent + Qdrant as tool (easiest setup)
2. **Hybrid Search RAG** — Official Qdrant node with dense+sparse+RRF (best quality)

---

## Pattern 1: Simple RAG Agent (LangChain)

### Overview
Uses n8n's built-in LangChain nodes. The AI Agent automatically decides when to query the knowledge base and formulates answers from retrieved context.

```
[Chat Trigger] → [AI Agent] → [Respond to User]
                    ↑ LLM (Gemini/GPT-4o)
                    ↑ Window Buffer Memory
                    ↑ Qdrant Vector Store Tool
                         ↑ OpenAI Embeddings
```

### Workflow JSON
```json
{
  "name": "RAG Chat Agent - Qdrant",
  "nodes": [
    {
      "id": "chat-trigger",
      "name": "Chat Trigger",
      "type": "@n8n/n8n-nodes-langchain.chatTrigger",
      "position": [-600, 0],
      "parameters": { "options": {} },
      "webhookId": "your-webhook-id",
      "typeVersion": 1.1
    },
    {
      "id": "ai-agent",
      "name": "AI Agent",
      "type": "@n8n/n8n-nodes-langchain.agent",
      "position": [-400, 0],
      "parameters": {
        "text": "={{ $json.chatInput }}",
        "promptType": "define",
        "options": {
          "systemMessage": "You are a helpful knowledge assistant. Use the knowledge_base tool to search for relevant information before answering questions.\n\nAlways:\n- Search the knowledge base first for any question about company matters\n- Cite your sources (mention the source_context and created_at from retrieved results)\n- If the knowledge base doesn't contain relevant information, say so clearly\n- Never make up facts not present in the retrieved context"
        }
      },
      "typeVersion": 1.7
    },
    {
      "id": "llm-gemini",
      "name": "Gemini Flash",
      "type": "@n8n/n8n-nodes-langchain.lmChatGoogleGemini",
      "position": [-550, 200],
      "parameters": {
        "modelName": "models/gemini-2.0-flash-exp",
        "options": { "maxOutputTokens": 8192, "temperature": 0.3 }
      },
      "credentials": { "googlePalmApi": { "id": "gemini-cred", "name": "Google Gemini" } },
      "typeVersion": 1
    },
    {
      "id": "memory",
      "name": "Window Buffer Memory",
      "type": "@n8n/n8n-nodes-langchain.memoryBufferWindow",
      "position": [-400, 200],
      "parameters": { "contextWindowLength": 20 },
      "typeVersion": 1.3
    },
    {
      "id": "qdrant-tool",
      "name": "Qdrant: Knowledge Base Tool",
      "type": "@n8n/n8n-nodes-langchain.vectorStoreQdrant",
      "position": [-250, 200],
      "parameters": {
        "mode": "retrieve-as-tool",
        "topK": 15,
        "toolName": "knowledge_base",
        "toolDescription": "Search the company knowledge base including Slack conversations, meeting transcripts, and documents. Use this for any question about company processes, decisions, or historical information.",
        "qdrantCollection": {
          "__rl": true,
          "mode": "id",
          "value": "CONFIGURE_ME_COLLECTION"
        },
        "options": {}
      },
      "credentials": { "qdrantApi": { "id": "qdrant-cred", "name": "Qdrant" } },
      "typeVersion": 1
    },
    {
      "id": "embeddings-retrieval",
      "name": "OpenAI Embeddings (Retrieval)",
      "type": "@n8n/n8n-nodes-langchain.embeddingsOpenAi",
      "position": [-100, 300],
      "parameters": {
        "model": "text-embedding-3-large",
        "options": {}
      },
      "credentials": { "openAiApi": { "id": "openai-cred", "name": "OpenAI" } },
      "typeVersion": 1
    },
    {
      "id": "respond",
      "name": "Respond to User",
      "type": "n8n-nodes-base.set",
      "position": [-200, 0],
      "parameters": {
        "options": {},
        "assignments": {
          "assignments": [
            { "name": "output", "value": "={{ $json.output }}", "type": "string" }
          ]
        }
      },
      "typeVersion": 3.4
    }
  ],
  "connections": {
    "Chat Trigger": { "main": [[{ "node": "AI Agent", "type": "main", "index": 0 }]] },
    "AI Agent": { "main": [[{ "node": "Respond to User", "type": "main", "index": 0 }]] },
    "Gemini Flash": { "ai_languageModel": [[{ "node": "AI Agent", "type": "ai_languageModel", "index": 0 }]] },
    "Window Buffer Memory": { "ai_memory": [[{ "node": "AI Agent", "type": "ai_memory", "index": 0 }]] },
    "Qdrant: Knowledge Base Tool": { "ai_tool": [[{ "node": "AI Agent", "type": "ai_tool", "index": 0 }]] },
    "OpenAI Embeddings (Retrieval)": { "ai_embedding": [[{ "node": "Qdrant: Knowledge Base Tool", "type": "ai_embedding", "index": 0 }]] }
  }
}
```

---

## Pattern 2: Hybrid Search RAG (Maximum Quality)

### Overview
Uses the Official Qdrant Node to run dense + sparse search with RRF fusion, then passes context to an LLM manually. More control, better retrieval quality.

**Requirements**: Collection must be configured with both dense and sparse vectors. Requires a sparse encoding service (see notes below).

```
[Chat Trigger]
    → [Parallel: Dense Embed + Sparse Encode]
    → [Merge embeddings]
    → [Qdrant: Query Points (hybrid/RRF)]
    → [Code: format context]
    → [OpenAI / Gemini: generate answer]
    → [Respond]
```

### Workflow JSON (Simplified)
```json
{
  "name": "Hybrid RAG Pipeline - Qdrant",
  "nodes": [
    {
      "id": "webhook-in",
      "name": "Chat Webhook",
      "type": "n8n-nodes-base.webhook",
      "position": [-800, 0],
      "parameters": { "path": "chat", "options": {} },
      "typeVersion": 2
    },
    {
      "id": "dense-embed",
      "name": "Dense Embedding",
      "type": "n8n-nodes-base.httpRequest",
      "position": [-600, -80],
      "parameters": {
        "url": "https://api.openai.com/v1/embeddings",
        "method": "POST",
        "sendHeaders": true,
        "headerParameters": {
          "parameters": [{ "name": "Authorization", "value": "=Bearer {{ $env.OPENAI_API_KEY }}" }]
        },
        "sendBody": true,
        "bodyParameters": {
          "parameters": [
            { "name": "model", "value": "text-embedding-3-large" },
            { "name": "input", "value": "={{ $json.body.query }}" }
          ]
        },
        "options": {}
      },
      "typeVersion": 4.2
    },
    {
      "id": "sparse-encode",
      "name": "Sparse Encoding (BM25/SPLADE)",
      "type": "n8n-nodes-base.httpRequest",
      "position": [-600, 80],
      "parameters": {
        "url": "http://your-sparse-encoder-service/encode",
        "method": "POST",
        "sendBody": true,
        "bodyParameters": {
          "parameters": [
            { "name": "text", "value": "={{ $json.body.query }}" }
          ]
        },
        "options": {}
      },
      "typeVersion": 4.2
    },
    {
      "id": "merge-vectors",
      "name": "Merge Vectors",
      "type": "n8n-nodes-base.merge",
      "position": [-400, 0],
      "parameters": { "mode": "combine", "combineBy": "combineAll", "options": {} },
      "typeVersion": 3
    },
    {
      "id": "hybrid-search",
      "name": "Qdrant: Hybrid Search",
      "type": "n8n-nodes-qdrant.qdrant",
      "position": [-200, 0],
      "parameters": {
        "resource": "search",
        "operation": "queryPoints",
        "collectionName": "CONFIGURE_ME_COLLECTION",
        "prefetch": "=[{ \"query\": {{ $json.dense_vector }}, \"using\": \"dense\", \"limit\": 20 }, { \"query\": { \"indices\": {{ $json.sparse_indices }}, \"values\": {{ $json.sparse_values }} }, \"using\": \"sparse\", \"limit\": 20 }]",
        "query": "={ \"fusion\": \"rrf\" }",
        "limit": 10,
        "withPayload": true,
        "withVector": false
      },
      "credentials": { "qdrantApi": { "id": "qdrant-cred", "name": "Qdrant" } },
      "typeVersion": 1
    },
    {
      "id": "format-context",
      "name": "Format Context",
      "type": "n8n-nodes-base.code",
      "position": [0, 0],
      "parameters": {
        "language": "javaScript",
        "jsCode": "const results = $input.all();\nconst query = $('Chat Webhook').item.json.body.query;\n\nconst context = results.map((item, i) => {\n  const p = item.json.payload;\n  return `[i+1] p.source_type | p.source_context | p.created_at?.substring(0,10)\\np.text`;\n}).join('\\n\\n---\\n\\n');\n\nreturn [{ json: { query, context, source_count: results.length } }];"
      },
      "typeVersion": 2
    },
    {
      "id": "llm-answer",
      "name": "Generate Answer",
      "type": "n8n-nodes-base.httpRequest",
      "position": [200, 0],
      "parameters": {
        "url": "https://api.openai.com/v1/chat/completions",
        "method": "POST",
        "sendHeaders": true,
        "headerParameters": {
          "parameters": [{ "name": "Authorization", "value": "=Bearer {{ $env.OPENAI_API_KEY }}" }]
        },
        "sendBody": true,
        "bodyParameters": {
          "parameters": [
            { "name": "model", "value": "gpt-4o" },
            { "name": "messages", "value": "=[{ \"role\": \"system\", \"content\": \"You are a helpful assistant. Answer questions based only on the provided context. Cite sources.\" }, { \"role\": \"user\", \"content\": \"Context:\\n{{ $json.context }}\\n\\nQuestion: {{ $json.query }}\" }]" },
            { "name": "max_tokens", "value": "=2000" }
          ]
        },
        "options": {}
      },
      "typeVersion": 4.2
    },
    {
      "id": "respond-webhook",
      "name": "Webhook Response",
      "type": "n8n-nodes-base.respondToWebhook",
      "position": [400, 0],
      "parameters": {
        "respondWith": "json",
        "responseBody": "={{ { \"answer\": $json.choices[0].message.content, \"sources_used\": $('Format Context').item.json.source_count } }}"
      },
      "typeVersion": 1
    }
  ]
}
```

---

## Sparse Encoder Service

For production hybrid search, run a simple FastAPI service alongside n8n:

```python
# sparse_encoder.py - Deploy as Docker container
from fastapi import FastAPI
from fastembed import SparseTextEmbedding

app = FastAPI()
model = SparseTextEmbedding(model_name="Qdrant/bm25")

@app.post("/encode")
async def encode(request: dict):
    text = request["text"]
    result = list(model.embed([text]))[0]
    return {
        "indices": result.indices.tolist(),
        "values": result.values.tolist()
    }
```

Add to docker-compose.yml:
```yaml
sparse-encoder:
  build: ./sparse_encoder
  ports:
    - "8001:8001"
  networks: ['demo']
```

Then in n8n, call `http://sparse-encoder:8001/encode`.

---

## Multi-Collection RAG (Multiple Sources)

When you have separate collections for different source types, use multiple Qdrant tool nodes:

```json
{
  "qdrant-tool-slack": {
    "toolName": "slack_messages",
    "toolDescription": "Search Slack channel messages and conversations",
    "qdrantCollection": "acme-slack-messages"
  },
  "qdrant-tool-meetings": {
    "toolName": "meeting_transcripts", 
    "toolDescription": "Search meeting transcripts and call recordings",
    "qdrantCollection": "acme-fireflies-transcripts"
  },
  "qdrant-tool-docs": {
    "toolName": "company_documents",
    "toolDescription": "Search company documents, policies, and guides",
    "qdrantCollection": "acme-gdrive-documents"
  }
}
```

The AI Agent will intelligently route queries to the appropriate tool(s) based on context.

FILE:docs/RAG-RETRIEVAL.md
# RAG Retrieval Patterns in n8n + Qdrant

## Dense vs Sparse vs Hybrid Search

Understanding the three retrieval modes is fundamental to building effective RAG pipelines.

---

## Dense Search (Semantic / Vector Search)

### What It Is
Dense search converts the query and all documents into high-dimensional vectors (embeddings) using a neural model. Similarity is measured by cosine similarity or dot product between the query vector and stored vectors.

### Strengths
- Captures semantic meaning, synonyms, paraphrases
- Works across languages (multilingual models)
- Finds conceptually related content even with no keyword overlap

### Weaknesses
- Can miss exact keyword matches (especially technical terms, IDs, names)
- Embedding quality varies by domain
- Requires re-embedding if model changes

### When to Use Dense Only
- General Q&A over prose documents
- Multi-language content
- Finding concepts described differently across docs
- Meeting transcripts, Slack conversations, support tickets

### n8n Implementation

**Option A: LangChain Vector Store (simplest)**
```
[Chat Trigger]
    → [AI Agent]
         ↑ [Qdrant Vector Store: retrieve-as-tool]
              ↑ [OpenAI Embeddings: same model as ingestion]
```

**Option B: Official Qdrant Node (more control)**
```
[Webhook / Chat input]
    → [HTTP Request: OpenAI Embeddings API]
    → [Code: extract embedding array from response]
    → [Qdrant Node: Query Points]
         query: ={{ $json.embedding }}
         limit: 10
         withPayload: true
         scoreThreshold: 0.65
    → [Set: format results for LLM context]
    → [LLM: generate response with context]
```

**Score threshold guidance**:
- 0.85+ = very high confidence (tight semantic match)
- 0.70–0.85 = strong match (recommended default)
- 0.55–0.70 = moderate match (exploratory search)
- Below 0.55 = likely noise, filter out

---

## Sparse Search (Keyword / BM25)

### What It Is
Sparse search uses high-dimensional vectors where each dimension corresponds to a vocabulary term. Most dimensions are zero; non-zero values represent term frequency/importance scores. SPLADE and BM25 are the most common sparse encoder models.

### Strengths
- Excellent for exact term matching (product codes, names, technical jargon)
- Interpretable — you can see which terms drove the match
- No information loss on rare/specialized terms

### Weaknesses
- No semantic understanding — "automobile" doesn't match "car"
- Requires sparse encoder to be run at query time
- Needs collection configured with sparse vector support

### When to Use Sparse Only
- Searching by exact field values, IDs, names
- Code search (exact function/class names)
- Legal/medical documents with precise terminology
- When users type exact terms from the source data

### Collection Setup for Sparse
The collection must be created with sparse vector config:
```json
{
  "sparse_vectors": {
    "sparse": {
      "index": { "on_disk": false }
    }
  }
}
```

### n8n Implementation

Since n8n doesn't have a native sparse encoder node, use one of:

**Option A: HTTP Request to a SPLADE/BM25 API**
```
[Query input]
    → [HTTP Request: POST to sparse-encoder-service/encode]
         body: { "text": "={{ $json.query }}" }
    → [Code: extract indices + values from response]
    → [Qdrant Node: Query Points]
         using: "sparse"
         query: { indices: =..., values: =... }
```

**Option B: Code Node with simple BM25**
```javascript
// Simple BM25-style sparse encoding in n8n Code node
const text = $input.first().json.query;
const tokens = text.toLowerCase().split(/\W+/).filter(t => t.length > 2);
const termFreq = {};
tokens.forEach(t => { termFreq[t] = (termFreq[t] || 0) + 1; });

// Map terms to vocabulary indices (requires vocabulary lookup)
// For production, call a proper SPLADE API instead
const indices = Object.keys(termFreq).map(t => hashTerm(t));
const values = Object.values(termFreq).map(f => Math.log(1 + f));

return [{ json: { sparse_indices: indices, sparse_values: values } }];
```

For production, deploy a FastAPI service wrapping `fastembed` or `splade` and call it via HTTP Request node.

---

## Hybrid Search (Dense + Sparse + RRF Fusion)

### What It Is
Hybrid search runs both dense and sparse queries simultaneously, then combines their ranked result lists using Reciprocal Rank Fusion (RRF) or relative score fusion (RSF). The result is a unified ranked list that captures both semantic meaning and keyword relevance.

### Why It's Better
Hybrid search consistently outperforms either method alone across most real-world datasets. It captures:
- Semantic similarity (dense) AND exact keyword matching (sparse)
- Handles both "find me something about X concept" AND "find docs mentioning product code ABC-123"

### RRF Formula
RRF score = Σ(1 / (rank_k + k)) where k=60 is standard
This naturally de-emphasizes outliers and rewards results that rank well across both methods.

### When to Use Hybrid
- **Default recommendation** for production RAG systems
- When your data mixes prose and technical content
- Slack/Discord channels (casual language + technical terms)
- Support tickets (user language + product identifiers)
- Meeting transcripts with action items and names

### Collection Setup
Must have both dense and sparse vectors:
```json
{
  "vectors": {
    "dense": { "size": 3072, "distance": "Cosine" }
  },
  "sparse_vectors": {
    "sparse": { "index": { "on_disk": false } }
  }
}
```

At ingest time, you must store BOTH vectors per point:
```json
{
  "id": "point-uuid",
  "vector": {
    "dense": [0.12, -0.34, ...],
    "sparse": {
      "indices": [102, 5843, 921],
      "values": [0.82, 0.61, 0.44]
    }
  },
  "payload": { ... }
}
```

### n8n Hybrid Search Implementation

```
[Query input: "what did the team decide about the API redesign?"]
    → [Parallel branches]:
         Branch A: [HTTP Request: Dense Embedding API]
                       → get dense vector
         Branch B: [HTTP Request: Sparse Encoder API]  
                       → get sparse indices + values
    → [Merge: combineAll]
    → [Qdrant Node: Query Points]
         prefetch: [
           {
             query: ={{ $json.dense_vector }},
             using: "dense",
             limit: 20
           },
           {
             query: {
               indices: ={{ $json.sparse_indices }},
               values: ={{ $json.sparse_values }}
             },
             using: "sparse",
             limit: 20
           }
         ],
         query: { fusion: "rrf" },
         limit: 10,
         withPayload: true
    → [Set: format context chunks]
    → [LLM: answer with context]
```

---

## RAG Agent Pipeline (LangChain Pattern)

The simplest production RAG setup using n8n's LangChain nodes:

```
[When chat message received]
    → [AI Agent]
         ↑ [LLM: Gemini Flash / GPT-4o]
         ↑ [Window Buffer Memory: contextWindowLength=40]
         ↑ [Qdrant Vector Store: retrieve-as-tool]
              toolName: "knowledge_base"
              toolDescription: "Search for relevant information from company documents and conversations"
              topK: 15
              ↑ [OpenAI Embeddings: text-embedding-3-large]
```

**AI Agent system prompt pattern**:
```
You are a helpful assistant with access to a knowledge base of [source description].

Use the knowledge_base tool to retrieve relevant information before answering.
Always cite which source you used (document title, date, channel).
If the knowledge base doesn't contain the answer, say so clearly.
Do not hallucinate or invent information not present in the retrieved context.
```

---

## Retrieval Configuration Reference

### topK / limit Values
| Use Case | Recommended topK |
|----------|-----------------|
| Single focused question | 5–8 |
| Complex multi-part question | 10–15 |
| Summarization / synthesis | 15–25 |
| AI Agent with re-ranking | 20–30 |

### Score Thresholds (Dense Search)
| Threshold | Interpretation |
|-----------|----------------|
| 0.90+ | Near-duplicate match |
| 0.80–0.90 | Highly relevant |
| 0.70–0.80 | Relevant — good default cutoff |
| 0.60–0.70 | Loosely related |
| <0.60 | Likely noise |

### Filtered Retrieval Examples

Retrieve only from a specific Slack channel:
```json
{
  "filter": {
    "must": [
      { "key": "source_context", "match": { "value": "#engineering" } }
    ]
  }
}
```

Retrieve only recent content:
```json
{
  "filter": {
    "must": [
      { 
        "key": "created_at",
        "range": { "gte": "2024-06-01T00:00:00Z" }
      }
    ]
  }
}
```

Retrieve from multiple sources:
```json
{
  "filter": {
    "should": [
      { "key": "source_type", "match": { "value": "slack" } },
      { "key": "source_type", "match": { "value": "fireflies" } }
    ]
  }
}
```

---

## Context Assembly for LLM

After retrieval, format the chunks into a clean context block:

### Set Node Expression (context assembly)
```javascript
// In a Code node after Qdrant search
const results = $input.all();
const contextChunks = results.map((item, i) => {
  const p = item.json.payload;
  return `[i+1] Source: p.source_type | p.source_context | p.created_at?.substring(0,10)
p.text || p.content`;
}).join('\n\n---\n\n');

return [{ json: { context: contextChunks, source_count: results.length } }];
```

### LLM Prompt Pattern
```
Given the following context retrieved from our knowledge base:

{{ $json.context }}

Answer the following question:
{{ $('Chat Trigger').item.json.chatInput }}

If the context doesn't contain enough information, say "I don't have enough information about this in the knowledge base."
Always mention which sources you used.
```

---

## Query Points Groups (Diverse Results)

Prevent returning 10 chunks from the same document — get diverse results across documents:

```json
{
  "resource": "search",
  "operation": "queryPointsGroups",
  "collectionName": "my-collection",
  "query": "={{ $json.embedding }}",
  "groupBy": "doc_id",
  "groupSize": 2,
  "limit": 5
}
```
Returns top 2 chunks from each of the top 5 distinct documents — 10 results total, maximum diversity.

---

## Re-ranking (Post-Retrieval)

For high-stakes retrieval, add a re-ranking step after initial Qdrant search:

```
[Qdrant: Query Points, topK=30]
    → [HTTP Request: Cohere Re-rank API or similar]
         body: {
           query: "={{ $('Chat Trigger').item.json.chatInput }}",
           documents: "={{ $json.results.map(r => r.payload.text) }}",
           top_n: 8
         }
    → [Code: reassemble top_n results with original payloads]
    → [LLM: answer with re-ranked context]
```

Re-ranking consistently improves answer quality for complex questions and is recommended for production systems where accuracy is critical.

FILE:docs/CHUNKING-METADATA.md
# Chunking Strategy & Metadata Schema Design

## Why Metadata Matters

In Qdrant, every point has a **payload** (the metadata). Rich, consistent payload design is what separates a basic prototype from a production-ready RAG system. Good metadata enables:

- **Filtered search**: find only content from a specific channel, timeframe, or source
- **Targeted deletion**: remove all chunks for a document without knowing their IDs
- **Source attribution**: tell users where an answer came from
- **Hybrid retrieval**: combine semantic search with structured filters
- **Observability**: audit what's in your collection

---

## Universal Metadata Schema

Every point in every collection should have these fields:

```json
{
  "text": "The actual chunk text that was embedded",
  "doc_id": "Unique identifier for the parent document/record",
  "chunk_index": 0,
  "chunk_total": 5,
  "source_type": "slack | fireflies | gdrive | pdf | email | webhook",
  "source_context": "Channel name, folder path, or collection identifier",
  "source_url": "Direct link to the source record (optional)",
  "author": "Who created the content",
  "created_at": "2024-03-15T10:30:00Z",
  "ingested_at": "2024-03-15T11:00:00Z",
  "language": "en",
  "collection_version": "1"
}
```

Plus **AI-extracted enrichment metadata** (generated at ingest time):
```json
{
  "summary": "1-3 sentence summary of the full parent document",
  "keywords": ["api", "redesign", "backend", "latency"],
  "entities": {
    "people": ["Alice Chen", "Bob Martinez"],
    "companies": ["Acme Corp"],
    "products": ["PaymentAPI v2"]
  },
  "topics": ["engineering", "product-design", "performance"],
  "sentiment": "neutral",
  "action_items": ["Review API spec by Friday", "Schedule design review"],
  "overarching_theme": "API redesign discussion focusing on latency improvements"
}
```

---

## Source-Specific Metadata Schemas

### Slack Messages
```json
{
  "text": "chunk text",
  "doc_id": "C01CHANNEL-1709123456.789000",
  "source_type": "slack",
  "source_context": "#engineering",
  "author": "alice.chen",
  "created_at": "2024-03-15T10:30:00Z",
  "thread_id": "1709123400.000000",
  "is_thread_reply": false,
  "reaction_count": 3,
  "workspace": "acme-corp"
}
```

**doc_id formula for Slack**: `{channel_id}-{message_ts}` — globally unique, sortable by time.

### Fireflies Meeting Transcripts
```json
{
  "text": "chunk text from transcript segment",
  "doc_id": "fireflies-meeting-abc123",
  "source_type": "fireflies",
  "source_context": "weekly-engineering-standup",
  "created_at": "2024-03-15T09:00:00Z",
  "meeting_title": "Weekly Engineering Standup",
  "duration_minutes": 45,
  "participants": ["[email protected]", "[email protected]"],
  "meeting_type": "standup",
  "has_action_items": true
}
```

**Chunking strategy for transcripts**: Chunk by speaker turns or by fixed token windows with overlap that preserves sentence boundaries. Segment-level chunks (one exchange = one chunk) often outperform fixed token splitting for meeting data.

### Google Drive Documents
```json
{
  "text": "chunk text",
  "doc_id": "gdrive-1BxiMVs0XRA5nFMdKvBdBZjgmUUqptlbs74OgVE2upms",
  "source_type": "gdrive",
  "source_context": "/Company Docs/Engineering/",
  "author": "[email protected]",
  "created_at": "2024-02-01T00:00:00Z",
  "modified_at": "2024-03-10T14:22:00Z",
  "file_name": "API Design Guidelines.pdf",
  "file_type": "pdf",
  "file_size_bytes": 245000,
  "drive_folder": "Engineering"
}
```

### Email / Gmail
```json
{
  "text": "chunk text from email body",
  "doc_id": "gmail-18e1f2a3b4c5d6e7",
  "source_type": "email",
  "source_context": "[email protected]",
  "author": "[email protected]",
  "created_at": "2024-03-15T08:15:00Z",
  "subject": "Re: API v2 Launch Timeline",
  "thread_id": "18e1f2a3b4c5d6e7",
  "recipient_count": 5,
  "has_attachments": false
}
```

---

## Metadata Extraction with Information Extractor Node

### Node Configuration

```
[Information Extractor]
  text: ={{ $json.content }}
  model: Gemini Flash or GPT-4o-mini
  
  systemPromptTemplate: |
    You are an expert information extraction system. Extract structured
    metadata from the provided content. Return only the requested fields.
    If a field cannot be determined, omit it rather than guessing.
```

### Attributes Configuration by Use Case

#### For Slack / Chat Messages
```json
[
  { "name": "summary", "description": "1-2 sentence summary of what this message or thread is about" },
  { "name": "keywords", "description": "5-8 keywords as an array of strings" },
  { "name": "topics", "description": "Topic categories from: engineering, product, design, ops, hr, finance, general" },
  { "name": "sentiment", "description": "One of: positive, negative, neutral, frustrated, excited" },
  { "name": "has_question", "description": "Boolean: does the message contain a question?" },
  { "name": "has_decision", "description": "Boolean: does the message announce a decision?" },
  { "name": "action_items", "description": "Array of action items or tasks mentioned, empty array if none" }
]
```

#### For Meeting Transcripts
```json
[
  { "name": "overarching_theme", "description": "The main topic or purpose of this meeting segment" },
  { "name": "recurring_topics", "description": "Topics that come up repeatedly, as array of strings" },
  { "name": "pain_points", "description": "Problems or challenges mentioned, as array of strings" },
  { "name": "decisions_made", "description": "Concrete decisions or agreements reached, as array" },
  { "name": "action_items", "description": "Tasks assigned or volunteered for, include owner if mentioned" },
  { "name": "keywords", "description": "10 keywords that best represent this segment" },
  { "name": "sentiment", "description": "Overall tone: productive, tense, exploratory, conclusive" }
]
```

#### For Documents / PDFs
```json
[
  { "name": "title", "description": "Document title if not already known" },
  { "name": "summary", "description": "2-3 sentence executive summary" },
  { "name": "document_type", "description": "One of: policy, guide, report, spec, proposal, reference, other" },
  { "name": "keywords", "description": "10 most important keywords" },
  { "name": "entities", "description": "Named entities: people, companies, products, locations as nested object" },
  { "name": "topics", "description": "Topic taxonomy labels as array" },
  { "name": "target_audience", "description": "Who this document is intended for" },
  { "name": "version", "description": "Document version number if mentioned" }
]
```

---

## Flattening Metadata for Qdrant

The Information Extractor returns an `output` object. You need to flatten it for use in Data Loader metadata fields.

### Set Node Pattern (after Information Extractor)
```javascript
// Expression in Set node to flatten extracted metadata
{
  "meta_summary": "={{ $json.output.summary }}",
  "meta_keywords": "={{ $json.output.keywords }}",
  "meta_topics": "={{ $json.output.topics }}",
  "meta_sentiment": "={{ $json.output.sentiment }}",
  "meta_action_items": "={{ $json.output.action_items }}",
  "meta_entities": "={{ JSON.stringify($json.output.entities) }}"
}
```

Note: Qdrant payload supports arrays natively. Pass arrays directly for `keywords`, `topics`, `action_items`. Serialize complex nested objects to JSON string if needed.

---

## Chunking Decision Tree

```
Is the source structured with natural record boundaries?
    YES (Slack messages, emails, individual rows)
        → One chunk per record if under 512 tokens
        → If longer, split at 512 tokens with 50 token overlap
    
    NO (Long documents, transcripts, PDFs)
        → Is content conversational?
            YES (transcripts)
                → Split by speaker turn or sentence boundary
                → Max 1000 tokens per chunk, 150 token overlap
            NO (technical docs, reports)
                → Split at paragraph boundaries first
                → Max 1500 tokens per chunk, 200 token overlap
```

### Semantic Chunking (Advanced)
For highest retrieval quality, use semantic chunking: split text where the topic changes, not at fixed token boundaries. Implement via:
1. Split into sentences
2. Embed each sentence
3. Find breakpoints where cosine similarity between adjacent sentences drops below threshold
4. Group sentences into chunks at breakpoints

This requires a Code node with the embedding API. Performance improvement is 10-20% over fixed chunking but adds latency and cost.

---

## Payload Index Strategy

Always create these indexes after collection setup:

| Field | Index Type | Priority |
|-------|-----------|---------|
| `doc_id` | keyword | Critical — enables targeted deletion |
| `source_type` | keyword | High — multi-source filtering |
| `source_context` | keyword | High — channel/folder filtering |
| `created_at` | datetime | High — time-range filtering |
| `author` | keyword | Medium — per-author search |
| `topics` | keyword | Medium — topic filtering |
| `has_action_items` | bool | Low — task extraction workflows |
| `chunk_index` | integer | Low — debugging/auditing |

Create indexes via the Official Qdrant Node → Payload → Create Payload Index, or via the Qdrant REST API at collection setup time.

---

## Content Quality Filters (Pre-Ingestion)

Filter out low-value content before spending tokens on embedding:

```javascript
// Code node: filter before ingestion
const item = $input.first().json;
const text = item.content || '';

// Skip conditions
if (text.length < 20) return []; // Too short
if (text.trim().split(' ').length < 5) return []; // Less than 5 words
if (/^[\s\W]+$/.test(text)) return []; // Only whitespace/punctuation
if (text.startsWith('http') && text.split(' ').length < 3) return []; // Bare URLs

return [$input.first()];
```

This reduces noise, lowers embedding costs, and improves retrieval precision.

FILE:docs/INGESTION-PIPELINE.md
# Ingestion Pipeline Architecture

## Overview

This document covers the step-by-step architecture for building production ingestion pipelines that take large datasets (Slack channels, Fireflies transcripts, Google Drive folders, databases, APIs) and reliably ingest every record into Qdrant.

---

## Phase 1: Collection Setup (Run Once)

Before ingesting, ensure the collection exists with the correct configuration.

### Node Sequence
```
[Manual Trigger or Webhook]
    → [Qdrant: Collection Exists]
    → [IF: exists == false]
         → [Qdrant: Create Collection]
```

### Collection Config for Dense-Only
```json
{
  "vectors": {
    "size": 3072,
    "distance": "Cosine"
  }
}
```

### Collection Config for Hybrid (Dense + Sparse)
```json
{
  "vectors": {
    "dense": {
      "size": 3072,
      "distance": "Cosine"
    }
  },
  "sparse_vectors": {
    "sparse": {
      "index": {
        "on_disk": false
      }
    }
  }
}
```

### Create Payload Indexes After Collection Creation
Always create indexes on fields you'll filter by:
```
[Qdrant: Create Payload Index] → field: "doc_id", schema: "keyword"
[Qdrant: Create Payload Index] → field: "source_type", schema: "keyword"
[Qdrant: Create Payload Index] → field: "created_at", schema: "datetime"
[Qdrant: Create Payload Index] → field: "source_context", schema: "keyword"
```

---

## Phase 2: Source Data Extraction

### Slack Channel Ingestion
```
[Schedule Trigger: every 15min]
    → [Slack: Get Channel Messages] (paginate with cursor)
    → [Split In Batches: batchSize=50]
```

**Key fields to extract from Slack**:
- `ts` → use as `source_id` (unique message timestamp)
- `text` → content to embed
- `channel` → source_context
- `user` → author
- `thread_ts` → for threading context

### Fireflies Meeting Transcript Ingestion
```
[Webhook or Schedule Trigger]
    → [HTTP Request: GET /transcripts/{id}]
    → [Split In Batches: batchSize=1]
```

**Key fields from Fireflies**:
- `id` → doc_id
- `title` → document title
- `date` → created_at
- `sentences[]` → iterate per sentence or concatenate into segments
- `participants[]` → metadata array
- `topics[]` → pre-built topic tags

### Google Drive Folder Ingestion
```
[Manual Trigger or Schedule]
    → [Google Drive: List Files in Folder]
    → [Split In Batches: batchSize=5]
    → [Google Drive: Download File]
    → [Extract From File: text]
```

### Generic API / Database Ingestion
```
[Trigger]
    → [HTTP Request / DB Node: fetch records]
    → [Split In Batches: batchSize=20-100]
```

---

## Phase 3: Content Normalization

Use a **Set node** to normalize your source data into a consistent schema before metadata extraction and embedding.

### Standard Normalized Schema
```javascript
// Set node expressions
{
  "content": "={{ $json.text || $json.body || $json.transcript }}",
  "doc_id": "={{ $json.id || $json.ts || $json.fileId }}",
  "source_type": "slack",  // hardcode per pipeline
  "source_context": "={{ $json.channel || $json.folder_name }}",
  "author": "={{ $json.user || $json.speaker }}",
  "created_at": "={{ $json.ts ? new Date($json.ts * 1000).toISOString() : $json.date }}",
  "title": "={{ $json.title || $json.text?.substring(0, 100) }}"
}
```

**Deduplication check** (optional but recommended for re-runs):
```
[Set: normalize]
    → [Qdrant: Count Points with filter doc_id == current_id]
    → [IF: count > 0]
         → SKIP (continue to next batch item)
         → PROCESS (continue ingestion)
```

---

## Phase 4: AI Metadata Extraction

Run **before chunking** — extract metadata from the full document, then attach to every chunk.

### Information Extractor Node Config

**Input**: `={{ $json.content }}` (the full document text)

**Attributes to extract** (adapt per source type):

```json
[
  {
    "name": "summary",
    "description": "1-3 sentence summary of the content"
  },
  {
    "name": "keywords",
    "description": "Array of 5-10 keywords capturing the main topics"
  },
  {
    "name": "entities",
    "description": "Named entities mentioned: people, companies, products, locations"
  },
  {
    "name": "sentiment",
    "description": "Overall sentiment: positive, negative, neutral, mixed"
  },
  {
    "name": "topics",
    "description": "Array of topic categories from a standard taxonomy"
  },
  {
    "name": "action_items",
    "description": "Any action items, decisions, or follow-ups mentioned"
  },
  {
    "name": "language",
    "description": "ISO 639-1 language code of the content"
  }
]
```

**LLM to use**: Gemini Flash or GPT-4o-mini (fast, cost-efficient for extraction tasks)

### Merge Metadata Back

After extraction, use a **Merge node** (combineByPosition or combineAll) to join the original normalized record with the extracted metadata before passing to the chunking stage.

```
[Set: normalize] ─────────────────────────────────── [Merge: combineByPosition]
[Information Extractor] → [Set: flatten output] ──── [Merge]
                                                          ↓
                                              [Text Splitter + Embedding]
```

---

## Phase 5: Text Splitting

### Using LangChain Data Loader + Token Splitter

Connect in this order:
```
[Qdrant Vector Store (insert mode)]
    ↑ ai_document
[Data Loader (documentDefaultDataLoader)]
    ↑ ai_textSplitter
[Token Splitter]
    
[Data Loader parameters]
  dataType: "json" or "binary"
  binaryMode: "specificField" (for file downloads)
  
[Data Loader metadata] — inject all metadata fields here:
  file_id: ={{ $json.doc_id }}
  source_type: ={{ $json.source_type }}
  source_context: ={{ $json.source_context }}
  created_at: ={{ $json.created_at }}
  keywords: ={{ $json.metadata_extraction.keywords }}
  summary: ={{ $json.metadata_extraction.summary }}
  entities: ={{ $json.metadata_extraction.entities }}
  topics: ={{ $json.metadata_extraction.topics }}
```

### Chunk Size Guidelines by Source Type

| Source | chunkSize (tokens) | chunkOverlap | Rationale |
|--------|--------------------|--------------|-----------|
| Slack messages | 256 | 25 | Messages are already short; 1 message = 1-2 chunks |
| Fireflies sentences | 512 | 50 | Preserve sentence-level context |
| Meeting full transcript | 1500 | 200 | Capture full thought segments |
| PDF documents | 1000 | 150 | Balance context vs precision |
| Google Docs | 800 | 100 | Section-level granularity |
| Code files | 512 | 100 | Function-level chunks |
| Email threads | 600 | 75 | Per email + some context |

---

## Phase 6: Embedding + Upsert

### LangChain Path (Recommended for Most Cases)
```
[Data Loader] → [Qdrant Vector Store: insert]
                    ↑ ai_embedding
               [OpenAI Embeddings: text-embedding-3-large]
```

The LangChain Vector Store handles embedding + upsert atomically.

### Direct Upsert Path (When You Need Full Payload Control)
```
[Code Node: generate UUID for each chunk]
    → [HTTP Request: POST to embedding API]
    → [Code Node: parse embedding array]
    → [Qdrant Node: Upsert Points]
```

Use this path when:
- You need to compute sparse vectors alongside dense
- You have pre-computed embeddings from an external system
- You need to store custom vector names for multi-vector collections

---

## Phase 7: Rate Limiting & Flow Control

### For Large Datasets (10k+ records)

```
[Source]
    → [Split In Batches: batchSize=25]
    → [... process batch ...]
    → [Qdrant: Upsert]
    → [Wait: 1-2 seconds]  ← critical for API rate limits
    → [back to Split In Batches]
```

### Wait Node Configuration
- `resumeUnit`: `seconds`
- `resumeAmount`: 1–5 (tune based on API plan)
- Set on the Qdrant Vector Store node: `onError: continueRegularOutput` + `retryOnFail: true`

### Error Handling Pattern
```
[Qdrant Node]
    → main output: continue to Wait → Loop
    → error output (if onError=continueErrorOutput): 
          → [Set: log error record]
          → [Google Sheets / Slack: report failed record]
          → continue to Wait → Loop
```

---

## Phase 8: Completion Notification

Always send a completion signal after large ingestion jobs:

```
[Split In Batches: "done" output]
    → [Telegram / Slack: "Ingestion complete: X records processed"]
```

---

## Full Pipeline: Slack Channel to Qdrant

```
[Schedule: every hour]
    → [Slack: getMessages channel=#engineering, limit=200]
    → [IF: message not already indexed] (check via Qdrant Count Points)
    → [Split In Batches: 20]
    → [Set: normalize fields]
    → [Information Extractor: keywords, summary, sentiment]
    → [Merge: normalized + metadata]
    → [Qdrant Vector Store: insert mode]
         ↑ Token Splitter (512 tokens, 50 overlap)
         ↑ Data Loader (json, with metadata injection)
         ↑ OpenAI Embeddings (text-embedding-3-large)
    → [Wait: 1s]
    → [loop back]
    → [Slack: notify #ops "Ingestion complete"]
```

---

## Deletion Pipeline (With Safety Gate)

When source records are deleted or updated, remove old vectors:

```
[Webhook: delete event from source system]
    → [Set: extract doc_id from payload]
    → [Telegram: sendAndWait "Delete X vectors for doc_id Y? [Approve/Decline]"]
    → [IF: approved == true]
         → [Qdrant Node: Delete Points, filter: doc_id == Y]
         → [Telegram: "Deletion complete"]
    → [IF: declined]
         → [Telegram: "Deletion cancelled"]
```

For automated re-ingestion (update = delete + re-ingest):
```
[Source update event]
    → [Qdrant: Delete Points by doc_id]
    → [continue to ingestion pipeline with new content]
```

FILE:docs/NODE-REFERENCE.md
# n8n Qdrant Node Reference

## Official Qdrant Node (`n8n-nodes-qdrant`)

Install via: n8n Settings → Community Nodes → `n8n-nodes-qdrant`

Compatible with Qdrant 1.14.0+

---

## Collection Operations

### Create Collection
**When to use**: First-time setup or automated provisioning before ingestion.

```json
{
  "resource": "collection",
  "operation": "createCollection",
  "collectionName": "my-collection",
  "vectorsConfig": {
    "size": 3072,
    "distance": "Cosine"
  }
}
```

**Vector sizes by model**:
| Model | Size |
|-------|------|
| text-embedding-3-large | 3072 |
| text-embedding-3-small | 1536 |
| text-embedding-ada-002 | 1536 |
| Gemini text-embedding-004 | 768 |
| nomic-embed-text | 768 |

**Distance metrics**:
- `Cosine` — Best for normalized text embeddings (most common)
- `Dot` — Use when vectors are not normalized (faster)
- `Euclid` — Use for geometric/spatial data

### Collection Exists
**When to use**: Guard before Create Collection in automated workflows.

```json
{
  "resource": "collection",
  "operation": "collectionExists",
  "collectionName": "={{ $json.collection_name }}"
}
```
Returns `{ exists: true/false }` — branch with IF node.

### List Collections
**When to use**: Audit, UI dropdowns, dynamic routing.

### Delete Collection
**When to use**: Full reset / re-ingestion. Always gate with approval workflow.

### Get Collection
**When to use**: Check vector count, config, status before operations.

---

## Point Operations

### Upsert Points
**When to use**: Raw ingestion when you pre-compute embeddings outside n8n, or need full control over payload structure.

```json
{
  "resource": "point",
  "operation": "upsertPoints",
  "collectionName": "my-collection",
  "points": [
    {
      "id": "={{ $json.point_id }}",
      "vector": "={{ $json.embedding }}",
      "payload": {
        "text": "={{ $json.chunk_text }}",
        "source_id": "={{ $json.source_id }}",
        "doc_id": "={{ $json.doc_id }}",
        "chunk_index": "={{ $json.chunk_index }}",
        "source_type": "slack",
        "created_at": "={{ $json.timestamp }}",
        "keywords": "={{ $json.keywords }}"
      }
    }
  ]
}
```

**ID formats**: UUID string or unsigned integer. Use UUID for distributed systems.

### Delete Points
**When to use**: Remove all chunks belonging to a document when source is updated or deleted.

```json
{
  "resource": "point",
  "operation": "deletePoints",
  "collectionName": "my-collection",
  "filter": {
    "must": [
      {
        "key": "doc_id",
        "match": { "value": "={{ $json.doc_id }}" }
      }
    ]
  }
}
```

**Filter operators**: `must` (AND), `should` (OR), `must_not` (NOT)
**Match types**: `match.value` (exact), `range` (numeric), `match.any` (array membership)

### Scroll Points
**When to use**: Iterate all points for export, audit, re-embedding, or bulk update.

```json
{
  "resource": "point",
  "operation": "scrollPoints",
  "collectionName": "my-collection",
  "limit": 100,
  "withPayload": true,
  "withVector": false,
  "filter": {
    "must": [{ "key": "source_type", "match": { "value": "slack" } }]
  }
}
```

Use `offset` parameter with Loop to paginate through full collection.

### Count Points
**When to use**: Check ingestion progress, collection health monitoring.

### Retrieve Points
**When to use**: Fetch specific points by ID for lookup, deduplication checks.

### Batch Update Points
**When to use**: High-throughput ingestion — combine multiple upsert/delete/payload operations in one API call. Reduces round trips significantly for bulk loads.

---

## Search Operations

### Query Points (Core search operation)
**When to use**: All retrieval — dense, sparse, or hybrid.

#### Dense (semantic) search
```json
{
  "resource": "search",
  "operation": "queryPoints",
  "collectionName": "my-collection",
  "query": "={{ $json.query_embedding }}",
  "limit": 10,
  "withPayload": true,
  "scoreThreshold": 0.7
}
```

#### Sparse (keyword/BM25) search
Requires collection created with sparse vector config. Uses keyword-frequency vectors.
```json
{
  "resource": "search",
  "operation": "queryPoints",
  "collectionName": "my-collection",
  "using": "sparse",
  "query": {
    "indices": [102, 5843, 921],
    "values": [0.82, 0.61, 0.44]
  },
  "limit": 10
}
```
Sparse vectors must be computed via a sparse encoder model (e.g. SPLADE, BM25). In n8n, use an HTTP Request node to call a sparse encoder API, or use Code node to compute BM25 weights.

#### Hybrid search (dense + sparse with RRF fusion)
```json
{
  "resource": "search",
  "operation": "queryPoints",
  "collectionName": "my-collection",
  "prefetch": [
    {
      "query": "={{ $json.dense_embedding }}",
      "using": "dense",
      "limit": 20
    },
    {
      "query": {
        "indices": "={{ $json.sparse_indices }}",
        "values": "={{ $json.sparse_values }}"
      },
      "using": "sparse",
      "limit": 20
    }
  ],
  "query": { "fusion": "rrf" },
  "limit": 10,
  "withPayload": true
}
```

#### Filtered search
Add `filter` to any search to narrow results:
```json
{
  "filter": {
    "must": [
      { "key": "source_type", "match": { "value": "slack" } },
      { "key": "created_at", "range": { "gte": "2024-01-01T00:00:00Z" } }
    ]
  }
}
```

### Query Points In Batch
**When to use**: When an AI agent needs to run multiple searches simultaneously (e.g. multiple sub-questions). More efficient than sequential calls.

### Query Points Groups
**When to use**: When you want top-K results per document (avoid returning 5 chunks from the same doc). Set `groupBy: "doc_id"`, `groupSize: 2`.

---

## Payload Operations

### Set Payload
**When to use**: Enrich existing points post-ingestion (e.g. add classification labels, tags).

### Create Payload Index
**When to use**: Speed up filtered searches. Create indexes on frequently-filtered fields.

```json
{
  "resource": "payload",
  "operation": "createPayloadIndex",
  "collectionName": "my-collection",
  "fieldName": "source_type",
  "fieldSchema": "keyword"
}
```

**Schema types**: `keyword`, `integer`, `float`, `bool`, `text`, `datetime`, `uuid`

Always index: `doc_id`, `source_id`, `source_type`, `created_at`, `chunk_index`

---

## LangChain Vector Store Node (`@n8n/n8n-nodes-langchain.vectorStoreQdrant`)

### Mode: insert
Receives documents from upstream Document Loaders/Text Splitters and upserts into Qdrant.

**Required sub-nodes**:
- Embeddings node (connected via `ai_embedding` port)
- Text Splitter (connected via `ai_textSplitter` port on the Data Loader)
- Data Loader (connected via `ai_document` port)

**Key settings**:
- `qdrantCollection`: collection name or expression
- `mode`: `insert`
- `options.metadata`: add custom metadata fields to every chunk

**Always set**:
- `onError: continueRegularOutput` — prevents failures from stopping the batch
- `retryOnFail: true` — retry transient errors
- `executeOnce: false` — process each item individually

### Mode: retrieve
Direct similarity search, returns documents. Used in pure chain workflows (no agent).

### Mode: retrieve-as-tool
Exposes the vector store as a callable tool for AI Agent nodes.

**Key settings**:
- `toolName`: snake_case name the LLM will reference (e.g. `slack_messages`)
- `toolDescription`: natural language description for the LLM
- `topK`: number of results to return (10–30 for most use cases)

**Required sub-node**: Embeddings node (same model used at ingest time)

---

## Embeddings Nodes

| Node | Model | Best For |
|------|-------|---------|
| OpenAI Embeddings | text-embedding-3-large (3072d) | Best quality, production default |
| OpenAI Embeddings | text-embedding-3-small (1536d) | Cost-efficient, good quality |
| Google Gemini Embeddings | text-embedding-004 (768d) | When Gemini is primary LLM |
| Ollama Embeddings | nomic-embed-text | Fully local, no API cost |

**Critical**: Use the **exact same embedding model** at ingest time and retrieval time. Mismatches produce nonsense results.

---

## Text Splitter Nodes

### Token Splitter (`@n8n/n8n-nodes-langchain.textSplitterTokenSplitter`)
Splits by token count. Most reliable for LLM context windows.

| Use Case | chunkSize | chunkOverlap |
|----------|-----------|--------------|
| Slack messages | 256–512 | 50 |
| Meeting transcripts | 1000–2000 | 200 |
| Documents/PDFs | 512–1500 | 100–200 |
| Code files | 512 | 100 |
| Short records | 128–256 | 25 |

### Recursive Character Splitter
Splits on natural boundaries (paragraphs → sentences → words). Better for structured prose.

**Rule**: chunk overlap should be 10–15% of chunk size.

---

## Information Extractor Node (`@n8n/n8n-nodes-langchain.informationExtractor`)
Use this to generate rich metadata from full document text before chunking.

**Pattern**:
1. Extract full document text (Extract From File node or HTTP response)
2. Run through Information Extractor with structured attributes
3. Store output, pass into Data Loader metadata fields
4. Then chunk + embed + ingest

See `docs/CHUNKING-METADATA.md` for full metadata schema templates.

FILE:docs/TROUBLESHOOTING.md
# Troubleshooting & Best Practices

## Common Errors and Fixes

### "Collection not found"
- Run the collection setup workflow first
- Check collection name spelling (case-sensitive)
- Verify Qdrant credentials point to correct instance

### "Vector dimension mismatch"
- You're using a different embedding model than the collection was created with
- Fix: delete and recreate collection, or use `Update Collection` to change vector config
- Prevention: always store embedding model name in a workflow variable/env

### "Request timeout" on large batches
- Reduce `batchSize` in Split In Batches (try 10 or 5)
- Increase `Wait` node duration
- Check Qdrant Cloud plan limits

### "Embedding model not found" (OpenAI)
- Model name changed — use `text-embedding-3-large` not `text-embedding-ada-002`
- Check OpenAI API key has access

### Retrieved results are irrelevant
- Check you're using the same embedding model at ingest AND retrieval
- Check score threshold — too low allows noise
- Check collection has expected number of vectors (use Get Collection)
- Try increasing topK and see if better results appear further down

### Information Extractor returns empty fields
- Input text too short — needs at least a few sentences for good extraction
- LLM temperature should be 0 for extraction tasks
- Check LLM node is properly connected via `ai_languageModel` port

### Loop not terminating
- Split In Batches "done" output wasn't connected
- Wait node not connected back to Split In Batches input
- Check connections on both outputs of Split In Batches node

---

## Performance Optimization

### Large Dataset Ingestion (100k+ records)
1. Use `Batch Update Points` (Official Qdrant Node) instead of individual upserts — batches of 100 points per call
2. Pre-compute all embeddings via OpenAI Batch API (50% cost reduction, async)
3. Increase `batchSize` in Split In Batches to 50–100 when using Batch Update Points
4. Use n8n's queue mode for memory-intensive workflows

### Retrieval Latency
1. Ensure payload indexes exist on all filtered fields
2. Set `hnsw_ef` at query time for speed/accuracy tradeoff: lower = faster but less accurate
3. Use `Query Points Groups` when you expect many chunks from same document
4. Enable `on_disk: true` for large collections to reduce memory pressure

### Cost Optimization
1. Use `text-embedding-3-small` instead of large (1536d vs 3072d, ~5x cheaper) — quality difference is small for most use cases
2. Filter content before embedding (remove short/empty records)
3. Deduplication check before re-ingesting already-indexed records
4. Use OpenAI Batch API for large one-time ingestion jobs

---

## Workflow Design Best Practices

### Error Resilience
```
Always set on the Qdrant Vector Store node:
  onError: "continueRegularOutput"
  retryOnFail: true
  maxTries: 3
  waitBetweenTries: 1000 (ms)
```

### Idempotency
Design ingestion workflows to be safe to re-run:
1. Check if `doc_id` already exists before ingesting
2. If exists, delete old vectors then re-ingest (for update workflows)
3. Use deterministic IDs (never random UUIDs) based on source record IDs

### Observability
Log ingestion progress to a tracking sheet:
```
[After each batch]
    → [Google Sheets: append row]
         - timestamp
         - batch_number  
         - records_processed
         - collection_name
         - status (success/error)
```

### Testing
Always test with a small dataset first:
1. Set `batchSize: 1` and `returnAll: false` with `limit: 3` on source node
2. Verify payload in Qdrant Cloud UI or via Scroll Points
3. Test retrieval with known queries before scaling up

---

## Security Best Practices

1. **Never hardcode API keys** — use n8n credentials or `$env.YOUR_KEY`
2. **Scope Qdrant API keys** — use read-only keys for retrieval workflows
3. **Always confirm destructive operations** — use sendAndWait for deletions
4. **Audit log all deletions** — write to Google Sheets or Slack before executing
5. **Use separate collections per environment** — `prod-slack-messages` vs `dev-slack-messages`

---

## External References

- Qdrant Official n8n Node: https://github.com/qdrant/n8n-nodes-qdrant
- Qdrant n8n Platform Docs: https://qdrant.tech/documentation/platforms/n8n
- Qdrant n8n Tutorial: https://qdrant.tech/documentation/tutorials-build-essentials/qdrant-n8n/
- n8n Qdrant Vector Store Docs: https://docs.n8n.io/integrations/builtin/cluster-nodes/root-nodes/n8n-nodes-langchain.vectorstoreqdrant
- Qdrant API Reference: https://api.qdrant.tech/api-reference
- n8n Self-hosted AI Starter Kit: https://github.com/n8n-io/self-hosted-ai-starter-kit
- FastEmbed (sparse encoding): https://github.com/qdrant/fastembed

FILE:docs/workflows/collection-setup.md
# Collection Setup & Management Workflows

## One-Time Collection Provisioning Workflow

Run this ONCE before starting any ingestion. Idempotent — safe to re-run.

```
[Manual Trigger]
    → [Set: collection config]
    → [Qdrant: Collection Exists?]
    → [IF: exists == false]
         → [Qdrant: Create Collection]
         → [Create 5 Payload Indexes in parallel]
              - doc_id (keyword)
              - source_type (keyword)  
              - source_context (keyword)
              - created_at (datetime)
              - author (keyword)
         → [Notification: "Collection ready"]
    → [IF: exists == true]
         → [Notification: "Collection already exists, skipping"]
```

## Collection Config Reference

### Dense-Only Collection (Simple Setup)
Best for: single-language text, when you don't need keyword precision
```json
{
  "vectors": {
    "size": 3072,
    "distance": "Cosine",
    "on_disk": false,
    "hnsw_config": {
      "m": 16,
      "ef_construct": 100
    }
  },
  "optimizers_config": {
    "default_segment_number": 5
  }
}
```

### Hybrid Collection (Dense + Sparse)
Best for: production, mixed content, technical terminology
```json
{
  "vectors": {
    "dense": {
      "size": 3072,
      "distance": "Cosine"
    }
  },
  "sparse_vectors": {
    "sparse": {
      "index": {
        "on_disk": false,
        "full_scan_threshold": 5000
      }
    }
  }
}
```

### Multi-Vector Collection (Dense + Small Dense)
Best for: when you want fast approximate search + precise rerank
```json
{
  "vectors": {
    "dense-large": { "size": 3072, "distance": "Cosine" },
    "dense-small": { "size": 1536, "distance": "Cosine" }
  }
}
```

## Payload Index Creation (Official Qdrant Node)

Run after collection creation. Each is a separate node execution:

```json
[
  { "fieldName": "doc_id", "fieldSchema": "keyword" },
  { "fieldName": "source_type", "fieldSchema": "keyword" },
  { "fieldName": "source_context", "fieldSchema": "keyword" },
  { "fieldName": "created_at", "fieldSchema": "datetime" },
  { "fieldName": "author", "fieldSchema": "keyword" },
  { "fieldName": "chunk_index", "fieldSchema": "integer" },
  { "fieldName": "meta_topics", "fieldSchema": "keyword" },
  { "fieldName": "meta_sentiment", "fieldSchema": "keyword" }
]
```

## Collection Health Check Workflow

Use on schedule (daily) to monitor collection state:

```
[Schedule: daily]
    → [Qdrant: Get Collection]
    → [Code: parse stats]
    → [IF: vector_count > expected_minimum]
         → [Notification: "✅ Collection healthy: X vectors"]
    → [IF: status != "green"]
         → [Alert: "⚠️ Collection status: {{ $json.status }}"]
```

Parse from Get Collection response:
- `result.vectors_count` — total vector count
- `result.status` — `green`, `yellow`, `red`
- `result.optimizer_status` — optimization state

## Backup / Export Workflow

Periodically export collection for backup:

```
[Schedule: weekly]
    → [Qdrant: Scroll Points, limit=100, withPayload=true]
    → [Loop until offset is null]
    → [Aggregate: collect all points]
    → [Google Drive: Write JSON backup file]
```

Scroll pagination pattern:
```javascript
// Code node: handle scroll pagination
const response = $input.first().json;
const points = response.result?.points || [];
const nextOffset = response.result?.next_page_offset;

return [{
  json: {
    points,
    next_offset: nextOffset,
    has_more: nextOffset !== null
  }
}];
```

Connect `has_more == true` back to Scroll Points with `offset: ={{ $json.next_offset }}`.

---
name: qdrant-ingestion-best-practices
description: Use this skill whenever building, designing, or debugging a RAG pipeline using Qdrant as the vector store. Covers ingestion pipelines, chunking standards, metadata schema design, hybrid dense+sparse retrieval with RRF, access control patterns, embedding model selection (BGE-M3 for hybrid, text-embedding-3-small for dense-only), collection architecture, normalization, deduplication, idempotency, and operational standards. Triggers include: any mention of Qdrant, RAG pipeline, vector ingestion, chunking, embeddings, hybrid search, payload filters, or access-controlled retrieval.
---

# Qdrant Ingestion Best Practices

## Overview

This skill package provides comprehensive, production-grade guidance for building RAG (Retrieval-Augmented Generation) pipelines using Qdrant as the vector store. It covers everything from data ingestion and chunking to hybrid retrieval, metadata standards, and access control patterns.

All detailed guidance lives in the `guides/` subfolder. **Always read the relevant guide(s) before writing code or designing a pipeline.** Use the Quick Decision Guide below to determine which guides to load.

---

## Skill Structure

| Guide | Path | When to Read |
|---|---|---|
| **RAG Pipeline Overview** | `guides/01-rag-pipeline-overview.md` | Start here. Architecture, decisions, model selection. |
| **Metadata Schema Standards** | `guides/02-metadata-schema.md` | Designing chunk payloads and payload index strategy. |
| **Data Classification & Collections** | `guides/03-data-classification.md` | Multi-collection design, sensitivity tiers, tenancy. |
| **Source Normalization** | `guides/04-source-normalization.md` | Pre-processing rules by source type before chunking. |
| **Chunking Standards** | `guides/05-chunking-standards.md` | Chunk size, overlap, strategy by content type. |
| **Embedding Models** | `guides/06-embedding-models.md` | Dense vs hybrid model selection and configuration. |
| **Ingestion Pipeline** | `guides/07-ingestion-pipeline.md` | Full pipeline steps, idempotency, upsert patterns. |
| **Retrieval Architecture** | `guides/08-retrieval-architecture.md` | Hybrid search, RRF, reranking, filter application. |
| **Access Control Patterns** | `guides/09-access-control.md` | Payload-based filtering, separation of concerns. |
| **Operational Standards** | `guides/10-operational-standards.md` | Lifecycle, retention, observability, conformance. |
| **Quick Reference** | `QUICK_REFERENCE.md` | Cheat sheet: model dims, chunk sizes, RRF params. |

---

## Quick Decision Guide

### What embedding model should I use?
Read `guides/06-embedding-models.md` for full details. Quick answer:

```
Need hybrid (semantic + keyword)?
  → BAAI/BGE-M3  (dense 1024-dim + SPLADE sparse, single model pass)

Need dense-only (simpler pipeline)?
  → text-embedding-3-small  (OpenAI, 1536-dim, cost-efficient)
  → text-embedding-3-large  (OpenAI, 3072-dim, highest quality dense)
```

### What chunking strategy should I use?
Read `guides/05-chunking-standards.md` for full code. Quick answer:

```
Conversational (Slack, short messages) → 150–300 tokens, 30 overlap, sentence window
Email threads                          → Split at reply boundary first, then 200–400 tokens
Meeting transcripts                    → Split at speaker turns, 200 tokens, 20 overlap
Documents / PDFs                       → Hierarchical paragraph, 300–500 tokens, 50 overlap
Tasks / Tickets                        → One chunk per task, max 512 tokens
```

### How many collections do I need?
Read `guides/03-data-classification.md`. Justify collections by: security boundary, query pattern, scale/index tuning, or lifecycle difference. Do NOT create a collection per data source. Standard setup = 3 collections: `company_memory`, `restricted_memory`, `pii_memory`.

### Building from scratch?
Read guides in this order: `01 → 02 → 03 → 06 → 07 → 08`

### Improving retrieval quality?
Read: `08 → 05 → 06 → 10`

### Adding access control?
Read: `09 → 03 → 02` (focus on governance fields)

### Onboarding a new data source?
Read: `04 → 05 → 02 → 07`

### Debugging ingestion or metadata issues?
Read: `07 → 10 → 02`

---

## 10 Rules — Never Violate These

1. **Classify sensitivity at ingest time.** Never defer to retrieval time.
2. **Apply access control filters inside the Qdrant query.** Never post-filter retrieved results.
3. **Never embed agent or user permission lists in chunk payloads.** Permissions belong in your orchestration layer config only.
4. **Chunking must be deterministic.** Same normalized input → same chunks → same hashes → same doc_ids.
5. **All writes to Qdrant must use upsert semantics.** Raw inserts are prohibited in pipelines.
6. **Compute all hashes from normalized content.** Never from raw source payloads.
7. **Index every field used as a payload filter.** Unindexed filter fields cause full collection scans.
8. **Never fabricate metadata.** If a value cannot be determined, use empty array or null.
9. **`model_inferred` fields must not be sole basis for security decisions.**
10. **Use the same embedding model at query time as at ingest time.** Mixing models produces meaningless scores.

---

## Mandatory Pipeline Stage Order

Every ingestion pipeline must execute these stages in this exact order:

```
1. Source capture        — fetch raw content + metadata from source API
2. Normalization         — apply universal + source-specific rules  (→ guide 04)
3. Document hash         — SHA-256 of full normalized document text
4. Change detection      — compare hash to stored hash; skip steps 5–8 if unchanged
5. Chunking              — apply strategy for content type              (→ guide 05)
6. Chunk hashing         — SHA-256 per chunk from normalized chunk text
7. Embedding             — dense ± sparse vectors                       (→ guide 06)
8. Upsert to Qdrant      — full metadata payload                        (→ guide 07)
9. Stale chunk cleanup   — delete chunks whose chunk_index is now out of range
```

---

## Sensitivity Tiers → Collections

| Tier | Examples | Collection |
|---|---|---|
| `public` | Marketing, public docs | `company_memory` |
| `internal` | Slack, all-hands, project docs | `company_memory` |
| `restricted` | Executive email, finance, legal | `restricted_memory` |
| `confidential` | Salary, PII, health records | `pii_memory` |

Default when nothing matches: **`internal`**

---

## Retrieval Pipeline Summary

```
Query → Embed (BGE-M3: dense + sparse in one pass)
     → Dense search top-20  ─┐
     → Sparse search top-20  ─┤  (access filters applied inside each branch)
                              ▼
                         RRF fusion (k=60)
                              ▼
                         Top 10–15 results
                              ▼
                    Optional: cross-encoder rerank → top 5–8
                              ▼
                         Return with attribution
```

See `guides/08-retrieval-architecture.md` for full implementation code.

FILE:QUICK_REFERENCE.md
# Quick Reference — Qdrant Ingestion Best Practices

## Embedding Model Cheat Sheet

| Need | Model | Dim | Notes |
|---|---|---|---|
| Hybrid (default) | `BAAI/bge-m3` | 1024 dense + sparse | One pass, dense + SPLADE sparse |
| Dense-only (efficient) | `text-embedding-3-small` | 1536 | OpenAI API, cost-efficient |
| Dense-only (quality) | `text-embedding-3-large` | 3072 | OpenAI API, highest quality |
| Sparse companion | Qdrant BM25 `modifier=IDF` | sparse | Native, no extra model |
| Sparse companion (quality) | `Qdrant/bm42-all-minilm-l6-v2-attentions` | sparse | Attention-weighted, better than BM25 |

## Chunking Parameters Cheat Sheet

| Content Type | Tokens | Overlap | Strategy |
|---|---|---|---|
| Slack / short messages | 150–300 | 30 | Sentence window |
| Email threads | 200–400 | 30 | Reply-boundary split first |
| Meeting transcripts | 150–250 | 20 | Speaker-turn split first |
| Documents / PDFs | 300–500 | 50 | Hierarchical paragraph |
| Tasks / Tickets | ≤ 512 | N/A | One chunk per task |
| Code | 200–400 | 50 | Function/class boundary-aware |

**Token estimation:** 4 characters ≈ 1 token (deterministic heuristic, no tokenizer needed).

## Sensitivity Tiers

| Tier | Examples | Collection |
|---|---|---|
| `public` | Marketing, public docs | `company_memory` |
| `internal` | Slack, all-hands, project docs | `company_memory` |
| `restricted` | Executive email, finance, legal | `restricted_memory` |
| `confidential` | Salary, PII, health records | `pii_memory` |

**Default:** `internal` when nothing matches.

## RRF Parameters

```
Dense prefetch: top-20
Sparse prefetch: top-20
RRF constant k: 60
Final results: 10–15
After reranking: 5–8
```

## Required Payload Indexes

```python
# Always create these indexes on every collection
fields_to_index = [
    ("sensitivity", "keyword"),
    ("org_id", "keyword", is_tenant=True),
    ("workspace_id", "keyword"),
    ("allowed_groups", "keyword"),
    ("data_scope_tags", "keyword"),
    ("team_ids", "keyword"),
    ("source_type", "keyword"),
    ("is_pii", "bool"),
    ("created_at", "datetime"),
    ("content_type", "keyword"),
]
```

## 10 Rules to Never Break

1. Classify sensitivity at ingest time — never at retrieval time
2. Apply access control filters inside the Qdrant query — never post-retrieval
3. Never store agent IDs or agent permission lists in chunk payloads
4. Chunking must be deterministic — same input always produces same chunks
5. All Qdrant writes must use upsert semantics — never raw insert
6. Compute all hashes from normalized content — never from raw source
7. Index every field used as a payload filter
8. Never fabricate metadata — use empty array or null for unknown values
9. `model_inferred` fields must not be sole basis for security decisions
10. Same embedding model must be used at both ingest time and query time

## Pipeline Stage Order

```
1. Source capture
2. Normalization
3. Document hash (SHA-256 of full normalized text)
4. Change detection (compare to stored hash)
5. Chunking
6. Chunk hashing (SHA-256 per chunk)
7. Embedding (dense ± sparse)
8. Upsert to Qdrant
9. Delete stale chunks (if re-ingestion produced fewer chunks)
```

## Scope Tags Taxonomy

| Prefix | Example | When |
|---|---|---|
| None (domain) | `sales`, `finance`, `hr` | Content is in this business domain |
| `team:` | `team:sales_a` | Content scoped to a specific team |
| `category:` | `category:pipeline` | Specific data category |
| `region:` | `region:emea` | Regional scope |
| `project:` | `project:alpha` | Specific project |

Tags must be: lowercase_snake_case, source_asserted or system_derived, stable, from the approved list.

## Guide Index

| Guide | Topic |
|---|---|
| `guides/01-rag-pipeline-overview.md` | Architecture, principles, decision guide |
| `guides/02-metadata-schema.md` | Full payload schema, all fields, indexes |
| `guides/03-data-classification.md` | Sensitivity tiers, collection setup, HNSW tuning |
| `guides/04-source-normalization.md` | Normalization rules by source type |
| `guides/05-chunking-standards.md` | Chunk sizes, strategies, code examples |
| `guides/06-embedding-models.md` | Model selection, BGE-M3, OpenAI, sparse |
| `guides/07-ingestion-pipeline.md` | Full pipeline code, lifecycle, idempotency |
| `guides/08-retrieval-architecture.md` | Hybrid search, RRF, reranking, filters |
| `guides/09-access-control.md` | Separation of concerns, permission flow, anti-patterns |
| `guides/10-operational-standards.md` | Schema versioning, retention, monitoring, conformance |

FILE:guides/01-rag-pipeline-overview.md
# Guide 01 — RAG Pipeline Overview

## Purpose

This guide describes the end-to-end architecture of a production RAG pipeline using Qdrant. It defines the two halves of the system — **ingestion** and **retrieval** — and establishes the architectural principles that all other guides build on.

---

## The Two Halves of a RAG Pipeline

```
INGESTION SIDE                          RETRIEVAL SIDE
─────────────────────────────────       ──────────────────────────────────
Source System                           User / Agent Query
    │                                       │
    ▼                                       ▼
Source Capture (API / connector)        Query Embedding (same model as ingest)
    │                                       │
    ▼                                       ▼
Normalization                           Access Control Filter Resolution
    │                                       │
    ▼                                       ▼
Change Detection (hash compare)         Qdrant Query (filters embedded)
    │                                       │
    ▼                                   Dense Search ──┐
Chunking                                Sparse Search ──┤ RRF Fusion
    │                                       │           │
    ▼                                       ▼           │
Chunk Hashing                          Top-N Candidates◄┘
    │                                       │
    ▼                                       ▼
Embedding (dense + sparse or dense)    Optional Re-ranking (cross-encoder)
    │                                       │
    ▼                                       ▼
Metadata Assembly                      Result Assembly + Attribution
    │                                       │
    ▼                                       ▼
Qdrant Upsert                          Return to LLM / Agent
```

---

## Mandatory Processing Order (Ingestion)

Every ingestion pipeline MUST execute these stages in this exact order. Do not skip or reorder.

1. **Source capture** — Fetch raw content and metadata from the source system API.
2. **Normalization** — Apply universal + source-specific normalization rules. See `04-source-normalization.md`.
3. **Document hashing** — Compute `document_content_hash` (SHA-256) from the full normalized document text.
4. **Change detection** — Compare hash to the previously stored hash. If unchanged, refresh metadata only and skip steps 5–8.
5. **Chunking** — Apply the chunking strategy for the content type. See `05-chunking-standards.md`.
6. **Chunk hashing** — Compute `content_hash` (SHA-256) for each chunk from normalized chunk text.
7. **Embedding** — Generate dense and/or sparse vectors. See `06-embedding-models.md`.
8. **Storage** — Upsert chunks with full metadata payload to Qdrant. See `07-ingestion-pipeline.md`.

---

## Core Architectural Principles

### 1. Separation of Concerns: Data vs. Access Policy

The vector store holds **data characteristics** (what the content is, who authored it, what domain it belongs to). It does NOT hold **access policy** (which agents or users may access it). Access mappings belong exclusively in your orchestration layer (e.g., n8n, LangGraph, custom middleware).

**Prohibited in chunk metadata:**
- `allowed_agents`, `permitted_agents`, or any agent identity list
- Agent-level permission scopes embedded as payload fields
- Any field that dynamically changes based on who is querying

**Why:** If you embed agent permissions in chunks, every access change requires re-ingesting thousands of chunks. The orchestration layer can be updated instantly without touching the vector store.

### 2. Filters Must Be Applied Inside Qdrant Queries

Access control filters (sensitivity, group membership, scope tags) must be passed as Qdrant payload filter conditions within the query itself — never applied post-retrieval to a raw result set.

**Why post-filtering is dangerous:**
- Qdrant returns top-K candidates from the full vector space. If you filter afterwards, you may discard half the results, leaving fewer than expected — or returning none at all for highly restricted content.
- Post-filtering creates a window where restricted content is briefly retrieved before being discarded. This is a compliance risk.

### 3. Classify at Ingest Time, Never at Retrieval Time

Every chunk must have its sensitivity tier and scope tags assigned during ingestion. Retrieval pipelines must never attempt to classify content on the fly.

**Why:** Classification at retrieval time is too slow, inconsistent, and untestable. Ingest-time classification is auditable and deterministic.

### 4. Deterministic Ingestion

The same normalized source content must always produce the same chunks, the same hashes, and the same `doc_id`s. This is what makes the pipeline safe to re-run (idempotent) and what enables reliable change detection.

### 5. Upsert, Not Insert

All writes to Qdrant use upsert semantics. If a point with the same ID already exists, it is overwritten. This is the foundation of idempotent ingestion.

---

## Collection Architecture Principles

Do NOT create a collection per data source. Collections should be justified by one or more of:

| Criterion | Example |
|---|---|
| Security boundary | Confidential/PII data must not coexist with public data |
| Query pattern | Different collections may require different HNSW tuning |
| Scale / index tuning | Very large datasets benefit from isolated indexing |
| Lifecycle / retention | Data with different retention policies |

A typical production setup uses 3 collections:

| Collection | Contents | Access |
|---|---|---|
| `company_memory` | Public + internal tier (messages, transcripts, docs, tasks) | All staff |
| `restricted_memory` | Restricted tier (email, finance, legal, HR non-PII) | Authorized roles only |
| `pii_memory` | Confidential tier (PII: salary, health, tax IDs) | HR leadership, compliance |

See `03-data-classification.md` for the full sensitivity tier model.

---

## Choosing Retrieval Mode

| Scenario | Recommended Mode |
|---|---|
| Content contains proper nouns, IDs, codes, product names, exact phrases | **Hybrid** (dense + sparse) |
| Users search with natural language, vague intent | **Dense-only** acceptable, hybrid preferred |
| Need to balance semantic understanding with exact keyword matching | **Hybrid** with RRF |
| Latency-critical, simple domain | **Dense-only** with `text-embedding-3-small` |
| Complex enterprise knowledge base | **Hybrid** with BGE-M3 |

See `06-embedding-models.md` and `08-retrieval-architecture.md` for implementation.

---

## Evaluation Is Not Optional

Every retrieval architecture and chunking configuration must be validated before going to production. Minimum requirements:

- Minimum 50 representative queries with known expected results
- Precision and recall benchmarks at top-5 and top-10
- Latency benchmarks (mean, p95, p99)
- Citation accuracy: do returned chunks contain the claimed answer?
- Source coverage: does the system find content from all active source types?
- Regression tests: do changes to chunking or embedding degrade existing queries?

Document your evaluation results and link them to your pipeline version.

FILE:guides/04-source-normalization.md
# Guide 04 — Source Normalization

## Purpose

Normalization ensures that minor formatting differences in the same underlying content do not produce different chunks, hash mismatches, or misleading metadata. All hashes (`document_content_hash`, `content_hash`) must be computed from **normalized** content, never from raw source payloads.

The normalization logic must be versioned independently from the schema, and the `normalizer_version` field recorded on every chunk.

---

## Why Normalization Matters

Without consistent normalization:
- The same message edited to fix a typo produces a different hash → unnecessary re-ingestion
- Whitespace differences between source systems cause false "content changed" detections
- Emoji codes (``:thumbsup:``) and actual emoji (👍) produce different embeddings for the same meaning
- Duplicate quoted-reply content in email inflates chunk count and confuses retrieval

---

## Universal Normalization Rules

Apply these to ALL content from ALL source types, in this order:

1. **Collapse whitespace** — Replace all consecutive whitespace characters (spaces, tabs, non-breaking spaces, zero-width spaces) with a single space.

2. **Strip leading/trailing whitespace** — From the full document text and from each logical segment before chunking.

3. **Normalize Unicode** — Convert to NFC (Canonical Decomposition, Canonical Composition) form. This resolves composed vs. decomposed character variants.

4. **Strip HTML/markup** — Remove HTML tags, markdown rendering artifacts, inline CSS. Preserve only the text content. Do not strip structural markers (headings, list markers) that inform chunking hierarchy.

5. **Normalize timestamps** — Convert all timestamps to ISO 8601 UTC: `2024-04-30T18:00:00Z`.

6. **Normalize author/participant names** — Map to canonical directory names. If a user is referenced as "Jane", "Jane Smith", "jsmith", or "Jane S." in different parts of the content, normalize to their canonical display name from your directory.

```python
import unicodedata
import re

def universal_normalize(text: str) -> str:
    """Apply all universal normalization rules."""
    # Step 1: Normalize Unicode to NFC
    text = unicodedata.normalize("NFC", text)
    
    # Step 2: Strip HTML tags
    text = re.sub(r"<[^>]+>", " ", text)
    
    # Step 3: Collapse all whitespace to single space
    text = re.sub(r"\s+", " ", text)
    
    # Step 4: Strip leading/trailing whitespace
    text = text.strip()
    
    return text
```

---

## Source-Specific Normalization Rules

Apply these AFTER the universal rules, for each source type.

### Slack

```python
def normalize_slack(text: str, user_directory: dict) -> str:
    """Normalize Slack message content."""
    # Replace user mention IDs with display names: <@U012AB3CD> → @Jane Smith
    def replace_mention(match):
        uid = match.group(1)
        return f"@{user_directory.get(uid, uid)}"
    text = re.sub(r"<@([A-Z0-9]+)>", replace_mention, text)
    
    # Replace channel links: <#C03AB|general> → #general
    text = re.sub(r"<#[A-Z0-9]+\|([^>]+)>", r"#\1", text)
    
    # Replace Slack URLs: <https://example.com|Click here> → Click here (https://example.com)
    text = re.sub(r"<(https?://[^|>]+)\|([^>]+)>", r"\2 (\1)", text)
    
    # Strip bare URLs in angle brackets: <https://example.com> → https://example.com
    text = re.sub(r"<(https?://[^>]+)>", r"\1", text)
    
    # Replace emoji shortcodes: :thumbsup: → 👍 (or strip if emoji not desired)
    # Use emoji library for replacement, or simply strip shortcodes:
    text = re.sub(r":[a-z0-9_+-]+:", "", text)
    
    # Strip bot-generated footers (customize patterns for your workspace)
    # e.g., "Sent via Zapier", "This is an automated message"
    text = re.sub(r"(?i)(sent via|this is an automated message).*$", "", text, flags=re.MULTILINE)
    
    return universal_normalize(text)

# Handle edited messages: always use latest text, flag is_edited=True in metadata
def get_slack_message_text(message: dict) -> tuple[str, bool]:
    is_edited = "edited" in message
    text = message.get("text", "")
    return text, is_edited
```

### Email

```python
def normalize_email_body(raw_body: str) -> str:
    """Normalize email body, stripping quoted replies and signatures."""
    
    # Strip quoted reply blocks (lines starting with >)
    lines = raw_body.split("\n")
    body_lines = []
    in_quoted = False
    for line in lines:
        stripped = line.strip()
        if stripped.startswith(">"):
            in_quoted = True
            continue
        # Common reply delimiters
        if any(stripped.startswith(d) for d in [
            "On ", "From:", "-----Original Message-----",
            "________________________________", "Sent from my"
        ]):
            break  # Everything after this is quoted reply
        if not in_quoted:
            body_lines.append(line)
    
    body = "\n".join(body_lines)
    
    # Strip email signatures (lines after -- on its own line)
    body = re.sub(r"\n--\s*\n.*$", "", body, flags=re.DOTALL)
    
    # Strip legal disclaimers (common patterns)
    body = re.sub(
        r"(?i)(this email and any attachments|confidentiality notice|disclaimer:).*$",
        "",
        body,
        flags=re.DOTALL,
    )
    
    return universal_normalize(body)

def normalize_email_headers(from_addr: str, to_addrs: list, subject: str) -> dict:
    """Normalize email header fields to consistent format."""
    return {
        "from": from_addr.strip().lower(),
        "to": [addr.strip().lower() for addr in to_addrs],
        "subject": subject.strip(),
    }
```

### Meeting Transcripts (Fireflies / similar)

```python
FILLER_WORDS = {
    "um", "uh", "er", "ah", "like", "you know", "sort of", "kind of",
    "basically", "literally", "actually", "right", "okay", "so"
}

def normalize_transcript_segment(
    text: str,
    speaker_name: str,
    canonical_names: dict,
    confidence_threshold: float = 0.7,
    segment_confidence: float = 1.0,
) -> str:
    """Normalize a single speaker turn in a meeting transcript."""
    
    # Normalize speaker name to canonical directory name
    canonical_speaker = canonical_names.get(speaker_name, speaker_name)
    
    # Strip filler words if confidence is below threshold
    if segment_confidence < confidence_threshold:
        words = text.split()
        words = [w for w in words if w.lower().strip(".,?!") not in FILLER_WORDS]
        text = " ".join(words)
    
    # Prepend speaker attribution
    normalized = f"[{canonical_speaker}]: {text}"
    
    return universal_normalize(normalized)

def build_transcript_context_header(meeting: dict) -> str:
    """Build the context header prepended to every transcript chunk."""
    return (
        f"Meeting: {meeting['title']} | "
        f"Date: {meeting['date']} | "
        f"Attendees: {', '.join(meeting['attendee_names'])}"
    )
```

### Google Drive Documents

```python
def normalize_gdrive_document(raw_content: str, doc_type: str) -> str:
    """Normalize Google Drive document content."""
    
    if doc_type == "doc":
        # Strip suggestion markup: {+added text+} and {-removed text-}
        raw_content = re.sub(r"\{\+[^}]*\+\}", "", raw_content)
        raw_content = re.sub(r"\{-[^}]*-\}", "", raw_content)
        
        # Strip comment anchors: [[COMMENT: ...]]
        raw_content = re.sub(r"\[\[COMMENT:[^\]]*\]\]", "", raw_content)
    
    # Export Google Docs as Markdown (use Google API export with MIME type text/markdown)
    # Preserve heading structure as structural markers for chunking:
    # ## Heading 2 → keep for hierarchical chunking boundary detection
    
    return universal_normalize(raw_content)
```

### ClickUp Tasks

```python
def normalize_clickup_task(task: dict) -> str:
    """Normalize ClickUp task into a single text block."""
    
    parts = []
    
    # Task title
    parts.append(f"Task: {task.get('name', '')}")
    
    # Task description
    if task.get("description"):
        desc = task["description"]
        # Strip ClickUp markup: @mentions, /commands, etc.
        desc = re.sub(r"@\w+", "", desc)
        desc = re.sub(r"/\w+", "", desc)
        parts.append(desc)
    
    # All comments, in chronological order
    for comment in sorted(task.get("comments", []), key=lambda c: c.get("date", "")):
        author = comment.get("user", {}).get("username", "Unknown")
        text = comment.get("comment_text", "")
        if text:
            text = re.sub(r"@\w+", "", text)  # Strip mentions
            parts.append(f"[{author}]: {text}")
    
    combined = "\n\n".join(parts)
    return universal_normalize(combined)
```

---

## Normalization Checklist

Before writing the normalization function for a new source:

- [ ] Unicode normalized to NFC
- [ ] HTML/markup stripped, structure preserved for chunking
- [ ] Source-specific user IDs replaced with display names
- [ ] Source-specific markup and noise removed
- [ ] Timestamps in ISO 8601 UTC
- [ ] Quoted/repeated content stripped (email, threaded messages)
- [ ] Signatures and disclaimers stripped
- [ ] Content verified to be deterministic: same input always produces same output
- [ ] `normalizer_version` bumped if normalization logic changes

FILE:guides/03-data-classification.md
# Guide 03 — Data Classification & Collections

## Purpose

This guide defines the four-tier sensitivity model, collection architecture patterns, and classification logic for determining where each chunk belongs in Qdrant.

---

## Four-Tier Sensitivity Model

Every chunk must be assigned exactly one sensitivity tier at ingest time. Classification must never be deferred to retrieval time.

| Tier | Description | Target Collection |
|---|---|---|
| `public` | Explicitly approved for external distribution. Marketing materials, published docs, public blog posts. | `company_memory` |
| `internal` | Intended for all staff but not for external distribution. Public Slack channels, all-hands transcripts, internal project docs. | `company_memory` |
| `restricted` | Limited to specific groups. Executive email, finance reports, legal contracts, HR communications (non-PII). | `restricted_memory` |
| `confidential` | Contains PII or requires highest protection. Salary data, performance reviews, health records, tax identifiers. | `pii_memory` |

**Default tier:** When no rule matches and no classifier flags the content, default to `internal`.

---

## Classification Logic (Precedence Order)

Apply in this exact order — stop at the first match:

1. **Explicit override** — A human-applied classification label on the source content. Highest precedence. Always respected.

2. **Rule-based classification** — Deterministic rules based on source metadata:
   - Folder location (e.g., `/Finance/Reports/` → `restricted`)
   - Distribution list (e.g., executives-only DL → `restricted`)
   - Channel type (e.g., `private` Slack channel with HR members → `restricted`)
   - Sender/recipient patterns (e.g., external domain sender → review)
   - Document sharing scope (e.g., `specific` share → at least `restricted`)

3. **Content-based classifier** — For content that cannot be classified by metadata alone, scan for signals:
   - Salary figures, compensation data → `confidential`
   - Tax identifiers, SSN, health information → `confidential`
   - Legal contract language → `restricted`
   - Explicitly internal communications → `internal`
   - This is `model_inferred` — log the classification event for audit.

4. **Default** — `internal` if nothing matches.

---

## Collection Architecture

### Standard Three-Collection Pattern

```python
from qdrant_client import QdrantClient, models

client = QdrantClient(url="http://localhost:6333")

# Shared HNSW config for all collections
hnsw_config = models.HnswConfigDiff(m=16, ef_construct=100)

# Dense-only collection (text-embedding-3-small)
def create_collection_dense(name: str):
    client.create_collection(
        collection_name=name,
        vectors_config=models.VectorParams(
            size=1536,
            distance=models.Distance.COSINE,
            hnsw_config=hnsw_config,
        ),
        quantization_config=models.ScalarQuantization(
            scalar=models.ScalarQuantizationConfig(
                type=models.ScalarType.INT8,
                quantile=0.99,
                always_ram=True,
            )
        ),
    )

# Hybrid collection (BGE-M3: dense 1024-dim + sparse SPLADE)
def create_collection_hybrid(name: str):
    client.create_collection(
        collection_name=name,
        vectors_config={
            "dense": models.VectorParams(
                size=1024,  # BGE-M3 dense dimension
                distance=models.Distance.COSINE,
                hnsw_config=hnsw_config,
            )
        },
        sparse_vectors_config={
            "sparse": models.SparseVectorParams(
                modifier=models.Modifier.IDF  # Enable IDF for BM25-style sparse
            )
        },
    )

# Create the three standard collections
create_collection_hybrid("company_memory")
create_collection_hybrid("restricted_memory")
create_collection_hybrid("pii_memory")
```

### HNSW Tuning by Collection Size

| Collection Size | Recommended m | Recommended ef_construct |
|---|---|---|
| < 100K chunks | 16 | 100 |
| 100K – 1M chunks | 32 | 200 |
| > 1M chunks | 64 | 400 |

Increasing `m` improves recall but increases memory and build time. Increase only when measured recall drops below target.

### Quantization

For production at scale, use scalar quantization (INT8) to reduce memory by ~75% with minimal quality loss:

```python
quantization_config=models.ScalarQuantization(
    scalar=models.ScalarQuantizationConfig(
        type=models.ScalarType.INT8,
        quantile=0.99,    # Clips outlier values for better quantization
        always_ram=True,  # Keep quantized vectors in RAM for speed
    )
)
```

For very large collections (>10M chunks) where memory is critical, binary quantization reduces by 32x but requires careful quality validation:

```python
quantization_config=models.BinaryQuantization(
    binary=models.BinaryQuantizationConfig(always_ram=True)
)
```

---

## Multi-Tenancy Pattern

If you're building a SaaS system with multiple organizations sharing a single Qdrant cluster, use the `is_tenant=True` flag on the `org_id` index rather than creating separate collections per tenant:

```python
# Create the tenant-aware payload index
client.create_payload_index(
    collection_name="company_memory",
    field_name="org_id",
    field_schema=models.KeywordIndexParams(
        type="keyword",
        is_tenant=True,  # Co-locates vectors for the same org for performance
    ),
)
```

**Why:** Creating one collection per tenant exhausts RAM and cluster resources quickly. Qdrant's `is_tenant` optimization co-locates tenant data in the HNSW graph, providing performance close to a dedicated collection without the resource cost.

**Always apply `org_id` as the outermost filter on every query** to enforce tenant isolation:

```python
client.query_points(
    collection_name="company_memory",
    query=dense_vector,
    using="dense",
    query_filter=models.Filter(
        must=[
            models.FieldCondition(
                key="org_id",
                match=models.MatchValue(value="org-456")
            ),
            # ... additional filters
        ]
    ),
    limit=20,
)
```

---

## Classification Rules by Source Type

### Slack Channels
| Condition | Sensitivity |
|---|---|
| Public channel, no HR/Finance/Legal members | `internal` |
| Private channel with Finance or Legal members | `restricted` |
| HR channel discussing non-PII matters | `restricted` |
| Any channel with salary, SSN, health data | `confidential` |
| Any public-facing content | `public` |

### Email
| Condition | Sensitivity |
|---|---|
| External recipient in To/CC | `restricted` (at minimum) |
| Executive distribution list only | `restricted` |
| Finance or Legal content | `restricted` |
| Content with salary, compensation, health data | `confidential` |
| All-staff announcement | `internal` |

### Google Drive
| Condition | Sensitivity |
|---|---|
| Share scope = `anyone` | `public` |
| Share scope = `domain` | `internal` |
| Share scope = `specific`, folder = `/Finance/` or `/Legal/` | `restricted` |
| Share scope = `specific`, folder = `/HR/Compensation/` | `confidential` |

### Meeting Transcripts (Fireflies)
| Condition | Sensitivity |
|---|---|
| All-hands or team meeting, general topic | `internal` |
| Meeting with external attendees discussing internal matters | `restricted` |
| Executive strategy discussions | `restricted` |
| HR meeting with compensation or performance data | `confidential` |

### ClickUp Tasks
| Condition | Sensitivity |
|---|---|
| Project tasks in general spaces | `internal` |
| Tasks in Finance or Legal spaces | `restricted` |
| Tasks containing PII in description or comments | `confidential` |

---

## Sensitivity Change / Reclassification

When a source record's sensitivity tier changes (e.g., an internal document is reclassified as restricted), the ingestion pipeline must:

1. Detect the change via `document_content_hash` or source metadata comparison during the next sync cycle.
2. Re-ingest the affected chunks with the updated `sensitivity`, target collection, and any updated `data_scope_tags`.
3. Delete old chunks from the prior collection (e.g., `company_memory`).
4. Write new chunks to the correct collection (e.g., `restricted_memory`).
5. Log the reclassification event to the audit log: `old_sensitivity`, `new_sensitivity`, `doc_id`, timestamp.

Unlike access policy changes (which only require updating your orchestration layer config), sensitivity reclassification requires pipeline action because sensitivity determines **collection placement** — a hard architectural boundary in Qdrant.

FILE:guides/07-ingestion-pipeline.md
# Guide 07 — Ingestion Pipeline

## Purpose

This guide provides a complete, production-grade ingestion pipeline implementation — covering source capture, change detection, upsert patterns, deduplication, idempotency, and lifecycle management.

---

## Complete Pipeline Implementation

```python
import hashlib
import uuid
from datetime import datetime, timezone
from qdrant_client import QdrantClient, models
from typing import Optional

class IngestionPipeline:
    
    def __init__(self, qdrant_client: QdrantClient, embed_fn, config: dict):
        self.client = qdrant_client
        self.embed = embed_fn  # Function: list[str] → list[{dense, sparse}] or list[list[float]]
        self.config = config
        self.run_id = str(uuid.uuid4())
    
    def ingest_document(
        self,
        source_content: str,
        source_metadata: dict,
        collection_name: str,
        normalizer,
        chunker,
        tagger,
        classifier,
    ) -> dict:
        """
        Full pipeline for a single source document.
        Returns a summary of what was done.
        """
        
        # Step 1: Normalization
        normalized_text = normalizer.normalize(source_content, source_metadata)
        
        # Step 2: Document-level hash (change detection)
        doc_hash = hashlib.sha256(normalized_text.encode("utf-8")).hexdigest()
        
        # Step 3: Change detection — check if we've seen this document before
        existing_hash = self._get_existing_hash(
            source_metadata["source_system"],
            source_metadata["source_record_id"],
            collection_name,
        )
        
        if existing_hash == doc_hash:
            # Content unchanged — only refresh metadata timestamps
            self._refresh_metadata_only(source_metadata, collection_name)
            return {"status": "unchanged", "doc_hash": doc_hash}
        
        # Step 4: Classification
        sensitivity = classifier.classify(normalized_text, source_metadata)
        
        # Step 5: Scope tagging
        data_scope_tags = tagger.derive_tags(source_metadata)
        
        # Step 6: Chunking
        chunks = chunker.chunk(normalized_text, source_metadata)
        new_total = len(chunks)
        
        # Step 7: Build payloads, hash chunks, embed, upsert
        points = []
        for i, chunk_text in enumerate(chunks):
            chunk_hash = hashlib.sha256(chunk_text.encode("utf-8")).hexdigest()
            
            # Check for within-source duplicate
            if self._is_duplicate_chunk(chunk_hash, source_metadata["source_record_id"], collection_name):
                continue  # Update metadata in place rather than writing new point
            
            doc_id = f"{source_metadata['source_type']}_{source_metadata['source_record_id']}_{i}"
            
            payload = self._build_payload(
                source_metadata=source_metadata,
                doc_id=doc_id,
                chunk_text=chunk_text,
                chunk_index=i,
                chunk_total=new_total,
                chunk_hash=chunk_hash,
                doc_hash=doc_hash,
                sensitivity=sensitivity,
                data_scope_tags=data_scope_tags,
                chunk_strategy=chunker.strategy_name,
            )
            
            # Step 8: Embedding
            embedding = self.embed([chunk_text])[0]
            
            # Build Qdrant point
            if isinstance(embedding, dict):
                # Hybrid: {dense: [...], sparse: {indices: [...], values: [...]}}
                vector = {
                    "dense": embedding["dense"],
                    "sparse": models.SparseVector(
                        indices=embedding["sparse"]["indices"],
                        values=embedding["sparse"]["values"],
                    )
                }
            else:
                # Dense-only: [...]
                vector = embedding
            
            points.append(models.PointStruct(
                id=self._stable_uuid(doc_id),
                vector=vector,
                payload=payload,
            ))
        
        # Step 9: Upsert all points
        if points:
            self.client.upsert(
                collection_name=collection_name,
                points=points,
                wait=True,  # Wait for confirmation before returning
            )
        
        # Step 10: Delete stale chunks (if re-ingestion produced fewer chunks)
        old_total = self._get_old_chunk_total(
            source_metadata["source_record_id"], collection_name
        )
        if old_total and old_total > new_total:
            self._delete_stale_chunks(
                source_metadata, collection_name, new_total, old_total
            )
        
        return {
            "status": "ingested",
            "doc_id_base": f"{source_metadata['source_type']}_{source_metadata['source_record_id']}",
            "chunks_written": len(points),
            "chunks_total": new_total,
            "sensitivity": sensitivity,
            "run_id": self.run_id,
        }
    
    def _stable_uuid(self, doc_id: str) -> str:
        """Generate a stable UUID from a doc_id string (deterministic)."""
        return str(uuid.uuid5(uuid.NAMESPACE_DNS, doc_id))
    
    def _build_payload(self, **kwargs) -> dict:
        """Assemble the complete chunk payload."""
        now = datetime.now(timezone.utc).isoformat()
        source_metadata = kwargs["source_metadata"]
        
        return {
            "doc_id": kwargs["doc_id"],
            "source_type": source_metadata["source_type"],
            "source_system": source_metadata["source_system"],
            "source_record_id": source_metadata["source_record_id"],
            "integration_id": source_metadata.get("integration_id", "unknown"),
            "parent_doc_id": source_metadata.get("parent_doc_id", kwargs["doc_id"]),
            "document_content_hash": kwargs["doc_hash"],
            "chunk_index": kwargs["chunk_index"],
            "chunk_total": kwargs["chunk_total"],
            "content_hash": kwargs["chunk_hash"],
            "created_at": source_metadata.get("created_at", now),
            "ingested_at": now,
            "modified_at": source_metadata.get("modified_at", now),
            "source_last_modified": source_metadata.get("source_last_modified", now),
            "author_id": source_metadata.get("author_id", ""),
            "author_name": source_metadata.get("author_name", ""),
            "author_email": source_metadata.get("author_email"),
            "author_department": source_metadata.get("author_department"),
            "project_ids": source_metadata.get("project_ids", []),
            "team_ids": source_metadata.get("team_ids", []),
            "workspace_id": source_metadata["workspace_id"],
            "org_id": source_metadata["org_id"],
            "sensitivity": kwargs["sensitivity"],
            "allowed_groups": source_metadata.get("allowed_groups", []),
            "owner_user_ids": source_metadata.get("owner_user_ids", []),
            "is_pii": source_metadata.get("is_pii", False),
            "retention_days": source_metadata.get("retention_days", 1095),
            "data_scope_tags": kwargs["data_scope_tags"],
            "language": source_metadata.get("language", "en"),
            "content_type": source_metadata.get("content_type", "document"),
            "content_subtype": source_metadata.get("content_subtype"),
            "title": source_metadata.get("title"),
            "summary": None,  # Populate separately if using a summarization model
            "token_count": len(kwargs["chunk_text"]) // 4,
            "chunk_strategy": kwargs["chunk_strategy"],
            "embedding_model": self.config["embedding_model"],
            "sparse_model": self.config.get("sparse_model", "none"),
            "schema_version": "2.2",
            "ingestion_run_id": self.run_id,
            "connector_version": self.config["connector_version"],
            "normalizer_version": self.config["normalizer_version"],
            "chunker_version": self.config["chunker_version"],
            "source_metadata": source_metadata.get("source_specific", {}),
        }
    
    def _get_existing_hash(self, source_system: str, record_id: str, collection: str) -> Optional[str]:
        """Look up the stored document_content_hash for this record."""
        try:
            results = self.client.scroll(
                collection_name=collection,
                scroll_filter=models.Filter(
                    must=[
                        models.FieldCondition(key="source_system", match=models.MatchValue(value=source_system)),
                        models.FieldCondition(key="source_record_id", match=models.MatchValue(value=record_id)),
                        models.FieldCondition(key="chunk_index", match=models.MatchValue(value=0)),
                    ]
                ),
                with_payload=True,
                limit=1,
            )
            if results[0]:
                return results[0][0].payload.get("document_content_hash")
        except Exception:
            pass
        return None
    
    def _is_duplicate_chunk(self, content_hash: str, source_record_id: str, collection: str) -> bool:
        """Check if a chunk with this content_hash already exists for this source record."""
        results = self.client.scroll(
            collection_name=collection,
            scroll_filter=models.Filter(
                must=[
                    models.FieldCondition(key="content_hash", match=models.MatchValue(value=content_hash)),
                    models.FieldCondition(key="source_record_id", match=models.MatchValue(value=source_record_id)),
                ]
            ),
            limit=1,
        )
        return len(results[0]) > 0
    
    def _get_old_chunk_total(self, source_record_id: str, collection: str) -> Optional[int]:
        """Get the previously stored chunk_total for this record."""
        results = self.client.scroll(
            collection_name=collection,
            scroll_filter=models.Filter(
                must=[
                    models.FieldCondition(key="source_record_id", match=models.MatchValue(value=source_record_id)),
                    models.FieldCondition(key="chunk_index", match=models.MatchValue(value=0)),
                ]
            ),
            with_payload=True,
            limit=1,
        )
        if results[0]:
            return results[0][0].payload.get("chunk_total")
        return None
    
    def _delete_stale_chunks(
        self, source_metadata: dict, collection: str, new_total: int, old_total: int
    ):
        """Delete chunks whose chunk_index is now out of range after re-ingestion."""
        stale_ids = []
        for stale_index in range(new_total, old_total):
            doc_id = f"{source_metadata['source_type']}_{source_metadata['source_record_id']}_{stale_index}"
            stale_ids.append(self._stable_uuid(doc_id))
        
        if stale_ids:
            self.client.delete(
                collection_name=collection,
                points_selector=models.PointIdsList(points=stale_ids),
            )
    
    def _refresh_metadata_only(self, source_metadata: dict, collection: str):
        """For unchanged content, update only the ingested_at and source_last_modified timestamps."""
        now = datetime.now(timezone.utc).isoformat()
        self.client.set_payload(
            collection_name=collection,
            payload={
                "ingested_at": now,
                "source_last_modified": source_metadata.get("source_last_modified", now),
                "ingestion_run_id": self.run_id,
            },
            points=models.Filter(
                must=[
                    models.FieldCondition(
                        key="source_record_id",
                        match=models.MatchValue(value=source_metadata["source_record_id"])
                    )
                ]
            ),
        )
    
    def delete_document(self, source_type: str, source_record_id: str, collection: str):
        """Hard-delete all chunks for a source record (e.g., content deleted from source)."""
        self.client.delete(
            collection_name=collection,
            points_selector=models.FilterSelector(
                filter=models.Filter(
                    must=[
                        models.FieldCondition(key="source_type", match=models.MatchValue(value=source_type)),
                        models.FieldCondition(key="source_record_id", match=models.MatchValue(value=source_record_id)),
                    ]
                )
            ),
        )
```

---

## Lifecycle Event Handling

| Event | Action |
|---|---|
| **Create** | Run full ingestion pipeline → upsert chunks |
| **Update** | Detect via `document_content_hash` → re-normalize, re-chunk, upsert, delete stale chunks |
| **Delete** | Hard-delete all chunks for the `source_record_id` |
| **Move / reclassify** | Metadata refresh + collection migration if sensitivity tier changes |
| **Access revoked** | Hard-delete if the content should no longer be retrievable |

### Collection Migration (Sensitivity Change)

```python
def migrate_chunks_to_collection(
    client: QdrantClient,
    source_record_id: str,
    old_collection: str,
    new_collection: str,
    new_sensitivity: str,
    new_data_scope_tags: list[str],
    audit_logger,
):
    """Move chunks from one collection to another when sensitivity changes."""
    
    # 1. Retrieve all chunks for this record from the old collection
    results, _ = client.scroll(
        collection_name=old_collection,
        scroll_filter=models.Filter(
            must=[models.FieldCondition(
                key="source_record_id",
                match=models.MatchValue(value=source_record_id)
            )]
        ),
        with_payload=True,
        with_vectors=True,
        limit=1000,
    )
    
    if not results:
        return
    
    old_sensitivity = results[0].payload.get("sensitivity")
    
    # 2. Update sensitivity and scope tags
    now = datetime.now(timezone.utc).isoformat()
    new_points = []
    for point in results:
        updated_payload = {
            **point.payload,
            "sensitivity": new_sensitivity,
            "data_scope_tags": new_data_scope_tags,
            "ingested_at": now,
        }
        new_points.append(models.PointStruct(
            id=point.id,
            vector=point.vector,
            payload=updated_payload,
        ))
    
    # 3. Write to new collection
    client.upsert(collection_name=new_collection, points=new_points, wait=True)
    
    # 4. Delete from old collection
    client.delete(
        collection_name=old_collection,
        points_selector=models.FilterSelector(
            filter=models.Filter(
                must=[models.FieldCondition(
                    key="source_record_id",
                    match=models.MatchValue(value=source_record_id)
                )]
            )
        ),
    )
    
    # 5. Audit log
    audit_logger.log_reclassification(
        source_record_id=source_record_id,
        old_sensitivity=old_sensitivity,
        new_sensitivity=new_sensitivity,
        timestamp=now,
        doc_ids=[p.payload["doc_id"] for p in results],
    )
```

---

## Deduplication

### Within-Source Deduplication
Before writing a chunk, check if a point with the same `content_hash` and `parent_doc_id` already exists in the target collection. On match, update metadata in place rather than writing a new point.

### Cross-Source Deduplication
Exact duplicates across different `source_type` values are retained as separate chunks because they represent distinct provenance. At retrieval time, near-duplicate results from different sources should be collapsed or down-ranked.

---

## Error Handling and Retry

```python
import time
from functools import wraps

def with_exponential_backoff(max_retries: int = 5, base_delay: float = 1.0):
    def decorator(fn):
        @wraps(fn)
        def wrapper(*args, **kwargs):
            for attempt in range(max_retries):
                try:
                    return fn(*args, **kwargs)
                except Exception as e:
                    if attempt == max_retries - 1:
                        # Route to dead-letter queue after max retries
                        dead_letter_queue.push({"fn": fn.__name__, "args": args, "error": str(e)})
                        raise
                    delay = base_delay * (2 ** attempt)
                    time.sleep(delay)
        return wrapper
    return decorator
```

Failed lifecycle events (after max retries) must be routed to a dead-letter queue for manual review.

---

## Idempotency Checklist

Before deploying an ingestion pipeline to production, verify:

- [ ] `doc_id` format is deterministic — no timestamps, no random components
- [ ] All writes use `upsert`, never raw `insert`
- [ ] Same input run twice produces the same Qdrant state (no duplicate points)
- [ ] Stale chunk cleanup runs after every re-ingestion
- [ ] `document_content_hash` change detection is working (run a golden doc test)
- [ ] `ingestion_run_id` is a fresh UUID per run (not per chunk)

FILE:guides/09-access-control.md
# Guide 09 — Access Control Patterns

## Purpose

This guide defines the architectural principles and implementation patterns for access control in Qdrant-based RAG systems. It explains the separation of concerns between the data layer and the policy layer, and provides concrete patterns for enforcing permissions at retrieval time.

---

## The Fundamental Principle

> **Metadata describes what data is. It does not encode which agents or users may access it.**

Agent-to-data access mappings belong exclusively in your orchestration layer (middleware, n8n, LangGraph, custom API gateway). They must never be stored in chunk payloads.

---

## Why NOT to Store Permissions in Chunks

A common temptation is to add an `allowed_agents` field to each chunk. This is explicitly wrong for three reasons:

**1. Re-ingestion risk:** If an agent's access scope changes (e.g., a new AI agent is granted access to finance data), every affected chunk would need to be updated in the vector store. At scale this is operationally dangerous, slow, and error-prone.

**2. Privilege escalation surface:** If an LLM can influence the values of access control fields (even indirectly, through prompt injection), and those fields are authoritative for access decisions, you have a security vulnerability.

**3. Tight coupling:** Embedding access policy in the data layer creates a hard dependency between your AI agent configuration and your vector store schema, making both harder to evolve independently.

---

## The Correct Architecture: Two-Layer Separation

```
┌─────────────────────────────────────────────────────────────────┐
│  ORCHESTRATION LAYER (middleware / n8n / LangGraph)             │
│                                                                 │
│  Stores: agent-to-data access mappings                         │
│  - Which (user_id, agent_id) pairs can access which scopes     │
│  - Which collections each agent may query                      │
│  - Which sensitivity tiers are permitted per role              │
│                                                                 │
│  Responsibilities:                                              │
│  1. Validate agent identity                                     │
│  2. Look up user identity from session/token                   │
│  3. Resolve permission scope for (user, agent) pair            │
│  4. Build Qdrant query filters from resolved scope             │
│  5. Execute query — agent cannot modify filters                │
│  6. Return results — agent sees only permitted content         │
└────────────────────────────┬────────────────────────────────────┘
                             │ Builds filter from scope
                             ▼
┌─────────────────────────────────────────────────────────────────┐
│  VECTOR STORE (Qdrant)                                          │
│                                                                 │
│  Stores: data characteristics only                             │
│  - sensitivity                                                  │
│  - team_ids                                                     │
│  - allowed_groups                                              │
│  - data_scope_tags                                             │
│  - org_id / workspace_id                                       │
│                                                                 │
│  Does NOT store:                                                │
│  - allowed_agents                                              │
│  - permitted_agents                                            │
│  - agent-level permission lists                                │
└─────────────────────────────────────────────────────────────────┘
```

---

## Permission Resolution Flow

```
1. User sends message → Orchestration layer receives it
2. Orchestration generates request_id, maps to user_id in session store
3. Request (with request_id, agent_id, query) sent to AI agent
4. AI agent calls retrieval tool → passes request_id, agent_id, query text only
5. Orchestration layer:
   a. Looks up user_id from session store using request_id
   b. Validates agent_id is permitted for this user
   c. Resolves permission scope for (user_id, agent_id):
      - permitted sensitivity tiers
      - permitted team_ids
      - permitted data_scope_tags
      - permitted collections
      - org_id / workspace_id
   d. Builds Qdrant filter from resolved scope
   e. Executes Qdrant query with filters as hard constraints
6. Results returned — AI agent never sees out-of-scope content
```

---

## Permission Scope Object

Your orchestration layer should resolve a permission scope that looks like this:

```python
# Example permission scope for a sales agent user
permission_scope = {
    "org_id": "org-456",
    "workspace_id": "ws-123",
    "permitted_collections": ["company_memory"],  # No access to restricted_memory
    "permitted_sensitivity": ["public", "internal"],
    "team_ids": ["team_sales_a", "team_sales_b"],  # Team-restricted
    "allowed_groups": ["g_all_staff", "g_sales"],
    "data_scope_tags": ["sales", "region:emea"],   # Only sales/EMEA content
}

# Example permission scope for an executive user
permission_scope_exec = {
    "org_id": "org-456",
    "workspace_id": "ws-123",
    "permitted_collections": ["company_memory", "restricted_memory"],
    "permitted_sensitivity": ["public", "internal", "restricted"],
    "team_ids": None,       # No team restriction — sees all team content
    "allowed_groups": None, # No group restriction
    "data_scope_tags": None, # No scope restriction — sees all domains
}
```

---

## Tiered Authorization Model

Implement access control as layers, where each layer narrows the scope:

| Layer | Enforced By | Mechanism |
|---|---|---|
| **Layer 1: User → Agent** | Orchestration layer | Agent Access Registry: is this agent_id permitted for this user_id? |
| **Layer 2: Agent → Collection** | Orchestration layer | Which collections may this (user, agent) pair query? |
| **Layer 3: Agent → Data** | Qdrant payload filters | sensitivity, team_ids, data_scope_tags, allowed_groups |
| **Layer 4: Composite** | Intersection of all above | Most restrictive rule wins |

---

## What Ingestion Pipelines Must NOT Store

The following fields are explicitly prohibited in chunk payloads:

```python
# PROHIBITED — never add these to a chunk payload
PROHIBITED_FIELDS = [
    "allowed_agents",
    "permitted_agents",
    "agent_access_list",
    "allowed_ai_agents",
    "agent_ids",
    # Any field that lists which AI agents may access this chunk
]
```

If you find these fields in an existing payload schema, they must be removed and replaced with the correct pattern (`data_scope_tags` + orchestration-layer mapping).

---

## Agent Access Changes Do Not Require Re-Ingestion

Because access mappings live in the orchestration layer (not in chunk metadata), access changes are instant and zero-cost to the vector store:

| Event | Required Action |
|---|---|
| New agent created and granted access to sales data | Update orchestration layer config: add `data_scope_tags: ["sales"]` to new agent's permitted scope. **No re-ingestion.** |
| Existing agent's access expanded to include finance | Update orchestration config. **No re-ingestion.** |
| Agent's access revoked | Update orchestration config. **No re-ingestion.** |
| Document reclassified to higher sensitivity tier | Re-ingest + migrate to correct collection. (See `03-data-classification.md`) |

The last row is the important exception: **sensitivity changes do require re-ingestion** because sensitivity determines collection placement, which is a hard boundary.

---

## Metadata Fields Used as Filter Primitives

These fields in the chunk payload are the filter inputs consumed by the orchestration layer. They must be accurately populated at ingest time and indexed in Qdrant:

| Field | Role | Ingest Requirement |
|---|---|---|
| `sensitivity` | Primary collection gate | Classified at ingest per Section 3 of `03-data-classification.md`. Never deferred. |
| `team_ids` | Team-scoped filter | Populated from source system wherever available. Never fabricated. |
| `allowed_groups` | Group-level filter | Populated from source system permissions at ingest. |
| `data_scope_tags` | Semantic scope filter | Populated using the controlled taxonomy in `02-metadata-schema.md`. |
| `org_id` / `workspace_id` | Tenant isolation | Always populated. Never omitted. |

---

## Audit Trail

Every retrieval request that passes through access control should produce an audit record. Your ingestion pipeline supports this by ensuring filter primitive fields are accurately populated — they are recorded in the audit log alongside each request.

Minimum audit record per retrieval request:

```python
audit_record = {
    "request_id": "...",          # Trace ID for this request
    "user_id": "...",             # Resolved from session store
    "agent_id": "...",            # Validated against Agent Access Registry
    "timestamp": "...",           # ISO 8601 UTC
    "permission_scope_applied": { # The resolved scope that was enforced
        "collections": ["company_memory"],
        "sensitivity": ["public", "internal"],
        "team_ids": ["team_sales_a"],
        "data_scope_tags": ["sales"],
    },
    "query_text": "...",          # Query submitted by the agent
    "doc_ids_returned": ["..."],  # Which chunks were returned
    "chunks_count": 7,
}
```

Given any `request_id`, operators must be able to reconstruct: who asked, which agent was acting, what filters were applied, and which chunks were returned.

---

## Security Anti-Patterns to Avoid

| Anti-Pattern | Why It's Wrong | Correct Approach |
|---|---|---|
| Storing `allowed_agents` in chunk payload | Re-ingestion required on every access change; privilege escalation risk | Store agent→scope mappings in orchestration layer config |
| Filtering access post-retrieval | Content briefly retrieved before being discarded; reduces result quality | Apply filters inside Qdrant query |
| Using `model_inferred` sensitivity as sole basis for access decisions | Variable confidence; untestable; may misclassify sensitive content | Require `source_asserted` or `system_derived` classification; log and review model-inferred |
| Skipping `data_scope_tags` when source context is known | Chunks become broadly accessible by default | Always populate scope tags when organizational context is determinable |
| Creating one collection per agent | Exhausts cluster resources; forces re-ingestion on access changes | Use scope tags + orchestration-layer filtering on a shared collection |

FILE:guides/05-chunking-standards.md
# Guide 05 — Chunking Standards

## Purpose

Chunking determines the retrieval unit — the piece of text returned to the LLM when a query matches. Chunks that are too large dilute relevance; chunks that are too small lack context. This guide defines chunk sizes, overlap, strategies, and determinism requirements by content type.

---

## The Golden Rule of Chunking

**The same normalized source input must always produce the same chunk set.** Given identical normalized text and the same `chunker_version`, the pipeline must produce:
- Identical chunk boundaries
- Identical chunk text
- Identical `content_hash` values
- Identical `doc_id` values

Non-deterministic chunking (e.g., using random seeds, model sampling) is prohibited in production pipelines.

---

## Chunking Parameters by Content Type

| Content Type | Token Range | Overlap | Strategy |
|---|---|---|---|
| Messages (Slack, short DMs) | 150–300 | 30 | Sentence window |
| Long email threads | 200–400 | 30 | Split at reply boundaries first, then sentence window within each reply |
| Meeting transcripts | 150–250 | 20 | Split at speaker turns first, then chunk within each speaker turn |
| Documents (Drive, PDFs, long-form) | 300–500 | 50 | Hierarchical: paragraph-level chunking with heading structure preserved |
| Tasks and tickets (ClickUp, Jira, etc.) | Up to 512 | N/A | One chunk per task; split only if token limit exceeded |
| Code files | 200–400 | 50 | Function/class boundary-aware; never split in the middle of a function |
| Structured data (tables, CSV rows) | N/A | N/A | One row or logical unit per chunk; include column headers in every chunk |

**Token count approximation:** Use the fixed heuristic of **4 characters per token** for fast, deterministic token estimation without requiring a tokenizer in the ingestion pipeline.

---

## Chunking Strategies Explained

### 1. Sentence Window (Messages, Short Content)

Chunk at sentence boundaries. Use sliding window with overlap to preserve context across chunk edges.

```python
import re

def chunk_sentence_window(
    text: str,
    max_tokens: int = 256,
    overlap_tokens: int = 30,
    chars_per_token: int = 4,
) -> list[str]:
    """Chunk text using a sliding sentence window."""
    
    max_chars = max_tokens * chars_per_token
    overlap_chars = overlap_tokens * chars_per_token
    
    # Split into sentences (simple heuristic; use a proper sentence splitter for production)
    sentences = re.split(r"(?<=[.!?])\s+", text.strip())
    
    chunks = []
    current_chunk = []
    current_len = 0
    
    for sentence in sentences:
        sent_len = len(sentence)
        
        if current_len + sent_len > max_chars and current_chunk:
            chunks.append(" ".join(current_chunk))
            # Keep overlap: retain last N chars worth of sentences
            overlap_kept = []
            overlap_len = 0
            for s in reversed(current_chunk):
                if overlap_len + len(s) <= overlap_chars:
                    overlap_kept.insert(0, s)
                    overlap_len += len(s)
                else:
                    break
            current_chunk = overlap_kept
            current_len = overlap_len
        
        current_chunk.append(sentence)
        current_len += sent_len
    
    if current_chunk:
        chunks.append(" ".join(current_chunk))
    
    return chunks
```

### 2. Reply-Boundary Chunking (Email Threads)

Split first at reply boundaries, then apply sentence window within each reply segment.

```python
REPLY_DELIMITERS = [
    r"^On .+wrote:$",
    r"^-----Original Message-----",
    r"^From:.+Sent:",
    r"^>{1,}\s",
]

def chunk_email_thread(text: str, max_tokens: int = 350) -> list[str]:
    """Split email thread at reply boundaries, then chunk each segment."""
    delimiter_pattern = "|".join(REPLY_DELIMITERS)
    
    # Split at reply delimiters
    segments = re.split(delimiter_pattern, text, flags=re.MULTILINE)
    segments = [s.strip() for s in segments if s.strip()]
    
    chunks = []
    for segment in segments:
        if len(segment) / 4 <= max_tokens:
            chunks.append(segment)
        else:
            chunks.extend(chunk_sentence_window(segment, max_tokens=max_tokens))
    
    return chunks
```

### 3. Speaker-Turn Chunking (Meeting Transcripts)

Split at speaker turn boundaries (identified by speaker attribution markers), then chunk within turns if they are long. Speaker attribution is **critical** — never split a speaker attribution from the text that follows it.

```python
def chunk_transcript(
    transcript_segments: list[dict],
    max_tokens: int = 200,
    overlap_tokens: int = 20,
    context_header: str = "",
) -> list[dict]:
    """
    Chunk a meeting transcript at speaker turns.
    Each segment is dict: {"speaker": "Jane", "text": "...", "start_ms": 0}
    """
    chunks = []
    
    for segment in transcript_segments:
        speaker = segment["speaker"]
        text = segment["text"]
        turn_text = f"[{speaker}]: {text}"
        
        if len(turn_text) / 4 <= max_tokens:
            # Fits in one chunk
            chunks.append({
                "text": turn_text,
                "context_header": context_header,
                "speaker": speaker,
                "start_ms": segment.get("start_ms", 0),
            })
        else:
            # Split long turn with overlap
            sub_chunks = chunk_sentence_window(text, max_tokens=max_tokens, overlap_tokens=overlap_tokens)
            for i, sub in enumerate(sub_chunks):
                chunks.append({
                    "text": f"[{speaker}]: {sub}",
                    "context_header": context_header,
                    "speaker": speaker,
                    "start_ms": segment.get("start_ms", 0),
                    "sub_chunk_index": i,
                })
    
    return chunks
```

### 4. Hierarchical Paragraph Chunking (Documents)

Preserve document structure. Chunk at paragraph boundaries. Include the nearest heading as a context header for each chunk, so chunks are self-contained without requiring the reader to know which section they came from.

```python
def chunk_document_hierarchical(
    markdown_text: str,
    max_tokens: int = 400,
    overlap_tokens: int = 50,
) -> list[dict]:
    """
    Chunk a Markdown document hierarchically.
    Splits at paragraph boundaries, tracks heading context.
    """
    lines = markdown_text.split("\n")
    chunks = []
    current_heading = ""
    current_paragraphs = []
    current_len = 0
    
    for line in lines:
        # Track heading context
        if line.startswith("#"):
            if current_paragraphs and current_len > 0:
                _flush_doc_chunk(current_paragraphs, current_heading, max_tokens, overlap_tokens, chunks)
                current_paragraphs = []
                current_len = 0
            current_heading = line.lstrip("#").strip()
            continue
        
        # Empty line = paragraph boundary
        if not line.strip():
            if current_len > 0 and current_len >= (max_tokens * 4 * 0.7):  # 70% full → flush
                _flush_doc_chunk(current_paragraphs, current_heading, max_tokens, overlap_tokens, chunks)
                current_paragraphs = []
                current_len = 0
            continue
        
        current_paragraphs.append(line)
        current_len += len(line)
    
    if current_paragraphs:
        _flush_doc_chunk(current_paragraphs, current_heading, max_tokens, overlap_tokens, chunks)
    
    return chunks

def _flush_doc_chunk(paragraphs, heading, max_tokens, overlap_tokens, chunks):
    text = " ".join(paragraphs)
    header = f"[Section: {heading}] " if heading else ""
    if len(text) / 4 <= max_tokens:
        chunks.append({"text": text, "section_heading": heading, "context_header": header})
    else:
        for sub in chunk_sentence_window(text, max_tokens, overlap_tokens):
            chunks.append({"text": sub, "section_heading": heading, "context_header": header})
```

---

## Context Headers

For threaded conversations, meeting transcripts, and email threads, prepend a **context header** to each chunk. The context header is separate from the chunk body's token count.

The context header ensures the chunk is self-contained — a retrieval result makes sense without the reader having to navigate back to the source.

**Format by type:**

| Source | Context Header Format |
|---|---|
| Slack thread | `[Channel: #sales-emea] [Thread from: Jane Smith, Apr 30 2024]` |
| Email thread | `[Subject: Q2 Budget Review] [From: [email protected], Apr 30 2024]` |
| Meeting transcript | `[Meeting: Q3 Forecast Review \| Date: 2024-04-30 \| Attendees: Jane, Bob, Alice]` |
| Document section | `[Document: Product Roadmap Q4 \| Section: Engineering Priorities]` |

---

## Chunk Metadata Fields

Every chunk must carry these fields in its payload:

```python
def build_chunk_payload(
    parent_metadata: dict,
    chunk_text: str,
    chunk_index: int,
    chunk_total: int,
    chunk_strategy: str,
    context_header: str = "",
) -> dict:
    import hashlib
    
    content_hash = hashlib.sha256(chunk_text.encode("utf-8")).hexdigest()
    token_count = len(chunk_text) // 4  # 4 chars per token approximation
    
    payload = {
        **parent_metadata,  # Inherit all parent document metadata
        "doc_id": f"{parent_metadata['source_type']}_{parent_metadata['source_record_id']}_{chunk_index}",
        "chunk_index": chunk_index,
        "chunk_total": chunk_total,
        "content_hash": content_hash,
        "token_count": token_count,
        "chunk_strategy": chunk_strategy,
        "context_header": context_header,
    }
    return payload
```

---

## Chunking Rules (Summary)

1. Every chunk carries the full universal metadata payload (`chunk_index`, `chunk_total`, `parent_doc_id` — no exceptions).
2. Chunks must not split mid-sentence. Always break at sentence or structural boundaries.
3. Context headers are prepended but do NOT count toward the chunk's token limit.
4. Token counts are approximated at **4 characters per token** — do not call a tokenizer for this.
5. The `chunk_strategy` field must accurately reflect the strategy used.
6. Chunk size and overlap values must be validated with retrieval testing before production deployment.
7. When re-ingesting a document that now produces **fewer chunks** than before, delete the out-of-range chunks (those with `chunk_index` ≥ new total) from Qdrant.

---

## Common Mistakes

| Mistake | Impact | Fix |
|---|---|---|
| Splitting a speaker attribution from its text | Destroys attribution in retrieval | Always keep `[Speaker]:` + text together |
| Including quoted email content in chunks | Duplicate information in the index | Strip quoted content during normalization |
| Using model token counters in the chunking loop | Non-determinism if model version changes | Use 4 chars/token heuristic |
| Creating chunks of 50 tokens or less | Too small to carry useful context | Enforce minimum chunk size of 100 tokens |
| Ignoring section headings in documents | Chunks lack context, retrieval suffers | Always prepend heading as context header |
| Not deleting stale chunks after re-ingestion | Index accumulates orphan chunks | Always compare old vs. new `chunk_total` |

FILE:guides/10-operational-standards.md
# Guide 10 — Operational Standards

## Purpose

This guide covers the operational requirements for running a production Qdrant RAG system: schema versioning, data retention, monitoring, conformance testing, and observability.

---

## Schema Versioning and Migration

### Version Management

The `schema_version` field enables backward-compatible evolution. When the schema is updated:

1. Increment the version number in the schema definition
2. Update all ingestion pipelines to write the new version
3. Provide a migration script for existing chunks
4. Configure the retrieval layer to gracefully handle chunks written under prior schema versions during the migration window

### Migration Pattern

```python
def migrate_schema(
    client: QdrantClient,
    collection: str,
    from_version: str,
    to_version: str,
    migration_fn: callable,
    migration_run_id: str,
    batch_size: int = 100,
):
    """
    Migrate all chunks in a collection from one schema version to another.
    Safe to re-run (idempotent): skips chunks already on the target version.
    """
    offset = None
    migrated = 0
    skipped = 0
    
    while True:
        results, next_offset = client.scroll(
            collection_name=collection,
            scroll_filter=models.Filter(
                must=[
                    models.FieldCondition(
                        key="schema_version",
                        match=models.MatchValue(value=from_version)
                    )
                ]
            ),
            with_payload=True,
            limit=batch_size,
            offset=offset,
        )
        
        if not results:
            break
        
        for point in results:
            new_payload = migration_fn(point.payload)
            new_payload["schema_version"] = to_version
            new_payload["migration_run_id"] = migration_run_id
            
            client.set_payload(
                collection_name=collection,
                payload=new_payload,
                points=[point.id],
            )
            migrated += 1
        
        offset = next_offset
        if offset is None:
            break
    
    return {"migrated": migrated, "skipped": skipped}
```

### Migration Note: v2.1 → v2.2 (data_scope_tags)

During migration from v2.1 to v2.2, chunks without `data_scope_tags` will have no scope tags. Configure your orchestration layer to treat absent `data_scope_tags` as a pass-through (no tag filter applied) to avoid breaking retrieval for untagged content. Backfill high-value collections first. Record the `migration_run_id` on all backfilled chunks.

---

## Data Retention

```python
from datetime import datetime, timezone, timedelta

def run_retention_cleanup(client: QdrantClient, collection: str, batch_size: int = 500):
    """
    Delete chunks whose retention period has expired.
    retention_days counts from created_at.
    """
    now = datetime.now(timezone.utc)
    deleted_count = 0
    
    # Find expired chunks
    # Strategy: compute cutoff dates for common retention windows
    # (Alternative: use Qdrant datetime range filter if created_at is indexed as datetime)
    
    offset = None
    while True:
        results, next_offset = client.scroll(
            collection_name=collection,
            with_payload=True,
            limit=batch_size,
            offset=offset,
        )
        
        if not results:
            break
        
        expired_ids = []
        for point in results:
            payload = point.payload
            created_str = payload.get("created_at")
            retention_days = payload.get("retention_days", 1095)
            
            if not created_str:
                continue
            
            created_at = datetime.fromisoformat(created_str.replace("Z", "+00:00"))
            expires_at = created_at + timedelta(days=retention_days)
            
            if now > expires_at:
                expired_ids.append(point.id)
                # Log deletion event
                _log_deletion(
                    doc_id=payload.get("doc_id"),
                    parent_doc_id=payload.get("parent_doc_id"),
                    deletion_timestamp=now.isoformat(),
                    reason="retention_expired",
                )
        
        if expired_ids:
            client.delete(
                collection_name=collection,
                points_selector=models.PointIdsList(points=expired_ids),
            )
            deleted_count += len(expired_ids)
        
        offset = next_offset
        if offset is None:
            break
    
    return {"deleted": deleted_count}

def _log_deletion(doc_id, parent_doc_id, deletion_timestamp, reason):
    """Audit log for deletion events."""
    print(f"DELETED doc_id={doc_id} parent={parent_doc_id} at={deletion_timestamp} reason={reason}")
    # Write to your audit log store
```

**Retention defaults by content type:**

| Content Type | Default Retention |
|---|---|
| Operational / conversational (Slack, email, transcripts) | 3 years (1,095 days) |
| Project documentation | 5 years (1,825 days) |
| Financial / legal records | 7 years (2,555 days) — check regulatory requirements |
| PII content | Per data privacy regulations (often shorter) |

---

## Conformance Testing

Every ingestion pipeline must pass automated conformance checks before being promoted to production. These are mandatory gates in CI/CD.

### Mandatory Conformance Checks

```python
class PipelineConformanceTests:
    
    def test_metadata_completeness(self, client, collection, sample_size=1000):
        """All required fields must be non-null on every chunk."""
        REQUIRED_FIELDS = [
            "doc_id", "source_type", "source_system", "source_record_id",
            "document_content_hash", "chunk_index", "chunk_total", "content_hash",
            "created_at", "ingested_at", "modified_at", "author_id", "author_name",
            "workspace_id", "org_id", "sensitivity", "allowed_groups", "owner_user_ids",
            "is_pii", "retention_days", "data_scope_tags", "language", "content_type",
            "token_count", "chunk_strategy", "embedding_model", "schema_version",
            "ingestion_run_id", "connector_version", "normalizer_version", "chunker_version",
        ]
        
        results, _ = client.scroll(collection_name=collection, limit=sample_size, with_payload=True)
        
        failures = []
        for point in results:
            for field in REQUIRED_FIELDS:
                if point.payload.get(field) is None:
                    failures.append({"doc_id": point.payload.get("doc_id"), "missing_field": field})
        
        assert not failures, f"Metadata completeness failures: {failures}"
    
    def test_data_scope_tags_coverage(self, client, collection, source_type, threshold=0.80):
        """At least 80% of newly ingested chunks per source type must have ≥1 data_scope_tag."""
        from datetime import datetime, timedelta, timezone
        
        cutoff = (datetime.now(timezone.utc) - timedelta(days=1)).isoformat()
        
        results, _ = client.scroll(
            collection_name=collection,
            scroll_filter=models.Filter(
                must=[
                    models.FieldCondition(key="source_type", match=models.MatchValue(value=source_type)),
                    models.FieldCondition(key="ingested_at", range=models.DatetimeRange(gte=cutoff)),
                ]
            ),
            with_payload=True,
            limit=5000,
        )
        
        if not results:
            return  # No recent ingestion to check
        
        tagged = sum(1 for p in results if p.payload.get("data_scope_tags"))
        coverage = tagged / len(results)
        
        assert coverage >= threshold, (
            f"data_scope_tags coverage for {source_type} is {coverage:.1%}, "
            f"below {threshold:.0%} threshold. Investigate tagging pipeline."
        )
    
    def test_deterministic_chunking(self, client, golden_docs: list, pipeline):
        """Re-ingest golden documents and verify chunks are byte-identical."""
        for golden_doc in golden_docs:
            previous_chunks = golden_doc["expected_chunks"]
            actual_chunks = pipeline.chunk(golden_doc["normalized_text"], golden_doc["metadata"])
            
            assert len(actual_chunks) == len(previous_chunks), (
                f"Chunk count mismatch for {golden_doc['id']}: "
                f"expected {len(previous_chunks)}, got {len(actual_chunks)}"
            )
            for i, (expected, actual) in enumerate(zip(previous_chunks, actual_chunks)):
                assert expected == actual, f"Chunk {i} mismatch for {golden_doc['id']}"
    
    def test_idempotent_rerun(self, client, collection, source_record_id, pipeline):
        """Re-run pipeline for a document and verify Qdrant point count doesn't increase."""
        before_count = self._count_chunks(client, collection, source_record_id)
        pipeline.ingest_document(...)  # Re-ingest
        after_count = self._count_chunks(client, collection, source_record_id)
        
        assert after_count == before_count, (
            f"Idempotency failure: chunk count went from {before_count} to {after_count}"
        )
    
    def test_duplicate_suppression(self, client, collection):
        """No duplicate content_hash values within the same parent_doc_id."""
        # Check for duplicate content_hashes within the same parent
        # Implementation: scroll and group by parent_doc_id, check content_hash uniqueness
        pass
    
    def test_delete_propagation(self, client, collection, source_record_id, pipeline):
        """Delete a source record and verify chunks are removed within propagation SLA."""
        pipeline.delete_document(source_type="slack", source_record_id=source_record_id, collection=collection)
        remaining = self._count_chunks(client, collection, source_record_id)
        assert remaining == 0, f"Delete propagation failed: {remaining} chunks remain"
    
    def test_lineage_fields_complete(self, client, collection, sample_size=1000):
        """ingestion_run_id, connector_version, normalizer_version, chunker_version must be set."""
        LINEAGE_FIELDS = ["ingestion_run_id", "connector_version", "normalizer_version", "chunker_version"]
        results, _ = client.scroll(collection_name=collection, limit=sample_size, with_payload=True)
        failures = [
            point.payload.get("doc_id")
            for point in results
            if any(not point.payload.get(f) for f in LINEAGE_FIELDS)
        ]
        assert not failures, f"Lineage field gaps in {len(failures)} chunks"
    
    def test_access_control_smoke(self, query_fn, test_cases: list):
        """
        For each agent configuration, verify that returned chunks satisfy expected filters.
        This validates the end-to-end filter chain, not just chunk payload.
        """
        for case in test_cases:
            results = query_fn(
                query=case["query"],
                permission_scope=case["permission_scope"],
            )
            for result in results:
                p = result.payload
                assert p["sensitivity"] in case["permitted_sensitivity"], (
                    f"Access control violation: chunk {p['doc_id']} has sensitivity "
                    f"{p['sensitivity']}, not in {case['permitted_sensitivity']}"
                )
    
    def _count_chunks(self, client, collection, source_record_id):
        results, _ = client.scroll(
            collection_name=collection,
            scroll_filter=models.Filter(
                must=[models.FieldCondition(
                    key="source_record_id",
                    match=models.MatchValue(value=source_record_id)
                )]
            ),
            limit=1000,
        )
        return len(results)
```

---

## Monitoring and Alerting

Instrument your pipeline for these signals:

| Signal | Alert Threshold | Action |
|---|---|---|
| Ingestion failure rate | > 1% of documents in a run | Investigate connector or pipeline error |
| Schema validation errors | Any | Block pipeline promotion; fix immediately |
| `data_scope_tags` coverage | < 80% of newly ingested chunks for any source type | Investigate tagging pipeline within one sprint |
| Retrieval p95 latency | Exceeds SLA (e.g., 500ms for hybrid) | Scale compute or tune HNSW parameters |
| Collection point count | Within 20% of collection capacity | Plan scaling |
| Stale data detection | Most recent `ingested_at` for active source > configurable threshold | Check connector health |
| Dead-letter queue depth | > configurable threshold | Review failed lifecycle events |
| Duplicate `content_hash` count | Any within same `parent_doc_id` | Deduplication regression |

---

## Golden Document Registry

Maintain a registry of golden documents for each content type. These are used for determinism testing and regression validation.

Each golden document must include:
- The raw source input
- The expected normalized output
- The expected chunk set (chunk text + metadata)
- The `chunker_version` and `normalizer_version` that produced this output

**Required coverage patterns (per source type):**
- Edited/deleted messages within threads
- Forwarded emails with nested quoted replies
- Documents with duplicate content or broken formatting
- Transcript segments with filler words and low-confidence speech
- Tasks with extensive comment threads
- Content that has been moved or reclassified
- Content where `data_scope_tags` derivation is ambiguous

Version the golden document registry alongside `chunker_version` and `normalizer_version`. When either bumps, update the expected outputs and document why the output changed.

---

## Replay Capability

The ingestion system must support replaying ingestion for:
- A specific document (by `source_record_id`)
- All documents from a specific source type
- All documents ingested within a date range
- All documents associated with a specific `ingestion_run_id`

Replay must be idempotent: replaying always produces the same final state in Qdrant as the original run (assuming the source content hasn't changed).

```python
def replay_ingestion(
    pipeline: IngestionPipeline,
    scope: dict,  # {"source_record_id": "..."} or {"source_type": "slack"} or {"run_id": "..."}
):
    """
    Replay ingestion for a specific scope.
    Fetches source records from the source system and re-ingests them.
    """
    source_records = fetch_source_records(scope)
    results = []
    for record in source_records:
        result = pipeline.ingest_document(record["content"], record["metadata"], ...)
        results.append(result)
    return results
```

FILE:guides/06-embedding-models.md
# Guide 06 — Embedding Models

## Purpose

This guide covers embedding model selection, configuration, and usage for both dense-only and hybrid (dense + sparse) RAG pipelines built on Qdrant.

---

## Decision: Dense-Only vs. Hybrid

Choose **hybrid** (dense + sparse) when your content contains:
- Proper nouns, product names, model numbers, or identifiers
- Technical jargon, acronyms, or domain-specific terminology
- Code identifiers, error codes, or numeric values
- Queries where exact keyword match matters alongside semantic similarity

Choose **dense-only** when:
- Content is purely natural language with no specialized terminology
- Latency is critical and you need the simplest possible pipeline
- Your embedding budget favors one API call over two model passes
- You're prototyping and want to validate the concept first

**Recommended default:** Hybrid with BGE-M3. The single-model-pass architecture (one call produces both dense and sparse vectors) makes it operationally simple.

---

## Recommended Models

### Hybrid Pipeline: BAAI/BGE-M3

**Use when:** You need both semantic and keyword search. This is the recommended default for enterprise RAG.

- Dense dimension: **1024**
- Sparse: SPLADE-style output (produced in the same model pass)
- Distance metric: **Cosine**
- Language support: Multilingual (100+ languages)
- Context window: 8,192 tokens
- One model pass produces both vector types

```python
from FlagEmbedding import BGEM3FlagModel

model = BGEM3FlagModel("BAAI/bge-m3", use_fp16=True)

def embed_hybrid_bgem3(texts: list[str]) -> list[dict]:
    """
    Embed a list of texts using BGE-M3.
    Returns list of dicts with 'dense' and 'sparse' vectors.
    """
    output = model.encode(
        texts,
        batch_size=12,
        max_length=512,
        return_dense=True,
        return_sparse=True,
        return_colbert_vecs=False,  # Set True if using late interaction reranking
    )
    
    results = []
    for i in range(len(texts)):
        dense_vector = output["dense_vecs"][i].tolist()
        sparse_weights = output["lexical_weights"][i]  # dict: {token_id: weight}
        
        # Convert sparse weights to Qdrant SparseVector format
        indices = list(sparse_weights.keys())
        values = list(sparse_weights.values())
        
        results.append({
            "dense": dense_vector,
            "sparse": {"indices": indices, "values": values},
        })
    
    return results

# Alternatively, using Qdrant's built-in BM25 with IDF for sparse
# (simpler, no external model needed for sparse):
# Configure the collection with modifier=IDF and send raw tokens for sparse
```

**Collection setup for BGE-M3:**

```python
from qdrant_client import QdrantClient, models

client = QdrantClient(url="http://localhost:6333")

client.create_collection(
    collection_name="my_collection",
    vectors_config={
        "dense": models.VectorParams(
            size=1024,
            distance=models.Distance.COSINE,
        )
    },
    sparse_vectors_config={
        "sparse": models.SparseVectorParams(
            modifier=models.Modifier.IDF  # Optional: use IDF weighting for sparse
        )
    },
)
```

---

### Dense-Only Pipeline: OpenAI text-embedding-3-small

**Use when:** You want a simple, cost-efficient dense embedding with strong performance. No local model hosting required.

- Dense dimension: **1536**
- Distance metric: **Cosine**
- Language support: Strong multilingual support
- API-based (no local GPU required)
- Supports Matryoshka truncation (can use 512-dim for faster/cheaper retrieval)

```python
from openai import OpenAI

openai_client = OpenAI()

def embed_dense_openai_small(texts: list[str], dimensions: int = 1536) -> list[list[float]]:
    """
    Embed texts using text-embedding-3-small.
    Set dimensions=512 for faster retrieval at some quality cost.
    """
    response = openai_client.embeddings.create(
        model="text-embedding-3-small",
        input=texts,
        dimensions=dimensions,
    )
    return [item.embedding for item in response.data]
```

**Collection setup for text-embedding-3-small:**

```python
client.create_collection(
    collection_name="my_collection",
    vectors_config=models.VectorParams(
        size=1536,  # or 512 if using Matryoshka truncation
        distance=models.Distance.COSINE,
        hnsw_config=models.HnswConfigDiff(m=16, ef_construct=100),
    ),
    quantization_config=models.ScalarQuantization(
        scalar=models.ScalarQuantizationConfig(
            type=models.ScalarType.INT8,
            quantile=0.99,
            always_ram=True,
        )
    ),
)
```

---

### Dense-Only (High Quality): OpenAI text-embedding-3-large

**Use when:** Maximum embedding quality is needed and cost is not the primary constraint.

- Dense dimension: **3072**
- Distance metric: **Cosine**
- Supports Matryoshka truncation down to 256 dimensions

```python
def embed_dense_openai_large(texts: list[str], dimensions: int = 3072) -> list[list[float]]:
    response = openai_client.embeddings.create(
        model="text-embedding-3-large",
        input=texts,
        dimensions=dimensions,
    )
    return [item.embedding for item in response.data]
```

---

### Sparse-Only Options (for Hybrid without BGE-M3)

If you're using a different dense model but still want hybrid retrieval, you can use these for the sparse side:

| Model | Notes |
|---|---|
| Qdrant BM25 with `modifier=IDF` | Native Qdrant BM25, no external model needed. Good baseline. |
| `BAAI/bge-m3` SPLADE output | Best quality sparse vectors, but requires BGE-M3. |
| `prithivida/Splade_PP_en_v1` | SPLADE model via FastEmbed. Good quality for English. |
| `Qdrant/bm42-all-minilm-l6-v2-attentions` | BM42 — attention-weighted sparse. Better than BM25 for chunked text. |

**BM42 setup (recommended sparse-only companion to any dense model):**

```python
from fastembed import SparseTextEmbedding

sparse_model = SparseTextEmbedding(model_name="Qdrant/bm42-all-minilm-l6-v2-attentions")

def embed_sparse_bm42(texts: list[str]) -> list[dict]:
    embeddings = list(sparse_model.embed(texts))
    return [
        {"indices": e.indices.tolist(), "values": e.values.tolist()}
        for e in embeddings
    ]
```

---

## Query vs. Document Embedding

For BGE-M3, use `encode_queries` for query embedding and `encode` for document embedding. This applies the E5-style instruction prefix that improves retrieval quality:

```python
# At ingestion time — embed documents
doc_embeddings = model.encode(documents, ...)

# At retrieval time — embed queries
query_embeddings = model.encode_queries(
    queries,
    return_dense=True,
    return_sparse=True,
)
```

For OpenAI models, the same model and API endpoint is used for both documents and queries.

---

## Batching and Performance

| Scenario | Recommendation |
|---|---|
| BGE-M3 local inference | Batch size 8–16 on GPU, 2–4 on CPU |
| OpenAI embeddings | Batch up to 2,048 texts per API call |
| Large ingestion runs | Process in batches; implement exponential backoff on rate limit errors |
| Real-time query embedding | Single text per call is fine; consider caching frequent queries |

```python
def embed_in_batches(texts: list[str], batch_size: int = 16, embed_fn) -> list:
    """Embed texts in batches with progress tracking."""
    all_results = []
    for i in range(0, len(texts), batch_size):
        batch = texts[i:i + batch_size]
        results = embed_fn(batch)
        all_results.extend(results)
    return all_results
```

---

## Model Selection Summary

| Use Case | Dense Model | Sparse Model | Collection dims |
|---|---|---|---|
| Enterprise RAG (recommended) | BGE-M3 (1024-dim) | BGE-M3 SPLADE | 1024 dense + sparse |
| Simple/budget RAG | text-embedding-3-small | Qdrant BM25 IDF | 1536 dense + sparse |
| Highest quality dense-only | text-embedding-3-large | None | 3072 dense |
| Standard dense-only | text-embedding-3-small | None | 1536 dense |
| Multilingual enterprise | BGE-M3 | BGE-M3 SPLADE | 1024 dense + sparse |

---

## Important: Use the Same Model at Query Time

The embedding model used during retrieval (query time) **must match** the model used during ingestion. Mixing models produces meaningless similarity scores.

Record the model in every chunk's payload:
```json
{
  "embedding_model": "BAAI/bge-m3",
  "sparse_model": "BAAI/bge-m3-splade"
}
```

Store the model identifier in your pipeline configuration and validate it at startup.

FILE:guides/08-retrieval-architecture.md
# Guide 08 — Retrieval Architecture

## Purpose

This guide covers the full retrieval pipeline: hybrid search with RRF, access control filter application, reranking, result assembly, and quality requirements.

---

## The Core Rule: Filters Inside the Query

Access control filters must be applied **within** the Qdrant query — not post-retrieval.

**Why filtering post-retrieval is wrong:**
- Qdrant returns top-K candidates from the full vector space before your code sees them
- If you filter afterward, you may discard most results, getting fewer than requested
- Post-filtering creates a window where restricted content is briefly retrieved, which is a compliance risk
- It wastes compute fetching content that will be discarded

**Always pass filters as Qdrant payload filter conditions, built server-side by your orchestration layer.**

---

## Hybrid Retrieval Pipeline (Recommended)

The standard pipeline for enterprise RAG using BGE-M3:

```
User/Agent Query
       │
       ▼
  Embed query with BGE-M3
  (produces dense + sparse in one pass)
       │
       ├─────────────────────────────────────────────┐
       ▼                                             ▼
  Dense Search (top-20)                    Sparse Search (top-20)
  + access control filters                 + access control filters
       │                                             │
       └──────────────────┬──────────────────────────┘
                          ▼
              RRF Fusion (k=60)
              Combined ranking
                          │
                          ▼
              Top 10–15 candidates
                          │
                          ▼
         Optional: Cross-encoder reranking
         (BGE-Reranker-v2 → top 5–8)
                          │
                          ▼
              Return results + attribution
```

---

## Implementation

### Dense-Only Query

```python
from qdrant_client import QdrantClient, models

def retrieve_dense(
    client: QdrantClient,
    collection: str,
    query_vector: list[float],
    access_filters: models.Filter,
    top_k: int = 20,
) -> list:
    """Single dense vector search with access control filters."""
    
    results = client.query_points(
        collection_name=collection,
        query=query_vector,
        using="dense",
        query_filter=access_filters,  # Filters applied INSIDE Qdrant query
        limit=top_k,
        with_payload=True,
    )
    return results.points
```

### Hybrid Query with RRF (Recommended)

```python
def retrieve_hybrid_rrf(
    client: QdrantClient,
    collection: str,
    query_dense: list[float],
    query_sparse: dict,  # {"indices": [...], "values": [...]}
    access_filters: models.Filter,
    prefetch_k: int = 20,
    final_k: int = 10,
) -> list:
    """
    Hybrid retrieval: dense + sparse with RRF fusion.
    Filters are applied inside each prefetch branch.
    """
    
    results = client.query_points(
        collection_name=collection,
        prefetch=[
            # Dense branch
            models.Prefetch(
                query=query_dense,
                using="dense",
                filter=access_filters,  # Filters inside the prefetch
                limit=prefetch_k,
            ),
            # Sparse branch
            models.Prefetch(
                query=models.SparseVector(
                    indices=query_sparse["indices"],
                    values=query_sparse["values"],
                ),
                using="sparse",
                filter=access_filters,  # Same filters applied to sparse branch
                limit=prefetch_k,
            ),
        ],
        query=models.FusionQuery(fusion=models.Fusion.RRF),  # RRF fusion
        limit=final_k,
        with_payload=True,
    )
    return results.points
```

### Multi-Collection Query (Searching Across Multiple Sensitivity Tiers)

```python
def retrieve_across_collections(
    client: QdrantClient,
    permitted_collections: list[str],
    query_dense: list[float],
    query_sparse: dict,
    access_filters: models.Filter,
    top_k: int = 10,
) -> list:
    """
    Query multiple collections in parallel and merge results.
    Use when a user has access to multiple sensitivity tiers.
    """
    import asyncio
    
    all_results = []
    for collection in permitted_collections:
        results = retrieve_hybrid_rrf(
            client, collection, query_dense, query_sparse,
            access_filters, prefetch_k=20, final_k=top_k
        )
        all_results.extend(results)
    
    # Apply RRF across collection results
    all_results.sort(key=lambda r: r.score, reverse=True)
    return all_results[:top_k]
```

---

## Building Access Control Filters

Access filters are constructed by your orchestration layer based on the resolved permission scope for the current user/agent pair. **The AI agent never constructs these filters.**

```python
def build_access_filter(permission_scope: dict) -> models.Filter:
    """
    Build a Qdrant filter from the resolved permission scope.
    
    permission_scope = {
        "org_id": "org-456",
        "workspace_id": "ws-123",
        "permitted_sensitivity": ["internal", "public"],
        "team_ids": ["team_sales_a"],           # None = no team restriction
        "allowed_groups": ["g_all_staff"],       # None = no group restriction
        "data_scope_tags": ["sales", "region:emea"],  # None = no scope restriction
    }
    """
    
    must_conditions = [
        # Always apply org and workspace isolation first
        models.FieldCondition(
            key="org_id",
            match=models.MatchValue(value=permission_scope["org_id"])
        ),
        models.FieldCondition(
            key="workspace_id",
            match=models.MatchValue(value=permission_scope["workspace_id"])
        ),
        # Sensitivity tier filter
        models.FieldCondition(
            key="sensitivity",
            match=models.MatchAny(any=permission_scope["permitted_sensitivity"])
        ),
    ]
    
    # Optional: team-scoped filter
    if permission_scope.get("team_ids"):
        must_conditions.append(
            models.FieldCondition(
                key="team_ids",
                match=models.MatchAny(any=permission_scope["team_ids"])
            )
        )
    
    # Optional: group-scoped filter
    if permission_scope.get("allowed_groups"):
        must_conditions.append(
            models.FieldCondition(
                key="allowed_groups",
                match=models.MatchAny(any=permission_scope["allowed_groups"])
            )
        )
    
    # Optional: data scope tag filter
    if permission_scope.get("data_scope_tags"):
        must_conditions.append(
            models.FieldCondition(
                key="data_scope_tags",
                match=models.MatchAny(any=permission_scope["data_scope_tags"])
            )
        )
    
    return models.Filter(must=must_conditions)
```

---

## RRF Parameters

Reciprocal Rank Fusion score: `score(d) = Σ 1/(k + rank_i(d))` for each ranked list i.

| Parameter | Default | Notes |
|---|---|---|
| `k` | 60 | Smoothing constant. Higher k = less differentiation at the top. 60 is the standard. |
| Dense prefetch | 20 | Retrieve top-20 from dense search before fusion |
| Sparse prefetch | 20 | Retrieve top-20 from sparse search before fusion |
| Final results | 10–15 | After RRF fusion, take top-10 to top-15 |
| After reranking | 5–8 | Final results passed to LLM after cross-encoder reranking |

**Only change k if measured retrieval quality degrades** and you have evidence that a different k improves it.

---

## Optional: Cross-Encoder Reranking

After RRF fusion, optionally apply a cross-encoder reranker to the top-N candidates. This produces higher quality final rankings but adds latency.

```python
from sentence_transformers import CrossEncoder

reranker = CrossEncoder("BAAI/bge-reranker-v2-m3")  # Recommended for multilingual

def rerank_results(query: str, candidates: list, top_k: int = 7) -> list:
    """
    Apply cross-encoder reranking to RRF candidates.
    Only use on small candidate sets (top 10–15 from RRF).
    """
    if not candidates:
        return candidates
    
    pairs = [(query, c.payload.get("text", "")) for c in candidates]
    scores = reranker.predict(pairs)
    
    ranked = sorted(
        zip(scores, candidates),
        key=lambda x: x[0],
        reverse=True,
    )
    
    return [c for _, c in ranked[:top_k]]
```

**Reranker options:**
| Model | Notes |
|---|---|
| `BAAI/bge-reranker-v2-m3` | Multilingual, recommended for enterprise |
| `BAAI/bge-reranker-large` | High quality English-focused |
| `cross-encoder/ms-marco-MiniLM-L-6-v2` | Fast, English, lower latency |

---

## Result Assembly and Attribution

Every result returned to the LLM must include human-readable attribution. Never return a chunk without its source attribution.

```python
def assemble_results(raw_points: list) -> list[dict]:
    """Format Qdrant results for LLM consumption."""
    assembled = []
    
    for point in raw_points:
        p = point.payload
        assembled.append({
            "text": p.get("text", ""),
            "context_header": p.get("context_header", ""),
            "attribution": {
                "source": p.get("source_type"),
                "title": p.get("title"),
                "author": p.get("author_name"),
                "created_at": p.get("created_at"),
                "doc_id": p.get("doc_id"),
            },
            "score": point.score,
        })
    
    return assembled
```

---

## Near-Duplicate Suppression

Near-duplicate chunks should be collapsed so the top results are not dominated by repeated content.

```python
def suppress_near_duplicates(
    results: list,
    similarity_threshold: float = 0.95,
) -> list:
    """
    Remove results that are near-duplicates of higher-ranked results.
    Uses content_hash for exact duplicates, parent_doc_id for near-duplicates.
    """
    seen_hashes = set()
    seen_parent_ids = {}  # parent_doc_id → best chunk score
    deduplicated = []
    
    for result in results:
        content_hash = result.payload.get("content_hash", "")
        parent_id = result.payload.get("parent_doc_id", "")
        
        # Exact duplicate: same content_hash
        if content_hash in seen_hashes:
            continue
        seen_hashes.add(content_hash)
        
        # Near-duplicate: same parent document, score within threshold of the best
        if parent_id in seen_parent_ids:
            best_score = seen_parent_ids[parent_id]
            if result.score >= best_score * similarity_threshold:
                continue  # Too similar, skip this one
        else:
            seen_parent_ids[parent_id] = result.score
        
        deduplicated.append(result)
    
    return deduplicated
```

---

## Retrieval Quality Requirements

| Requirement | Standard |
|---|---|
| Minimum context | Returned chunks must contain enough surrounding context to be useful without navigating to source |
| Attribution | Every result includes source system, document/conversation title, author, and timestamp |
| Near-duplicate handling | Top results must not be dominated by repeated content from the same source document |
| Version freshness | When multiple versions of the same document exist, the current version outranks superseded versions |
| Latency | p95 retrieval latency must stay within defined SLA (typically <500ms for hybrid, <200ms for dense-only) |

---

## Retrieval Path Observability

Log the following for every retrieval request:

```python
{
    "request_id": "...",          # Trace ID for the full request
    "query_text": "...",          # Query submitted
    "collection": "...",          # Collection(s) queried
    "filters_applied": {          # Filters applied (sensitivity, team_ids, scope_tags)
        "sensitivity": ["internal"],
        "team_ids": ["team_sales_a"],
        "data_scope_tags": ["sales"],
    },
    "dense_candidates": 20,       # Candidates from dense search
    "sparse_candidates": 18,      # Candidates from sparse search
    "rrf_candidates": 15,         # After RRF fusion
    "after_reranking": 7,         # After reranking
    "after_dedup": 6,             # After near-duplicate suppression
    "doc_ids_returned": ["..."],  # Final doc_ids returned to LLM
    "latency_ms": 145,            # Total retrieval latency
}
```

FILE:guides/02-metadata-schema.md
# Guide 02 — Metadata Schema Standards

## Purpose

Every chunk written to Qdrant must carry a complete, well-structured metadata payload. This guide defines the universal schema, field confidence levels, payload indexing requirements, and source-specific extension namespacing.

The metadata schema is the **contract** between your ingestion pipeline and your retrieval + access control layers. Gaps in metadata cannot be corrected at retrieval time.

---

## Schema Version

Current version: **2.2**

Every chunk must include `schema_version: "2.2"`. When the schema evolves, bump this field and provide a migration guide.

---

## Metadata Confidence Levels

Every field has an implicit confidence level. This determines whether it can be used for filtering and security decisions.

| Level | Meaning | Can Filter On? | Can Use for Security? |
|---|---|---|---|
| `source_asserted` | Value came directly from the source system API | ✅ Yes | ✅ Yes |
| `system_derived` | Value computed deterministically by the pipeline | ✅ Yes | ✅ Yes |
| `model_inferred` | Value produced by an ML classifier or heuristic | ⚠️ Log only | ❌ No (unless separately validated) |

**Rule:** Never use `model_inferred` fields as the sole basis for sensitivity classification, access control, or routing decisions.

---

## Universal Payload Schema

Every chunk in every collection must carry all required fields below. No exceptions.

### Identity and Provenance

| Field | Type | Required | Description |
|---|---|---|---|
| `doc_id` | string | ✅ | Globally unique chunk identifier. Format: `{source}_{entity_id}_{chunk_index}`. Must be deterministic. |
| `source_type` | string | ✅ | Canonical source system name: `slack`, `fireflies`, `clickup`, `email`, `gdrive`, etc. |
| `source_system` | string | ✅ | Specific instance or workspace identifier (e.g., workspace name or URL). |
| `source_record_id` | string | ✅ | Native ID of the source record in the originating system. |
| `integration_id` | string | ✅ | Identifier of the integration pipeline that produced this chunk. |
| `parent_doc_id` | string | ✅ | ID of the parent document or conversation this chunk belongs to. |
| `document_content_hash` | string | ✅ | SHA-256 of the full normalized source document. Used for change detection. |

**Identifier Precedence:**
- Source identity: `source_system` + `source_record_id` uniquely identifies the original record.
- Content equivalence: `document_content_hash` detects whether source content changed.
- Chunk identity & upsert key: `doc_id` is the authoritative key for all Qdrant upsert/delete operations.
- Chunk deduplication: `content_hash` detects duplicate chunks within a source.

### Chunk Hierarchy

| Field | Type | Required | Description |
|---|---|---|---|
| `chunk_index` | integer | ✅ | Zero-based position of this chunk within its parent document. |
| `chunk_total` | integer | ✅ | Total chunks produced from the parent document. |
| `content_hash` | string | ✅ | SHA-256 of the normalized chunk text. Used for within-source deduplication. |

### Timestamps

| Field | Type | Required | Description |
|---|---|---|---|
| `created_at` | ISO 8601 | ✅ | When the original content was created in the source system. |
| `ingested_at` | ISO 8601 | ✅ | When this chunk was written to Qdrant. |
| `modified_at` | ISO 8601 | ✅ | When the original content was last modified in the source system. |
| `source_last_modified` | ISO 8601 | ✅ | Source system's own last-modified timestamp. |

All timestamps must be stored in **UTC** using ISO 8601 format: `2024-11-01T14:30:00Z`.

### Authorship

| Field | Type | Required | Description |
|---|---|---|---|
| `author_id` | string | ✅ | Unique ID of the content author in the source system. |
| `author_name` | string | ✅ | Display name of the content author. |
| `author_email` | string | Conditional | Required when available from the source system. |
| `author_department` | string | Conditional | Required when organizational directory mapping is available. |

### Organizational Context

| Field | Type | Required | Description |
|---|---|---|---|
| `project_ids` | string[] | Optional | List of project IDs. Empty array if source doesn't model projects. |
| `team_ids` | string[] | Optional | List of team IDs. Critical for permission scope resolution. Never fabricate. |
| `workspace_id` | string | ✅ | Workspace or tenant identifier. |
| `org_id` | string | ✅ | Top-level organization identifier. All retrieval is scoped to this value. |

### Governance and Classification

| Field | Type | Required | Description |
|---|---|---|---|
| `sensitivity` | enum | ✅ | One of: `public`, `internal`, `restricted`, `confidential`. Determines target collection. |
| `allowed_groups` | string[] | ✅ | Group IDs permitted to retrieve this chunk. Empty array for public content. |
| `owner_user_ids` | string[] | ✅ | User IDs with ownership rights. Empty array when not determinable. |
| `is_pii` | boolean | ✅ | Whether this chunk contains personally identifiable information. |
| `retention_days` | integer | ✅ | Days to retain before scheduled deletion. |
| `data_scope_tags` | string[] | ✅ | Structured labels describing data domain and organizational scope. See Scope Tags section below. |

### Content Descriptors

| Field | Type | Required | Description |
|---|---|---|---|
| `language` | string | ✅ | ISO 639-1 language code: `en`, `es`, `fr`, etc. |
| `content_type` | string | ✅ | Primary category: `message`, `transcript`, `document`, `task`, `email`. |
| `content_subtype` | string | Conditional | Refinement: `thread_reply`, `speaker_segment`, `attachment`. |
| `title` | string | Conditional | Document or conversation title, when available. |
| `summary` | string | Conditional | Natural-language summary. Confidence: `model_inferred`. Do NOT use for filtering. |

### Processing and Lineage

| Field | Type | Required | Description |
|---|---|---|---|
| `token_count` | integer | ✅ | Approximate token count of chunk text (use 4 chars/token heuristic). |
| `chunk_strategy` | string | ✅ | Chunking strategy used (e.g., `sentence_window`, `speaker_turn`, `paragraph`). |
| `embedding_model` | string | ✅ | Dense embedding model identifier. |
| `sparse_model` | string | ✅ | Sparse embedding model identifier. Use `"none"` for dense-only pipelines. |
| `schema_version` | string | ✅ | Metadata schema version. Current: `"2.2"`. |
| `ingestion_run_id` | string | ✅ | Unique identifier for the ingestion job or run. |
| `connector_version` | string | ✅ | Version of the source connector used. |
| `normalizer_version` | string | ✅ | Version of the normalization logic applied. |
| `chunker_version` | string | ✅ | Version of the chunking logic applied. |

---

## Data Scope Tags

`data_scope_tags` is the primary field for semantic-scope filtering. Tags describe **what the data is** — not who may access it.

Use the following controlled taxonomy. Custom tags require explicit definition and documentation before use.

| Namespace | Examples | When to Assign |
|---|---|---|
| Domain (no prefix) | `sales`, `support`, `finance`, `legal`, `hr`, `engineering`, `marketing`, `operations` | Content relates to this business domain. Derived from source metadata, folder, channel name. |
| Team scope (`team:`) | `team:sales_a`, `team:emea_support` | Content scoped to a specific team. Derived from `team_ids`. |
| Data category (`category:`) | `category:pipeline`, `category:forecast`, `category:compensation`, `category:contract` | Content falls into a specific data category with distinct access patterns. |
| Region (`region:`) | `region:emea`, `region:apac`, `region:americas` | Content is regionally scoped. |
| Project (`project:`) | `project:alpha`, `project:crm_migration` | Content associated with a specific named project. Derived from `project_ids`. |

**Tagging Rules:**
- Tags must use **lowercase snake_case**. No free-text values.
- Tags must be **stable**: same source record ingested twice → same tags.
- When organizational context cannot be determined, use an **empty array** (not fabricated tags).
- An empty `data_scope_tags` results in the broadest access (additive model). Do not speculate — when in doubt, omit the tag.
- A chunk may carry multiple tags across multiple namespaces.
- Do not add `model_inferred` tags as sole basis for security-relevant filtering.

---

## Source-Specific Metadata Namespacing

Source-specific fields must NOT be injected into the top-level payload. All source-specific fields go under a `source_metadata` object keyed by source type:

```json
{
  "source_metadata": {
    "slack": { "channel_id": "C03AB", "thread_ts": "1714500000.000000" },
    "email": null,
    "fireflies": null
  }
}
```

Only populate the namespace relevant to the current chunk's source. See source-specific schemas below.

### Slack Source Metadata

| Field | Type | Required | Description |
|---|---|---|---|
| `channel_id` | string | ✅ | Slack channel identifier. |
| `channel_name` | string | ✅ | Human-readable channel name. Used to derive `data_scope_tags`. |
| `channel_type` | enum | ✅ | `public`, `private`, `dm`, `group_dm`. |
| `thread_ts` | string | Conditional | Parent thread timestamp. Null if top-level. |
| `message_ts` | string | ✅ | Message timestamp. |
| `reaction_count` | integer | ✅ | Number of emoji reactions. |
| `reply_count` | integer | Conditional | Number of thread replies (top-level messages only). |
| `is_edited` | boolean | ✅ | Whether this message has been edited. |
| `ordering_index` | integer | ✅ | Zero-based position within thread or channel window. |

### Fireflies (Meeting Transcript) Source Metadata

| Field | Type | Required | Description |
|---|---|---|---|
| `meeting_id` | string | ✅ | Fireflies meeting identifier. |
| `meeting_title` | string | ✅ | Meeting title or subject. Used to derive `data_scope_tags`. |
| `meeting_date` | ISO 8601 | ✅ | Date and time the meeting occurred. |
| `attendee_ids` | string[] | ✅ | Identifiers of all meeting attendees. |
| `attendee_names` | string[] | ✅ | Display names of all attendees. |
| `speaker_id` | string | ✅ | Identifier of the speaker for this chunk. |
| `speaker_name` | string | ✅ | Display name of the speaker for this chunk. |
| `meeting_duration_mins` | integer | ✅ | Total meeting duration in minutes. |
| `transcript_segment_start_ms` | integer | ✅ | Millisecond offset of this chunk's start within the transcript. |

### ClickUp Source Metadata

| Field | Type | Required | Description |
|---|---|---|---|
| `task_id` | string | ✅ | ClickUp task identifier. |
| `task_name` | string | ✅ | Task title. Used to derive `data_scope_tags`. |
| `task_status` | string | ✅ | Current task status: `open`, `in_progress`, `done`. |
| `list_id` | string | ✅ | ClickUp list identifier. |
| `list_name` | string | ✅ | List name. |
| `space_id` | string | ✅ | ClickUp space identifier. |
| `assignee_ids` | string[] | Conditional | Identifiers of assigned users. |
| `due_date` | ISO 8601 | Conditional | Task due date. Null if not set. |
| `priority` | string | Conditional | Task priority level. |
| `custom_fields` | object | Conditional | Key-value map of custom fields. Evaluate for tag derivation. |

### Email Source Metadata

| Field | Type | Required | Description |
|---|---|---|---|
| `message_id` | string | ✅ | RFC 5322 Message-ID. |
| `thread_id` | string | ✅ | Email thread/conversation identifier. |
| `subject` | string | ✅ | Email subject line. Used to derive `data_scope_tags`. |
| `from_address` | string | ✅ | Sender email address. |
| `to_addresses` | string[] | ✅ | Recipient email addresses. |
| `cc_addresses` | string[] | Conditional | CC recipients. |
| `has_attachments` | boolean | ✅ | Whether the email includes attachments. |
| `email_folder` | string | Conditional | Folder or label in source mailbox. |
| `is_reply` | boolean | ✅ | Whether this email is a reply in a thread. |

### Google Drive Source Metadata

| Field | Type | Required | Description |
|---|---|---|---|
| `file_id` | string | ✅ | Google Drive file identifier. |
| `file_type` | string | ✅ | Document type: `doc`, `sheet`, `slide`, `pdf`. |
| `mime_type` | string | ✅ | MIME type of the file. |
| `folder_path` | string | ✅ | Full folder path in Drive. Critical for sensitivity classification and tag derivation. |
| `last_editor_id` | string | Conditional | ID of last person to edit the file. |
| `version` | string | Conditional | Document version number or revision ID. |
| `share_scope` | enum | Conditional | `anyone`, `domain`, `specific`. Informs `allowed_groups` population. |

---

## Reference Payload Example

```json
{
  "doc_id": "slack_C03AB_1714500000_0",
  "source_type": "slack",
  "source_system": "workspace-name",
  "source_record_id": "1714500000.000000",
  "integration_id": "slack-ingester-v3",
  "parent_doc_id": "slack_C03AB_thread_1714499900",
  "document_content_hash": "a3f1c29e...",
  "chunk_index": 0,
  "chunk_total": 1,
  "content_hash": "b7d2e44f...",
  "created_at": "2024-04-30T18:00:00Z",
  "ingested_at": "2024-04-30T18:05:00Z",
  "modified_at": "2024-04-30T18:00:00Z",
  "source_last_modified": "2024-04-30T18:00:00Z",
  "author_id": "U012AB3CD",
  "author_name": "Jane Smith",
  "author_email": "[email protected]",
  "author_department": "Sales",
  "project_ids": [],
  "team_ids": ["team_sales_a", "team_sales_b"],
  "workspace_id": "workspace-123",
  "org_id": "org-456",
  "sensitivity": "internal",
  "allowed_groups": ["g_all_staff"],
  "owner_user_ids": [],
  "is_pii": false,
  "retention_days": 1095,
  "data_scope_tags": ["sales", "team:sales_a", "team:sales_b", "category:pipeline", "region:emea"],
  "language": "en",
  "content_type": "message",
  "content_subtype": "thread_reply",
  "title": null,
  "summary": null,
  "token_count": 187,
  "chunk_strategy": "sentence_window",
  "embedding_model": "BAAI/bge-m3",
  "sparse_model": "BAAI/bge-m3-splade",
  "schema_version": "2.2",
  "ingestion_run_id": "run_2024043001",
  "connector_version": "1.4.2",
  "normalizer_version": "2.1.0",
  "chunker_version": "3.0.1",
  "source_metadata": {
    "slack": {
      "channel_id": "C03AB",
      "channel_name": "sales-emea-pipeline",
      "channel_type": "public",
      "thread_ts": "1714499900.000000",
      "message_ts": "1714500000.000000",
      "reaction_count": 2,
      "is_edited": false,
      "ordering_index": 3
    }
  }
}
```

---

## Payload Indexing Requirements

**Every field used as a filter in Qdrant queries MUST have a payload index created.** Filtering on unindexed fields causes full collection scans and degrades performance.

```python
from qdrant_client import QdrantClient, models

client = QdrantClient(url="...")

# Index sensitivity (most important — used in every query)
client.create_payload_index(
    collection_name="company_memory",
    field_name="sensitivity",
    field_schema=models.PayloadSchemaType.KEYWORD,
)

# Index org_id for tenant isolation (use is_tenant=True for multi-tenant)
client.create_payload_index(
    collection_name="company_memory",
    field_name="org_id",
    field_schema=models.KeywordIndexParams(type="keyword", is_tenant=True),
)

# Index allowed_groups for group-based access
client.create_payload_index(
    collection_name="company_memory",
    field_name="allowed_groups",
    field_schema=models.PayloadSchemaType.KEYWORD,
)

# Index data_scope_tags for scope-based filtering
client.create_payload_index(
    collection_name="company_memory",
    field_name="data_scope_tags",
    field_schema=models.PayloadSchemaType.KEYWORD,
)

# Index team_ids for team-scoped queries
client.create_payload_index(
    collection_name="company_memory",
    field_name="team_ids",
    field_schema=models.PayloadSchemaType.KEYWORD,
)

# Index timestamps for recency filtering
client.create_payload_index(
    collection_name="company_memory",
    field_name="created_at",
    field_schema=models.PayloadSchemaType.DATETIME,
)

# Index source_type for source-specific queries
client.create_payload_index(
    collection_name="company_memory",
    field_name="source_type",
    field_schema=models.PayloadSchemaType.KEYWORD,
)
```

**Fields to always index:** `sensitivity`, `org_id`, `workspace_id`, `allowed_groups`, `data_scope_tags`, `team_ids`, `source_type`, `is_pii`, `created_at`, `content_type`.

---
name: astro-advanced
description: >
  Comprehensive skill for building, configuring, and troubleshooting Astro projects.
  Use this skill whenever the user mentions Astro, .astro files, Astro config, Astro islands,
  partial hydration, client directives (client:load, client:visible, client:idle), Astro SSR,
  Astro adapters (Node, Vercel, Cloudflare, Netlify), Astro content collections, Astro layouts,
  hybrid rendering, astro.config.mjs, or any framework integration with Astro (Vue, React, Svelte).
  Also trigger when the user is debugging broken Astro builds, hydration mismatches, routing issues,
  base path problems, or deployment failures on static/serverless hosts. Even if the user doesn't
  say "Astro" explicitly, trigger if they reference .astro file syntax, frontmatter fences (---),
  or island architecture concepts. This skill covers project setup, rendering modes (SSG/SSR/hybrid),
  SSR caching, SEO, layouts, templates, Vue island integration, content collections, data fetching,
  deployment, auth, performance, and troubleshooting.
---

# Astro Advanced Skill

This skill provides production-grade guidance for Astro projects — from initial scaffolding through
deployment, caching, and performance tuning. It covers the patterns that actually matter in real
projects and the mistakes that actually happen.

## How to use this skill

1. **Read this file first** for the core workflow and decision tree.
2. **Consult the reference files** in `references/` for deep dives on specific topics:
   - `references/setup-and-structure.md` — Project creation, file structure, config, adapters
   - `references/rendering-modes.md` — SSG vs SSR vs Hybrid, when to use each, caching strategies
   - `references/seo.md` — Meta tags, Open Graph, JSON-LD, sitemaps, canonical URLs
   - `references/islands-and-vue.md` — Island architecture, client directives, Vue/React/Svelte integration
   - `references/content-and-data.md` — Content collections, data fetching, dynamic routes
   - `references/deployment.md` — Adapters, static hosts, serverless, environment variables
   - `references/performance.md` — Image optimization, bundle analysis, hydration control
   - `references/troubleshooting.md` — Common errors and fixes organized by symptom

## Core decision tree

When helping with an Astro project, follow this sequence:

### 1. Identify the rendering strategy first
This is the single most important decision in any Astro project. Everything else flows from it.

- **Pure static site (blog, docs, marketing)?** → SSG (default). No adapter needed.
- **Needs user-specific data, auth, or real-time content?** → SSR with an adapter.
- **Mostly static but a few dynamic pages?** → Hybrid mode. Set `output: 'static'` in config and use `export const prerender = false` on dynamic pages.

### 2. Pick the right adapter
Only needed for SSR or hybrid:
- **Vercel** → `@astrojs/vercel` (serverless or edge)
- **Netlify** → `@astrojs/netlify`
- **Cloudflare Pages** → `@astrojs/cloudflare`
- **Self-hosted Node** → `@astrojs/node` (standalone or middleware)

### 3. Set up integrations
Add only what you need. Each integration is a potential build-time dependency:

```bash
# Common additions
npx astro add vue        # Vue islands
npx astro add tailwind   # Tailwind CSS
npx astro add mdx        # MDX support
npx astro add sitemap    # Auto sitemap generation
```

### 4. Establish content strategy
- **Few pages, hand-authored** → Regular `.astro` pages in `/src/pages`
- **Blog/docs with structured content** → Content collections with Zod schemas
- **CMS-driven** → Fetch at build time (SSG) or runtime (SSR)

### 5. Apply SEO from the start
Don't bolt it on later. See `references/seo.md` for the full pattern, but at minimum:
- Create a reusable `<SEO>` component for head tags
- Set up canonical URLs
- Add structured data (JSON-LD) for key pages
- Generate sitemap via `@astrojs/sitemap`

## Key patterns to always follow

### Layout pattern
Every page should use a layout. Layouts handle `<html>`, `<head>`, and shared chrome:

```astro
---
// src/layouts/Base.astro
import SEO from '../components/SEO.astro';
const { title, description } = Astro.props;
---
<html lang="en">
  <head>
    <SEO title={title} description={description} />
  </head>
  <body>
    <nav><!-- shared nav --></nav>
    <slot />
    <footer><!-- shared footer --></footer>
  </body>
</html>
```

### Island pattern
Static by default. Only hydrate what needs interactivity:

```astro
<!-- Static: renders HTML, ships zero JS -->
<Card title="Hello" />

<!-- Interactive: hydrates on load -->
<Counter client:load />

<!-- Interactive: hydrates when visible (lazy) -->
<ImageGallery client:visible />
```

**The #1 Astro mistake**: forgetting `client:*` on a component that needs interactivity,
then wondering why click handlers don't work.

### SSR caching pattern
SSR without caching is just a slow website. Always pair SSR with a caching strategy:

```ts
// In an SSR endpoint or page
return new Response(JSON.stringify(data), {
  headers: {
    "Cache-Control": "public, s-maxage=60, stale-while-revalidate=300",
    "Content-Type": "application/json"
  }
});
```

## When things go wrong

Read `references/troubleshooting.md` for a symptom-based guide. The top 5 issues:

1. **"Component doesn't do anything"** → Missing `client:*` directive
2. **Build fails after adding integration** → Version mismatch, check `package.json`
3. **SSR returns 500** → Missing adapter or wrong `output` mode in config
4. **Broken links after deploy** → Base path not set, or trailing slash mismatch
5. **Hydration mismatch errors** → Server/client HTML differs (conditional rendering, dates, randomness)

## File output conventions

When generating Astro project files:
- Always include the frontmatter fence (`---`) even if empty
- Use `.astro` extension for Astro components and pages
- Place pages in `src/pages/`, components in `src/components/`, layouts in `src/layouts/`
- Use TypeScript in frontmatter when the project uses TS
- Include `astro.config.mjs` with only the integrations and settings actually needed

FILE:references/troubleshooting.md
# Troubleshooting Guide

Organized by symptom. Find what's broken, then follow the fix.

## Table of Contents
1. [Component doesn't do anything (no interactivity)](#no-interactivity)
2. [Hydration mismatch errors](#hydration-mismatch)
3. [Build fails after adding integration](#integration-build-fail)
4. [SSR returns 500 or blank page](#ssr-500)
5. [Broken links / 404 after deploy](#broken-links)
6. ["window is not defined" / "document is not defined"](#window-undefined)
7. [Content collection errors](#content-errors)
8. [Styles not applying](#styles-broken)
9. [Dynamic routes not working](#dynamic-routes)
10. [Environment variables undefined](#env-undefined)
11. [Slow builds](#slow-builds)
12. [CORS errors from API calls](#cors)
13. [Images not loading](#images-broken)
14. [TypeScript errors](#typescript)
15. [Dev server issues](#dev-server)

---

## 1. Component doesn't do anything (no interactivity) {#no-interactivity}

**Symptom**: Component renders HTML but buttons, inputs, event handlers don't work.

**Cause**: Missing `client:*` directive. Without it, Astro renders static HTML and ships zero JS.

**Fix**:
```astro
<!-- Before: static HTML only -->
<Counter />

<!-- After: hydrated and interactive -->
<Counter client:load />
```

**Checklist if you already have a client directive**:
- Is the directive on the right component? (Check parent vs child)
- Is the component a `.astro` file? (Astro components can't be hydrated — only framework components like `.vue`, `.tsx`, `.svelte`)
- Is the integration installed? (`npx astro add vue`)

---

## 2. Hydration mismatch errors {#hydration-mismatch}

**Symptom**: Console warns about hydration mismatch. UI might flicker or revert after load.

**Cause**: HTML rendered on the server differs from what the client renders.

**Common triggers**:
- Using `Date.now()`, `Math.random()`, or `new Date()` in the render template
- Reading `localStorage` or `window.innerWidth` during render
- Conditional rendering based on browser-only values
- Different timezone between server and client

**Fix pattern**:
```vue
<script setup>
import { ref, onMounted } from 'vue';

// Use a safe default for SSR
const width = ref(1024);

// Update with real value after mount (client only)
onMounted(() => {
  width.value = window.innerWidth;
});
</script>
```

**Nuclear option**: Use `client:only="vue"` to skip SSR entirely for that component. Only do this if you can't fix the mismatch.

---

## 3. Build fails after adding integration {#integration-build-fail}

**Symptom**: `npm run build` or `npm run dev` crashes after installing a new integration.

**Diagnosis**:
```bash
# Check for version conflicts
npm ls --depth=0

# Clear caches
rm -rf node_modules/.astro
rm -rf node_modules/.vite

# Reinstall
rm -rf node_modules
npm install
```

**Common causes**:
- **Integration version incompatible with Astro version**: Check the integration's docs for supported Astro versions
- **Conflicting peer dependencies**: Two integrations requiring different versions of a shared dependency
- **Config not updated**: `npx astro add` updates config automatically; manual install doesn't

**Fix**: Use `npx astro add <integration>` instead of `npm install`. It handles both package.json and astro.config.mjs.

---

## 4. SSR returns 500 or blank page {#ssr-500}

**Symptom**: SSR pages return HTTP 500 or render nothing.

**Checklist**:
1. **Is `output` set correctly?**
   ```js
   // astro.config.mjs
   export default defineConfig({
     output: 'server', // Must be 'server' for full SSR
   });
   ```

2. **Is an adapter installed?**
   ```bash
   npx astro add node  # or vercel, netlify, cloudflare
   ```

3. **Check server logs** — the actual error is in the terminal/server output, not the browser.

4. **Does the page have `export const prerender = false`?** If you're using hybrid mode (output: 'static'), SSR pages need this.

5. **Are environment variables available?** SSR runs on the server — check server-side env vars, not just build-time ones.

6. **Is `Astro.redirect()` returning correctly?**
   ```astro
   ---
   // Must RETURN the redirect
   if (!user) return Astro.redirect('/login');
   // Without return, execution continues
   ---
   ```

---

## 5. Broken links / 404 after deploy {#broken-links}

**Symptom**: Site works locally but links 404 in production.

**Check these in order**:

1. **Base path**:
   ```js
   // If hosting at https://example.com/docs/
   export default defineConfig({
     base: '/docs/',
   });
   ```
   All internal links must use the base. Use `import.meta.env.BASE_URL` for dynamic base.

2. **Trailing slash mismatch**:
   ```js
   export default defineConfig({
     trailingSlash: 'always', // Match your host's behavior
   });
   ```
   Netlify adds trailing slashes by default. Vercel doesn't. Mismatch = redirect loops or 404s.

3. **Case sensitivity**: macOS doesn't care about case; Linux does. `About.astro` creates `/About` on Linux, which is different from `/about`.

4. **SSR pages on static host**: If you deployed SSR pages to a static host (GitHub Pages, S3), they won't exist. Either prerender them or use an SSR-capable host.

---

## 6. "window is not defined" / "document is not defined" {#window-undefined}

**Symptom**: Build or SSR crashes with ReferenceError about browser globals.

**Cause**: Code that runs on the server is trying to use browser-only APIs.

**Where this happens**:
- Astro frontmatter (always runs on server)
- Framework component render functions (run on server during SSR)
- Third-party libraries that assume a browser environment

**Fixes**:

1. **Move to `onMounted` / `useEffect`** (only runs in browser):
   ```vue
   <script setup>
   import { onMounted } from 'vue';
   onMounted(() => {
     // Safe: only runs in browser
     const el = document.getElementById('my-element');
   });
   </script>
   ```

2. **Guard with typeof check**:
   ```ts
   if (typeof window !== 'undefined') {
     window.addEventListener('scroll', handler);
   }
   ```

3. **Use `client:only`** to skip SSR entirely:
   ```astro
   <MapComponent client:only="vue" />
   ```

4. **Lazy import browser-only libraries**:
   ```ts
   let lib;
   if (typeof window !== 'undefined') {
     lib = await import('browser-only-lib');
   }
   ```

---

## 7. Content collection errors {#content-errors}

### "Collection X not found"
- Does `src/content/config.ts` exist?
- Does it export the collection? `export const collections = { blog };`
- Does the folder name match? `src/content/blog/` for collection `blog`
- Run `npx astro sync` to regenerate types

### Schema validation error
```
[ERROR] blog/my-post.md frontmatter does not match collection schema
```
- Check the error details — it tells you which field failed
- Common: using `z.date()` instead of `z.coerce.date()` for frontmatter dates
- Common: missing required fields in frontmatter
- Common: wrong type (string where number expected)

### Content not updating in dev
- Restart the dev server after changing `config.ts`
- Run `npx astro sync` after schema changes
- Clear `.astro` cache: `rm -rf node_modules/.astro`

---

## 8. Styles not applying {#styles-broken}

### Scoped styles not working on child components
Astro scoped styles only apply to HTML in the current `.astro` file, not children.

```astro
<style>
  /* Only affects <h1> directly in THIS file, not in child components */
  h1 { color: red; }
</style>
```

**Fix**: Use `:global()` to target child HTML, or use `is:global` on the style tag:
```astro
<style>
  :global(.child-class) { color: red; }
</style>

<!-- Or make the entire block global -->
<style is:global>
  .applies-everywhere { color: red; }
</style>
```

### Tailwind not working
- Is the integration installed? `npx astro add tailwind`
- Is `@tailwind` imported in a global CSS file or layout?
- Check `tailwind.config.mjs` content paths include `.astro` files:
  ```js
  content: ['./src/**/*.{astro,html,js,jsx,md,mdx,svelte,ts,tsx,vue}']
  ```

### Styles flash or disappear
- FOUC (Flash of Unstyled Content): Make sure CSS is in `<head>`, not dynamically injected
- Check that scoped styles aren't conflicting with global styles

---

## 9. Dynamic routes not working {#dynamic-routes}

### SSG: "getStaticPaths() is required"
```
[ERROR] [GetStaticPaths] `getStaticPaths()` function is required for dynamic routes.
```
Static mode requires all paths to be known at build time:
```astro
---
export async function getStaticPaths() {
  return [
    { params: { slug: 'post-1' } },
    { params: { slug: 'post-2' } },
  ];
}
---
```

### SSR: Params are undefined
Check that the file naming matches:
- `[slug].astro` → `Astro.params.slug`
- `[...path].astro` → `Astro.params.path`
- `[category]/[id].astro` → `Astro.params.category`, `Astro.params.id`

---

## 10. Environment variables undefined {#env-undefined}

### In frontmatter (server):
```ts
// All variables available
console.log(import.meta.env.SECRET_KEY);     // ✓
console.log(import.meta.env.PUBLIC_API_URL); // ✓
```

### In client-side island:
```ts
// ONLY PUBLIC_ prefixed variables
console.log(import.meta.env.PUBLIC_API_URL); // ✓
console.log(import.meta.env.SECRET_KEY);     // undefined (by design)
```

### In production (SSR):
- Set variables on the hosting platform, not just in `.env`
- `.env` files are used during build. SSR runtime needs actual environment variables.
- On Vercel/Netlify: set in dashboard
- On Node: set in shell or use dotenv

---

## 11. Slow builds {#slow-builds}

**Diagnosis**: Run with timing:
```bash
time npm run build
```

**Common causes**:
- **Too many pages**: 10,000+ SSG pages = long builds. Consider hybrid (SSR for long-tail pages).
- **Heavy data fetching in getStaticPaths**: API calls during build. Cache or parallelize.
- **Unoptimized images**: Large images being processed at build time. Pre-optimize before adding to project.
- **Expensive integrations**: Some integrations add build steps. Test by removing one at a time.

**Quick fixes**:
- Limit pages in dev: `if (import.meta.env.DEV) return posts.slice(0, 10);`
- Use incremental builds if your host supports them (Vercel, Netlify)
- Pre-optimize images before import

---

## 12. CORS errors from API calls {#cors}

**Symptom**: Browser console shows CORS errors when fetching from an API.

**Key insight**: CORS only applies to browser-to-server requests. Server-to-server requests (Astro frontmatter, endpoints) are never blocked by CORS.

**Fix**: Move the API call from the client island to an Astro server endpoint:

```ts
// src/pages/api/proxy.ts — Astro endpoint (server-side, no CORS)
export const GET: APIRoute = async ({ url }) => {
  const res = await fetch(`https://external-api.com/data?q=url.searchParams.get('q')`);
  return new Response(await res.text(), {
    headers: { 'Content-Type': 'application/json' },
  });
};
```

Then call your own endpoint from the island:
```ts
// In your Vue/React island
const res = await fetch('/api/proxy?q=search');
```

---

## 13. Images not loading {#images-broken}

### Images from `src/assets/` not found
```astro
---
// Must use import for processed images
import hero from '../assets/hero.jpg';
---
<Image src={hero} alt="Hero" />
```

### Remote images not loading
Add the domain to allowed list:
```js
// astro.config.mjs
export default defineConfig({
  image: {
    domains: ['cdn.example.com'],
  },
});
```

### Images 404 after deploy
- Check `base` path in config
- Ensure images in `/public/` are deployed (some hosts ignore certain directories)
- For processed images, check that the build output includes `_astro/` directory

---

## 14. TypeScript errors {#typescript}

### "Cannot find module 'astro:content'"
Run `npx astro sync` to generate type definitions.

### Props type mismatch
```astro
---
// Define strict props interface
interface Props {
  title: string;
  count: number;  // If passing from parent, ensure it's a number, not string
}
const { title, count } = Astro.props;
---
```

### Integration types missing
```bash
# Regenerate all types
npx astro sync
```

---

## 15. Dev server issues {#dev-server}

### Port already in use
```bash
# Find what's using port 4321
lsof -i :4321
# Kill it, or use a different port:
npx astro dev --port 3000
```

### HMR (Hot Module Replacement) not working
- Some changes require a full restart (config changes, new content collections)
- Clear the Vite cache: `rm -rf node_modules/.vite`
- Check for circular imports

### Dev server crashes on save
- Usually a syntax error in `.astro` file — check terminal output
- Malformed frontmatter (missing closing `---`)
- Invalid TypeScript in frontmatter

---

## General debugging strategy

1. **Read the terminal error** — Astro's error messages are usually clear and point to the exact file/line
2. **Check `View Source`** — See the actual HTML output (not DevTools, which shows post-hydration DOM)
3. **Build locally first** — `npm run build && npm run preview` catches issues before deploying
4. **Isolate the problem** — Comment out components until you find which one causes the issue
5. **Check the Astro docs** — `docs.astro.build` has excellent troubleshooting sections
6. **Check GitHub issues** — `github.com/withastro/astro/issues` for known bugs

FILE:references/content-and-data.md
# Content Collections & Data Fetching

## Table of Contents
1. [Content collections overview](#overview)
2. [Defining collections](#defining)
3. [Querying collections](#querying)
4. [Data fetching patterns](#data-fetching)
5. [Dynamic routes with data](#dynamic-routes)
6. [Common problems](#problems)

---

## Content collections overview

Content collections are Astro's system for organizing and validating structured content (blog posts, docs, products, etc.). They live in `src/content/` and use Zod schemas for type safety.

Why use them instead of raw markdown in `/src/pages`:
- Schema validation catches content errors at build time
- TypeScript types are auto-generated from schemas
- Query APIs for filtering, sorting, pagination
- Separates content from presentation

---

## Defining collections

### Directory structure
```
src/content/
├── config.ts          # Schema definitions (REQUIRED)
├── blog/
│   ├── first-post.md
│   ├── second-post.mdx
│   └── third-post.md
├── authors/
│   ├── alice.json
│   └── bob.json
└── products/
    └── widget.yaml
```

### Schema definition
```ts
// src/content/config.ts
import { defineCollection, z, reference } from 'astro:content';

const blog = defineCollection({
  type: 'content', // Markdown/MDX (has a body to render)
  schema: z.object({
    title: z.string(),
    description: z.string(),
    publishDate: z.coerce.date(), // Coerces string to Date
    updatedDate: z.coerce.date().optional(),
    author: reference('authors'), // Reference to another collection
    tags: z.array(z.string()).default([]),
    draft: z.boolean().default(false),
    image: z.string().optional(),
  }),
});

const authors = defineCollection({
  type: 'data', // JSON/YAML (structured data, no body)
  schema: z.object({
    name: z.string(),
    email: z.string().email(),
    avatar: z.string().url(),
    bio: z.string().optional(),
  }),
});

export const collections = { blog, authors };
```

### Content file format
```md
---
title: "My First Post"
description: "An introduction to Astro"
publishDate: 2024-01-15
author: alice
tags: ["astro", "webdev"]
draft: false
---

# Hello World

This is the body content rendered as HTML.
```

---

## Querying collections

```astro
---
import { getCollection, getEntry } from 'astro:content';

// Get all published posts, sorted by date
const posts = (await getCollection('blog', ({ data }) => {
  return !data.draft; // Filter out drafts
})).sort((a, b) =>
  b.data.publishDate.valueOf() - a.data.publishDate.valueOf()
);

// Get a single entry by slug
const post = await getEntry('blog', 'first-post');

// Resolve a reference
const author = await getEntry(post.data.author);

// Render content to HTML
const { Content, headings } = await post.render();
---

<h1>{post.data.title}</h1>
<p>By {author.data.name}</p>
<Content />
```

### Useful query patterns

**Pagination**:
```astro
---
const allPosts = await getCollection('blog');
const pageSize = 10;
const page = Number(Astro.params.page) || 1;
const totalPages = Math.ceil(allPosts.length / pageSize);
const posts = allPosts.slice((page - 1) * pageSize, page * pageSize);
---
```

**Filter by tag**:
```astro
---
const tag = Astro.params.tag;
const posts = await getCollection('blog', ({ data }) =>
  data.tags.includes(tag)
);
---
```

**Group by year**:
```ts
const postsByYear = posts.reduce((acc, post) => {
  const year = post.data.publishDate.getFullYear();
  (acc[year] ||= []).push(post);
  return acc;
}, {} as Record<number, typeof posts>);
```

---

## Data fetching patterns

### SSG: Fetch at build time

In static mode, all fetching happens during the build. Data is baked into HTML.

```astro
---
// This runs at BUILD TIME, not in the browser
const res = await fetch('https://api.example.com/products');
const products = await res.json();
---

{products.map(p => <ProductCard product={p} />)}
```

**Important**: API rate limits can bite you during builds. If you have 1000 pages each fetching from an API, that's 1000 requests during build.

Mitigations:
- Batch API calls where possible
- Use content collections for local data
- Cache API responses during build (Astro caches `fetch` by default in SSG)

### SSR: Fetch at request time

In SSR mode, fetching happens on every request.

```astro
---
// This runs on EVERY REQUEST
const query = Astro.url.searchParams.get('q') || '';
const res = await fetch(`https://api.example.com/search?q=query`);
const results = await res.json();
---

<SearchResults results={results} />
```

### Fetch in API endpoints

```ts
// src/pages/api/products.ts
import type { APIRoute } from 'astro';

export const GET: APIRoute = async ({ url }) => {
  const category = url.searchParams.get('category');
  const products = await db.getProducts({ category });

  return new Response(JSON.stringify(products), {
    headers: {
      'Content-Type': 'application/json',
      'Cache-Control': 'public, s-maxage=120',
    },
  });
};
```

### Client-side fetching from islands

For interactive data loading (search, infinite scroll), fetch from within the island:

```vue
<script setup>
import { ref, onMounted } from 'vue';

const data = ref(null);
const loading = ref(true);

onMounted(async () => {
  const res = await fetch('/api/products');
  data.value = await res.json();
  loading.value = false;
});
</script>
```

---

## Dynamic routes with data

### SSG dynamic routes
Must provide all paths at build time:

```astro
---
// src/pages/blog/[slug].astro
import { getCollection } from 'astro:content';

export async function getStaticPaths() {
  const posts = await getCollection('blog');
  return posts.map(post => ({
    params: { slug: post.slug },
    props: { post },
  }));
}

const { post } = Astro.props;
const { Content } = await post.render();
---

<Layout title={post.data.title}>
  <Content />
</Layout>
```

### SSR dynamic routes
Read params at request time:

```astro
---
// src/pages/blog/[slug].astro (SSR)
import { getEntry } from 'astro:content';

const { slug } = Astro.params;
const post = await getEntry('blog', slug);

if (!post) {
  return Astro.redirect('/404');
}

const { Content } = await post.render();
---
```

### Catch-all routes
```astro
---
// src/pages/[...path].astro
const { path } = Astro.params;
// path = "a/b/c" for URL /a/b/c
---
```

---

## Common problems

### "Collection not found"
- Check that `src/content/config.ts` exists and exports the collection
- Make sure the collection name in `getCollection('name')` matches the folder name
- Run `astro sync` to regenerate types after changing schemas

### Schema validation errors
```
[ERROR] blog/my-post.md: "publishDate" Expected date, received string
```
Fix: Use `z.coerce.date()` instead of `z.date()` for frontmatter dates.

### Content not updating
- Content collections are cached. Restart the dev server after structural changes
- Run `astro sync` after modifying `config.ts`

### Slow builds with large collections
- Avoid fetching external data inside `getStaticPaths()` if possible
- Use pagination to limit pages generated per build
- Consider switching data-heavy pages to SSR

### MDX components not rendering
- Make sure `@astrojs/mdx` is installed and in `astro.config.mjs`
- Import components in the MDX file or pass them via the layout

### Reference resolution
```ts
// Get the referenced entry
const author = await getEntry(post.data.author);
// author.data.name, author.data.email, etc.
```
References only store the collection name and slug. You must call `getEntry()` to resolve them.

FILE:references/seo.md
# SEO Best Practices for Astro

## Table of Contents
1. [Why Astro is great for SEO](#why-astro)
2. [Reusable SEO component](#seo-component)
3. [Open Graph & social meta](#open-graph)
4. [Structured data (JSON-LD)](#structured-data)
5. [Sitemap & robots.txt](#sitemap)
6. [Canonical URLs](#canonical-urls)
7. [Performance signals](#performance)
8. [Common SEO mistakes in Astro](#mistakes)

---

## Why Astro is great for SEO

Astro outputs static HTML by default — no JavaScript required for content rendering. This means:
- Search engines see full content on first crawl (no hydration needed)
- Core Web Vitals are excellent out of the box (minimal JS = fast LCP, low TBT)
- No flash of unstyled/empty content

The main risk is undoing these advantages by shipping too much client-side JS through islands.

---

## Reusable SEO component

Create a single component that handles all head meta. Use it in every layout.

```astro
---
// src/components/SEO.astro
interface Props {
  title: string;
  description: string;
  image?: string;
  canonicalURL?: string;
  type?: 'website' | 'article';
  publishedDate?: string;
  noindex?: boolean;
}

const {
  title,
  description,
  image = '/og-default.png',
  canonicalURL = new URL(Astro.url.pathname, Astro.site).href,
  type = 'website',
  publishedDate,
  noindex = false,
} = Astro.props;

const siteName = 'Your Site Name';
const fullTitle = `title | siteName`;
const absoluteImage = new URL(image, Astro.site).href;
---

<!-- Primary meta -->
<title>{fullTitle}</title>
<meta name="description" content={description} />
{noindex && <meta name="robots" content="noindex, nofollow" />}
<link rel="canonical" href={canonicalURL} />

<!-- Open Graph -->
<meta property="og:type" content={type} />
<meta property="og:title" content={title} />
<meta property="og:description" content={description} />
<meta property="og:image" content={absoluteImage} />
<meta property="og:url" content={canonicalURL} />
<meta property="og:site_name" content={siteName} />
{publishedDate && <meta property="article:published_time" content={publishedDate} />}

<!-- Twitter -->
<meta name="twitter:card" content="summary_large_image" />
<meta name="twitter:title" content={title} />
<meta name="twitter:description" content={description} />
<meta name="twitter:image" content={absoluteImage} />
```

Usage in a layout:
```astro
---
import SEO from '../components/SEO.astro';
const { title, description } = Astro.props;
---
<html lang="en">
  <head>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />
    <SEO title={title} description={description} />
  </head>
  <body>
    <slot />
  </body>
</html>
```

---

## Open Graph & social meta

### Dynamic OG images

For blog posts, generate unique OG images. Options:
1. **Static images per post** — Place in `/public/og/` and reference by slug
2. **Dynamic generation** — Use `@vercel/og`, Satori, or a similar library in an API route

```ts
// src/pages/og/[slug].png.ts — Dynamic OG image endpoint
import type { APIRoute } from 'astro';
import { getEntry } from 'astro:content';
// Use satori + sharp or @resvg/resvg-js to render HTML to PNG

export const GET: APIRoute = async ({ params }) => {
  const post = await getEntry('blog', params.slug!);
  // Generate and return PNG...
};
```

### Social preview checklist
- OG image is exactly 1200×630px
- Title is under 60 characters (truncation varies by platform)
- Description is under 155 characters
- Test with: Facebook Sharing Debugger, Twitter Card Validator, LinkedIn Post Inspector

---

## Structured data (JSON-LD)

Add structured data to help search engines understand your content.

### Article schema
```astro
---
const articleSchema = {
  "@context": "https://schema.org",
  "@type": "Article",
  "headline": title,
  "description": description,
  "image": image,
  "datePublished": publishedDate,
  "dateModified": modifiedDate,
  "author": {
    "@type": "Person",
    "name": authorName,
    "url": authorURL
  },
  "publisher": {
    "@type": "Organization",
    "name": "Your Site",
    "logo": {
      "@type": "ImageObject",
      "url": "https://example.com/logo.png"
    }
  }
};
---
<script type="application/ld+json" set:html={JSON.stringify(articleSchema)} />
```

### Breadcrumb schema
```astro
---
const breadcrumbs = {
  "@context": "https://schema.org",
  "@type": "BreadcrumbList",
  "itemListElement": [
    { "@type": "ListItem", "position": 1, "name": "Home", "item": "https://example.com" },
    { "@type": "ListItem", "position": 2, "name": "Blog", "item": "https://example.com/blog" },
    { "@type": "ListItem", "position": 3, "name": title },
  ]
};
---
<script type="application/ld+json" set:html={JSON.stringify(breadcrumbs)} />
```

### FAQ schema
```astro
---
const faqSchema = {
  "@context": "https://schema.org",
  "@type": "FAQPage",
  "mainEntity": faqs.map(faq => ({
    "@type": "Question",
    "name": faq.question,
    "acceptedAnswer": {
      "@type": "Answer",
      "text": faq.answer
    }
  }))
};
---
<script type="application/ld+json" set:html={JSON.stringify(faqSchema)} />
```

---

## Sitemap & robots.txt

### Sitemap
```bash
npx astro add sitemap
```

```js
// astro.config.mjs
import sitemap from '@astrojs/sitemap';

export default defineConfig({
  site: 'https://example.com', // REQUIRED for sitemap
  integrations: [
    sitemap({
      filter: (page) => !page.includes('/admin/'), // Exclude pages
      changefreq: 'weekly',
      priority: 0.7,
      lastmod: new Date(),
    }),
  ],
});
```

### robots.txt
Place in `/public/robots.txt`:
```
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/

Sitemap: https://example.com/sitemap-index.xml
```

---

## Canonical URLs

Every page needs a canonical URL to prevent duplicate content issues.

**The pattern**: Use `Astro.site` + `Astro.url.pathname` to build canonical URLs.

```astro
---
const canonicalURL = new URL(Astro.url.pathname, Astro.site);
---
<link rel="canonical" href={canonicalURL.href} />
```

**Watch out for**:
- Trailing slash inconsistency (`/about` vs `/about/` are different URLs)
- Query parameters (canonical should not include `?page=2` etc.)
- `www` vs non-`www` (pick one and redirect the other)
- HTTP vs HTTPS (always canonical to HTTPS)

Set `trailingSlash` in config to match your host's behavior:
```js
export default defineConfig({
  trailingSlash: 'always', // or 'never'
});
```

---

## Performance signals (SEO-relevant)

Google uses Core Web Vitals as ranking signals. Astro makes these easy to ace.

**LCP (Largest Contentful Paint)**:
- Use `<Image />` from `astro:assets` for optimized images
- Preload the LCP image: `<link rel="preload" as="image" href={heroImage} />`
- Avoid lazy-loading above-the-fold images

**CLS (Cumulative Layout Shift)**:
- Always set `width` and `height` on images
- Avoid injecting content above existing content after load
- Reserve space for dynamic islands

**INP (Interaction to Next Paint)**:
- Ship minimal JS (Astro's default behavior)
- Use `client:visible` or `client:idle` instead of `client:load` where possible
- Avoid heavy computation in event handlers

---

## Common SEO mistakes in Astro

1. **Forgetting `site` in config** → `Astro.site` is undefined, canonical URLs and sitemap break silently.

2. **No canonical URLs** → Duplicate content across routes (with/without trailing slash, www/non-www).

3. **Same meta on every page** → Every page should have a unique `<title>` and `<meta name="description">`.

4. **Missing OG image** → Social shares look broken. Always provide a fallback default.

5. **Over-hydrating destroys performance** → Adding `client:load` to everything ships tons of JS, killing Core Web Vitals. Use the lightest directive possible.

6. **Dynamic SSR pages without meta** → SSR pages still need proper `<title>` and meta tags. Fetch the data first, then render the head.

7. **Not testing with `View Source`** → Islands that render client-side only (missing SSR support) may show empty HTML to crawlers. Always check the raw HTML output.

FILE:references/rendering-modes.md
# Rendering Modes & SSR Caching

## Table of Contents
1. [SSG (Static Site Generation)](#ssg)
2. [SSR (Server-Side Rendering)](#ssr)
3. [Hybrid mode](#hybrid)
4. [SSR caching strategies](#caching)
5. [Middleware](#middleware)
6. [API routes / endpoints](#endpoints)
7. [Decision matrix](#decision-matrix)

---

## SSG (Static Site Generation) — the default

Every `.astro` page is pre-rendered at build time into static HTML. No server needed at runtime.

```js
// astro.config.mjs
export default defineConfig({
  output: 'static', // This is the default; you can omit it
});
```

**When to use SSG**:
- Blog, docs, marketing sites, portfolios
- Content that changes at deploy time, not per-request
- Maximum performance (every page is a cached HTML file)

**Limitations**:
- No per-request data (user sessions, search results, real-time content)
- Dynamic routes must define all paths at build time via `getStaticPaths()`
- Large sites with thousands of pages = long build times

---

## SSR (Server-Side Rendering)

Pages are rendered on-demand per request. Requires an adapter.

```js
// astro.config.mjs
import node from '@astrojs/node';

export default defineConfig({
  output: 'server',
  adapter: node({ mode: 'standalone' }),
});
```

**When to use SSR**:
- User-specific content (dashboards, account pages)
- Real-time or frequently changing data
- Auth-gated pages
- Search results, filtered listings
- Content from a CMS that updates without redeploying

**Key behavior differences from SSG**:
- `Astro.request` contains the actual request (headers, cookies, URL)
- `Astro.cookies` is available for reading/writing cookies
- `Astro.redirect()` works for server-side redirects
- No `getStaticPaths()` — params come from the live URL

```astro
---
// SSR page: reads cookies, fetches user-specific data
const token = Astro.cookies.get('auth_token')?.value;
if (!token) return Astro.redirect('/login');

const user = await fetchUser(token);
---
<h1>Welcome, {user.name}</h1>
```

---

## Hybrid mode

The practical sweet spot for most real projects. Static by default, SSR where needed.

```js
// astro.config.mjs — Astro 4+ approach
export default defineConfig({
  output: 'static',  // Default: pages are static
  adapter: node({ mode: 'standalone' }), // Needed for the SSR pages
});
```

Then opt individual pages OUT of prerendering:

```astro
---
// src/pages/dashboard.astro — this page renders on every request
export const prerender = false;
---
```

Or if you set `output: 'server'` (SSR default), opt individual pages IN to prerendering:

```astro
---
// src/pages/about.astro — this page is pre-rendered at build time
export const prerender = true;
---
```

**Hybrid is the right choice when**:
- Most pages are static (marketing, blog, docs)
- A few pages need dynamic data (login, dashboard, search)
- You want fast static pages without giving up server capabilities

---

## SSR Caching Strategies

SSR without caching is just a slow website. Every SSR page should have a caching plan.

### Level 1: HTTP Cache-Control headers

The simplest and most effective approach. Set headers and let the CDN/browser handle it.

```astro
---
// In an .astro page
Astro.response.headers.set(
  'Cache-Control',
  'public, s-maxage=60, stale-while-revalidate=300'
);

const data = await fetchExpensiveData();
---
<h1>{data.title}</h1>
```

**Header cheat sheet**:

| Header | Meaning |
|--------|---------|
| `public, max-age=3600` | Browser + CDN cache for 1 hour |
| `public, s-maxage=60` | CDN caches 60s, browser doesn't |
| `public, s-maxage=60, stale-while-revalidate=300` | CDN serves stale for 5min while refreshing |
| `private, max-age=0` | No caching (user-specific data) |
| `no-store` | Never cache anywhere |

### Level 2: Middleware-based caching

Use Astro middleware to apply caching logic globally or per-route:

```ts
// src/middleware.ts
import { defineMiddleware } from 'astro:middleware';

export const onRequest = defineMiddleware(async ({ url, request }, next) => {
  const response = await next();

  // Cache static-ish pages for 5 minutes at the CDN
  if (url.pathname.startsWith('/blog/')) {
    response.headers.set(
      'Cache-Control',
      'public, s-maxage=300, stale-while-revalidate=600'
    );
  }

  // Never cache user-specific pages
  if (url.pathname.startsWith('/dashboard')) {
    response.headers.set('Cache-Control', 'private, no-store');
  }

  return response;
});
```

### Level 3: In-memory / external cache for expensive operations

For data that's expensive to fetch but doesn't change often:

```ts
// src/lib/cache.ts
const cache = new Map<string, { data: any; expires: number }>();

export async function cachedFetch<T>(
  key: string,
  fetcher: () => Promise<T>,
  ttlSeconds = 60
): Promise<T> {
  const cached = cache.get(key);
  if (cached && cached.expires > Date.now()) {
    return cached.data as T;
  }

  const data = await fetcher();
  cache.set(key, {
    data,
    expires: Date.now() + ttlSeconds * 1000,
  });
  return data;
}
```

Usage in a page:
```astro
---
import { cachedFetch } from '../lib/cache';

const posts = await cachedFetch('recent-posts', async () => {
  return await cms.getPosts({ limit: 10 });
}, 120); // Cache for 2 minutes
---
```

### Level 4: Edge caching (CDN-specific)

When deploying to Vercel, Cloudflare, or Netlify, use their edge caching:

**Vercel**: Uses `s-maxage` headers automatically. Add `stale-while-revalidate` for instant responses.

**Cloudflare**: Use Cache API in Workers for fine-grained control:
```ts
// Works in Cloudflare adapter context
const cache = caches.default;
const cached = await cache.match(request);
if (cached) return cached;
```

### Cache invalidation strategies

- **Time-based** (TTL): Set `max-age` or `s-maxage`. Simple, predictable.
- **On-demand**: Purge CDN cache via API when content changes (webhook from CMS).
- **Stale-while-revalidate**: Serve stale content immediately, refresh in background. Best UX.
- **Cache busting**: Append version/hash to URLs for assets.

---

## Middleware

Middleware runs before every request in SSR mode. Use it for auth, logging, caching headers, redirects.

```ts
// src/middleware.ts
import { defineMiddleware, sequence } from 'astro:middleware';

const auth = defineMiddleware(async ({ cookies, url, redirect }, next) => {
  const token = cookies.get('session')?.value;

  if (url.pathname.startsWith('/admin') && !token) {
    return redirect('/login');
  }

  return next();
});

const timing = defineMiddleware(async (context, next) => {
  const start = performance.now();
  const response = await next();
  const duration = performance.now() - start;
  response.headers.set('X-Response-Time', `duration.toFixed(0)ms`);
  return response;
});

// Chain multiple middleware
export const onRequest = sequence(auth, timing);
```

---

## API Routes / Endpoints

Files in `src/pages/api/` that export HTTP method handlers:

```ts
// src/pages/api/search.ts
import type { APIRoute } from 'astro';

export const GET: APIRoute = async ({ url }) => {
  const query = url.searchParams.get('q');
  const results = await search(query);

  return new Response(JSON.stringify(results), {
    headers: {
      'Content-Type': 'application/json',
      'Cache-Control': 'public, s-maxage=30',
    },
  });
};

export const POST: APIRoute = async ({ request }) => {
  const body = await request.json();
  // Process the submission
  return new Response(JSON.stringify({ ok: true }), {
    status: 200,
  });
};
```

---

## Decision Matrix

| Scenario | Mode | Caching |
|----------|------|---------|
| Blog / docs | SSG | N/A (static files) |
| Marketing site with contact form | Hybrid (form endpoint is SSR) | Static pages cached by CDN |
| E-commerce product pages | Hybrid or SSR | `s-maxage=300, stale-while-revalidate` |
| User dashboard | SSR | `private, no-store` |
| API-driven listing with filters | SSR | `s-maxage=60` + in-memory cache |
| CMS preview mode | SSR | `no-store` for preview, cached for published |

FILE:references/setup-and-structure.md
# Setup & Project Structure

## Table of Contents
1. [Creating a project](#creating-a-project)
2. [File structure explained](#file-structure)
3. [astro.config.mjs](#config)
4. [Adapters](#adapters)
5. [TypeScript setup](#typescript)
6. [Common setup mistakes](#common-mistakes)

---

## Creating a project

```bash
# Interactive setup
npm create astro@latest

# With a specific template
npm create astro@latest -- --template blog
npm create astro@latest -- --template docs
npm create astro@latest -- --template minimal

# With a community template
npm create astro@latest -- --template github-user/repo
```

After creation:
```bash
cd my-project
npm install
npm run dev     # Dev server on localhost:4321
npm run build   # Production build to ./dist
npm run preview # Preview production build locally
```

---

## File structure

```
my-project/
├── public/              # Static assets (served as-is, no processing)
│   ├── favicon.svg
│   ├── robots.txt
│   └── og-image.png
├── src/
│   ├── pages/           # File-based routing (REQUIRED for routes)
│   │   ├── index.astro          # → /
│   │   ├── about.astro          # → /about
│   │   ├── blog/
│   │   │   ├── index.astro      # → /blog
│   │   │   └── [slug].astro     # → /blog/:slug (dynamic)
│   │   └── api/
│   │       └── search.ts        # → /api/search (API endpoint)
│   ├── components/      # Reusable UI (Astro, Vue, React, etc.)
│   ├── layouts/         # Page shells with <slot />
│   ├── content/         # Content collections (markdown, MDX, JSON)
│   │   └── config.ts    # Collection schemas
│   ├── styles/          # Global styles
│   ├── lib/             # Utility functions, API clients
│   └── middleware.ts    # Request middleware (SSR only)
├── astro.config.mjs     # Astro configuration
├── tsconfig.json        # TypeScript config
└── package.json
```

### Key distinctions that trip people up

**`/public` vs `src/assets/`**:
- `/public` → Copied verbatim to build output. Use for files that need a stable URL (favicons, robots.txt, downloadable PDFs).
- `src/assets/` → Processed by Astro's build pipeline. Use for images (gets optimized), CSS (gets bundled), etc.

**`/src/pages` vs `/src/components`**:
- Pages create routes. Every `.astro`, `.md`, or `.mdx` file in `/src/pages` becomes a URL.
- Components don't create routes. They're imported and used inside pages or other components.
- A component in `/src/pages` IS a route whether you intended it or not.

**Dynamic routes**:
```
src/pages/blog/[slug].astro     → /blog/my-post
src/pages/[...path].astro       → catch-all (404 page, etc.)
src/pages/[lang]/[...slug].astro → /en/about, /es/about
```

For SSG, dynamic routes need `getStaticPaths()`:
```astro
---
export async function getStaticPaths() {
  const posts = await getCollection('blog');
  return posts.map(post => ({
    params: { slug: post.slug },
    props: { post }
  }));
}
const { post } = Astro.props;
---
```

For SSR, dynamic routes read params at runtime:
```astro
---
const { slug } = Astro.params;
const post = await getEntry('blog', slug);
if (!post) return Astro.redirect('/404');
---
```

---

## astro.config.mjs

Minimal config:
```js
import { defineConfig } from 'astro/config';

export default defineConfig({});
```

Common production config:
```js
import { defineConfig } from 'astro/config';
import vue from '@astrojs/vue';
import tailwind from '@astrojs/tailwind';
import sitemap from '@astrojs/sitemap';
import node from '@astrojs/node';

export default defineConfig({
  site: 'https://example.com',        // Required for sitemap, canonical URLs
  base: '/',                           // Set if hosting at a subpath: '/docs/'
  output: 'static',                    // 'static' | 'server'
  adapter: node({ mode: 'standalone' }), // Only for SSR/hybrid
  integrations: [
    vue(),
    tailwind(),
    sitemap(),
  ],
  vite: {
    // Vite config overrides if needed
  },
  trailingSlash: 'always',            // 'always' | 'never' | 'ignore'
});
```

### Config options that matter most

| Option | What it does | When you need it |
|--------|-------------|-----------------|
| `site` | Sets the canonical base URL | Always in production (sitemap, OG tags) |
| `base` | Subpath prefix for all routes | Hosting at `/docs/` or behind a reverse proxy |
| `output` | Rendering mode | `'server'` for SSR, `'static'` for SSG (default) |
| `adapter` | Server runtime | Required when `output` is `'server'` |
| `trailingSlash` | URL format | Match your host's behavior to avoid redirects |

---

## Adapters

Adapters tell Astro how to run your server code. Only needed for SSR or hybrid.

### Node.js (self-hosted)
```bash
npx astro add node
```
```js
// astro.config.mjs
import node from '@astrojs/node';
export default defineConfig({
  output: 'server',
  adapter: node({ mode: 'standalone' }), // or 'middleware'
});
```
- `standalone` → Runs its own HTTP server (port 4321 by default)
- `middleware` → Exports an Express/Fastify-compatible handler

### Vercel
```bash
npx astro add vercel
```
```js
import vercel from '@astrojs/vercel';
export default defineConfig({
  output: 'server',
  adapter: vercel(),
});
```

### Netlify
```bash
npx astro add netlify
```

### Cloudflare Pages
```bash
npx astro add cloudflare
```

---

## TypeScript setup

Astro supports TypeScript out of the box. The frontmatter section of `.astro` files is TypeScript by default.

```json
// tsconfig.json (Astro's recommended)
{
  "extends": "astro/tsconfigs/strict"  // or "base" for looser checks
}
```

Type-safe props:
```astro
---
interface Props {
  title: string;
  description?: string;
  tags: string[];
}
const { title, description = 'Default', tags } = Astro.props;
---
```

---

## Common setup mistakes

**Installing integrations manually instead of using `npx astro add`**:
The `astro add` command updates both `package.json` AND `astro.config.mjs`. Manual npm install only does half the job.

**Putting components in `/src/pages` accidentally**:
Everything in pages becomes a route. If you have `src/pages/Button.astro`, it creates a `/Button` page.

**Forgetting `site` in config**:
Without it, `Astro.site` returns undefined, sitemap generation fails, and canonical URLs break.

**Wrong Node.js version**:
Astro requires Node 18+. Check with `node --version`.

**Mixing package managers**:
Pick one (npm, pnpm, yarn) and stick with it. Delete lock files from other managers.

FILE:references/islands-and-vue.md
# Islands Architecture & Vue Integration

## Table of Contents
1. [How islands work](#how-islands-work)
2. [Client directives explained](#client-directives)
3. [Choosing the right directive](#choosing-directives)
4. [Vue integration setup](#vue-setup)
5. [Vue component patterns](#vue-patterns)
6. [React and Svelte islands](#other-frameworks)
7. [Common island mistakes](#mistakes)
8. [Advanced patterns](#advanced)

---

## How islands work

Astro's island architecture is its core differentiator. The mental model:

1. **Every component renders to static HTML on the server** (build time for SSG, request time for SSR).
2. **By default, zero JavaScript is shipped for any component.**
3. **Adding a `client:*` directive tells Astro to also ship the JS and hydrate that component.**

This means a page with 20 components but only 2 `client:*` directives ships JS for only those 2 components. The rest are pure HTML.

```
┌──────────────────────────────────────────────┐
│  Page (static HTML shell)                    │
│                                              │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐   │
│  │ Header   │  │ Article  │  │ Sidebar  │   │
│  │ (static) │  │ (static) │  │ (static) │   │
│  └──────────┘  └──────────┘  └──────────┘   │
│                                              │
│  ┌──────────────────┐  ┌─────────────────┐   │
│  │ Search Bar       │  │ Comment Form    │   │
│  │ (island: Vue)    │  │ (island: React) │   │
│  │ client:load      │  │ client:visible  │   │
│  └──────────────────┘  └─────────────────┘   │
│                                              │
└──────────────────────────────────────────────┘
```

---

## Client directives explained

| Directive | When JS loads | When component hydrates | Use case |
|-----------|-------------|----------------------|---------|
| `client:load` | Immediately on page load | As soon as JS loads | Critical interactive UI (nav menus, auth forms) |
| `client:idle` | After page is idle (`requestIdleCallback`) | When browser is idle | Non-critical interactivity (newsletter signup, chat widget) |
| `client:visible` | When component enters viewport | When scrolled into view | Below-fold content (comments, carousels, charts) |
| `client:media="(query)"` | When media query matches | On match | Mobile-only menus, responsive components |
| `client:only="vue"` | Immediately | Client-only (no SSR HTML) | Components that can't SSR (uses `window`, DOM APIs) |

### Hydration flow

```
Server:  Render HTML → Send to browser
Browser: Display HTML immediately (fast!)
         ↓
         Load JS for island (based on directive timing)
         ↓
         Hydrate: Attach event listeners to existing HTML
         ↓
         Component is now interactive
```

**The hydration contract**: The HTML generated on the server must match what the client would render. If they differ, you get hydration mismatch errors.

---

## Choosing the right directive

**Decision flow**:

```
Does the component need interactivity?
├── No  → Don't use any client directive (static HTML)
├── Yes →
│   Is it above the fold / immediately needed?
│   ├── Yes → client:load
│   └── No  →
│       Is it below the fold?
│       ├── Yes → client:visible
│       └── No  →
│           Can it wait until the page is idle?
│           ├── Yes → client:idle
│           └── No  → client:load
│
│   Does it ONLY work in the browser (uses window/document)?
│   └── Yes → client:only="framework"
```

**Performance ranking** (best to worst):
1. No directive (zero JS)
2. `client:visible` (loads only if user scrolls to it)
3. `client:idle` (defers until page is idle)
4. `client:media` (loads only on matching viewport)
5. `client:load` (loads immediately)
6. `client:only` (no SSR HTML + immediate load)

---

## Vue integration setup

### Installation
```bash
npx astro add vue
```

This automatically:
- Installs `@astrojs/vue` and `vue`
- Adds the integration to `astro.config.mjs`

### Manual setup (if needed)
```bash
npm install @astrojs/vue vue
```

```js
// astro.config.mjs
import vue from '@astrojs/vue';

export default defineConfig({
  integrations: [
    vue({
      appEntrypoint: '/src/pages/_app',  // Optional: custom app setup
      jsx: false,                         // Enable JSX in Vue if needed
    }),
  ],
});
```

### Custom app entrypoint (for plugins, stores)
```ts
// src/pages/_app.ts
import type { App } from 'vue';
import { createPinia } from 'pinia';

export default (app: App) => {
  app.use(createPinia());
  // Register global components, directives, etc.
};
```

---

## Vue component patterns

### Basic Vue island
```vue
<!-- src/components/Counter.vue -->
<script setup lang="ts">
import { ref } from 'vue';

interface Props {
  initialCount?: number;
}

const props = withDefaults(defineProps<Props>(), {
  initialCount: 0,
});

const count = ref(props.initialCount);
</script>

<template>
  <div class="counter">
    <button @click="count--">-</button>
    <span>{{ count }}</span>
    <button @click="count++">+</button>
  </div>
</template>
```

Using it in Astro:
```astro
---
import Counter from '../components/Counter.vue';
---

<!-- Static: renders HTML but buttons won't work -->
<Counter initialCount={5} />

<!-- Interactive: buttons work after hydration -->
<Counter client:load initialCount={5} />
```

### Props serialization rules

Props passed from Astro to framework islands are serialized (converted to JSON). This means:

**Works** (serializable):
- Strings, numbers, booleans
- Arrays, plain objects
- `null`, `undefined`
- Dates (serialized as ISO strings)

**Does NOT work** (not serializable):
- Functions, callbacks
- Class instances
- Symbols
- Circular references
- Reactive Vue refs/computed

```astro
---
import MyVue from '../components/MyVue.vue';

// GOOD: plain data
const items = [{ id: 1, name: 'First' }];

// BAD: functions can't cross the boundary
const onClick = () => console.log('nope');
---

<MyVue client:load items={items} />
<!-- <MyVue client:load onClick={onClick} /> ← This won't work -->
```

### Composables and state management

Vue composables work normally inside hydrated islands:

```vue
<!-- src/components/SearchBar.vue -->
<script setup>
import { ref, watch } from 'vue';
import { useDebounceFn } from '@vueuse/core'; // VueUse works

const query = ref('');
const results = ref([]);

const search = useDebounceFn(async (q) => {
  if (!q) { results.value = []; return; }
  const res = await fetch(`/api/search?q=encodeURIComponent(q)`);
  results.value = await res.json();
}, 300);

watch(query, (q) => search(q));
</script>

<template>
  <input v-model="query" placeholder="Search..." />
  <ul v-if="results.length">
    <li v-for="r in results" :key="r.id">{{ r.title }}</li>
  </ul>
</template>
```

### Pinia stores across islands

Each island is an independent Vue app. Pinia stores are NOT shared between islands by default.

If you need shared state between Vue islands, use:
1. **Custom events** (`window.dispatchEvent` / `addEventListener`)
2. **Shared external store** (nanostores — works across any framework)
3. **URL state** (query params, hash)

```ts
// src/stores/cart.ts — using nanostores (framework-agnostic)
import { atom, map } from 'nanostores';

export const $cartItems = map<Record<string, number>>({});
export const $cartCount = atom(0);

export function addToCart(id: string) {
  const items = $cartItems.get();
  $cartItems.setKey(id, (items[id] || 0) + 1);
  $cartCount.set(Object.values($cartItems.get()).reduce((a, b) => a + b, 0));
}
```

---

## React and Svelte islands

The same patterns apply. Install the integration, add `client:*` directives.

```bash
npx astro add react
npx astro add svelte
```

**Mixing frameworks on the same page is fully supported**:
```astro
---
import VueCounter from '../components/Counter.vue';
import ReactChart from '../components/Chart.tsx';
import SvelteToggle from '../components/Toggle.svelte';
---

<VueCounter client:load />
<ReactChart client:visible data={chartData} />
<SvelteToggle client:idle />
```

Each island hydrates independently. They share no runtime state (unless using nanostores or similar).

---

## Common island mistakes

### 1. Forgetting the client directive
```astro
<!-- Bug: renders HTML but nothing is interactive -->
<Counter />

<!-- Fix: add the appropriate directive -->
<Counter client:load />
```
This is the #1 Astro support question. The component renders HTML fine but click handlers, reactive state, and lifecycle hooks don't run.

### 2. Using client:only when you don't need to
```astro
<!-- Avoid: No SSR HTML, shows blank until JS loads -->
<HeroSection client:only="vue" />

<!-- Better: SSR HTML shows immediately, then hydrates -->
<HeroSection client:load />
```
Only use `client:only` when the component truly cannot render on the server (uses `window`, `document`, canvas, WebGL, etc.).

### 3. Hydration mismatch errors
The server HTML must match the client render. Common causes:
- **Dates/timestamps**: Server and client format differently
- **Random values**: `Math.random()` in render
- **Conditional rendering based on `window`**: Server has no `window`
- **Browser-only APIs in template**: `localStorage.getItem()` in template

Fix: Move browser-only logic to `onMounted` (Vue) or `useEffect` (React):
```vue
<script setup>
import { ref, onMounted } from 'vue';

const theme = ref('light'); // Safe default for SSR

onMounted(() => {
  theme.value = localStorage.getItem('theme') || 'light';
});
</script>
```

### 4. Over-hydrating the page
If every component has `client:load`, you've defeated Astro's purpose. Audit:
- Does this component actually need JS? (Maybe it's just displaying data)
- Can it use `client:visible` instead of `client:load`?
- Can the interactive part be extracted into a smaller island?

### 5. Passing non-serializable props
```astro
---
// Bug: functions can't be serialized across the island boundary
const handleClick = () => alert('hi');
---
<MyVue client:load onClick={handleClick} />
```
Move the handler inside the Vue component instead.

---

## Advanced patterns

### Nested islands
An Astro component can contain multiple islands. An island can contain Astro children via slots:

```astro
---
import VueWrapper from '../components/Wrapper.vue';
import StaticContent from '../components/StaticContent.astro';
---

<VueWrapper client:load>
  <!-- This static content is passed as a slot -->
  <StaticContent />
</VueWrapper>
```

But a hydrated island cannot contain another hydrated island from a different framework. Keep islands flat.

### Island communication
Use custom events for cross-island communication:

```vue
<!-- Island A: emits event -->
<script setup>
function notify() {
  window.dispatchEvent(new CustomEvent('cart-updated', { detail: { count: 5 } }));
}
</script>
```

```vue
<!-- Island B: listens for event -->
<script setup>
import { onMounted, onUnmounted, ref } from 'vue';

const count = ref(0);
function handler(e) { count.value = e.detail.count; }

onMounted(() => window.addEventListener('cart-updated', handler));
onUnmounted(() => window.removeEventListener('cart-updated', handler));
</script>
```

### Lazy-loading heavy islands
Combine `client:visible` with dynamic imports for code-splitting:

```astro
---
// The Vue component JS only loads when scrolled into view
import HeavyChart from '../components/HeavyChart.vue';
---
<HeavyChart client:visible data={bigDataset} />
```

Astro automatically code-splits each island, so each `client:visible` component only downloads its JS when triggered.

FILE:references/deployment.md
# Deployment & Adapters

## Table of Contents
1. [Deployment decision tree](#decision-tree)
2. [Static deployment](#static)
3. [SSR deployment](#ssr)
4. [Environment variables](#env)
5. [Base path / subpath hosting](#base-path)
6. [Auth & protected pages](#auth)
7. [Common deployment failures](#failures)

---

## Deployment decision tree

```
Is every page static (SSG)?
├── Yes → Deploy to ANY static host
│   ├── Netlify, Vercel, Cloudflare Pages
│   ├── GitHub Pages, AWS S3 + CloudFront
│   └── Any web server (nginx, Apache)
│
└── No (SSR or Hybrid) → Need a server runtime
    ├── Vercel → @astrojs/vercel
    ├── Netlify → @astrojs/netlify
    ├── Cloudflare → @astrojs/cloudflare
    ├── AWS Lambda → @astrojs/node + serverless wrapper
    └── Self-hosted → @astrojs/node (standalone)
```

---

## Static deployment

Build produces a `dist/` folder of HTML, CSS, JS, and assets. Upload it anywhere.

```bash
npm run build
# Output: ./dist/
```

### Netlify
```toml
# netlify.toml
[build]
  command = "npm run build"
  publish = "dist"
```

### Vercel (static)
No adapter needed. Vercel auto-detects Astro:
```json
// vercel.json (optional, for customization)
{
  "buildCommand": "npm run build",
  "outputDirectory": "dist"
}
```

### GitHub Pages
```yaml
# .github/workflows/deploy.yml
name: Deploy to GitHub Pages
on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pages: write
      id-token: write
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npm ci
      - run: npm run build
      - uses: actions/upload-pages-artifact@v3
        with:
          path: dist
      - uses: actions/deploy-pages@v4
```

For GitHub Pages, set the base path:
```js
// astro.config.mjs
export default defineConfig({
  site: 'https://username.github.io',
  base: '/repo-name/',
});
```

### Cloudflare Pages (static)
Connect your Git repo in the Cloudflare dashboard. Build command: `npm run build`, output: `dist`.

---

## SSR deployment

### Vercel (serverless)
```bash
npx astro add vercel
```

```js
// astro.config.mjs
import vercel from '@astrojs/vercel';

export default defineConfig({
  output: 'server',
  adapter: vercel({
    // Options
    imageService: true,     // Use Vercel's image optimization
    isr: {
      expiration: 60,       // ISR: revalidate every 60 seconds
    },
  }),
});
```

### Netlify (serverless)
```bash
npx astro add netlify
```

```js
import netlify from '@astrojs/netlify';

export default defineConfig({
  output: 'server',
  adapter: netlify(),
});
```

### Cloudflare Workers
```bash
npx astro add cloudflare
```

```js
import cloudflare from '@astrojs/cloudflare';

export default defineConfig({
  output: 'server',
  adapter: cloudflare({
    mode: 'directory', // or 'advanced'
  }),
});
```

### Node.js (self-hosted)
```bash
npx astro add node
```

```js
import node from '@astrojs/node';

export default defineConfig({
  output: 'server',
  adapter: node({
    mode: 'standalone', // Runs its own HTTP server
    // mode: 'middleware' // Exports handler for Express/Fastify
  }),
});
```

Running in production:
```bash
npm run build
node ./dist/server/entry.mjs
# Server starts on port 4321
```

With a process manager:
```bash
# PM2
pm2 start ./dist/server/entry.mjs --name my-astro-app

# Or with environment variables
HOST=0.0.0.0 PORT=3000 node ./dist/server/entry.mjs
```

### Docker
```dockerfile
FROM node:20-alpine AS build
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build

FROM node:20-alpine AS runtime
WORKDIR /app
COPY --from=build /app/dist ./dist
COPY --from=build /app/node_modules ./node_modules
COPY --from=build /app/package.json ./

ENV HOST=0.0.0.0
ENV PORT=4321
EXPOSE 4321

CMD ["node", "./dist/server/entry.mjs"]
```

---

## Environment variables

### Defining variables
```bash
# .env (development)
PUBLIC_API_URL=https://api.dev.example.com
SECRET_API_KEY=sk-dev-12345

# .env.production (production build)
PUBLIC_API_URL=https://api.example.com
SECRET_API_KEY=sk-prod-67890
```

### Accessing variables

**Server-side (frontmatter, endpoints, middleware)**:
```ts
// Access ALL variables (public and secret)
const apiKey = import.meta.env.SECRET_API_KEY;
const apiUrl = import.meta.env.PUBLIC_API_URL;
```

**Client-side (islands, browser JS)**:
```ts
// Only PUBLIC_ prefixed variables are available
const apiUrl = import.meta.env.PUBLIC_API_URL;
// import.meta.env.SECRET_API_KEY → undefined (intentionally hidden)
```

### Platform-specific env setup

**Vercel**: Set in project Settings → Environment Variables
**Netlify**: Set in Site Settings → Environment Variables
**Cloudflare**: Set in Workers & Pages → Settings → Variables
**Node**: Pass via shell or `.env` file

The `PUBLIC_` prefix convention is enforced by Astro. Never put secrets in `PUBLIC_` variables.

---

## Base path / subpath hosting

When hosting at a subpath like `https://example.com/docs/`:

```js
// astro.config.mjs
export default defineConfig({
  base: '/docs/',
  site: 'https://example.com',
});
```

### What `base` affects
- All generated links are prefixed with `/docs/`
- `Astro.url` includes the base
- Static assets are served from `/docs/`
- The dev server mounts at `localhost:4321/docs/`

### What breaks without `base`
- Links point to `/` instead of `/docs/` → 404
- CSS/JS assets 404 because paths are wrong
- Images in `/public/` load from the wrong path

### Using base-aware paths
```astro
---
// Always use Astro's URL utilities for base-aware paths
const homeURL = new URL('/', Astro.url);
---

<!-- DON'T hardcode paths -->
<a href="/">Home</a>  <!-- Wrong if base is /docs/ -->

<!-- DO use Astro's base-aware utilities -->
<a href={`import.meta.env.BASE_URL`}>Home</a>
```

### Reverse proxy setup (nginx)
```nginx
location /docs/ {
    proxy_pass http://localhost:4321/docs/;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
}
```

---

## Auth & protected pages

### Middleware-based auth (SSR only)
```ts
// src/middleware.ts
import { defineMiddleware } from 'astro:middleware';

const protectedRoutes = ['/dashboard', '/admin', '/settings'];

export const onRequest = defineMiddleware(async ({ url, cookies, redirect }, next) => {
  const isProtected = protectedRoutes.some(r => url.pathname.startsWith(r));

  if (isProtected) {
    const session = cookies.get('session')?.value;

    if (!session) {
      return redirect(`/login?redirect=encodeURIComponent(url.pathname)`);
    }

    // Validate session (check DB, verify JWT, etc.)
    const user = await validateSession(session);
    if (!user) {
      cookies.delete('session');
      return redirect('/login');
    }

    // Make user available to the page
    Astro.locals.user = user;
  }

  return next();
});
```

### Auth API endpoints
```ts
// src/pages/api/auth/login.ts
import type { APIRoute } from 'astro';

export const POST: APIRoute = async ({ request, cookies, redirect }) => {
  const formData = await request.formData();
  const email = formData.get('email') as string;
  const password = formData.get('password') as string;

  const user = await authenticateUser(email, password);

  if (!user) {
    return new Response('Invalid credentials', { status: 401 });
  }

  const session = await createSession(user.id);
  cookies.set('session', session.id, {
    httpOnly: true,
    secure: true,
    sameSite: 'lax',
    path: '/',
    maxAge: 60 * 60 * 24 * 7, // 1 week
  });

  return redirect('/dashboard');
};
```

### Auth in static mode (DON'T)

You cannot do server-side auth in static mode. The HTML is pre-built. Options:
1. Switch protected pages to SSR (hybrid mode)
2. Use client-side auth (check token in island, redirect via JS) — but the HTML is still in the build output
3. Use an external auth gateway (Cloudflare Access, Netlify Identity)

---

## Common deployment failures

### "Cannot find adapter"
```
[ERROR] Cannot use `output: 'server'` without an adapter.
```
Fix: Install the correct adapter: `npx astro add vercel` (or node, netlify, etc.)

### Environment variables undefined in production
- Check the variable is set on the hosting platform
- For client-side access, variable must start with `PUBLIC_`
- For SSR, make sure the runtime environment has the variable (not just build time)

### SSR pages return 404 on static host
You deployed an SSR app to a static host (GitHub Pages, S3). Either:
- Switch those pages to `export const prerender = true`
- Or deploy to a host that supports SSR (Vercel, Netlify, Cloudflare Workers, Node)

### Broken assets after deploy
- Check `base` in config matches the deployment path
- Verify `trailingSlash` setting matches the host's behavior
- Clear CDN cache after deploy

### Build succeeds locally but fails on host
- Node version mismatch (Astro needs 18+)
- Missing environment variables
- Different package manager lock file
- Case-sensitivity in file names (macOS is case-insensitive, Linux isn't)

### "window is not defined" during build
A component uses browser APIs during SSR. Fix:
- Wrap in `client:only="framework"` directive
- Move browser code to `onMounted` / `useEffect`
- Guard with `if (typeof window !== 'undefined')`

FILE:references/performance.md
# Performance Optimization

## Table of Contents
1. [Image optimization](#images)
2. [Hydration control](#hydration)
3. [Bundle analysis](#bundle)
4. [Font loading](#fonts)
5. [Prefetching](#prefetching)
6. [Build performance](#build)

---

## Image optimization

Astro has built-in image optimization via `astro:assets`. Use it — raw `<img>` tags are a performance miss.

### The `<Image />` component
```astro
---
import { Image } from 'astro:assets';
import heroImage from '../assets/hero.jpg';
---

<!-- Optimized: resized, converted to WebP/AVIF, width/height set -->
<Image src={heroImage} alt="Hero" />

<!-- With explicit dimensions -->
<Image src={heroImage} alt="Hero" width={800} height={400} />

<!-- Remote images (must be configured) -->
<Image
  src="https://example.com/photo.jpg"
  alt="Remote"
  width={600}
  height={300}
  inferSize  <!-- Or use inferSize to auto-detect -->
/>
```

### Configure remote image domains
```js
// astro.config.mjs
export default defineConfig({
  image: {
    domains: ['example.com', 'cdn.example.com'],
    // Or allow all remotes (less secure):
    // remotePatterns: [{ protocol: 'https' }],
  },
});
```

### The `<Picture />` component
For responsive art direction:
```astro
---
import { Picture } from 'astro:assets';
import hero from '../assets/hero.jpg';
---

<Picture
  src={hero}
  formats={['avif', 'webp']}
  alt="Hero"
  widths={[400, 800, 1200]}
  sizes="(max-width: 600px) 400px, (max-width: 900px) 800px, 1200px"
/>
```

### Critical image tips
- **Never lazy-load above-the-fold images** — it delays LCP
- **Always set width and height** — prevents layout shift (CLS)
- **Use `loading="eager"` for LCP image**, `loading="lazy"` for everything else
- **Preload the hero image** in the layout `<head>`:
  ```html
  <link rel="preload" as="image" href={heroSrc} />
  ```

---

## Hydration control

The fewer islands you hydrate, the faster your site. Audit every `client:*` directive.

### Hydration audit checklist
For each component with a `client:*` directive, ask:
1. Does this component actually need JavaScript? (Display-only components don't)
2. Is this component above the fold? (If not, use `client:visible`)
3. Is this component critical to the page? (If not, use `client:idle`)
4. Does the entire component need to be interactive? (Extract the interactive part into a smaller island)

### Splitting large components
Instead of hydrating a large component:
```astro
<!-- Bad: hydrates the entire product card just for the "Add to Cart" button -->
<ProductCard client:load product={product} />
```

Split into static + interactive:
```astro
<!-- Good: static HTML for display, tiny island for the button -->
<ProductCard product={product} />
<AddToCartButton client:idle productId={product.id} />
```

### Measuring hydration cost
Check the Network tab in DevTools:
- Filter by JS files
- Look at which islands load the most JS
- Each island creates its own chunk — find the heavy ones

---

## Bundle analysis

### Inspect the build output
```bash
npm run build
# Check dist/ size
du -sh dist/
du -sh dist/_astro/  # JS + CSS bundles
```

### Use Vite's bundle visualizer
```js
// astro.config.mjs
export default defineConfig({
  vite: {
    build: {
      rollupOptions: {
        plugins: [
          // Add visualizer plugin
        ],
      },
    },
  },
});
```

Or install and use `vite-bundle-visualizer`:
```bash
npx vite-bundle-visualizer
```

### Common bundle bloat sources
- **Large framework islands**: A single Vue/React component pulls in the entire framework runtime
- **Unused library imports**: `import _ from 'lodash'` instead of `import debounce from 'lodash/debounce'`
- **All-in-one component libraries**: Import only the components you use
- **Client-side routing libraries**: Astro doesn't need SPA routing; use standard `<a>` tags

---

## Font loading

Fonts are a common LCP blocker. Load them efficiently.

### Self-host fonts (preferred)
```css
/* src/styles/fonts.css */
@font-face {
  font-family: 'MyFont';
  src: url('/fonts/myfont.woff2') format('woff2');
  font-weight: 400;
  font-style: normal;
  font-display: swap; /* Show fallback immediately, swap when loaded */
}
```

### Preload critical fonts
```astro
<!-- In your layout <head> -->
<link
  rel="preload"
  href="/fonts/myfont.woff2"
  as="font"
  type="font/woff2"
  crossorigin
/>
```

### Font subsetting
Only include the characters you need:
```bash
# Using glyphhanger
npx glyphhanger --whitelist="US_ASCII" --subset=myfont.ttf
```

### Google Fonts the right way
Don't use the default `<link>` tag. Instead, self-host:
```bash
# Download fonts locally with fontsource
npm install @fontsource/inter
```

```ts
// In your layout frontmatter
import '@fontsource/inter/400.css';
import '@fontsource/inter/700.css';
```

---

## Prefetching

Astro has built-in link prefetching to speed up navigation.

```js
// astro.config.mjs
export default defineConfig({
  prefetch: {
    prefetchAll: false,          // Don't prefetch everything
    defaultStrategy: 'viewport', // Prefetch links when they enter viewport
  },
});
```

### Per-link control
```html
<!-- Prefetch on hover (default) -->
<a href="/about" data-astro-prefetch>About</a>

<!-- Prefetch when visible in viewport -->
<a href="/blog" data-astro-prefetch="viewport">Blog</a>

<!-- Don't prefetch -->
<a href="/heavy-page" data-astro-prefetch="false">Heavy Page</a>
```

---

## Build performance

### Speed up large SSG builds

**Limit getStaticPaths in development**:
```ts
export async function getStaticPaths() {
  const posts = await getCollection('blog');

  // In dev, only build a few pages for speed
  if (import.meta.env.DEV) {
    return posts.slice(0, 5).map(post => ({
      params: { slug: post.slug },
      props: { post },
    }));
  }

  return posts.map(post => ({
    params: { slug: post.slug },
    props: { post },
  }));
}
```

**Deduplicate API calls**: Astro caches `fetch()` calls during static builds. Use the same URL to benefit from deduplication:
```ts
// These two calls hit the API only once
const res1 = await fetch('https://api.example.com/posts');
const res2 = await fetch('https://api.example.com/posts');
```

**Parallelize where possible**: Astro builds pages in parallel. But if your data fetching is sequential, that bottlenecks the build.

### Reduce build output size
- Use `<Image />` instead of raw `<img>` (smaller optimized images)
- Remove unused CSS (Astro scopes styles by default, but global CSS can bloat)
- Check for duplicate dependencies: `npm ls | grep duplicated`

---
name: astro-starlight
description: >
  Use this skill whenever the user wants to create, configure, customize, or troubleshoot an Astro Starlight documentation site. Triggers include any mention of 'Starlight', 'Astro docs site', 'documentation site with Astro', '@astrojs/starlight', or requests to build a docs website. Also trigger when the user mentions sidebar configuration for docs, MDX components in documentation, Pagefind search setup, doc site theming/styling, deploying a docs site to a subpath, or versioned documentation. Use this skill even if the user just says 'docs site' or 'documentation website' — Starlight is the recommended approach. Covers: project scaffolding, content structure, sidebar navigation, MDX/Markdown authoring, CSS/Tailwind theming, component usage, i18n, deployment, search, and all common pitfalls.
---

# Astro Starlight Skill

Build production-grade documentation sites with Astro Starlight. This skill covers everything from initial scaffolding to advanced customization and deployment.

## When to use this skill

- Creating a new documentation site from scratch
- Adding Starlight to an existing Astro project
- Configuring sidebar navigation (manual, autogenerated, or mixed)
- Styling/theming a Starlight site (custom CSS, Tailwind, dark mode)
- Using MDX components inside docs (Tabs, Cards, Asides, Code, etc.)
- Deploying to subpaths (`/docs/`), Vercel, Netlify, Cloudflare Pages
- Setting up search (Pagefind, Algolia)
- Internationalization (i18n)
- Overriding built-in components
- Troubleshooting common issues (404s, broken links, sidebar mismatches)

## Quick orientation

Before writing any code, read the relevant reference file(s) from `references/`:

| Task | Read first |
|---|---|
| New project or project structure | `references/project-setup.md` |
| Sidebar, navigation, frontmatter | `references/sidebar-and-content.md` |
| Styling, theming, Tailwind | `references/styling-and-theming.md` |
| MDX components, custom components | `references/components.md` |
| Deployment, subpaths, search, i18n | `references/deployment-and-advanced.md` |
| Something broken? | `references/troubleshooting.md` |

For most tasks, read `project-setup.md` first, then the topic-specific file.

## Core workflow

### 1. Scaffold the project

```bash
npm create astro@latest -- --template starlight
```

This gives you a working project with:
```
my-docs/
├── astro.config.mjs        # Starlight integration config
├── src/
│   ├── content/
│   │   └── docs/            # All doc pages live here
│   │       └── index.mdx
│   └── content.config.ts    # Content collection schema
├── public/                  # Static assets (favicon, images)
└── package.json
```

### 2. Design folder structure FIRST

This is the most important step. Your folder structure = your URL structure = your sidebar structure. Plan it before writing content. See `references/sidebar-and-content.md` for patterns.

### 3. Configure `astro.config.mjs`

The Starlight integration is configured here. Minimum viable config:

```js
import { defineConfig } from 'astro/config';
import starlight from '@astrojs/starlight';

export default defineConfig({
  integrations: [
    starlight({
      title: 'My Docs',
      sidebar: [
        {
          label: 'Guides',
          autogenerate: { directory: 'guides' },
        },
      ],
    }),
  ],
});
```

### 4. Write content in `src/content/docs/`

Every `.md` or `.mdx` file needs frontmatter with at least `title`:

```yaml
---
title: Getting Started
description: How to get started with our product.
sidebar:
  order: 1
---
```

### 5. Build and deploy

```bash
npm run build    # Outputs to dist/
npm run preview  # Preview the build locally
```

## Key principles

1. **Design folder structure before writing content.** Restructuring later means broken links and sidebar chaos.
2. **Keep sidebar config explicit** rather than fully auto-generated — you'll want control over ordering and grouping.
3. **Use `.mdx` only when you need component imports.** Plain `.md` is simpler and avoids hydration issues.
4. **Test deployment early**, especially if hosting at a subpath. Don't wait until the end.
5. **Keep CSS overrides minimal.** Work with Starlight's design tokens (CSS custom properties) rather than fighting its defaults.
6. **Use frontmatter `sidebar.order`** to control page ordering within autogenerated groups.

## Important gotchas (read these)

- **Subpath deployment is the #1 source of bugs.** If deploying to `/docs/`, you must set both `site` and `base` in `astro.config.mjs`. See `references/deployment-and-advanced.md`.
- **MDX imports only work in `.mdx` files**, not `.md`. If you get import errors, check your file extension.
- **Sidebar config and filesystem must agree.** A page that exists on disk but isn't in the sidebar config (or vice versa) causes confusing 404s or missing nav items.
- **Tailwind + Starlight requires the `@astrojs/starlight-tailwind` compatibility package.** Without it, styles will fight each other.
- **Pagefind search breaks on subpaths** if `base` isn't configured correctly.
- **Custom components need `client:load`** (or another client directive) if they use JavaScript/interactivity. Without it, they render as static HTML only.

## Reference files

Read these for detailed guidance. Each is self-contained and covers one topic area thoroughly:

- `references/project-setup.md` — Scaffolding, project structure, `astro.config.mjs` anatomy, content collections
- `references/sidebar-and-content.md` — Sidebar config patterns, frontmatter reference, content authoring, ordering
- `references/styling-and-theming.md` — Custom CSS, Tailwind integration, theming, dark mode, Expressive Code
- `references/components.md` — Built-in components (Tabs, Cards, Asides, etc.), custom components in MDX, hydration
- `references/deployment-and-advanced.md` — Deployment targets, subpath hosting, search, i18n, versioning, auth, SSR
- `references/troubleshooting.md` — Diagnosis and fixes for the most common Starlight issues

## Useful links

- Docs: https://starlight.astro.build
- GitHub: https://github.com/withastro/starlight
- Config reference: https://starlight.astro.build/reference/configuration/
- Frontmatter reference: https://starlight.astro.build/reference/frontmatter/
- Component list: https://starlight.astro.build/components/using-components/
- Community plugins: https://starlight.astro.build/resources/plugins/

FILE:references/deployment-and-advanced.md
# Deployment & Advanced Topics

## Table of Contents
1. [Deployment basics](#deployment)
2. [Subpath hosting (the #1 headache)](#subpath)
3. [Search configuration](#search)
4. [Internationalization (i18n)](#i18n)
5. [Versioned documentation](#versioning)
6. [Authentication / private docs](#auth)
7. [SSR (server-side rendering)](#ssr)
8. [Plugins](#plugins)

---

## 1. Deployment basics {#deployment}

Starlight produces static HTML by default. Build with:

```bash
npm run build     # Outputs to dist/
npm run preview   # Local preview of built site
```

### Platform-specific guides

**Vercel:**
```bash
npx astro add vercel
```
Then push to GitHub and connect to Vercel. Works with zero config for root-path deployments.

**Netlify:**
```bash
npx astro add netlify
```
Or create `netlify.toml`:
```toml
[build]
  command = "npm run build"
  publish = "dist"
```

**Cloudflare Pages:**
```bash
npx astro add cloudflare
```
Set build command to `npm run build` and output directory to `dist`.

**GitHub Pages:**
Set in `astro.config.mjs`:
```js
export default defineConfig({
  site: 'https://username.github.io',
  base: '/repo-name/',  // Only if not using custom domain
  // ...
});
```

Full Astro deployment guide: https://docs.astro.build/en/guides/deploy/

### Pre-deployment checklist

1. Set `site` in `astro.config.mjs` to your production URL
2. If using a subpath, set `base` (see section 2)
3. Run `npm run build` and check for errors
4. Run `npm run preview` and test all pages
5. Test navigation, search, and links
6. Verify images/assets load correctly
7. Check dark mode works

## 2. Subpath hosting (the #1 headache) {#subpath}

Hosting at `https://example.com/docs/` instead of the root is the single most common source of deployment bugs.

### Configuration

```js
// astro.config.mjs
export default defineConfig({
  site: 'https://example.com',
  base: '/docs/',
  integrations: [
    starlight({
      title: 'My Docs',
      // ... rest of config
    }),
  ],
});
```

Both `site` and `base` are required. `base` must start and end with `/`.

### What breaks and why

| Symptom | Cause | Fix |
|---|---|---|
| All links 404 after deploy | `base` not set | Add `base: '/docs/'` to config |
| Assets (CSS/JS) 404 | `base` not matching server path | Ensure `base` matches your reverse proxy or hosting path |
| Sidebar links go to wrong URLs | Mixing absolute and relative links | Use Starlight's built-in link handling; avoid hardcoded absolute paths |
| Search returns results but links are broken | Pagefind not aware of base path | Rebuild after setting `base`; Pagefind picks it up automatically |
| Images missing | Referencing from `public/` with wrong path | Use `~/assets/` imports for optimized images, or prefix public paths with the base |

### Reverse proxy setup (nginx example)

```nginx
location /docs/ {
    proxy_pass http://localhost:3000/docs/;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
}
```

The proxy path must match the `base` path exactly.

### The failure pattern to avoid

1. Build docs locally at root `/` — everything works
2. Add reverse proxy at `/docs`
3. Deploy
4. Everything breaks: links, assets, search, sidebar

Prevention: **Set `base` from day one if you plan to use a subpath.** Test with `npm run preview` before deploying.

### Using Starlight at a subpath of an existing Astro site

If your main Astro site lives at `/` and you want docs at `/docs/`:

```js
// astro.config.mjs
export default defineConfig({
  integrations: [
    starlight({
      title: 'Docs',
      // Starlight handles the /docs/ prefix internally
    }),
  ],
});
```

Starlight will serve from `/docs/` based on the content collection location. See the manual setup guide for details.

## 3. Search configuration {#search}

### Pagefind (default)

Pagefind is enabled by default. It indexes all pages at build time and provides a built-in search UI.

**To disable:**
```js
starlight({ pagefind: false })
```

**To customize ranking:**
```js
starlight({
  pagefind: {
    ranking: {
      pageLength: 0.75,
      termFrequency: 1.2,
      termSaturation: 1.2,
      termSimilarity: 0.9,
    },
  },
})
```

**To search across multiple sites:**
```js
starlight({
  pagefind: {
    mergeIndex: [{
      bundlePath: 'https://other-site.com/_pagefind',
    }],
  },
})
```

**Excluding pages from search:**
```yaml
---
title: Hidden Page
pagefind: false
---
```

### Pagefind + subpaths

Pagefind should work with subpaths if `base` is configured correctly. If search breaks:
1. Ensure `base` is set in `astro.config.mjs`
2. Do a full rebuild (`npm run build`)
3. Check that `_pagefind/` directory exists in your build output

### Algolia DocSearch

To use Algolia instead of Pagefind:
1. Disable Pagefind: `pagefind: false`
2. Install Algolia integration or use a community plugin
3. Override the `Search` component:
```js
starlight({
  pagefind: false,
  components: {
    Search: './src/components/AlgoliaSearch.astro',
  },
})
```

## 4. Internationalization (i18n) {#i18n}

### Setup

```js
starlight({
  defaultLocale: 'en',
  locales: {
    en: { label: 'English' },
    es: { label: 'Español' },
    ja: { label: '日本語', lang: 'ja' },
    ar: { label: 'العربية', dir: 'rtl' },
  },
})
```

### Content structure for i18n

```
src/content/docs/
├── en/
│   ├── index.md
│   └── guides/
│       └── intro.md
├── es/
│   ├── index.md
│   └── guides/
│       └── intro.md
└── ja/
    └── index.md
```

### Root locale (no `/en/` prefix)

Serve the default language at the root:

```js
locales: {
  root: { label: 'English', lang: 'en' },
  es: { label: 'Español' },
},
```

Now English pages are at `/getting-started/` (not `/en/getting-started/`) and Spanish at `/es/getting-started/`.

### Translating sidebar labels

```js
sidebar: [{
  label: 'Guides',
  translations: { es: 'Guías', ja: 'ガイド' },
  items: [
    {
      slug: 'guides/intro',
      label: 'Introduction',
      translations: { es: 'Introducción' },
    },
  ],
}],
```

### i18n content collection

For UI string translations, add the i18n collection:

```ts
// src/content.config.ts
import { docsLoader, i18nLoader } from '@astrojs/starlight/loaders';
import { docsSchema, i18nSchema } from '@astrojs/starlight/schema';

export const collections = {
  docs: defineCollection({ loader: docsLoader(), schema: docsSchema() }),
  i18n: defineCollection({ loader: i18nLoader(), schema: i18nSchema() }),
};
```

### Fallback content

If a translation doesn't exist, Starlight shows the default locale version with a notice. This is automatic.

## 5. Versioned documentation {#versioning}

Starlight does **not** have built-in versioning. This is a known limitation and a common pain point for teams.

### Workarounds

**Option A: Folder-based versions**

```
src/content/docs/
├── v1/
│   ├── getting-started.md
│   └── api.md
├── v2/
│   ├── getting-started.md
│   └── api.md
└── latest/           # Symlink or copy of current version
```

Sidebar config:
```js
sidebar: [
  {
    label: 'v2 (Latest)',
    autogenerate: { directory: 'v2' },
  },
  {
    label: 'v1',
    collapsed: true,
    autogenerate: { directory: 'v1' },
  },
],
```

Problems with this approach:
- Content duplication
- Sidebar gets long with many versions
- No automatic version switcher

**Option B: Separate builds per version**

Deploy each version as a separate Starlight site:
- `docs.example.com/v1/` — separate Starlight build
- `docs.example.com/v2/` — separate Starlight build

Use Pagefind's `mergeIndex` to search across versions.

**Option C: Community plugins**

Check the Starlight plugin showcase for versioning plugins: https://starlight.astro.build/resources/plugins/

## 6. Authentication / private docs {#auth}

Starlight is static-first, which makes auth non-trivial.

### Options

**Hosting-level auth (simplest):**
- Vercel: Use [Vercel Authentication](https://vercel.com/docs/security/password-protection)
- Netlify: Use [Netlify Identity](https://docs.netlify.com/security/secure-access-to-sites/)
- Cloudflare: Use [Cloudflare Access](https://developers.cloudflare.com/cloudflare-one/applications/configure-apps/)

**SSR mode with middleware:**
Set `prerender: false` in Starlight config, add an SSR adapter, and use Astro middleware:

```js
// astro.config.mjs
import node from '@astrojs/node';

export default defineConfig({
  output: 'server',
  adapter: node({ mode: 'standalone' }),
  integrations: [
    starlight({
      prerender: false,
      // ...
    }),
  ],
});
```

```ts
// src/middleware.ts
import { defineMiddleware } from 'astro:middleware';

export const onRequest = defineMiddleware(async (context, next) => {
  // Check auth cookie/token
  const isAuthenticated = checkAuth(context.request);
  if (!isAuthenticated && context.url.pathname.startsWith('/private/')) {
    return context.redirect('/login');
  }
  return next();
});
```

**Mixed public/private:**
- Keep public docs as static (default)
- Use `src/pages/` for private routes with middleware-based auth

## 7. SSR (server-side rendering) {#ssr}

By default, Starlight prerenders all pages to static HTML. To render on demand:

```js
starlight({
  prerender: false,
})
```

Requires an SSR adapter (Node, Vercel, Netlify, Cloudflare, etc.).

Note: Pagefind search requires prerendered pages. If `prerender: false`, you must disable Pagefind and use an alternative search solution.

## 8. Plugins {#plugins}

Starlight has a plugin API for extending functionality.

### Using plugins

```js
import starlightPlugin from 'some-starlight-plugin';

starlight({
  plugins: [starlightPlugin()],
})
```

### Popular community plugins

Check the showcase: https://starlight.astro.build/resources/plugins/

Common categories:
- Blog integration
- Versioning
- API documentation generators
- Additional search providers
- Analytics
- Custom sidebar enhancements

### Plugin API

Plugins can:
- Modify the Starlight config
- Add custom CSS
- Override components
- Add Astro integrations
- Hook into the build process

See: https://starlight.astro.build/reference/plugins/

FILE:references/components.md
# Components

## Table of Contents
1. [Built-in components overview](#built-in)
2. [Using components in MDX](#mdx-usage)
3. [Using components in Markdoc](#markdoc-usage)
4. [Built-in component catalog](#catalog)
5. [Custom components](#custom)
6. [Overriding Starlight components](#overriding)
7. [Common component pitfalls](#pitfalls)

---

## 1. Built-in components overview {#built-in}

Starlight ships with a set of UI components designed for documentation. Import them from `@astrojs/starlight/components` in `.mdx` files.

These components only work in `.mdx` files (not plain `.md`). If you need components, rename your file from `.md` to `.mdx`.

## 2. Using components in MDX {#mdx-usage}

Import and use like JSX:

```mdx
---
title: My Page
---

import { Tabs, TabItem } from '@astrojs/starlight/components';

<Tabs>
  <TabItem label="npm">npm install my-package</TabItem>
  <TabItem label="pnpm">pnpm add my-package</TabItem>
  <TabItem label="yarn">yarn add my-package</TabItem>
</Tabs>
```

You can also import your own custom components:

```mdx
import CustomCard from '../../components/CustomCard.astro';

<CustomCard title="Hello">
  This is custom content.
</CustomCard>
```

## 3. Using components in Markdoc {#markdoc-usage}

If using Markdoc (`.mdoc` files) with the Starlight Markdoc preset, built-in components are available without imports using `{% %}` tag syntax:

```markdoc
{% card title="Stars" icon="star" %}
Sirius, Vega, Betelgeuse
{% /card %}
```

## 4. Built-in component catalog {#catalog}

### Tabs

Multi-tab content blocks:

```mdx
import { Tabs, TabItem } from '@astrojs/starlight/components';

<Tabs>
  <TabItem label="Stars">Sirius, Vega</TabItem>
  <TabItem label="Planets">Jupiter, Saturn</TabItem>
</Tabs>
```

Synchronized tabs (tabs with the same `syncKey` switch together):
```mdx
<Tabs syncKey="pkg">
  <TabItem label="npm">npm run build</TabItem>
  <TabItem label="pnpm">pnpm build</TabItem>
</Tabs>

<!-- Later on the same page, these stay synced: -->
<Tabs syncKey="pkg">
  <TabItem label="npm">npm run dev</TabItem>
  <TabItem label="pnpm">pnpm dev</TabItem>
</Tabs>
```

### Cards

```mdx
import { Card, CardGrid } from '@astrojs/starlight/components';

<CardGrid>
  <Card title="Feature One" icon="rocket">
    Description of feature one.
  </Card>
  <Card title="Feature Two" icon="star">
    Description of feature two.
  </Card>
</CardGrid>
```

### Link Cards

Cards that are clickable links:

```mdx
import { LinkCard } from '@astrojs/starlight/components';

<LinkCard
  title="Getting Started"
  description="Learn the basics."
  href="/getting-started/"
/>
```

### Asides

Callout boxes. Can also be created with `:::note` syntax in plain Markdown.

```mdx
import { Aside } from '@astrojs/starlight/components';

<Aside type="tip" title="Did you know?">
  You can use components in MDX!
</Aside>
```

Types: `note`, `tip`, `caution`, `danger`.

### Steps

Numbered step-by-step instructions:

```mdx
import { Steps } from '@astrojs/starlight/components';

<Steps>
1. Install the package
   ```bash
   npm install my-package
   ```

2. Configure your project
   Edit `config.js` with your settings.

3. Deploy
   Push to your hosting provider.
</Steps>
```

### Code

Enhanced code blocks with Expressive Code features:

```mdx
import { Code } from '@astrojs/starlight/components';

export const exampleCode = `const greeting = 'Hello';
console.log(greeting);`;

<Code code={exampleCode} lang="js" title="example.js" />
```

Useful when you need to pass dynamic code from variables.

### File Tree

Visual file/folder structure:

```mdx
import { FileTree } from '@astrojs/starlight/components';

<FileTree>
- src/
  - content/
    - docs/
      - index.md
      - **getting-started.md** (highlighted)
  - components/
- astro.config.mjs
- package.json
</FileTree>
```

Bold items with `**name**`. Add comments with parentheses.

### Icons

```mdx
import { Icon } from '@astrojs/starlight/components';

<Icon name="star" size="1.5rem" />
```

See the icons reference for available names: https://starlight.astro.build/reference/icons/

### Link Buttons

```mdx
import { LinkButton } from '@astrojs/starlight/components';

<LinkButton href="/getting-started/" variant="primary" icon="right-arrow">
  Get Started
</LinkButton>
```

Variants: `primary`, `secondary`, `minimal`.

### Badges

```mdx
import { Badge } from '@astrojs/starlight/components';

<Badge text="New" variant="tip" />
<Badge text="Deprecated" variant="caution" />
```

Variants: `note`, `tip`, `caution`, `danger`, `success`, `default`.

## 5. Custom components {#custom}

### Creating an Astro component for docs

```astro
<!-- src/components/DocsCallout.astro -->
---
interface Props {
  type?: 'info' | 'warning';
  title: string;
}

const { type = 'info', title } = Astro.props;
---

<div class={`callout callout-type`}>
  <h3>{title}</h3>
  <slot />
</div>

<style>
  .callout {
    padding: 1rem;
    border-radius: 0.5rem;
    margin: 1rem 0;
  }
  .callout-info { background: var(--sl-color-blue-low); }
  .callout-warning { background: var(--sl-color-orange-low); }
</style>
```

Use in MDX:
```mdx
import DocsCallout from '../../components/DocsCallout.astro';

<DocsCallout title="Important" type="warning">
  Don't forget to configure your base path!
</DocsCallout>
```

### Interactive components (React, Vue, Svelte, etc.)

For components that need client-side JavaScript, you need a `client:*` directive:

```mdx
import Counter from '../../components/Counter.jsx';

{/* Static — rendered at build time only, no interactivity */}
<Counter />

{/* Interactive — hydrated on the client */}
<Counter client:load />
```

Client directives:
- `client:load` — Hydrate immediately on page load (most common)
- `client:idle` — Hydrate when browser is idle
- `client:visible` — Hydrate when component scrolls into view
- `client:media` — Hydrate when a CSS media query matches
- `client:only="react"` — Skip server rendering, client only

### The `not-content` class

Starlight applies default content styles (margins, typography) to everything inside a doc page. If your component's layout conflicts:

```html
<div class="not-content">
  <!-- Your component's custom layout here -->
  <p>Not affected by Starlight content styles.</p>
</div>
```

## 6. Overriding Starlight components {#overriding}

You can replace any built-in Starlight component with your own:

```js
// astro.config.mjs
starlight({
  components: {
    // Replace the default header
    Header: './src/components/MyHeader.astro',
    // Replace social links
    SocialLinks: './src/components/MySocialLinks.astro',
    // Replace the theme toggle
    ThemeSelect: './src/components/MyThemeSelect.astro',
  },
})
```

Full list of overridable components: https://starlight.astro.build/reference/overrides/

To extend (not fully replace) a built-in component, import and wrap the original:

```astro
---
// src/components/MyHeader.astro
import type { Props } from '@astrojs/starlight/props';
import Default from '@astrojs/starlight/components/Header.astro';
---

<div class="my-header-wrapper">
  <Default {...Astro.props}><slot /></Default>
  <div class="custom-banner">Announcement here</div>
</div>
```

## 7. Common component pitfalls {#pitfalls}

**Problem: "Cannot use import statement" or import errors**
- You're trying to import in a `.md` file. Rename it to `.mdx`.

**Problem: Component renders but has no interactivity**
- Missing `client:load` (or another client directive). Without it, framework components render as static HTML.

**Problem: Component doesn't match Starlight's theme**
- Use Starlight's CSS custom properties (`var(--sl-color-*)`) instead of hardcoded colors.
- Check dark mode: use `[data-theme='dark']` selectors or Starlight's `--sl-color-*` which auto-switch.

**Problem: Hydration mismatch warnings**
- Ensure your interactive component produces the same HTML on server and client.
- Avoid `window` or `document` references during server rendering. Use `client:only="react"` for client-only components.

**Problem: Component layout breaks inside docs**
- Add `class="not-content"` to the wrapper element to opt out of Starlight's content typography styles.

**Problem: MDX component not found at runtime**
- Check the import path. Relative paths (`../../components/...`) are relative to the MDX file's location.
- Ensure the component file actually exports a default export (for Astro components, this is automatic).

FILE:references/troubleshooting.md
# Troubleshooting Starlight

Organized by symptom. Find your problem, understand the cause, apply the fix.

## Table of Contents
1. [Build & startup errors](#build-errors)
2. [404 errors and broken links](#404s)
3. [Sidebar issues](#sidebar)
4. [Styling and theming problems](#styling)
5. [MDX and component errors](#mdx)
6. [Search not working](#search)
7. [Deployment issues](#deployment)
8. [Performance issues](#performance)
9. [Dark mode issues](#dark-mode)
10. [i18n issues](#i18n)
11. [Upgrade issues](#upgrade)
12. [Nuclear options (when nothing else works)](#nuclear)

---

## 1. Build & startup errors {#build-errors}

### "Cannot find module '@astrojs/starlight'"

**Cause:** Starlight not installed or node_modules out of date.
**Fix:**
```bash
npm install @astrojs/starlight
# or if node_modules seems corrupt:
rm -rf node_modules package-lock.json
npm install
```

### "Content collection 'docs' is not defined"

**Cause:** Missing or misconfigured `src/content.config.ts`.
**Fix:** Create the file:
```ts
import { defineCollection } from 'astro:content';
import { docsLoader } from '@astrojs/starlight/loaders';
import { docsSchema } from '@astrojs/starlight/schema';

export const collections = {
  docs: defineCollection({ loader: docsLoader(), schema: docsSchema() }),
};
```

### "Invalid frontmatter" or schema validation errors

**Cause:** A page's frontmatter doesn't match the expected schema.
**Fix:**
- Ensure every page has at least `title` in frontmatter
- Check for YAML syntax errors (wrong indentation, missing quotes on special characters)
- If using custom schema extensions, verify the Zod schema matches your data

### Build hangs or is extremely slow

**Cause:** Usually a large number of pages or unoptimized images.
**Fix:**
- Use `~/assets/` for images (Astro optimizes them) instead of `public/`
- Check for circular imports in MDX files
- Consider if you have unused dependencies in your config
- Try `npm run build -- --verbose` for more info

### "port already in use" when running dev

**Fix:**
```bash
# Use a different port
npm run dev -- --port 3001
# Or kill the existing process
lsof -ti:4321 | xargs kill
```

## 2. 404 errors and broken links {#404s}

### Page exists but returns 404

**Diagnosis checklist:**
1. Is the file in `src/content/docs/`? Files elsewhere won't be picked up.
2. Does the file have valid frontmatter with `title`?
3. Is the URL correct? `src/content/docs/guides/intro.md` → `/guides/intro/`
4. If using `base` in config, is the full path including base correct?
5. Is the file extension `.md` or `.mdx`? Other extensions aren't processed by default.
6. Check for `draft: true` in frontmatter — draft pages are excluded from production builds.

### All pages 404 after deployment

**Cause:** Almost always a `base` path mismatch.
**Fix:**
- If deploying at root: remove `base` from config (or set to `'/'`)
- If deploying at `/docs/`: set `base: '/docs/'` (with leading AND trailing slashes)
- Rebuild and redeploy after changing `base`

### Internal links in content lead to 404

**Cause:** Incorrect link format in Markdown.
**Fix:** Use relative links from the current page's perspective:
```markdown
<!-- From src/content/docs/guides/intro.md -->

<!-- Link to src/content/docs/reference/api.md -->
[API Reference](/reference/api/)

<!-- Or relative -->
[API Reference](../reference/api/)
```

### Assets (images, CSS, JS) return 404

**Cause:** Wrong path or `base` not accounted for.
**Fix:**
- For images in `src/assets/`: use `~/assets/image.png` in frontmatter, or import in MDX
- For images in `public/`: use absolute paths from root, e.g. `/images/photo.jpg`
- If using `base: '/docs/'`, public assets need the base prefix: `/docs/images/photo.jpg`
- Recommendation: prefer `src/assets/` with imports — Astro handles path prefixing automatically

## 3. Sidebar issues {#sidebar}

### Page not showing in sidebar

**Possible causes:**
- Using manual sidebar and forgot to add the page
- Using autogenerate but file isn't in the specified directory
- Page has `sidebar.hidden: true` in frontmatter
- Page has `draft: true` in frontmatter (excluded in production)

**Fix:** Check your sidebar config and the page's frontmatter.

### Sidebar shows pages in wrong order

**Cause:** Autogenerated sidebar sorts alphabetically by filename.
**Fix:** Add `sidebar.order` to each page's frontmatter:
```yaml
---
title: First Page
sidebar:
  order: 1
---
```

### Sidebar link doesn't match the page it should

**Cause:** `slug` in sidebar config doesn't match the file path.
**Fix:** The slug should be the file path relative to `src/content/docs/`, without the extension:
- File: `src/content/docs/guides/intro.md`
- Slug: `guides/intro`

### Duplicate entries in sidebar

**Cause:** Page appears in both an autogenerated group AND a manual entry.
**Fix:** Use only one approach per page. If you need manual control over one page in an autogenerated group, either switch the whole group to manual, or use `sidebar.hidden: true` and add the manual entry elsewhere.

### Nested autogenerated groups have wrong labels

**Cause:** Subdirectory names become group labels.
**Fix:** Either rename the directory or switch to manual sidebar for that section.

## 4. Styling and theming problems {#styling}

### Custom CSS not taking effect

**Checklist:**
1. File path in `customCss` is correct (relative to project root, starts with `./`)
2. CSS syntax is valid
3. Specificity: Starlight uses cascade layers. Unlayered CSS wins. If using layers, check order.
4. Browser cache: hard refresh (Ctrl+Shift+R / Cmd+Shift+R)
5. Dev server: restart after config changes (`npm run dev`)

### Tailwind classes not working

**Checklist:**
1. Is `@astrojs/starlight-tailwind` installed?
2. Is the Tailwind CSS file listed FIRST in `customCss`?
3. Is `tailwindcss` Vite plugin added?
4. Tailwind utility classes work in `.astro` components but NOT directly in Markdown content
5. For Markdown content styling, use `customCss` or override components

### Fonts not loading

**Steps:**
1. Install: `npm install @fontsource/your-font`
2. Add to config: `customCss: ['@fontsource/your-font', './src/styles/custom.css']`
3. Set in CSS:
```css
:root {
  --sl-font: 'Your Font', sans-serif;
}
```

### Content area too narrow

```css
:root {
  --sl-content-width: 50rem; /* Default is 40rem */
}
```

### Colors look wrong

- Check that you're overriding BOTH light and dark mode variants
- Use `[data-theme='dark']` and `[data-theme='light']` selectors
- Or use Starlight's HSL-based color tokens which auto-adjust

## 5. MDX and component errors {#mdx}

### "Cannot use import statement outside a module"

**Cause:** Trying to use `import` in a `.md` file.
**Fix:** Rename the file to `.mdx`.

### Component renders but has no interactivity (clicks don't work, state doesn't update)

**Cause:** Missing client directive on a framework component (React, Vue, Svelte).
**Fix:** Add `client:load` (or another directive):
```mdx
import MyButton from '../../components/MyButton.jsx';

<MyButton client:load />
```

### "Hydration mismatch" warnings in console

**Cause:** Server-rendered HTML doesn't match what the client expects.
**Fix:**
- Don't access `window`, `document`, or `localStorage` during server rendering
- Use `client:only="react"` to skip server rendering entirely
- Ensure component output is deterministic (no random values, no Date.now())

### Component imported but nothing renders

**Checklist:**
1. Import path is correct (relative to the MDX file)
2. Component has a default export
3. If using self-closing tag (`<Component />`), ensure it's valid JSX
4. Check the browser console for errors
5. Ensure you're in an `.mdx` file, not `.md`

### MDX syntax errors (unexpected token, etc.)

**Common causes:**
- Using `<` or `>` in text without escaping (conflicts with JSX)
- HTML comments `<!-- -->` don't work in MDX; use `{/* JSX comments */}`
- Curly braces `{` `}` are interpreted as JSX expressions; escape with `\{`
- Angle brackets in generic types: use `{'<T>'}` or backtick code spans

## 6. Search not working {#search}

### Search returns no results

**Checklist:**
1. Did you run `npm run build`? Pagefind indexes at build time, not in dev.
2. Is `pagefind` enabled in config? (It is by default)
3. Check if pages have `pagefind: false` in frontmatter
4. Look for `_pagefind/` in your build output directory

### Search UI doesn't appear

- If `pagefind: false` is set in config, search UI is hidden
- If you've overridden the `Search` component, check your override

### Search works locally but not after deployment

**Common cause:** `base` path mismatch.
**Fix:**
1. Ensure `base` in config matches your deployment path
2. Rebuild after setting `base`
3. Verify `_pagefind/` exists at the correct path in your deployment

### Search results link to wrong URLs

**Cause:** `base` changed after initial build, or reverse proxy rewrites paths.
**Fix:** Full rebuild with correct `base` setting.

## 7. Deployment issues {#deployment}

### "Mixed content" warnings (HTTP/HTTPS)

**Fix:** Ensure `site` in config uses `https://`:
```js
site: 'https://example.com',  // not http://
```

### GitHub Pages: site works at root but not in subdirectory

**Fix:**
```js
site: 'https://username.github.io',
base: '/repo-name/',
```

And configure GitHub Actions to build with the correct base.

### Vercel/Netlify: build succeeds but site is blank

**Checklist:**
1. Build output directory is `dist/` (default for Astro)
2. No build errors (check deploy logs)
3. `src/content/docs/` has at least one file with valid frontmatter
4. Check for runtime errors in browser console

### Assets served with wrong MIME type

**Cause:** Hosting platform not configured to serve static files correctly.
**Fix:** Platform-specific; usually a config issue. Check the platform's documentation for static site settings.

### Trailing slashes cause 404s

**Fix:** Configure Astro's `trailingSlash` option:
```js
export default defineConfig({
  trailingSlash: 'always',  // or 'never' or 'ignore'
  // ...
});
```

Match this with your hosting platform's settings.

## 8. Performance issues {#performance}

### Large build output / slow builds

- Optimize images: use `src/assets/` with imports (Astro auto-optimizes)
- Minimize custom JS: Starlight is static by default; adding `client:load` to many components increases bundle
- Check for unused dependencies
- Use `npm run build -- --verbose` to find bottlenecks

### Slow page loads

- Check for large unoptimized images
- Minimize third-party scripts added via `head` config
- Use `client:idle` or `client:visible` instead of `client:load` for non-critical interactive components

## 9. Dark mode issues {#dark-mode}

### Dark mode toggle missing

- Check if `ThemeSelect` component has been overridden with an empty component
- Check for CSS that hides the theme selector

### Custom colors don't work in dark mode

- Override BOTH themes:
```css
:root[data-theme='light'] { --sl-color-accent: #4f46e5; }
:root[data-theme='dark'] { --sl-color-accent: #818cf8; }
```

### Expressive Code blocks don't follow dark mode

- In config: `expressiveCode.useStarlightDarkModeSwitch: true`
- Must have at least one dark AND one light theme in `expressiveCode.themes`

## 10. i18n issues {#i18n}

### Translated pages not showing up

- Ensure files are in the correct locale directory: `src/content/docs/es/page.md`
- Check that the locale is configured in `starlight.locales`
- Verify file has valid frontmatter with `title`

### Language switcher not appearing

- At least two locales must be configured
- Content must exist for at least two languages

### Fallback content shows unexpected language

- Check `defaultLocale` setting
- Starlight falls back to the default locale when a translation doesn't exist

## 11. Upgrade issues {#upgrade}

### After upgrade, build fails with type errors

**Fix:**
```bash
# Clear Astro's type cache
rm -rf .astro/
npm run build
```

### After upgrade, styles look different

- Check the changelog for breaking changes in CSS custom properties
- Clear browser cache
- Review your custom CSS for properties that may have been renamed

### Peer dependency conflicts after upgrade

**Fix:**
```bash
npx @astrojs/upgrade
```
This handles version coordination. If manual resolution is needed:
```bash
rm -rf node_modules package-lock.json
npm install
```

## 12. Nuclear options (when nothing else works) {#nuclear}

### Full reset

```bash
rm -rf node_modules package-lock.json .astro dist
npm install
npm run build
```

### Recreate content collection types

```bash
rm -rf .astro/
npm run dev
```

### Validate config in isolation

Create a minimal `astro.config.mjs` with just title and one page. If that works, add config back piece by piece to find the culprit.

### Check versions

```bash
npx astro --version
npm list @astrojs/starlight
```

Ensure Astro and Starlight versions are compatible. Use `npx @astrojs/upgrade` to sync.

### Get help

- GitHub Issues: https://github.com/withastro/starlight/issues
- Astro Discord: https://astro.build/chat (use `#starlight` channel)
- Stack Overflow: tag with `astro` and `starlight`

When reporting issues, include:
- Astro and Starlight versions
- Relevant config (`astro.config.mjs`)
- Error messages (full text)
- Steps to reproduce
- Operating system and Node.js version

FILE:references/styling-and-theming.md
# Styling & Theming

## Table of Contents
1. [Custom CSS basics](#custom-css)
2. [Starlight CSS custom properties](#css-props)
3. [Cascade layers](#cascade-layers)
4. [Tailwind CSS integration](#tailwind)
5. [Dark mode](#dark-mode)
6. [Expressive Code (syntax highlighting)](#expressive-code)
7. [Common styling pitfalls](#pitfalls)

---

## 1. Custom CSS basics {#custom-css}

Add custom styles by creating a CSS file and registering it in config:

```css
/* src/styles/custom.css */
:root {
  --sl-content-width: 50rem;
  --sl-text-5xl: 3.5rem;
}
```

```js
// astro.config.mjs
starlight({
  customCss: ['./src/styles/custom.css'],
})
```

You can list multiple CSS files. They're loaded in order, so later files override earlier ones.

You can also import npm packages:
```js
customCss: ['./src/styles/custom.css', '@fontsource/inter'],
```

## 2. Starlight CSS custom properties {#css-props}

Starlight exposes a comprehensive set of CSS custom properties for theming. Override them in your custom CSS.

### Color tokens

The two main color scales are **accent** (links, highlights) and **gray** (backgrounds, text, borders). Each has shades from 50 (lightest) to 950 (darkest).

```css
:root {
  /* Accent colors */
  --sl-color-accent-low: #1a1a4e;
  --sl-color-accent: #4f46e5;
  --sl-color-accent-high: #c7d2fe;

  /* You can also override individual Hue/Chroma values */
  --sl-hue-accent: 234;
  --sl-hue-gray: 240;
}
```

### Typography tokens

```css
:root {
  --sl-font: 'Inter', sans-serif;
  --sl-font-mono: 'JetBrains Mono', monospace;
  --sl-text-xs: 0.75rem;
  --sl-text-sm: 0.875rem;
  --sl-text-base: 1rem;
  --sl-text-lg: 1.125rem;
  --sl-text-xl: 1.25rem;
  --sl-text-2xl: 1.5rem;
  --sl-text-3xl: 1.875rem;
  --sl-text-4xl: 2.25rem;
  --sl-text-5xl: 3rem;
  --sl-text-6xl: 3.75rem;
}
```

### Layout tokens

```css
:root {
  --sl-content-width: 40rem;        /* Max width of content area */
  --sl-sidebar-width: 18.75rem;     /* Sidebar width */
  --sl-content-pad-x: 1rem;         /* Horizontal padding */
  --sl-nav-height: 3.5rem;          /* Top nav bar height */
}
```

Full list of properties: https://github.com/withastro/starlight/blob/main/packages/starlight/style/props.css

### Targeting dark mode

```css
:root[data-theme='dark'] {
  --sl-color-accent-low: #1e1b4b;
  --sl-color-accent: #818cf8;
}

:root[data-theme='light'] {
  --sl-color-accent-low: #eef2ff;
  --sl-color-accent: #4f46e5;
}
```

## 3. Cascade layers {#cascade-layers}

Starlight uses CSS cascade layers internally. Custom unlayered CSS automatically overrides Starlight defaults (because unlayered CSS beats layered CSS).

If you use layers in your own CSS, define the order explicitly:

```css
@layer my-reset, starlight, my-overrides;
```

- `my-reset` — applied before Starlight (Starlight can still override it)
- `starlight` — Starlight's own styles
- `my-overrides` — your overrides that always win

## 4. Tailwind CSS integration {#tailwind}

### New project with Tailwind

```bash
npm create astro@latest -- --template starlight/tailwind
```

### Adding Tailwind to existing Starlight project

1. Add Tailwind:
```bash
npx astro add tailwind
```

2. Install compatibility package:
```bash
npm install @astrojs/starlight-tailwind
```

3. Replace `src/styles/global.css` contents:
```css
@layer base, starlight, theme, components, utilities;
@import '@astrojs/starlight-tailwind';
@import 'tailwindcss/theme.css' layer(theme);
@import 'tailwindcss/utilities.css' layer(utilities);
```

4. Update `astro.config.mjs`:
```js
import tailwindcss from '@tailwindcss/vite';

export default defineConfig({
  integrations: [
    starlight({
      title: 'My Docs',
      customCss: ['./src/styles/global.css'],  // Must be first!
    }),
  ],
  vite: { plugins: [tailwindcss()] },
});
```

### Customizing Tailwind theme for Starlight

In `src/styles/global.css`, use `@theme` to override Starlight's colors:

```css
@layer base, starlight, theme, components, utilities;
@import '@astrojs/starlight-tailwind';
@import 'tailwindcss/theme.css' layer(theme);
@import 'tailwindcss/utilities.css' layer(utilities);

@theme {
  --font-sans: 'Inter';
  --font-mono: 'JetBrains Mono';

  /* Accent colors — indigo is closest to Starlight defaults */
  --color-accent-50: var(--color-indigo-50);
  --color-accent-100: var(--color-indigo-100);
  /* ... through 950 */

  /* Gray scale — zinc is closest to Starlight defaults */
  --color-gray-50: var(--color-zinc-50);
  --color-gray-100: var(--color-zinc-100);
  /* ... through 950 */
}
```

### Common Tailwind + Starlight conflicts

**Problem: Tailwind resets fight Starlight typography**
- The `@astrojs/starlight-tailwind` package handles this. Without it, Tailwind's Preflight reset removes Starlight's content styling.

**Problem: Tailwind classes don't work in .md/.mdx content**
- Tailwind utility classes work in Astro components. In Markdown content, Starlight applies its own content styles. Use `customCss` or custom components instead.

**Problem: Dark mode classes conflict**
- The Starlight Tailwind compatibility CSS configures `dark:` variants to sync with Starlight's theme switcher. Don't try to configure `darkMode` separately in Tailwind.

## 5. Dark mode {#dark-mode}

Starlight supports light/dark mode out of the box with a toggle in the header.

### Customizing dark mode colors

```css
:root[data-theme='dark'] {
  --sl-color-bg: #0f172a;
  --sl-color-bg-nav: #1e293b;
}

:root[data-theme='light'] {
  --sl-color-bg: #ffffff;
  --sl-color-bg-nav: #f8fafc;
}
```

### Disabling dark mode

There's no single config flag. You can hide the toggle by overriding the `ThemeSelect` component:

```js
// astro.config.mjs
starlight({
  components: {
    ThemeSelect: './src/components/EmptyThemeSelect.astro',
  },
})
```

```astro
<!-- src/components/EmptyThemeSelect.astro -->
<!-- Empty component — no theme toggle rendered -->
```

Then force a theme in CSS:
```css
:root { color-scheme: light; }
```

## 6. Expressive Code (syntax highlighting) {#expressive-code}

Starlight uses Expressive Code for code blocks. Enabled by default.

### Configuration

```js
starlight({
  expressiveCode: {
    themes: ['dracula', 'github-light'],
    styleOverrides: {
      borderRadius: '0.5rem',
      codeFontFamily: 'JetBrains Mono, monospace',
    },
    // Sync with Starlight's dark mode toggle
    useStarlightDarkModeSwitch: true,
    // Use Starlight's UI colors for code block chrome
    useStarlightUiThemeColors: true,
  },
})
```

### Disabling Expressive Code

```js
starlight({
  expressiveCode: false,
})
```

### Code block features in Markdown

Title: ````js title="config.js"````
Line highlighting: ````js {2-3}````
Insertions: ````js ins={3}````
Deletions: ````js del={1}````
Line numbers: Enabled via theme or config
File name tab: ````js title="src/index.js"````

## 7. Common styling pitfalls {#pitfalls}

**Problem: Custom CSS doesn't take effect**
- Check that the file path in `customCss` is correct (relative to project root).
- If using layers, ensure your layer order puts your overrides after `starlight`.
- Use browser devtools to check specificity conflicts.

**Problem: Fonts not loading**
- Install the font package: `npm install @fontsource/inter`
- Add to `customCss`: `customCss: ['@fontsource/inter', './src/styles/custom.css']`
- Set the font in CSS: `--sl-font: 'Inter', sans-serif;`

**Problem: Content area too narrow/wide**
- Override `--sl-content-width` in your custom CSS.

**Problem: Tailwind styles overriding Starlight in unexpected ways**
- Make sure `@astrojs/starlight-tailwind` is installed and imported.
- The global.css file must be listed first in `customCss`.
- Don't configure `darkMode` in Tailwind separately.

FILE:references/project-setup.md
# Project Setup & Structure

## Table of Contents
1. [Scaffolding a new project](#scaffolding)
2. [Adding Starlight to an existing Astro project](#manual-setup)
3. [Project structure anatomy](#project-structure)
4. [astro.config.mjs deep dive](#config)
5. [Content collections setup](#content-collections)
6. [Updating Starlight](#updating)

---

## 1. Scaffolding a new project {#scaffolding}

The fastest way to start:

```bash
# npm
npm create astro@latest -- --template starlight

# pnpm
pnpm create astro --template starlight

# yarn
yarn create astro --template starlight
```

For Starlight + Tailwind pre-configured:
```bash
npm create astro@latest -- --template starlight/tailwind
```

After scaffolding, start the dev server:
```bash
cd my-docs
npm run dev
```

## 2. Adding Starlight to an existing Astro project {#manual-setup}

Install the integration:
```bash
npx astro add starlight
```

This does three things:
- Installs `@astrojs/starlight`
- Adds the integration to `astro.config.mjs`
- Creates `src/content/docs/` with a starter page
- Creates `src/content.config.ts` with the docs collection

If the CLI doesn't do everything, manually:

1. Install: `npm install @astrojs/starlight`
2. Add to config (see section 4 below)
3. Create `src/content/docs/index.md` with a title in frontmatter
4. Create `src/content.config.ts` (see section 5 below)

## 3. Project structure anatomy {#project-structure}

```
my-docs/
├── astro.config.mjs          # Main config — Starlight lives here
├── src/
│   ├── content/
│   │   └── docs/              # ALL doc pages go here (file-based routing)
│   │       ├── index.mdx      # → yoursite.com/
│   │       ├── guides/
│   │       │   ├── intro.md   # → yoursite.com/guides/intro/
│   │       │   └── setup.md   # → yoursite.com/guides/setup/
│   │       └── reference/
│   │           └── api.md     # → yoursite.com/reference/api/
│   ├── content.config.ts      # Content collection schema definition
│   ├── assets/                # Optimized images (imported in content)
│   ├── styles/                # Custom CSS files
│   ├── components/            # Custom Astro/React/etc. components
│   └── pages/                 # Custom non-Starlight pages (optional)
├── public/                    # Static assets served as-is (favicon, robots.txt)
└── package.json
```

Key rules:
- **`src/content/docs/`** is where ALL documentation pages live. Subdirectories become URL path segments.
- **`src/assets/`** is for images you want Astro to optimize. Reference them in content with `~/assets/image.png`.
- **`public/`** is for files served verbatim (favicon, CNAME, etc.). Reference them with absolute paths like `/favicon.svg`.
- **`src/pages/`** is optional — use it for custom pages that don't follow Starlight's layout.

## 4. astro.config.mjs deep dive {#config}

Full annotated example:

```js
import { defineConfig } from 'astro/config';
import starlight from '@astrojs/starlight';

export default defineConfig({
  // IMPORTANT for subpath deployments:
  site: 'https://example.com',
  // base: '/docs/',  // Uncomment ONLY if hosting at a subpath

  integrations: [
    starlight({
      // REQUIRED — your site's title
      title: 'My Docs',

      // Optional — meta description
      description: 'Documentation for my awesome project',

      // Sidebar configuration (see sidebar-and-content.md for details)
      sidebar: [
        {
          label: 'Guides',
          items: [
            { slug: 'guides/intro' },
            { slug: 'guides/setup' },
          ],
        },
        {
          label: 'Reference',
          autogenerate: { directory: 'reference' },
        },
      ],

      // Social links in the header
      social: [
        { icon: 'github', label: 'GitHub', href: 'https://github.com/...' },
      ],

      // Edit page links
      editLink: {
        baseUrl: 'https://github.com/your-org/your-repo/edit/main/',
      },

      // Custom CSS (see styling-and-theming.md)
      customCss: [
        './src/styles/custom.css',
      ],

      // Logo
      logo: {
        src: './src/assets/logo.svg',
        // Or separate dark/light: { light: '...', dark: '...' }
      },

      // Table of contents config
      tableOfContents: { minHeadingLevel: 2, maxHeadingLevel: 3 },

      // Show last updated date (requires git history)
      lastUpdated: true,

      // Pagination (prev/next links)
      pagination: true,

      // Favicon
      favicon: '/favicon.svg',

      // Head tags (analytics, etc.)
      head: [
        {
          tag: 'script',
          attrs: { src: 'https://analytics.example.com/script.js', defer: true },
        },
      ],

      // Search — true by default (Pagefind)
      // Set false to disable, or pass PagefindOptions object
      pagefind: true,

      // Expressive Code (syntax highlighting) config
      expressiveCode: {
        // themes: ['dracula', 'github-light'],
        styleOverrides: { borderRadius: '0.5rem' },
      },

      // Component overrides
      // components: {
      //   SocialLinks: './src/components/MySocialLinks.astro',
      // },

      // Plugins
      // plugins: [myPlugin()],
    }),
  ],
});
```

### Key config properties

| Property | Type | Default | Purpose |
|---|---|---|---|
| `title` | `string` | (required) | Site title in nav bar and metadata |
| `description` | `string` | — | Default meta description |
| `sidebar` | `SidebarItem[]` | auto from filesystem | Navigation structure |
| `customCss` | `string[]` | `[]` | CSS file paths to include |
| `logo` | `LogoConfig` | — | Site logo |
| `social` | `SocialLink[]` | — | Header social icons |
| `editLink` | `{ baseUrl: string }` | — | "Edit this page" links |
| `tableOfContents` | `false \| { min, max }` | `{ min: 2, max: 3 }` | Right-side ToC |
| `lastUpdated` | `boolean` | `false` | Show git-based last updated |
| `pagefind` | `boolean \| PagefindOptions` | `true` | Search configuration |
| `expressiveCode` | `boolean \| object` | `true` | Code block highlighting |
| `head` | `HeadConfig[]` | `[]` | Custom `<head>` tags |
| `locales` | `object` | — | i18n setup |
| `prerender` | `boolean` | `true` | Static vs SSR rendering |
| `components` | `Record<string, string>` | — | Override built-in components |

## 5. Content collections setup {#content-collections}

Starlight requires a content collection for docs. The config file:

```ts
// src/content.config.ts
import { defineCollection } from 'astro:content';
import { docsLoader, i18nLoader } from '@astrojs/starlight/loaders';
import { docsSchema, i18nSchema } from '@astrojs/starlight/schema';

export const collections = {
  docs: defineCollection({ loader: docsLoader(), schema: docsSchema() }),
  // Optional — for multilingual UI translations:
  i18n: defineCollection({ loader: i18nLoader(), schema: i18nSchema() }),
};
```

You can extend the schema with custom frontmatter fields:
```ts
import { docsSchema } from '@astrojs/starlight/schema';
import { z } from 'astro:content';

schema: docsSchema({
  extend: z.object({
    category: z.string().optional(),
    difficulty: z.enum(['beginner', 'intermediate', 'advanced']).optional(),
  }),
}),
```

## 6. Updating Starlight {#updating}

Use the Astro upgrade tool:
```bash
npx @astrojs/upgrade
```

This updates Starlight and all Astro packages together, handling peer dependency coordination.

Check the changelog for breaking changes: https://github.com/withastro/starlight/blob/main/packages/starlight/CHANGELOG.md

FILE:references/sidebar-and-content.md
# Sidebar Navigation & Content Authoring

## Table of Contents
1. [Folder structure design](#folder-structure)
2. [Sidebar configuration patterns](#sidebar-config)
3. [Frontmatter reference](#frontmatter)
4. [Content authoring (Markdown vs MDX)](#authoring)
5. [Page ordering strategies](#ordering)
6. [Common sidebar pitfalls](#pitfalls)

---

## 1. Folder structure design {#folder-structure}

Your folder structure inside `src/content/docs/` directly determines your URL structure and default sidebar. Design it before writing content.

### Recommended pattern for most projects

```
src/content/docs/
├── index.mdx                 # Landing page (/)
├── getting-started.md        # /getting-started/
├── guides/
│   ├── installation.md       # /guides/installation/
│   ├── configuration.md      # /guides/configuration/
│   └── deployment.md         # /guides/deployment/
├── reference/
│   ├── api.md                # /reference/api/
│   ├── cli.md                # /reference/cli/
│   └── config.md             # /reference/config/
└── tutorials/
    ├── quickstart.md          # /tutorials/quickstart/
    └── advanced.md            # /tutorials/advanced/
```

### Rules of thumb
- Keep nesting to 2 levels max. Deeper nesting makes navigation confusing.
- Use descriptive folder names — they become sidebar group labels by default.
- Put overview/index pages at the folder root, not in a subfolder.
- Name files with lowercase kebab-case: `getting-started.md`, not `GettingStarted.md`.

## 2. Sidebar configuration patterns {#sidebar-config}

Configure in `astro.config.mjs` under `starlight({ sidebar: [...] })`.

### Pattern A: Fully manual (recommended for most projects)

You control every entry and its order:

```js
sidebar: [
  { slug: 'getting-started' },
  {
    label: 'Guides',
    items: [
      { slug: 'guides/installation' },
      { slug: 'guides/configuration' },
      { slug: 'guides/deployment' },
    ],
  },
  {
    label: 'Reference',
    items: [
      { slug: 'reference/api' },
      { slug: 'reference/cli' },
    ],
  },
],
```

When using `slug`, the page's `title` frontmatter becomes the sidebar label automatically. Override with `label`:
```js
{ slug: 'guides/installation', label: 'Install' },
```

Shorthand — just use a string for internal links:
```js
items: ['guides/installation', 'guides/configuration'],
```

### Pattern B: Autogenerated (good for reference sections)

Automatically includes all pages in a directory:

```js
sidebar: [
  {
    label: 'Reference',
    autogenerate: { directory: 'reference' },
  },
],
```

Pages are sorted alphabetically by filename by default. Control order with frontmatter `sidebar.order` (see below).

Subdirectories become nested groups automatically.

### Pattern C: Mixed (most common in practice)

Manual for curated sections, auto for reference:

```js
sidebar: [
  {
    label: 'Getting Started',
    items: [
      { slug: 'getting-started' },
      { slug: 'guides/installation' },
    ],
  },
  {
    label: 'API Reference',
    autogenerate: { directory: 'reference' },
  },
],
```

### Collapsible groups

Groups expand by default. Make them collapsed:

```js
{
  label: 'Advanced',
  collapsed: true,
  items: [...],
},
```

For autogenerated groups with collapsed subgroups:
```js
{
  label: 'Reference',
  autogenerate: { directory: 'reference', collapsed: true },
},
```

### External links in sidebar

```js
{ label: 'GitHub', link: 'https://github.com/your/repo' },
{ label: 'API Playground', link: '/playground/', attrs: { target: '_blank' } },
```

### Sidebar badges

Add visual badges to sidebar items:

```js
{ slug: 'guides/new-feature', badge: 'New' },
{ slug: 'reference/deprecated', badge: { text: 'Deprecated', variant: 'caution' } },
```

Badge variants: `note`, `tip`, `caution`, `danger`, `success`, `default`.

### Sidebar translations (i18n)

```js
{
  label: 'Guides',
  translations: { 'es': 'Guías', 'fr': 'Guides' },
  items: [
    {
      label: 'Getting Started',
      translations: { 'es': 'Primeros pasos' },
      slug: 'guides/getting-started',
    },
  ],
},
```

## 3. Frontmatter reference {#frontmatter}

Every page in `src/content/docs/` needs YAML frontmatter. Required field: `title`.

### Common frontmatter fields

```yaml
---
title: Page Title                  # REQUIRED
description: SEO meta description  # Optional but recommended
sidebar:
  label: Short Name                # Override sidebar display name
  order: 1                         # Lower = higher in autogenerated groups
  hidden: false                    # true to hide from autogenerated sidebar
  badge:
    text: New
    variant: tip
template: doc                      # 'doc' (default) or 'splash' (landing page)
tableOfContents:
  minHeadingLevel: 2
  maxHeadingLevel: 3
lastUpdated: 2024-01-15            # Override git date, or false to hide
editUrl: false                     # false to hide edit link, or custom URL
pagefind: true                     # false to exclude from search
draft: false                       # true to exclude from production builds
prev: true                         # false to hide, string to override text
next: true                         # same as prev
---
```

### Hero section (for landing pages)

```yaml
---
title: My Project
template: splash
hero:
  title: 'Welcome to My Project'
  tagline: The best docs you'll ever read.
  image:
    file: ~/assets/hero.png
    alt: Hero image description
  actions:
    - text: Get Started
      link: /getting-started/
      icon: right-arrow
    - text: View on GitHub
      link: https://github.com/...
      icon: external
      variant: minimal
---
```

### Banner

```yaml
---
title: My Page
banner:
  content: |
    We just released v2.0!
    <a href="/changelog/">See what's new</a>
---
```

## 4. Content authoring {#authoring}

### Markdown (.md) vs MDX (.mdx)

| Feature | `.md` | `.mdx` |
|---|---|---|
| Frontmatter | Yes | Yes |
| Standard Markdown | Yes | Yes |
| Component imports | No | Yes |
| JSX expressions | No | Yes |
| Aside syntax (`:::`) | Yes | Yes |

**Rule: Use `.md` by default. Switch to `.mdx` only when you need component imports.**

### Aside syntax (works in both .md and .mdx)

```markdown
:::note
This is a note.
:::

:::tip
Helpful tip here.
:::

:::caution
Be careful!
:::

:::danger
This is dangerous.
:::

:::note[Custom Title]
A note with a custom title.
:::
```

### Code blocks

Starlight uses Expressive Code for rich code blocks:

````markdown
```js title="example.js" ins={3} del={1}
const old = 'removed';
const kept = 'unchanged';
const added = 'new line';
```
````

Features: titles, line highlighting, insertions/deletions, line numbers, file names, and more.

## 5. Page ordering strategies {#ordering}

### In autogenerated groups

Use `sidebar.order` in frontmatter:

```yaml
# src/content/docs/guides/intro.md
---
title: Introduction
sidebar:
  order: 1
---
```

```yaml
# src/content/docs/guides/setup.md
---
title: Setup
sidebar:
  order: 2
---
```

Lower numbers appear first. Pages without `order` are sorted alphabetically after ordered pages.

### In manual sidebar

Order matches the array order in `astro.config.mjs`. This is the most predictable approach.

### Naming convention trick

For autogenerated sidebars without frontmatter ordering, prefix filenames:
```
01-introduction.md
02-installation.md
03-configuration.md
```
This works because alphabetical sort respects the prefix. However, it makes URLs ugly (`/guides/01-introduction/`). Use `slug` frontmatter to override:
```yaml
---
title: Introduction
slug: guides/introduction
---
```

## 6. Common sidebar pitfalls {#pitfalls}

**Problem: Page exists but doesn't show in sidebar**
- If using manual sidebar: you forgot to add it to the `items` array.
- If using autogenerate: the file isn't in the directory specified by `autogenerate.directory`.
- Check for `sidebar.hidden: true` in the page's frontmatter.

**Problem: Sidebar link leads to 404**
- The `slug` in config doesn't match the actual file path relative to `src/content/docs/`.
- File extension mismatch: config references `guides/intro` but file is `guides/intro.mdx` — this should still work, but check for typos.

**Problem: Autogenerated order is wrong**
- Without `sidebar.order`, pages sort alphabetically by filename.
- Add `sidebar.order` to every page's frontmatter in that group.

**Problem: Nested autogenerated groups show unexpected headings**
- Subdirectories become groups with the directory name as label.
- To customize, add an `index.md` in the subdirectory with `sidebar.label` set in frontmatter, or switch to manual sidebar for that section.

**Problem: Mixing autogenerate with manual items in the same group**
- You can't. A group is either `items: [...]` (manual) or `autogenerate: { directory: '...' }`, not both.
- Workaround: use manual items and list everything explicitly, or restructure your directories.

---
name: directus_io
description: Use this skill for anything involving Directus — the open-source headless CMS and backend-as-a-service. Triggers include setting up Directus (Docker, Cloud, self-hosted), using the Directus JavaScript/TypeScript SDK, integrating Directus with frontend frameworks (especially Astro, but also Next.js, Nuxt, SvelteKit), building Directus Flows and automations, generating content with AI via Directus Automate, building custom extensions (hooks, endpoints, interfaces, layouts, modules, operations), data modeling and collections, access control and permissions, file management, real-time subscriptions, REST and GraphQL API usage, troubleshooting Directus errors, and any mention of 'Directus', 'directus.io', '@directus/sdk', 'Directus Cloud', 'Directus Flows', 'Directus Automate', or 'Data Studio'. Also trigger when users mention headless CMS integration with Astro/TypeScript and Directus is a likely fit, or when they ask about CMS-powered content pipelines, dynamic page generation from a CMS, or automating content workflows with a headless backend.
---

# Directus Skill

Directus is an open-source headless CMS and backend-as-a-service that wraps any SQL database with instant REST and GraphQL APIs, plus a no-code Data Studio for content management. It's built with Node.js and Vue.js, supports PostgreSQL, MySQL, SQLite, MariaDB, MS-SQL, OracleDB, and CockroachDB, and is fully extensible.

## How to Use This Skill

This skill is organized into focused reference files. Read the relevant reference before answering:

| Topic | Reference File | When to Read |
|---|---|---|
| **SDK & API Basics** | `references/sdk-and-api.md` | Any question about the JS/TS SDK, REST API, GraphQL, authentication, CRUD operations, filtering, or real-time subscriptions |
| **Astro Integration** | `references/astro-integration.md` | Integrating Directus with Astro — fetching data, dynamic routes, SSG/SSR, live preview, authentication, visual editing |
| **TypeScript Patterns** | `references/typescript-patterns.md` | Schema typing, type generation, advanced generics, type-safe SDK usage |
| **Flows & Automation** | `references/flows-and-automation.md` | Directus Flows, triggers, operations, scheduled tasks, AI content generation, webhook integrations |
| **Extensions** | `references/extensions.md` | Custom endpoints, hooks, interfaces, layouts, displays, modules, operations, panels, themes |
| **Data Modeling & Admin** | `references/data-modeling.md` | Collections, fields, relations (M2O, O2M, M2M, M2A), singletons, permissions, roles, file management, environment config |
| **Troubleshooting** | `references/troubleshooting.md` | Common errors, debugging, CORS issues, Docker problems, migration headaches, performance tips |

**Always read the relevant reference file before generating code or instructions.** For complex questions spanning multiple topics, read multiple references.

## Quick-Reference Essentials

These are the most common patterns you'll need. For anything beyond these basics, consult the reference files.

### Install the SDK
```bash
npm install @directus/sdk
```

### Create a Client (TypeScript)
```typescript
import { createDirectus, rest, staticToken } from '@directus/sdk';

// Define your schema for type safety
interface MySchema {
  posts: Post[];
  categories: Category[];
  global: GlobalSettings; // singleton (not an array)
}

interface Post {
  id: number;
  title: string;
  content: string;
  status: string;
  category: number | Category;
}

interface Category {
  id: number;
  name: string;
}

interface GlobalSettings {
  site_title: string;
  description: string;
}

const client = createDirectus<MySchema>('https://your-directus-url.com')
  .with(staticToken('your-token'))
  .with(rest());
```

### Read Items
```typescript
import { readItems, readItem, readSingleton } from '@directus/sdk';

// Get all posts
const posts = await client.request(readItems('posts'));

// Get one post with relational data
const post = await client.request(readItem('posts', 1, {
  fields: ['*', { category: ['name'] }],
}));

// Get singleton
const global = await client.request(readSingleton('global'));
```

### Astro Quick Setup
```typescript
// src/lib/directus.ts
import { createDirectus, rest } from '@directus/sdk';
const directus = createDirectus(import.meta.env.DIRECTUS_URL).with(rest());
export default directus;
```

### Docker Quick Start
```yaml
# docker-compose.yml
services:
  directus:
    image: directus/directus:latest
    ports:
      - 8055:8055
    volumes:
      - ./database:/directus/database
      - ./uploads:/directus/uploads
      - ./extensions:/directus/extensions
    environment:
      SECRET: 'your-random-secret'
      ADMIN_EMAIL: '[email protected]'
      ADMIN_PASSWORD: 'your-password'
      DB_CLIENT: 'sqlite3'
      DB_FILENAME: '/directus/database/data.db'
      WEBSOCKETS_ENABLED: 'true'
```

## Key Concepts to Remember

- **Composable Client**: The SDK client starts empty — you add features with `.with(rest())`, `.with(authentication())`, `.with(realtime())`, `.with(staticToken())`.
- **Schema Typing**: Always define a TypeScript schema interface for type-safe SDK usage. Regular collections are arrays, singletons are singular types.
- **Access Policies**: New collections are private by default. You must configure public read access or use authentication tokens.
- **Directus Assets**: Images/files are served at `{DIRECTUS_URL}/assets/{file-id}` with optional transformation query params like `?width=500&format=webp`.
- **Flows vs Extensions**: Flows are no-code automations configured in the Data Studio. Extensions are code-based additions (hooks, endpoints, etc.) that extend Directus itself.
- **Environment Variables**: Directus is heavily configured via env vars — `DB_CLIENT`, `SECRET`, `CORS_ENABLED`, `AUTH_PROVIDERS`, etc.

FILE:references/data-modeling.md
# Directus Data Modeling & Administration

## Table of Contents
1. [Collections & Fields](#collections--fields)
2. [Relations](#relations)
3. [Permissions & Access Control](#permissions--access-control)
4. [File Management](#file-management)
5. [Environment Configuration](#environment-configuration)
6. [Installation & Deployment](#installation--deployment)
7. [Schema Migration](#schema-migration)

---

## Collections & Fields

### Creating Collections
Collections map to database tables. Create them in the Data Studio under **Settings → Data Model** or via API.

**Common field types:**
| Type | Description | Use Case |
|---|---|---|
| `string` | Short text (varchar) | Titles, names, slugs |
| `text` | Long text | Descriptions, excerpts |
| `integer` | Whole number | IDs, counts |
| `float` / `decimal` | Decimal number | Prices, percentages |
| `boolean` | True/false | Toggles, flags |
| `datetime` | Date and time | Publish dates, timestamps |
| `date` | Date only | Birthdays, event dates |
| `time` | Time only | Schedules |
| `json` | JSON object | Structured data, settings |
| `uuid` | UUID string | Primary keys |
| `hash` | Hashed string | Passwords |
| `csv` | Comma-separated values | Simple lists |

### Interface Types
Interfaces control how fields appear in the editor:
- `input` — standard text input
- `input-rich-text-html` — WYSIWYG editor (stores HTML)
- `input-rich-text-md` — Markdown editor
- `input-code` — Code editor with syntax highlighting
- `select-dropdown` — Dropdown select
- `select-multiple-checkbox` — Checkboxes
- `boolean` — Toggle switch
- `datetime` — Date/time picker
- `file-image` — Image upload
- `file` — File upload
- `tags` — Tag input
- `slider` — Range slider
- `map` — Map coordinate picker
- `translations` — Multilingual content interface

### System Fields
Directus can automatically manage these optional fields:
- `status` — workflow status (draft, published, archived)
- `sort` — manual sort order
- `date_created` — auto-set on creation
- `date_updated` — auto-set on update
- `user_created` — auto-set to creating user
- `user_updated` — auto-set to updating user

### Singleton Collections
A singleton has exactly one item — used for global settings, homepage content, etc. Enable in collection settings: **Singleton → Treat as a single object**.

---

## Relations

### Many-to-One (M2O)
The most common relation. A foreign key field on the "many" side points to the "one" side.

**Example:** Each `post` belongs to one `category`.
- Field `category` on `posts` collection → references `categories.id`
- In the editor, shows as a dropdown or drawer selector

### One-to-Many (O2M)
The reverse of M2O. Displayed on the "one" side as a list of related items.

**Example:** A `category` has many `posts`.
- Configure as an O2M alias field on `categories` → references `posts.category`
- No new database column — it reads the existing M2O relation in reverse

### Many-to-Many (M2M)
Requires a junction collection with two foreign keys.

**Example:** `posts` and `tags` linked via `posts_tags`.
- Junction collection `posts_tags`:
  - `posts_id` → M2O to `posts`
  - `tags_id` → M2O to `tags`
- Directus auto-creates the junction if you set up M2M in the Data Model UI

### Many-to-Any (M2A)
A polymorphic relation where items can reference rows from multiple collections. Used for page builders and block-based content.

**Example:** A `page` has `blocks` that can be heroes, text sections, galleries, etc.
- Junction collection `pages_blocks`:
  - `pages_id` → M2O to `pages`
  - `collection` → string (which collection the item comes from)
  - `item` → string (the ID of the related item)
  - `sort` → integer (ordering)

### Translations
A special M2M pattern for multilingual content. Directus provides a dedicated translations interface that creates:
- A `languages` collection
- A junction collection (e.g., `posts_translations`) with fields for each translated field plus a `languages_code` foreign key

---

## Permissions & Access Control

### Concepts
- **Roles** — groups of users (e.g., Admin, Editor, Viewer)
- **Policies** — sets of permissions attached to a role
- **Permissions** — CRUD access per collection, optionally with field-level and item-level rules
- **Access Policies** — found in Settings → Access Policies

### Default State
New collections are completely private. No role (including Public) has any access until you grant it.

### Public Role
The Public role controls what unauthenticated users can access. For a public-facing site:
1. Go to **Settings → Access Policies → Public**
2. For each collection you want publicly readable, enable **Read** access
3. Optionally restrict which fields are visible

### Field-Level Permissions
Control which fields a role can read or write:
```
Role: Editor
Collection: posts
Create: ✅ (title, content, excerpt, status=draft only)
Read: ✅ (all fields)
Update: ✅ (title, content, excerpt — but NOT status)
Delete: ❌
```

### Item-Level Permissions (Custom Rules)
Restrict access to specific items based on field values:
```
Role: Author
Collection: posts
Read: Custom → user_created equals $CURRENT_USER
Update: Custom → user_created equals $CURRENT_USER
```

### Available Variables in Permission Rules
- `$CURRENT_USER` — the authenticated user's ID
- `$CURRENT_ROLE` — the authenticated user's role UUID
- `$NOW` — current timestamp

### Admin Role
Users with the admin role bypass all permission checks. There is always at least one admin user.

---

## File Management

### Storage Adapters
Directus supports multiple storage backends:

```env
# Local filesystem (default)
STORAGE_LOCATIONS="local"
STORAGE_LOCAL_ROOT="./uploads"

# Amazon S3
STORAGE_LOCATIONS="s3"
STORAGE_S3_KEY="your-access-key"
STORAGE_S3_SECRET="your-secret-key"
STORAGE_S3_BUCKET="your-bucket"
STORAGE_S3_REGION="us-east-1"

# Google Cloud Storage
STORAGE_LOCATIONS="gcs"
STORAGE_GCS_KEY_FILENAME="./service-account.json"
STORAGE_GCS_BUCKET="your-bucket"

# Cloudflare R2
STORAGE_LOCATIONS="s3"
STORAGE_S3_KEY="your-r2-key"
STORAGE_S3_SECRET="your-r2-secret"
STORAGE_S3_BUCKET="your-bucket"
STORAGE_S3_ENDPOINT="https://{account-id}.r2.cloudflarestorage.com"
STORAGE_S3_REGION="auto"
```

### Image Transformations
Directus automatically serves transformed images via query parameters:
```
GET /assets/{file-id}?width=300&height=200&fit=cover&format=webp&quality=80
```

Transformation presets can be configured to limit allowed transformations:
```env
ASSETS_TRANSFORM="presets"
ASSETS_TRANSFORM_IMAGE_MAX_DIMENSION="4096"
```

---

## Environment Configuration

### Essential Variables

```env
# Security (REQUIRED)
SECRET="a-long-random-string"

# Database
DB_CLIENT="pg"                    # pg, mysql, sqlite3, mssql, oracledb, cockroachdb
DB_HOST="localhost"
DB_PORT="5432"
DB_DATABASE="directus"
DB_USER="directus"
DB_PASSWORD="password"

# For SQLite
DB_CLIENT="sqlite3"
DB_FILENAME="./database.db"

# Admin User (first run only)
ADMIN_EMAIL="[email protected]"
ADMIN_PASSWORD="secure-password"
ADMIN_TOKEN="optional-static-token"

# Server
HOST="0.0.0.0"
PORT="8055"
PUBLIC_URL="https://your-directus-domain.com"

# CORS
CORS_ENABLED="true"
CORS_ORIGIN="https://your-frontend.com"
CORS_METHODS="GET,POST,PATCH,DELETE"

# WebSockets
WEBSOCKETS_ENABLED="true"

# Email
EMAIL_TRANSPORT="smtp"
EMAIL_SMTP_HOST="smtp.example.com"
EMAIL_SMTP_PORT="587"
EMAIL_SMTP_USER="user"
EMAIL_SMTP_PASSWORD="password"
EMAIL_FROM="[email protected]"

# Cache
CACHE_ENABLED="true"
CACHE_STORE="redis"
CACHE_REDIS="redis://localhost:6379"
CACHE_TTL="5m"

# Rate Limiting
RATE_LIMITER_ENABLED="true"
RATE_LIMITER_STORE="redis"
RATE_LIMITER_POINTS="50"
RATE_LIMITER_DURATION="1"

# Extensions
EXTENSIONS_PATH="./extensions"
EXTENSIONS_AUTO_RELOAD="false"

# Logging
LOG_LEVEL="info"                  # fatal, error, warn, info, debug, trace
LOG_STYLE="pretty"                # pretty, raw
```

---

## Installation & Deployment

### Docker (Recommended)
```yaml
# docker-compose.yml
services:
  database:
    image: postgres:16
    volumes:
      - pgdata:/var/lib/postgresql/data
    environment:
      POSTGRES_USER: directus
      POSTGRES_PASSWORD: directus
      POSTGRES_DB: directus

  cache:
    image: redis:7

  directus:
    image: directus/directus:latest
    ports:
      - 8055:8055
    volumes:
      - ./uploads:/directus/uploads
      - ./extensions:/directus/extensions
    depends_on:
      - database
      - cache
    environment:
      SECRET: 'change-this-to-a-random-value'
      ADMIN_EMAIL: '[email protected]'
      ADMIN_PASSWORD: 'your-password'

      DB_CLIENT: 'pg'
      DB_HOST: 'database'
      DB_PORT: '5432'
      DB_DATABASE: 'directus'
      DB_USER: 'directus'
      DB_PASSWORD: 'directus'

      CACHE_ENABLED: 'true'
      CACHE_STORE: 'redis'
      CACHE_REDIS: 'redis://cache:6379'

      WEBSOCKETS_ENABLED: 'true'

volumes:
  pgdata:
```

### npm (Self-Hosted)
```bash
npx create-directus-project my-project
cd my-project
npx directus start
```

### Directus Cloud
Sign up at directus.io/cloud — managed hosting from $15/month with auto-scaling, CDN, and backups.

### One-Click Deploy
- **Railway**: One-click deploy with PostgreSQL, Redis, and S3 storage
- **DigitalOcean**: App Platform deployment
- **Render**: Web service deployment

---

## Schema Migration

Directus supports schema snapshots and diffs for migrating data models between environments.

### Export Schema Snapshot
```bash
# CLI
npx directus schema snapshot ./snapshot.yaml

# API
GET /schema/snapshot
```

### Apply Schema to Another Instance
```bash
# Generate a diff
npx directus schema diff ./snapshot.yaml

# Apply the diff
npx directus schema apply ./snapshot.yaml
```

### API-Based Migration
```typescript
// Export from source
const snapshot = await fetch('https://source.directus.app/schema/snapshot', {
  headers: { Authorization: 'Bearer admin-token' },
}).then(r => r.json());

// Diff against target
const diff = await fetch('https://target.directus.app/schema/diff', {
  method: 'POST',
  headers: {
    Authorization: 'Bearer admin-token',
    'Content-Type': 'application/json',
  },
  body: JSON.stringify(snapshot),
}).then(r => r.json());

// Apply diff to target
await fetch('https://target.directus.app/schema/apply', {
  method: 'POST',
  headers: {
    Authorization: 'Bearer admin-token',
    'Content-Type': 'application/json',
  },
  body: JSON.stringify(diff),
});
```

### Best Practices
- Always snapshot before upgrading Directus versions
- Use schema migration in CI/CD pipelines to keep staging and production in sync
- Schema snapshots include collections, fields, and relations — not content data
- For content migration, use the API to export/import items

FILE:references/troubleshooting.md
# Directus Troubleshooting

## Table of Contents
1. [Installation & Startup Issues](#installation--startup-issues)
2. [Authentication & Token Errors](#authentication--token-errors)
3. [CORS & Network Errors](#cors--network-errors)
4. [SDK Errors](#sdk-errors)
5. [Docker Issues](#docker-issues)
6. [Database Issues](#database-issues)
7. [File Upload & Asset Issues](#file-upload--asset-issues)
8. [Extension Issues](#extension-issues)
9. [Performance Issues](#performance-issues)
10. [Migration & Upgrade Issues](#migration--upgrade-issues)
11. [Flow & Automation Issues](#flow--automation-issues)
12. [Framework Integration Issues](#framework-integration-issues)

---

## Installation & Startup Issues

### "SECRET must be set" Error
Directus requires a `SECRET` environment variable for encryption.
```env
SECRET="a-random-string-at-least-32-characters-long"
```
Generate one: `openssl rand -hex 32`

### "Port 8055 already in use"
Another process is using the port. Either stop it or change Directus's port:
```env
PORT=8056
```

### Directus Hangs on Startup
Common causes:
- Database connection failing — verify `DB_HOST`, `DB_PORT`, credentials
- Redis connection failing (if cache enabled) — verify `CACHE_REDIS` URL
- For Docker: ensure database service is healthy before Directus starts (use `depends_on` with health checks)

### "Cannot find module" Errors
After upgrading Directus or Node.js:
```bash
rm -rf node_modules
npm install
```

For Docker, pull the latest image:
```bash
docker compose pull
docker compose up -d
```

---

## Authentication & Token Errors

### "Invalid token" Error
Causes and fixes:
- **Token expired**: Access tokens have a short TTL. Refresh with `await client.refresh()` or re-login
- **Token not saved**: After generating a static token in the Data Studio, you MUST click Save on the user profile
- **Wrong token type**: Static tokens go in `staticToken()`, login tokens are managed by `authentication()`
- **SECRET changed**: If you change the `SECRET` env var, all existing tokens are invalidated

### "Forbidden" / 403 Error
The authenticated user (or Public role) doesn't have permission for the requested action.

Checklist:
1. Check **Settings → Access Policies** for the relevant role
2. Verify the collection has the needed CRUD permissions
3. Check for item-level permission rules that might filter out the item
4. If using the Public role, ensure read access is explicitly granted
5. For file access, ensure the Public role has read access to `directus_files`

### Login Returns "Invalid credentials"
- Verify email and password
- Check if the user exists and is active (not suspended)
- External auth provider users cannot use email/password login — they must use their provider's flow
- The `requestPasswordReset` endpoint throws a Forbidden error for external auth provider users

### Session/Cookie Auth Not Working
For SSR frameworks using cookie auth:
- `credentials: 'include'` must be set on both `authentication()` and `rest()`:
  ```typescript
  const client = createDirectus(url)
    .with(authentication('session', { credentials: 'include' }))
    .with(rest({ credentials: 'include' }));
  ```
- Ensure `CORS_CREDENTIALS=true` in Directus
- Cookies require matching domains or proper `SameSite` configuration

---

## CORS & Network Errors

### "Access-Control-Allow-Origin" Errors
Enable and configure CORS in Directus:
```env
CORS_ENABLED=true
CORS_ORIGIN=http://localhost:4321,https://your-site.com
CORS_METHODS=GET,POST,PATCH,DELETE
CORS_ALLOWED_HEADERS=Content-Type,Authorization
CORS_CREDENTIALS=true
```

Common mistakes:
- Using `CORS_ORIGIN=*` with `CORS_CREDENTIALS=true` — browsers reject this. List specific origins instead
- Forgetting to include both `http://localhost:PORT` for dev and production URL
- Missing `CORS_CREDENTIALS=true` when using cookie/session auth

### "Mixed Content" Errors
Browser blocks HTTP requests from HTTPS pages. Ensure both your frontend and Directus use HTTPS in production.

### Fetch Failures in SSG Build
During static build (e.g., `astro build`), your Directus server must be running and reachable. Verify `DIRECTUS_URL` in `.env` is correct and the server is up.

---

## SDK Errors

### "No URL provided" / "Request failed"
Ensure you're passing a valid URL to `createDirectus()`:
```typescript
// Wrong — missing protocol
const client = createDirectus('directus.example.com');

// Correct
const client = createDirectus('https://directus.example.com');
```

### "readItems is not a function" / Import Errors
Ensure you're importing from `@directus/sdk`, not an old SDK version:
```typescript
// Correct (SDK v13+)
import { createDirectus, rest, readItems } from '@directus/sdk';

// Wrong (old SDK)
import { Directus } from '@directus/sdk';
```

### TypeScript Type Errors with SDK
- Provide a Schema generic: `createDirectus<MySchema>(url)`
- Ensure your schema interface uses arrays for regular collections and singular types for singletons
- Relational fields should use union types: `number | RelatedType`
- After upgrading the SDK, check for breaking changes in type definitions

### "Cannot read property of undefined" on Response
The response might be empty or the collection name might be wrong. Always handle null:
```typescript
const posts = await client.request(readItems('posts'));
// posts could be an empty array — check length before accessing [0]
```

---

## Docker Issues

### Container Crashes on Startup
Check logs first:
```bash
docker compose logs directus
```

Common causes:
- Database not ready yet — add a health check:
  ```yaml
  database:
    image: postgres:16
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U directus"]
      interval: 5s
      timeout: 5s
      retries: 5

  directus:
    depends_on:
      database:
        condition: service_healthy
  ```
- Incorrect volume mounts — ensure paths exist on the host
- Incompatible env vars — check for typos

### Volumes / Permissions Errors
If Directus can't write to mounted volumes:
```bash
# Fix permissions (Linux/Mac)
sudo chown -R 1000:1000 ./uploads ./database ./extensions
```

Or in docker-compose:
```yaml
directus:
  user: "1000:1000"
```

### Extensions Not Loading in Docker
Ensure the extensions volume mount is correct:
```yaml
volumes:
  - ./extensions:/directus/extensions
```

And that built extensions (with `dist/` or `index.js`) are in the right structure:
```
extensions/
└── my-extension/
    ├── package.json
    └── dist/
        └── index.js
```

### Upgrading Directus in Docker
```bash
docker compose pull
docker compose down
docker compose up -d
```

Always snapshot your schema before upgrading:
```bash
# Before upgrade
docker compose exec directus npx directus schema snapshot /directus/snapshot.yaml
```

---

## Database Issues

### "Relation already exists" on Setup
If you're reusing a database that already has Directus tables, either:
- Use a fresh database
- Or run migrations: `npx directus database migrate:latest`

### Slow Queries
- Add database indexes for fields you frequently filter/sort on
- Use `fields` parameter to limit returned data
- Use `limit` to avoid fetching entire collections
- Enable caching: `CACHE_ENABLED=true`
- For PostgreSQL, run `ANALYZE` periodically

### Connection Pool Exhaustion
If you see "too many connections" errors:
```env
DB_POOL__MIN=0
DB_POOL__MAX=10
```

---

## File Upload & Asset Issues

### "Payload too large" on Upload
Increase the max payload size:
```env
MAX_PAYLOAD_SIZE="100mb"
```

For Docker with a reverse proxy, also configure the proxy's max body size (e.g., `client_max_body_size 100m;` in nginx).

### Assets Return 403
Ensure the accessing role has read permission on `directus_files`. For public access:
1. **Settings → Access Policies → Public**
2. Enable Read on `directus_files`

### Image Transformations Not Working
- Check `ASSETS_TRANSFORM` env var — must be `"all"` or `"presets"`
- Verify Sharp is installed (included in Docker image, may need manual install for npm setups)
- Very large images may exceed `ASSETS_TRANSFORM_IMAGE_MAX_DIMENSION`

---

## Extension Issues

### Extension Not Appearing
1. Verify the extension is built: check for `dist/index.js`
2. Verify directory structure: `extensions/{extension-name}/package.json`
3. Check `package.json` has correct `directus:extension` metadata
4. Restart Directus (or enable `EXTENSIONS_AUTO_RELOAD=true`)
5. Check logs for extension loading errors

### "Route doesn't exist" for Custom Endpoints
- Endpoints are available at `/{extension-name}/` (the directory name or custom `id`)
- Ensure the extension exported correctly — `export default` for the handler
- Check that the extension built successfully without errors
- Verify Express route paths are correct (leading slash matters)

### Hooks Not Firing
- Verify event names: `items.create` not `item.create`
- Collection-specific events: use `posts.items.create` or check `meta.collection` inside a generic handler
- Filter hooks must `return` the payload — forgetting this silently drops the data
- Check that the hook is registered in the correct lifecycle phase

---

## Performance Issues

### Slow API Responses
1. **Enable caching**: `CACHE_ENABLED=true` with Redis
2. **Limit fields**: Always specify `fields` in queries instead of returning everything
3. **Use pagination**: Never fetch unbounded result sets
4. **Add database indexes**: For frequently filtered/sorted fields
5. **Optimize relations**: Don't deeply nest relational queries if you don't need the data

### High Memory Usage
- Directus with Sharp (image processing) can use significant memory
- Set `ASSETS_TRANSFORM_IMAGE_MAX_DIMENSION` to limit max image size
- Use Redis for caching instead of in-memory
- For Docker, set memory limits:
  ```yaml
  directus:
    deploy:
      resources:
        limits:
          memory: 512M
  ```

### Slow Builds (SSG)
If `astro build` is slow due to many API calls:
- Reduce the number of collections/items fetched at build time
- Use incremental builds where supported
- Consider hybrid (SSR for dynamic pages, SSG for static)
- Cache Directus responses during build

---

## Migration & Upgrade Issues

### Breaking Changes Between Versions
Always check the [Directus release notes](https://github.com/directus/directus/releases) before upgrading. Major areas of change:
- SDK import paths and function signatures
- Environment variable names
- Database migration requirements
- Extension API changes

### Schema Migration Conflicts
If `schema apply` fails:
1. Take a fresh snapshot of the target
2. Compare manually with the source snapshot
3. Resolve conflicts in collection/field definitions
4. Apply incrementally if needed

### Data Loss Prevention
- Always back up your database before upgrading
- Test upgrades in a staging environment first
- Schema snapshots don't include content — back up your database separately

---

## Flow & Automation Issues

### Flow Not Triggering
- Check the flow's status is "Active"
- For Event Hooks: verify the correct scope (items.create, items.update) and collection are selected
- For Schedules: verify CRON syntax is correct
- For Webhooks: test the URL with curl
- For Manual: ensure the collection is selected in the trigger config

### Operations Failing Silently
- Check server logs: `docker compose logs directus`
- Use a "Log to Console" operation between steps to debug the data chain
- Web Request failures: check response status codes, verify URLs and headers
- Template variables: ensure they match exactly (e.g., `{{ operation_key.field }}`)

### Data Chain Variables Not Resolving
- Operation keys are case-sensitive
- Use `{{ $trigger }}` for trigger data, `{{ operation_key }}` for operation output
- Array access: `{{ operation_key[0].field }}`
- Check that the previous operation actually returned data

---

## Framework Integration Issues

### Astro: "getStaticPaths() requires at least one entry"
Your Directus query returned zero items. Causes:
- Collection is empty
- Filter excludes all items (e.g., nothing is "published")
- Public role doesn't have read access
- Token is invalid or missing

Handle gracefully:
```typescript
export async function getStaticPaths() {
  try {
    const pages = await directus.request(readItems('pages'));
    if (!pages.length) return [{ params: { slug: '404' }, props: {} }];
    return pages.map(p => ({ params: { slug: p.slug }, props: p }));
  } catch (error) {
    console.error('Failed to fetch pages:', error);
    return [{ params: { slug: '404' }, props: {} }];
  }
}
```

### Astro: Content Not Updating After Directus Changes
For SSG sites, you need to rebuild after content changes. Options:
- Set up a Directus Flow to trigger a deploy webhook on publish
- Use SSR mode (`output: 'server'`) for always-fresh content
- Use hybrid mode and mark content pages as SSR

### Next.js / Nuxt: ISR Cache Not Invalidating
Use `revalidate` (Next.js) or Directus webhooks to trigger revalidation:
```typescript
// Next.js API route for on-demand revalidation
export async function POST(request: Request) {
  const body = await request.json();
  revalidatePath(`/blog/body.slug`);
  return Response.json({ revalidated: true });
}
```

### Environment Variables Not Available
- In Astro: prefix with nothing (just use `import.meta.env.DIRECTUS_URL`)
- In Next.js: prefix with `NEXT_PUBLIC_` for client-side, no prefix for server-side
- In Nuxt: use `runtimeConfig` in `nuxt.config.ts`
- In plain Node.js: use `process.env.DIRECTUS_URL`
- Check `.env` file is in the project root and the dev server was restarted after changes

### "fetch is not defined" in Node.js
The Directus SDK uses `fetch` internally. Node.js 18+ includes it natively. For older versions:
```bash
npm install undici
```
Or use Node.js 18+.

FILE:references/flows-and-automation.md
# Directus Flows & Automation

## Table of Contents
1. [Flows Overview](#flows-overview)
2. [Trigger Types](#trigger-types)
3. [Operations](#operations)
4. [The Data Chain](#the-data-chain)
5. [AI Content Generation](#ai-content-generation)
6. [Scheduled Content Publishing](#scheduled-content-publishing)
7. [Deployment Webhooks](#deployment-webhooks)
8. [Content Approval Workflows](#content-approval-workflows)
9. [External Service Integration](#external-service-integration)
10. [Managing Flows via API](#managing-flows-via-api)

---

## Flows Overview

Directus Flows (also called Directus Automate) provide a no-code visual interface for building event-driven automations. A Flow consists of a **trigger** (what starts it) and one or more **operations** (what it does), connected in a chain.

Flows are configured in the Data Studio under **Settings → Flows**.

Key concepts:
- **Triggers** start the flow — they can be events, schedules, webhooks, or manual actions
- **Operations** are individual steps — reading data, making API calls, sending emails, running code, etc.
- **Data Chain** — each operation's output is available to subsequent operations via template syntax `{{ $trigger.body }}`, `{{ operationName.result }}`
- **Permissions** — flows run with configurable permissions: `$public`, `$trigger` (same as triggering user), `$full` (admin), or a specific role UUID

---

## Trigger Types

### Event Hook
Fires when something happens in Directus (item created, updated, deleted, etc.).

```
Type: Event Hook
Scope: items.create, items.update, items.delete
Collections: posts, pages (select which collections trigger the flow)
Response: Action (Non-Blocking) — runs in background
         Filter (Blocking) — can modify or reject the operation
```

**Action (Non-Blocking)** — the triggering operation completes immediately; the flow runs asynchronously. Use for notifications, logging, external sync.

**Filter (Blocking)** — the flow runs before the operation completes and can modify the payload or throw an error to cancel it. Use for validation, data enrichment before save.

### Schedule (CRON)
Fires on a time-based schedule using CRON syntax.
```
Type: Schedule (CRON)
Interval examples:
  */5 * * * *     — every 5 minutes
  0 * * * *       — every hour
  0 9 * * 1-5     — 9 AM weekdays
  0 0 * * *       — midnight daily
  0 0 1 * *       — first of each month
```

### Webhook
Fires when an external service POSTs to the flow's webhook URL.
```
Type: Webhook
Method: GET or POST
Async: true/false (respond immediately or wait for flow to complete)
```

The webhook URL is: `{DIRECTUS_URL}/flows/trigger/{flow-id}`

### Manual
Triggered by a user clicking a button in the Data Studio on a specific item.
```
Type: Manual Trigger
Collections: posts (which collections show the button)
```

The trigger payload includes `{{ $trigger.body.keys }}` — an array of selected item IDs.

### Another Flow
Fires when called by another flow's "Trigger Flow" operation. Used for modular, reusable flows.

---

## Operations

### Read Data
Reads items from a collection.
```
Collection: posts
Permissions: Full Access / From Trigger / From Role
Query: filter, fields, sort, limit
Result key: {{ read_data_operation_name }}
```

### Create Data
Creates a new item in a collection.

### Update Data
Updates items by ID or filter.
- Check "Emit Events" if you want this update to trigger other event-based flows.

### Delete Data
Deletes items by ID or filter.

### Web Request
Makes an HTTP request to an external service.
```
Method: GET, POST, PUT, PATCH, DELETE
URL: https://api.example.com/endpoint
Headers: { "Authorization": "Bearer {{$env.API_KEY}}", "Content-Type": "application/json" }
Body: JSON with template variables
```

### Send Email
Sends an email using Directus's configured email transport.
```
To: [email protected] (or {{ $trigger.payload.user_email }})
Subject: New post published
Body: HTML template with variables
```

### Send Notification
Sends an in-app notification to a Directus user.

### Run Script
Executes custom JavaScript. Has access to the data chain via the `data` parameter:
```javascript
module.exports = async function(data) {
  const triggerData = data.$trigger;
  const previousStep = data.previous_operation_key;

  // Do processing
  const result = { processed: true, count: 42 };

  return result;  // Available as {{ this_operation_key }}
}
```

### Transform Payload
Transforms data using a JSON template with Liquid-style syntax.

### Condition
Branches the flow based on a condition. If true, runs the "success" branch; if false, runs the "reject" branch.
```
Rule: {{ $trigger.payload.status }} == "published"
```

### Log to Console
Logs data to the server console (useful for debugging).

### Trigger Flow
Calls another flow, enabling modular/reusable flow patterns.

### Sleep
Pauses execution for a specified duration (in milliseconds).

---

## The Data Chain

Every trigger and operation adds data to the flow's data chain. Access data from any previous step using template syntax:

```
{{ $trigger }}                          — Full trigger payload
{{ $trigger.body }}                     — Request body (webhooks)
{{ $trigger.body.keys[0] }}             — First selected item ID (manual triggers)
{{ $trigger.payload }}                  — The item data (event hooks)
{{ $trigger.payload.title }}            — Specific field from trigger payload
{{ $accountability.user }}              — ID of user who triggered

{{ operation_key }}                     — Full output of a named operation
{{ operation_key.field_name }}          — Specific field from operation output
{{ operation_key[0].title }}            — Array item access

{{ $env.MY_SECRET }}                    — Environment variable
{{ $now }}                              — Current ISO timestamp
{{ $last }}                             — Output of the immediately previous operation
```

---

## AI Content Generation

### Generate Social Posts with OpenAI

**Flow setup:**
1. **Trigger**: Manual, on the `posts` collection
2. **Read Data** (operation key: `article`): Read the triggered item from `posts` with full access
   - Collection: `posts`
   - Key: `{{ $trigger.body.keys[0] }}`
3. **Web Request** (key: `generate`): POST to OpenAI
   - URL: `https://api.openai.com/v1/chat/completions`
   - Headers: `{ "Authorization": "Bearer {{ $env.OPENAI_API_KEY }}", "Content-Type": "application/json" }`
   - Body:
   ```json
   {
     "model": "gpt-4",
     "messages": [
       {
         "role": "system",
         "content": "You are a social media manager. Write compelling promotional posts based on the content provided."
       },
       {
         "role": "user",
         "content": "Write a Twitter post for: {{ article.title }}"
       }
     ]
   }
   ```
4. **Update Data**: Save the result back to the post
   - Collection: `posts`
   - Key: `{{ $trigger.body.keys[0] }}`
   - Payload: `{ "social_output": "{{ generate.data.choices[0].message.content }}" }`

### Generate SEO Metadata

Same pattern, but the prompt asks for JSON output with `meta_title`, `meta_description`, and `keywords`. Use a **Run Script** operation to parse the JSON and then **Update Data** to save each field.

### Generate Images with DALL·E

**Web Request** to `https://api.openai.com/v1/images/generations`:
```json
{
  "model": "dall-e-3",
  "prompt": "A professional blog header image for an article about {{ article.title }}",
  "n": 1,
  "size": "1792x1024"
}
```

Then use another **Web Request** or **Run Script** to download the generated image URL and import it into Directus Files.

### Auto-Translate Content

Use a flow triggered on `items.create` or `items.update` for your content collection, then:
1. Read the source content
2. Web Request to a translation API (DeepL, Google Translate, or an LLM)
3. Create/Update items in the translations junction collection

---

## Scheduled Content Publishing

For SSG sites that need to publish content at a scheduled time:

### Flow: Publish Scheduled Articles

1. **Trigger**: Schedule (CRON) — e.g., `*/15 * * * *` (every 15 minutes)
2. **Update Data**: Update items in `posts` where:
   - Filter: `status` equals `scheduled` AND `date_published` is less than or equal to `$NOW`
   - Payload: `{ "status": "published" }`
   - Check "Emit Events" so downstream flows (like deploy hooks) are triggered
3. **Condition**: Check if any items were updated
4. **Web Request**: POST to your hosting deploy hook URL

### For SSR/Dynamic Sites
No flow needed — just filter in your frontend query:
```typescript
const posts = await client.request(readItems('posts', {
  filter: {
    _and: [
      { status: { _eq: 'published' } },
      { date_published: { _lte: '$NOW' } },
    ],
  },
}));
```

---

## Deployment Webhooks

### Trigger Rebuild on Content Change

1. **Trigger**: Event Hook, `items.create` / `items.update` on content collections, Action (Non-Blocking)
2. **Condition**: Check `{{ $trigger.payload.status }}` equals `published`
3. **Web Request**: POST to deploy hook

**Netlify:**
```
POST https://api.netlify.com/build_hooks/{your-hook-id}
```

**Vercel:**
```
POST https://api.vercel.com/v1/integrations/deploy/{your-hook-id}/{your-project-id}
```

**Cloudflare Pages:**
```
POST https://api.cloudflare.com/client/v4/pages/webhooks/deploy_hooks/{your-hook-id}
```

---

## Content Approval Workflows

### Multi-Stage Review

1. **Trigger**: Event Hook on `items.update` for `posts` (Filter/Blocking)
2. **Condition**: Status changed to `in_review`
3. **Read Data**: Get the editor/reviewer assigned to this post
4. **Send Email**: Notify the reviewer with a link to the item
5. **Send Notification**: Also send an in-app notification

### Prevent Publishing Without Approval
Use a Filter (Blocking) trigger:

1. **Trigger**: Event Hook on `items.update`, Filter (Blocking)
2. **Condition**: Status is being changed to `published`
3. **Read Data**: Check if the item has an `approved_by` field set
4. **Condition**: If `approved_by` is null, reject the operation (return an error)

---

## External Service Integration

### Sync to External Systems
When content is published, sync to Algolia, Elasticsearch, or a custom service:

1. **Trigger**: Event Hook on `items.create` / `items.update`
2. **Condition**: Status is `published`
3. **Transform Payload**: Shape data for the external API
4. **Web Request**: POST/PUT to external service

### Inbound Webhook Processing
Accept data from external services:

1. **Trigger**: Webhook (POST)
2. **Run Script**: Validate and transform incoming data
3. **Create Data**: Save to a Directus collection
4. **Send Notification**: Alert relevant users

---

## Managing Flows via API

Flows can also be managed programmatically:

```typescript
import { readFlows, readFlow, createFlow, updateFlow, deleteFlow } from '@directus/sdk';

// List all flows
const flows = await client.request(readFlows());

// Read a specific flow
const flow = await client.request(readFlow('flow-uuid'));

// Trigger a webhook flow
await fetch('https://directus.example.com/flows/trigger/flow-uuid', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({ key: 'value' }),
});
```

FILE:references/extensions.md
# Directus Extensions

## Table of Contents
1. [Extension Types Overview](#extension-types-overview)
2. [Creating Extensions](#creating-extensions)
3. [Custom Endpoints](#custom-endpoints)
4. [Event Hooks](#event-hooks)
5. [Custom Operations](#custom-operations)
6. [App Extensions (Interfaces, Displays, Layouts, Modules, Panels)](#app-extensions)
7. [Bundles](#bundles)
8. [The Sandbox](#the-sandbox)
9. [Development Workflow](#development-workflow)

---

## Extension Types Overview

| Type | Category | Purpose |
|---|---|---|
| **Endpoints** | API | Register custom REST routes |
| **Hooks** | API | Run code on lifecycle/database events |
| **Operations** | API | Custom operations for use in Flows |
| **Interfaces** | App | Custom form inputs in the editor |
| **Displays** | App | Custom value renderers throughout the Data Studio |
| **Layouts** | App | Custom list/grid views for collection pages |
| **Modules** | App | Custom top-level pages in the Data Studio sidebar |
| **Panels** | App | Custom widgets for Insights dashboards |
| **Themes** | App | Custom visual themes for the Data Studio |
| **Bundles** | Both | Package multiple extensions together |

---

## Creating Extensions

### Scaffold an Extension
```bash
npx create-directus-extension@latest
```

You'll be prompted for:
- Extension type (endpoint, hook, interface, etc.)
- Name
- Language (JavaScript or TypeScript)

This creates a directory with:
```
my-extension/
├── src/
│   └── index.ts (or index.js)
├── package.json
└── tsconfig.json (if TypeScript)
```

### Build and Install
```bash
cd my-extension
npm install
npm run build
```

Copy the built extension to your Directus project's `extensions/` directory. The structure depends on how you're running Directus:

**Docker:**
```yaml
volumes:
  - ./extensions:/directus/extensions
```

**Self-hosted:**
Place in the `extensions/` directory at the root of your Directus project.

### Auto-Reload in Development
Set `EXTENSIONS_AUTO_RELOAD=true` in your Directus `.env` to automatically reload extensions when files change.

---

## Custom Endpoints

Endpoints add new API routes to your Directus instance. They are Express.js route handlers.

### Basic Endpoint
```typescript
// src/index.ts
import { defineEndpoint } from '@directus/extensions-sdk';

export default defineEndpoint((router, context) => {
  const { services, getSchema } = context;

  router.get('/', async (req, res) => {
    res.json({ message: 'Hello from custom endpoint!' });
  });

  router.get('/posts-count', async (req, res, next) => {
    try {
      const { ItemsService } = services;
      const schema = await getSchema();

      const postsService = new ItemsService('posts', {
        schema,
        accountability: req.accountability,
      });

      const posts = await postsService.readByQuery({
        aggregate: { count: '*' },
      });

      res.json({ count: posts[0].count });
    } catch (error) {
      next(error);
    }
  });
});
```

This endpoint is available at `/my-extension/` and `/my-extension/posts-count`.

### Endpoint with Custom ID
```typescript
export default {
  id: 'my-api',
  handler: (router, context) => {
    router.get('/', (req, res) => res.send('Available at /my-api'));
    router.post('/process', async (req, res, next) => {
      try {
        // process req.body
        res.json({ success: true });
      } catch (error) {
        next(error);
      }
    });
  },
};
```

### Context Object
The `context` parameter provides:

| Property | Description |
|---|---|
| `services` | All Directus service classes (ItemsService, FilesService, UsersService, etc.) |
| `getSchema` | Async function returning the current database schema |
| `database` | Knex database instance |
| `env` | Environment variables |
| `logger` | Pino logger instance |
| `emitter` | Event emitter for triggering hooks |

### Error Handling
Always use `next(error)` to pass errors to Directus's error handler:
```typescript
import { createError, ForbiddenError } from '@directus/errors';

const MyError = createError('MY_ERROR', 'Something went wrong.', 400);

router.get('/protected', async (req, res, next) => {
  if (!req.accountability?.user) {
    return next(new ForbiddenError());
  }
  // ...
});
```

---

## Event Hooks

Hooks run code when events occur — database operations, application lifecycle, or schedules.

### Basic Hook
```typescript
import { defineHook } from '@directus/extensions-sdk';

export default defineHook(({ filter, action, init, schedule, embed }) => {
  // FILTER: runs BEFORE event, can modify payload or reject
  filter('items.create', async (payload, meta, context) => {
    // Modify the payload before it's saved
    if (meta.collection === 'posts') {
      payload.slug = payload.title.toLowerCase().replace(/\s+/g, '-');
    }
    return payload;
  });

  // ACTION: runs AFTER event, for side effects
  action('items.create', async (meta, context) => {
    if (meta.collection === 'posts') {
      console.log(`Post created with ID: meta.key`);
    }
  });

  // INIT: runs during Directus startup
  init('app.after', async () => {
    console.log('Directus is ready!');
  });

  // SCHEDULE: runs on a CRON schedule
  schedule('0 * * * *', async () => {
    console.log('Hourly task running...');
  });

  // EMBED: inject HTML into Data Studio
  embed('head', '<style>body { --primary: #ff6600; }</style>');
});
```

### Filter Events
Filters are synchronous and blocking. They can modify data or throw errors:

| Event | Description |
|---|---|
| `items.create` | Before item creation |
| `items.update` | Before item update |
| `items.delete` | Before item deletion |
| `items.query` | Before reading items |
| `<collection>.items.create` | Collection-specific events |
| `request.not_found` | When a route isn't found |
| `request.error` | When a request errors |

```typescript
filter('items.create', async (payload, meta, context) => {
  const { collection, accountability } = meta;
  const { database, schema } = context;

  if (collection === 'posts' && !payload.title) {
    throw new Error('Title is required');
  }

  return payload; // must return the (possibly modified) payload
});
```

### Action Events
Actions are asynchronous and non-blocking:

| Event | Description |
|---|---|
| `items.create` | After item creation |
| `items.update` | After item update |
| `items.delete` | After item deletion |
| `items.sort` | After items are sorted |
| `server.start` | After server starts |
| `server.stop` | Before server stops |
| `auth.login` | After user logs in |
| `auth.logout` | After user logs out |
| `files.upload` | After file upload |

```typescript
action('items.update', async (meta, context) => {
  const { collection, keys, payload } = meta;
  const { services, getSchema } = context;

  if (collection === 'posts' && payload.status === 'published') {
    // Send notification, sync to external service, etc.
    const { MailService } = services;
    const schema = await getSchema();
    const mailService = new MailService({ schema });

    await mailService.send({
      to: '[email protected]',
      subject: 'New post published',
      text: `Post keys[0] has been published.`,
    });
  }
});
```

### Important Warnings
- Do NOT emit the same event you're handling inside a hook — this creates infinite loops
- Filter hooks on `items.query` / read events can severely impact performance if they do database reads
- System collections use unprefixed names: `users.create` (not `directus_users.create`)
- The `files` collection does not emit create/update events on file upload — use `files.upload` action instead
- The `collections` and `fields` system collections do not emit read events
- The `relations` collection does not emit delete events

---

## Custom Operations

Operations are custom steps usable within Directus Flows.

### Define an Operation
```typescript
// src/index.ts
import { defineOperationApi, defineOperationApp } from '@directus/extensions-sdk';

// API side — the logic
export const api = defineOperationApi({
  id: 'my-operation',
  handler: async (options, context) => {
    const { text, uppercase } = options;
    const result = uppercase ? text.toUpperCase() : text;
    return { processed: result };
  },
});

// App side — the UI in the Flow editor
export const app = defineOperationApp({
  id: 'my-operation',
  name: 'My Custom Operation',
  icon: 'bolt',
  description: 'Processes text',
  overview: ({ text }) => [
    { label: 'Text', text: text || 'Not configured' },
  ],
  options: [
    {
      field: 'text',
      name: 'Text Input',
      type: 'string',
      meta: { interface: 'input', width: 'full' },
    },
    {
      field: 'uppercase',
      name: 'Uppercase',
      type: 'boolean',
      meta: { interface: 'boolean', width: 'half' },
      schema: { default_value: false },
    },
  ],
});
```

---

## App Extensions

### Interfaces
Custom form inputs used in the item editor.

```typescript
import { defineInterface } from '@directus/extensions-sdk';

export default defineInterface({
  id: 'my-color-picker',
  name: 'Color Picker',
  icon: 'palette',
  description: 'A custom color input',
  component: MyColorPicker,        // Vue component
  types: ['string'],                // Directus field types this works with
  options: [                        // Configuration options
    {
      field: 'presets',
      name: 'Preset Colors',
      type: 'json',
      meta: { interface: 'tags' },
    },
  ],
});
```

### Displays
Render a value anywhere items are shown (tables, cards, etc.).

```typescript
import { defineDisplay } from '@directus/extensions-sdk';

export default defineDisplay({
  id: 'my-badge',
  name: 'Status Badge',
  icon: 'flag',
  description: 'Shows status as a colored badge',
  component: MyBadge,
  types: ['string'],
  options: [],
});
```

### Layouts
Custom views for collection list pages.

### Modules
Top-level pages in the Data Studio navigation sidebar.

### Panels
Dashboard widgets for the Insights module.

---

## Bundles

Bundles package multiple extensions into a single installable unit:

```json
// package.json
{
  "name": "my-directus-bundle",
  "directus:bundle": {
    "entries": [
      { "type": "endpoint", "name": "my-endpoint", "source": "src/endpoint/index.ts" },
      { "type": "hook", "name": "my-hook", "source": "src/hook/index.ts" },
      { "type": "interface", "name": "my-interface", "source": "src/interface/index.ts" }
    ]
  }
}
```

---

## The Sandbox

Directus runs API extensions (hooks, endpoints, operations) inside an isolated VM sandbox for security. This means:

- Extensions can't access the file system directly
- Extensions can't require arbitrary Node.js modules
- Extensions use the `@directus/extensions-sdk` to access services
- The sandbox provides controlled access to the Directus API

If you need to bypass the sandbox (for specific packages or functionality), you can disable it for specific extensions in Directus configuration — but this is not recommended for production.

---

## Development Workflow

### Recommended Setup

1. **Docker Compose** for Directus with mounted `extensions/` volume
2. **Extension development** in a separate directory with `npm run dev` for hot-reloading
3. **`EXTENSIONS_AUTO_RELOAD=true`** in Directus env for auto-detection of changes

### Docker Compose for Extension Dev
```yaml
services:
  directus:
    image: directus/directus:latest
    ports:
      - 8055:8055
    volumes:
      - ./database:/directus/database
      - ./uploads:/directus/uploads
      - ./extensions:/directus/extensions
    environment:
      SECRET: 'change-me'
      ADMIN_EMAIL: '[email protected]'
      ADMIN_PASSWORD: 'password'
      DB_CLIENT: 'sqlite3'
      DB_FILENAME: '/directus/database/data.db'
      EXTENSIONS_AUTO_RELOAD: 'true'
```

### Build & Test Loop
```bash
# In your extension directory
npm run build          # Build once
npm run dev            # Watch mode (auto-rebuild on changes)

# Directus auto-reloads the extension if EXTENSIONS_AUTO_RELOAD=true
```

### Debugging
- Use `context.logger.info()` or `context.logger.error()` in API extensions
- Use `console.log` in Run Script operations (visible in server logs)
- Check Docker logs: `docker compose logs -f directus`
- For app extensions, use Vue DevTools in the browser

FILE:references/sdk-and-api.md
# Directus SDK & API Reference

## Table of Contents
1. [SDK Setup & Composables](#sdk-setup--composables)
2. [Authentication](#authentication)
3. [CRUD Operations](#crud-operations)
4. [Filtering, Sorting & Pagination](#filtering-sorting--pagination)
5. [Relational Data & Fields](#relational-data--fields)
6. [File Management](#file-management)
7. [Real-Time Subscriptions](#real-time-subscriptions)
8. [GraphQL Usage](#graphql-usage)
9. [Raw REST API](#raw-rest-api)

---

## SDK Setup & Composables

The Directus SDK (`@directus/sdk`) uses a composable architecture. The client starts empty and gains features through `.with()`:

```bash
npm install @directus/sdk
```

### Available Composables

| Composable | Purpose |
|---|---|
| `rest()` | Enables REST API requests via `client.request()` |
| `graphql()` | Enables GraphQL queries via `client.query()` |
| `authentication(mode)` | Manages login/logout/refresh. Modes: `'json'`, `'session'`, `'cookie'` |
| `staticToken(token)` | Sets a fixed access token for all requests |
| `realtime(options)` | Enables WebSocket subscriptions |

### Client Creation Patterns

**Public (no auth):**
```typescript
import { createDirectus, rest } from '@directus/sdk';

const client = createDirectus<Schema>('https://directus.example.com')
  .with(rest());
```

**Static token (API key):**
```typescript
import { createDirectus, rest, staticToken } from '@directus/sdk';

const client = createDirectus<Schema>('https://directus.example.com')
  .with(staticToken('my-api-token'))
  .with(rest());
```

**Session-based auth (for SSR apps like Astro in server mode):**
```typescript
import { createDirectus, rest, authentication } from '@directus/sdk';

const client = createDirectus<Schema>('https://directus.example.com')
  .with(authentication('cookie'))
  .with(rest());
```

**JSON auth (for SPAs):**
```typescript
import { createDirectus, rest, authentication } from '@directus/sdk';

const client = createDirectus<Schema>('https://directus.example.com')
  .with(authentication('json'))
  .with(rest());
```

### Per-Request Token Override
```typescript
import { withToken, readItems } from '@directus/sdk';

const result = await client.request(
  withToken('temporary-token', readItems('posts'))
);
```

---

## Authentication

### Login / Logout / Refresh
```typescript
// Login (requires authentication() composable)
await client.login('[email protected]', 'password');

// Refresh tokens
await client.refresh();

// Logout
await client.logout();

// Get current user
import { readMe } from '@directus/sdk';
const me = await client.request(readMe());
```

### Generate Static Token
In the Directus Data Studio: Users → select user → Token field → Generate → Save. Use this token with `staticToken()` or in API headers as `Authorization: Bearer <token>`.

### SSO / External Auth
Directus supports OAuth2, OpenID Connect, LDAP, and SAML. Configure via environment variables:
```env
AUTH_PROVIDERS="google"
AUTH_GOOGLE_DRIVER="openid"
AUTH_GOOGLE_CLIENT_ID="your-client-id"
AUTH_GOOGLE_CLIENT_SECRET="your-client-secret"
AUTH_GOOGLE_ISSUER_URL="https://accounts.google.com/.well-known/openid-configuration"
AUTH_GOOGLE_REDIRECT_ALLOW_LIST="http://localhost:3000/auth/callback"
```

---

## CRUD Operations

All operations use `client.request(operationFunction())`.

### Create
```typescript
import { createItem, createItems } from '@directus/sdk';

// Single item
const newPost = await client.request(createItem('posts', {
  title: 'Hello World',
  content: '<p>My first post</p>',
  status: 'published',
}));

// Multiple items
const newPosts = await client.request(createItems('posts', [
  { title: 'Post A', status: 'draft' },
  { title: 'Post B', status: 'draft' },
]));
```

### Read
```typescript
import { readItems, readItem, readSingleton } from '@directus/sdk';

// All items (with query)
const posts = await client.request(readItems('posts', {
  fields: ['id', 'title', 'status'],
  filter: { status: { _eq: 'published' } },
  sort: ['-date_created'],
  limit: 10,
}));

// Single item by ID
const post = await client.request(readItem('posts', 42));

// Singleton collection (e.g., global settings)
const settings = await client.request(readSingleton('global'));
```

### Update
```typescript
import { updateItem, updateItems, updateSingleton } from '@directus/sdk';

// Single item
await client.request(updateItem('posts', 42, {
  title: 'Updated Title',
}));

// Multiple items by query
await client.request(updateItems('posts', {
  filter: { status: { _eq: 'draft' } },
  limit: -1,
}, {
  status: 'archived',
}));

// Singleton
await client.request(updateSingleton('global', {
  site_title: 'New Site Title',
}));
```

### Delete
```typescript
import { deleteItem, deleteItems } from '@directus/sdk';

// Single
await client.request(deleteItem('posts', 42));

// Multiple
await client.request(deleteItems('posts', [1, 2, 3]));
```

---

## Filtering, Sorting & Pagination

### Filter Operators

| Operator | Meaning |
|---|---|
| `_eq` | Equals |
| `_neq` | Not equals |
| `_gt`, `_gte` | Greater than (or equal) |
| `_lt`, `_lte` | Less than (or equal) |
| `_in` | In array |
| `_nin` | Not in array |
| `_contains` | String contains (case-sensitive) |
| `_icontains` | String contains (case-insensitive) |
| `_starts_with` | String starts with |
| `_ends_with` | String ends with |
| `_null` | Is null |
| `_nnull` | Is not null |
| `_between` | Between two values |
| `_empty` | Is empty (null or '') |
| `_nempty` | Is not empty |

### Compound Filters
```typescript
const results = await client.request(readItems('posts', {
  filter: {
    _and: [
      { status: { _eq: 'published' } },
      {
        _or: [
          { category: { _eq: 'news' } },
          { featured: { _eq: true } },
        ],
      },
    ],
  },
}));
```

### Filtering Relational Fields
```typescript
const results = await client.request(readItems('posts', {
  filter: {
    author: {
      name: { _eq: 'John' },
    },
  },
}));
```

### Sorting
```typescript
// Ascending
const sorted = await client.request(readItems('posts', {
  sort: ['title'],
}));

// Descending (prefix with -)
const sortedDesc = await client.request(readItems('posts', {
  sort: ['-date_created'],
}));

// Multiple sort fields
const multiSort = await client.request(readItems('posts', {
  sort: ['-featured', 'title'],
}));
```

### Pagination
```typescript
// Offset-based
const page2 = await client.request(readItems('posts', {
  limit: 10,
  offset: 10,
}));

// Get total count with items
const withCount = await client.request(readItems('posts', {
  limit: 10,
  offset: 0,
  meta: ['total_count', 'filter_count'],
}));
```

### Search
```typescript
const results = await client.request(readItems('posts', {
  search: 'keyword',
}));
```

---

## Relational Data & Fields

### Selecting Specific Fields
```typescript
// Flat fields
const posts = await client.request(readItems('posts', {
  fields: ['id', 'title'],
}));

// Wildcard (all top-level fields)
const posts = await client.request(readItems('posts', {
  fields: ['*'],
}));

// Nested relational fields
const posts = await client.request(readItems('posts', {
  fields: ['id', 'title', { author: ['name', 'avatar'] }],
}));

// Deep nested
const posts = await client.request(readItems('posts', {
  fields: [
    'id',
    'title',
    { author: ['name', { role: ['name'] }] },
    { tags: [{ tags_id: ['name', 'slug'] }] },
  ],
}));
```

### Aggregation
```typescript
import { aggregate } from '@directus/sdk';

const stats = await client.request(aggregate('posts', {
  aggregate: { count: '*' },
  groupBy: ['status'],
}));
```

---

## File Management

### Upload a File
```typescript
import { uploadFiles } from '@directus/sdk';

const formData = new FormData();
formData.append('file', fileBlob, 'photo.jpg');

const result = await client.request(uploadFiles(formData));
```

### Import a File by URL
```typescript
import { importFile } from '@directus/sdk';

const result = await client.request(importFile('https://example.com/photo.jpg'));
```

### Access File URLs
Files are served at: `{DIRECTUS_URL}/assets/{file-id}`

**Transformations via query params:**
```
/assets/{id}?width=300&height=200&fit=cover&format=webp&quality=80
```

| Param | Values |
|---|---|
| `width` | Pixel width |
| `height` | Pixel height |
| `fit` | `cover`, `contain`, `inside`, `outside` |
| `format` | `jpg`, `png`, `webp`, `avif`, `tiff` |
| `quality` | 1–100 |
| `withoutEnlargement` | `true` to prevent upscaling |

---

## Real-Time Subscriptions

Requires `WEBSOCKETS_ENABLED=true` in Directus config.

```typescript
import { createDirectus, realtime, authentication } from '@directus/sdk';

const client = createDirectus<Schema>('https://directus.example.com')
  .with(authentication())
  .with(realtime());

await client.connect();
await client.login('[email protected]', 'password');

// Subscribe to collection changes
const { subscription, unsubscribe } = await client.subscribe('posts', {
  query: { fields: ['id', 'title', 'status'] },
});

for await (const message of subscription) {
  console.log('Change:', message);
}
```

### Real-Time with Public Access
```typescript
const client = createDirectus<Schema>('https://directus.example.com')
  .with(realtime({ authMode: 'public' }));

await client.connect();
```

---

## GraphQL Usage

```typescript
import { createDirectus, graphql } from '@directus/sdk';

const client = createDirectus<Schema>('https://directus.example.com')
  .with(graphql());

const result = await client.query<{ posts: Post[] }>(`
  query {
    posts(filter: { status: { _eq: "published" } }) {
      id
      title
      content
    }
  }
`);
```

GraphQL endpoints:
- User collections: `POST /graphql`
- System collections: `POST /graphql/system`

---

## Raw REST API

When the SDK doesn't cover your use case, make custom requests:

```typescript
const result = await client.request(() => ({
  path: '/custom/my-endpoint',
  method: 'GET',
}));

// With body
const result = await client.request(() => ({
  path: '/custom/my-endpoint',
  method: 'POST',
  body: JSON.stringify({ key: 'value' }),
  headers: { 'Content-Type': 'application/json' },
}));
```

### REST Endpoints Reference

| Resource | Endpoint |
|---|---|
| Items | `GET/POST/PATCH/DELETE /items/{collection}` |
| Single Item | `GET/PATCH/DELETE /items/{collection}/{id}` |
| Singleton | `GET/PATCH /items/{collection}` |
| Files | `GET/POST /files` |
| Assets | `GET /assets/{id}` |
| Users | `GET/POST/PATCH/DELETE /users` |
| Auth | `POST /auth/login`, `POST /auth/refresh`, `POST /auth/logout` |
| Flows | `GET/POST/PATCH/DELETE /flows` |
| Activity | `GET /activity` |
| Schema | `GET /schema/snapshot`, `POST /schema/diff`, `POST /schema/apply` |

FILE:references/typescript-patterns.md
# Directus TypeScript Patterns

## Table of Contents
1. [Schema Definition](#schema-definition)
2. [Relation Types](#relation-types)
3. [Singleton Collections](#singleton-collections)
4. [Extending Core Collections](#extending-core-collections)
5. [Automatic Type Generation](#automatic-type-generation)
6. [Utility Types & Helpers](#utility-types--helpers)
7. [Typed SDK Requests](#typed-sdk-requests)
8. [Working with JavaScript (Non-TS)](#working-with-javascript-non-ts)

---

## Schema Definition

The SDK's type system revolves around a root schema interface that maps collection names to their item types. This is the single source of truth for all type checking and autocompletion.

```typescript
// src/lib/schema.ts

// Root schema — provided to createDirectus<Schema>()
export interface Schema {
  // Regular collections are ARRAYS
  posts: Post[];
  categories: Category[];
  authors: Author[];
  tags: Tag[];

  // Junction collections (for M2M) are also arrays
  posts_tags: PostTag[];

  // Singletons are SINGULAR types (not arrays)
  global_settings: GlobalSettings;

  // Custom fields on core collections are singular
  directus_users: CustomUser;
}

export interface Post {
  id: number;
  status: 'draft' | 'published' | 'archived';
  title: string;
  slug: string;
  content: string;
  excerpt: string | null;
  cover_image: string | null;           // file UUID
  date_created: string;                  // ISO 8601
  date_updated: string | null;
  // Relations — use union type: foreign key OR populated object
  author: number | Author;
  category: number | Category;
  tags: number[] | PostTag[];
}

export interface Author {
  id: number;
  name: string;
  bio: string | null;
  avatar: string | null;
}

export interface Category {
  id: number;
  name: string;
  slug: string;
  description: string | null;
}

export interface Tag {
  id: number;
  name: string;
  slug: string;
}

// Junction table for posts <-> tags (M2M)
export interface PostTag {
  id: number;
  posts_id: number | Post;
  tags_id: number | Tag;
}

export interface GlobalSettings {
  site_title: string;
  description: string;
  logo: string | null;
  social_links: Record<string, string>;
}

// Extend core DirectusUser type
export interface CustomUser {
  department: string;
  phone: string | null;
}
```

### Why Union Types for Relations?

Relational fields return either the raw foreign key (a number or string) or the fully populated object, depending on whether you included that relation in your `fields` query. The union type `number | Author` handles both cases:

```typescript
const post = await client.request(readItem('posts', 1));
// post.author is `number` (just the ID)

const post = await client.request(readItem('posts', 1, {
  fields: ['*', { author: ['name'] }],
}));
// post.author is `Author` (populated object)
```

---

## Relation Types

### Many-to-One (M2O)
A post belongs to one category. The `category` field stores the foreign key.
```typescript
interface Post {
  category: number | Category;
}
```

### One-to-Many (O2M)
A category has many posts. Represented as an array on the "one" side:
```typescript
interface Category {
  id: number;
  name: string;
  posts: number[] | Post[];  // O2M — array of IDs or populated items
}
```

### Many-to-Many (M2M)
Posts and tags, linked through a junction collection:
```typescript
// In the schema root:
export interface Schema {
  posts: Post[];
  tags: Tag[];
  posts_tags: PostTag[];  // junction MUST be listed
}

interface Post {
  tags: number[] | PostTag[];  // references the junction
}

interface PostTag {
  id: number;
  posts_id: number | Post;
  tags_id: number | Tag;
}
```

Fetching M2M data:
```typescript
const posts = await client.request(readItems('posts', {
  fields: ['title', { tags: [{ tags_id: ['name', 'slug'] }] }],
}));

// Access: posts[0].tags[0].tags_id.name
```

### Many-to-Any (M2A)
Used for page builder / block patterns where a field can reference items from multiple collections:
```typescript
interface Page {
  id: number;
  title: string;
  blocks: PageBlock[];
}

interface PageBlock {
  id: number;
  collection: string;          // which collection this block comes from
  item: BlockHero | BlockText | BlockGallery;
  sort: number;
}

interface BlockHero {
  headline: string;
  content: string;
  image: string | null;
}

interface BlockText {
  body: string;
}

interface BlockGallery {
  title: string;
  images: string[];
}
```

---

## Singleton Collections

Singletons store exactly one item — used for global settings, homepage content, etc. Define them as a singular type (not an array) in the schema:

```typescript
export interface Schema {
  global_settings: GlobalSettings;  // singular = singleton
  posts: Post[];                     // array = regular collection
}
```

Read/update singletons with dedicated functions:
```typescript
import { readSingleton, updateSingleton } from '@directus/sdk';

const settings = await client.request(readSingleton('global_settings'));
await client.request(updateSingleton('global_settings', {
  site_title: 'New Title',
}));
```

---

## Extending Core Collections

Directus has built-in system collections (`directus_users`, `directus_files`, etc.). If you add custom fields to these, include them in your schema:

```typescript
export interface Schema {
  // Your custom fields on directus_users (singular type)
  directus_users: CustomUser;
}

interface CustomUser {
  // Only YOUR custom fields go here — Directus provides the rest
  department: string;
  employee_id: string;
  phone: string | null;
}
```

---

## Automatic Type Generation

Rather than writing types by hand, you can generate them from a running Directus instance.

### Using directus-typescript-gen
```bash
npx directus-typescript-gen \
  --host http://localhost:8055 \
  --email [email protected] \
  --password your-password \
  --outFile src/lib/schema.d.ts
```

This produces a complete schema file matching your current data model.

### Using the Directus CMS Starter's Built-In Script
The official Astro CMS starter includes a type generation script:
```bash
pnpm run generate:types
```
This requires an admin token with permission to read system collections like `directus_fields`.

### Manual Approach via OpenAPI Spec
Export your OpenAPI spec from Directus (`GET /server/specs/oas`) and use it to derive types manually or with tools like `openapi-typescript`.

---

## Utility Types & Helpers

### Extract Item Type from Schema
```typescript
// Get the item type for a collection
type PostItem = Schema['posts'] extends (infer T)[] ? T : never;
// Result: Post
```

### Narrowing Relational Fields
After fetching with populated relations, narrow the type:
```typescript
function isPopulated<T>(value: number | T): value is T {
  return typeof value === 'object' && value !== null;
}

const post = await client.request(readItem('posts', 1, {
  fields: ['*', { author: ['name'] }],
}));

if (isPopulated(post.author)) {
  console.log(post.author.name); // TypeScript knows it's Author
}
```

### Creating Reusable Query Helpers
```typescript
// src/lib/queries.ts
import { readItems, readItem } from '@directus/sdk';
import directus from './directus';

export async function getPublishedPosts(limit = 20) {
  return directus.request(readItems('posts', {
    fields: ['id', 'slug', 'title', 'excerpt', 'date_created', 'cover_image',
      { author: ['name', 'avatar'] }],
    filter: { status: { _eq: 'published' } },
    sort: ['-date_created'],
    limit,
  }));
}

export async function getPostBySlug(slug: string) {
  const results = await directus.request(readItems('posts', {
    fields: ['*', { author: ['name', 'avatar'] }, { tags: [{ tags_id: ['name', 'slug'] }] }],
    filter: { slug: { _eq: slug }, status: { _eq: 'published' } },
    limit: 1,
  }));
  return results[0] ?? null;
}
```

---

## Typed SDK Requests

### Type Inference on Responses
The SDK infers return types based on your schema and the `fields` parameter. When you specify fields, the return type is narrowed:

```typescript
// Returns full Post type
const post = await client.request(readItem('posts', 1));

// Returns only { id: number; title: string }
const post = await client.request(readItem('posts', 1, {
  fields: ['id', 'title'],
}));
```

### Typing Custom Endpoints
When calling custom API endpoints, you can specify the return type:
```typescript
const result = await client.request<{ total: number; average: number }>(() => ({
  path: '/custom/stats',
  method: 'GET',
}));
```

---

## Working with JavaScript (Non-TS)

If you're not using TypeScript, the SDK still works — you just won't get type checking or autocompletion.

### JavaScript Setup
```javascript
// src/lib/directus.js
import { createDirectus, rest, staticToken } from '@directus/sdk';

const directus = createDirectus(process.env.DIRECTUS_URL)
  .with(staticToken(process.env.DIRECTUS_TOKEN))
  .with(rest());

export default directus;
```

### JavaScript CRUD
```javascript
import { readItems, createItem, updateItem, deleteItem } from '@directus/sdk';
import directus from './directus.js';

// Read
const posts = await directus.request(readItems('posts', {
  fields: ['id', 'title'],
  filter: { status: { _eq: 'published' } },
}));

// Create
const newPost = await directus.request(createItem('posts', {
  title: 'New Post',
  content: 'Hello world',
  status: 'draft',
}));

// Update
await directus.request(updateItem('posts', newPost.id, {
  status: 'published',
}));

// Delete
await directus.request(deleteItem('posts', newPost.id));
```

### JSDoc Type Hints (Optional)
You can add type hints in JS files without full TypeScript:
```javascript
/**
 * @typedef {Object} Post
 * @property {number} id
 * @property {string} title
 * @property {string} slug
 * @property {string} content
 * @property {'draft'|'published'|'archived'} status
 */

/** @type {import('@directus/sdk').DirectusClient} */
const directus = createDirectus('https://example.com').with(rest());
```

FILE:references/astro-integration.md
# Directus + Astro Integration Guide

## Table of Contents
1. [Project Setup](#project-setup)
2. [Directus Client Configuration](#directus-client-configuration)
3. [Fetching & Displaying Data](#fetching--displaying-data)
4. [Dynamic Routes & Pages](#dynamic-routes--pages)
5. [Blog with Relational Data](#blog-with-relational-data)
6. [Images & Assets](#images--assets)
7. [Live Preview](#live-preview)
8. [Authentication in Astro](#authentication-in-astro)
9. [SSR vs SSG Considerations](#ssr-vs-ssg-considerations)
10. [Dynamic Blocks / Page Builder](#dynamic-blocks--page-builder)
11. [Real-Time with Astro](#real-time-with-astro)
12. [Multilingual Content](#multilingual-content)
13. [Deployment & Build Hooks](#deployment--build-hooks)

---

## Project Setup

### Create Astro Project
```bash
npm create astro@latest my-site
cd my-site
npm install @directus/sdk
```

### Environment Variables
Create `.env` in the project root:
```env
DIRECTUS_URL=https://your-directus-project.directus.app
DIRECTUS_TOKEN=your-static-token
```

For TypeScript projects, add types in `src/env.d.ts`:
```typescript
/// <reference types="astro/client" />
interface ImportMetaEnv {
  readonly DIRECTUS_URL: string;
  readonly DIRECTUS_TOKEN: string;
}
```

---

## Directus Client Configuration

Create `src/lib/directus.ts`:

### Public-Only Access (SSG)
```typescript
import { createDirectus, rest } from '@directus/sdk';

// Import your schema types
import type { Schema } from './schema';

const directus = createDirectus<Schema>(import.meta.env.DIRECTUS_URL)
  .with(rest());

export default directus;
```

### Token-Based Access (SSG/SSR)
```typescript
import { createDirectus, rest, staticToken } from '@directus/sdk';
import type { Schema } from './schema';

const directus = createDirectus<Schema>(import.meta.env.DIRECTUS_URL)
  .with(staticToken(import.meta.env.DIRECTUS_TOKEN))
  .with(rest());

export default directus;
```

### Authenticated Access (SSR Only)
```typescript
import { createDirectus, rest, authentication } from '@directus/sdk';
import type { Schema } from './schema';

const directus = createDirectus<Schema>(import.meta.env.DIRECTUS_URL)
  .with(authentication('cookie'))
  .with(rest());

export default directus;
```

---

## Fetching & Displaying Data

### Basic Page with Singleton Data
```astro
---
// src/pages/index.astro
import Layout from '../layouts/Layout.astro';
import directus from '../lib/directus';
import { readSingleton } from '@directus/sdk';

const global = await directus.request(readSingleton('global'));
---

<Layout title={global.site_title}>
  <h1>{global.site_title}</h1>
  <p>{global.description}</p>
</Layout>
```

### List Page
```astro
---
// src/pages/blog/index.astro
import Layout from '../../layouts/Layout.astro';
import directus from '../../lib/directus';
import { readItems } from '@directus/sdk';

const posts = await directus.request(readItems('posts', {
  fields: ['id', 'slug', 'title', 'excerpt', 'date_created', 'cover_image'],
  filter: { status: { _eq: 'published' } },
  sort: ['-date_created'],
  limit: 20,
}));
---

<Layout title="Blog">
  <h1>Blog</h1>
  <ul>
    {posts.map((post) => (
      <li>
        <a href={`/blog/post.slug`}>
          <h2>{post.title}</h2>
          <p>{post.excerpt}</p>
          <time>{new Date(post.date_created).toLocaleDateString()}</time>
        </a>
      </li>
    ))}
  </ul>
</Layout>
```

---

## Dynamic Routes & Pages

### Static Generation (SSG) — `[slug].astro`
```astro
---
// src/pages/[slug].astro
import Layout from '../layouts/Layout.astro';
import directus from '../lib/directus';
import { readItems } from '@directus/sdk';

export async function getStaticPaths() {
  const pages = await directus.request(readItems('pages', {
    fields: ['slug', 'title', 'content'],
    filter: { status: { _eq: 'published' } },
  }));

  return pages.map((page) => ({
    params: { slug: page.slug },
    props: page,
  }));
}

const page = Astro.props;
---

<Layout title={page.title}>
  <h1>{page.title}</h1>
  <div set:html={page.content} />
</Layout>
```

### Blog Post with Relational Author
```astro
---
// src/pages/blog/[slug].astro
import Layout from '../../layouts/Layout.astro';
import directus from '../../lib/directus';
import { readItems } from '@directus/sdk';

export async function getStaticPaths() {
  const posts = await directus.request(readItems('posts', {
    fields: [
      'slug', 'title', 'content', 'date_created', 'cover_image',
      { author: ['name', 'avatar'] },
    ],
    filter: { status: { _eq: 'published' } },
  }));

  return posts.map((post) => ({
    params: { slug: post.slug },
    props: post,
  }));
}

const post = Astro.props;
const DIRECTUS_URL = import.meta.env.DIRECTUS_URL;
---

<Layout title={post.title}>
  {post.cover_image && (
    <img
      src={`DIRECTUS_URL/assets/post.cover_image?width=1200&format=webp`}
      alt={post.title}
    />
  )}
  <h1>{post.title}</h1>
  {typeof post.author === 'object' && post.author && (
    <p>By {post.author.name}</p>
  )}
  <time>{new Date(post.date_created).toLocaleDateString()}</time>
  <article set:html={post.content} />
</Layout>
```

### SSR Dynamic Route (No getStaticPaths)
For SSR mode (`output: 'server'` in astro.config), you don't need `getStaticPaths`:
```astro
---
// src/pages/blog/[slug].astro (SSR mode)
import directus from '../../lib/directus';
import { readItems } from '@directus/sdk';

const { slug } = Astro.params;

const posts = await directus.request(readItems('posts', {
  fields: ['*', { author: ['name'] }],
  filter: { slug: { _eq: slug }, status: { _eq: 'published' } },
  limit: 1,
}));

if (!posts.length) return Astro.redirect('/404');
const post = posts[0];
---
```

---

## Blog with Relational Data

### Many-to-Many Tags
Directus M2M relations use a junction collection (e.g., `posts_tags`).

```astro
---
const posts = await directus.request(readItems('posts', {
  fields: [
    'id', 'title', 'slug',
    { tags: [{ tags_id: ['name', 'slug'] }] },
  ],
  filter: { status: { _eq: 'published' } },
}));
---

{posts.map((post) => (
  <article>
    <h2>{post.title}</h2>
    <div class="tags">
      {post.tags?.map((junction) => (
        <span>{junction.tags_id.name}</span>
      ))}
    </div>
  </article>
))}
```

### Filter by Tag
```astro
---
// src/pages/tags/[tag].astro
export async function getStaticPaths() {
  const tags = await directus.request(readItems('tags', {
    fields: ['slug', 'name'],
  }));

  return tags.map((tag) => ({
    params: { tag: tag.slug },
    props: { tag },
  }));
}

const { tag } = Astro.props;

const posts = await directus.request(readItems('posts', {
  fields: ['slug', 'title'],
  filter: {
    tags: {
      tags_id: { slug: { _eq: tag.slug } },
    },
  },
}));
---
```

---

## Images & Assets

### Rendering Directus Images
```astro
---
const DIRECTUS_URL = import.meta.env.DIRECTUS_URL;
---

<!-- Basic image -->
<img src={`DIRECTUS_URL/assets/imageId`} alt="Description" />

<!-- With transformations -->
<img
  src={`DIRECTUS_URL/assets/imageId?width=800&height=450&fit=cover&format=webp&quality=80`}
  alt="Description"
  width="800"
  height="450"
  loading="lazy"
/>
```

### Responsive Image Helper
```typescript
// src/lib/image.ts
const DIRECTUS_URL = import.meta.env.DIRECTUS_URL;

export function getImageUrl(
  id: string,
  options?: { width?: number; height?: number; fit?: string; format?: string; quality?: number }
) {
  const params = new URLSearchParams();
  if (options?.width) params.set('width', String(options.width));
  if (options?.height) params.set('height', String(options.height));
  if (options?.fit) params.set('fit', options.fit);
  if (options?.format) params.set('format', options.format);
  if (options?.quality) params.set('quality', String(options.quality));

  const query = params.toString();
  return `DIRECTUS_URL/assets/idquery ? `?${query` : ''}`;
}
```

---

## Live Preview

### Directus Config
In Settings → Data Model → your collection, enable Live Preview with URL pattern:
```
http://localhost:4321/{slug}?preview=true
```

### Astro Config (requires SSR)
```javascript
// astro.config.mjs
import { defineConfig } from 'astro/config';

export default defineConfig({
  output: 'server',
});
```

### Preview-Aware Page
```astro
---
// src/pages/blog/[slug].astro
import directus from '../../lib/directus';
import { readItems, withToken } from '@directus/sdk';

const { slug } = Astro.params;
const preview = Astro.url.searchParams.get('preview') === 'true';
const token = Astro.url.searchParams.get('token');

let filter: any = {
  slug: { _eq: slug },
};

// In preview mode, don't filter by status
if (!preview) {
  filter.status = { _eq: 'published' };
}

let request = readItems('posts', {
  fields: ['*'],
  filter,
  limit: 1,
});

// Use token for draft content access
const posts = token
  ? await directus.request(withToken(token, request))
  : await directus.request(request);

if (!posts.length) return Astro.redirect('/404');
const post = posts[0];
---
```

### Visual Editing Support
Install the visual editing library for real-time field editing:
```bash
npm install @directus/visual-editing
```

---

## Authentication in Astro

### SSR Login Flow
```astro
---
// src/pages/login.astro
import directus from '../lib/directus';

if (Astro.request.method === 'POST') {
  const formData = await Astro.request.formData();
  const email = formData.get('email') as string;
  const password = formData.get('password') as string;

  try {
    const result = await directus.login(email, password);
    // Store tokens in cookies
    Astro.cookies.set('access_token', result.access_token, {
      httpOnly: true,
      secure: true,
      path: '/',
      maxAge: 60 * 15, // 15 minutes
    });
    Astro.cookies.set('refresh_token', result.refresh_token, {
      httpOnly: true,
      secure: true,
      path: '/',
      maxAge: 60 * 60 * 24 * 7, // 7 days
    });
    return Astro.redirect('/dashboard');
  } catch (e) {
    // handle error
  }
}
---

<form method="POST">
  <input type="email" name="email" required />
  <input type="password" name="password" required />
  <button type="submit">Login</button>
</form>
```

### Protected Routes via Middleware
```typescript
// src/middleware.ts
import { defineMiddleware } from 'astro:middleware';
import { createDirectus, rest, withToken, readMe } from '@directus/sdk';

export const onRequest = defineMiddleware(async (context, next) => {
  const protectedPaths = ['/dashboard', '/profile'];
  const isProtected = protectedPaths.some(p => context.url.pathname.startsWith(p));

  if (!isProtected) return next();

  const token = context.cookies.get('access_token')?.value;
  if (!token) return context.redirect('/login');

  try {
    const client = createDirectus(import.meta.env.DIRECTUS_URL).with(rest());
    const user = await client.request(withToken(token, readMe()));
    context.locals.user = user;
    return next();
  } catch {
    return context.redirect('/login');
  }
});
```

---

## SSR vs SSG Considerations

| Feature | SSG (Static) | SSR (Server) |
|---|---|---|
| Config | `output: 'static'` (default) | `output: 'server'` |
| Data freshness | Build-time only | Every request |
| Dynamic routes | Requires `getStaticPaths()` | Uses `Astro.params` directly |
| Auth / login flows | Not supported | Supported |
| Live preview | Not supported | Supported |
| Hosting | Any static host | Node.js server, serverless, or edge |
| Build hooks | Rebuild on content change | Not needed |

### Hybrid Mode
Astro supports `output: 'hybrid'` where pages default to static but individual pages can opt into SSR:
```astro
---
// This specific page renders on the server
export const prerender = false;
---
```

---

## Dynamic Blocks / Page Builder

Directus Many-to-Any (M2A) relationships enable page-builder patterns where pages contain ordered blocks of different types.

### Data Model
- Collection `pages`: slug, title
- Collection `block_hero`: headline, content, image, buttons (JSON)
- Collection `block_richtext`: content (WYSIWYG)
- Collection `block_gallery`: title, images (M2M to directus_files)
- Junction: `pages_blocks` (M2A linking pages → various block collections)

### Astro Implementation
```astro
---
// src/pages/[slug].astro
import directus from '../lib/directus';
import { readItems } from '@directus/sdk';
import Hero from '../components/blocks/Hero.astro';
import RichText from '../components/blocks/RichText.astro';
import Gallery from '../components/blocks/Gallery.astro';

export async function getStaticPaths() {
  const pages = await directus.request(readItems('pages', {
    fields: [
      'slug', 'title',
      {
        blocks: [
          'id', 'collection', 'sort',
          { item: { block_hero: ['headline', 'content', 'image', 'buttons'] } },
          { item: { block_richtext: ['content'] } },
          { item: { block_gallery: ['title', { images: [{ directus_files_id: ['id', 'title'] }] }] } },
        ],
      },
    ],
  }));

  return pages.map((page) => ({
    params: { slug: page.slug },
    props: page,
  }));
}

const page = Astro.props;

const blockComponents: Record<string, any> = {
  block_hero: Hero,
  block_richtext: RichText,
  block_gallery: Gallery,
};
---

<h1>{page.title}</h1>
{page.blocks
  ?.sort((a, b) => a.sort - b.sort)
  .map((block) => {
    const Component = blockComponents[block.collection];
    return Component ? <Component item={block.item} /> : null;
  })}
```

---

## Real-Time with Astro

Real-time features require client-side JavaScript. Use an Astro island:

```astro
---
// src/pages/live.astro
import LiveFeed from '../components/LiveFeed';
---

<LiveFeed client:only="react" />
```

```tsx
// src/components/LiveFeed.tsx (React island)
import { useEffect, useState } from 'react';
import { createDirectus, realtime } from '@directus/sdk';

export default function LiveFeed() {
  const [messages, setMessages] = useState([]);

  useEffect(() => {
    const client = createDirectus('https://directus.example.com')
      .with(realtime({ authMode: 'public' }));

    async function connect() {
      await client.connect();
      const { subscription } = await client.subscribe('messages', {
        query: { fields: ['id', 'text', 'date_created'] },
      });

      for await (const msg of subscription) {
        if (msg.event !== 'init') {
          setMessages(prev => [...prev, ...msg.data]);
        }
      }
    }

    connect();
  }, []);

  return (
    <ul>
      {messages.map((m) => <li key={m.id}>{m.text}</li>)}
    </ul>
  );
}
```

---

## Multilingual Content

Directus handles translations via a translations interface that creates a junction collection.

### Fetching Translated Content
```astro
---
const posts = await directus.request(readItems('posts', {
  fields: ['id', 'slug', { translations: ['title', 'content', 'languages_code'] }],
  deep: {
    translations: {
      _filter: { languages_code: { _eq: 'en-US' } },
    },
  },
}));
---

{posts.map((post) => {
  const t = post.translations?.[0];
  return t ? <h2>{t.title}</h2> : null;
})}
```

---

## Deployment & Build Hooks

### Triggering Astro Rebuilds on Content Change
Use Directus Flows to send a webhook when content is published:

1. Create a Flow with an Event Hook trigger on `items.create` / `items.update` for your content collections
2. Add a Condition operation to check `{{ $trigger.payload.status }}` equals `published`
3. Add a Web Request operation to POST to your hosting provider's deploy hook:
   - **Netlify**: `https://api.netlify.com/build_hooks/{hook-id}`
   - **Vercel**: `https://api.vercel.com/v1/integrations/deploy/{hook-id}`
   - **Cloudflare Pages**: Use their deploy hook URL

### CORS Configuration
If your Astro frontend and Directus backend are on different domains, configure CORS in Directus:
```env
CORS_ENABLED=true
CORS_ORIGIN=https://your-astro-site.com
CORS_METHODS=GET,POST,PATCH,DELETE
CORS_ALLOWED_HEADERS=Content-Type,Authorization
```

For local development:
```env
CORS_ENABLED=true
CORS_ORIGIN=http://localhost:4321
```

---
name: cognito
description: >
  Use this skill for ANY task involving AWS Cognito — user pools, identity pools, authentication flows,
  token handling, social/enterprise federation, MFA, Lambda triggers, hosted UI, or Cognito integration
  with API Gateway, AppSync, S3, DynamoDB, Amplify, or any AWS service. Trigger whenever the user mentions
  "Cognito", "user pool", "identity pool", "auth flow", "social login with AWS", "JWT tokens from AWS",
  "hosted UI", "managed login", "Cognito triggers", "OAuth with Cognito", "SAML federation", "MFA setup",
  "sign-up/sign-in", "password policy", "Cognito CDK", "Cognito CloudFormation", "Cognito Terraform",
  "Cognito SDK", "aws-amplify auth", "token refresh", "Cognito groups", "RBAC with Cognito",
  or any authentication/authorization task that could involve AWS Cognito — even if they don't name
  Cognito explicitly but describe a pattern it solves (e.g. "I need user auth for my AWS app").
---

# AWS Cognito Skill

This skill helps you build, configure, debug, and manage AWS Cognito resources — user pools, identity pools, app clients, Lambda triggers, federation, and integrations with other AWS services.

## Quick Decision: What Does the User Need?

1. **New Cognito setup from scratch** → Read `references/setup-guide.md`, then follow the setup workflow
2. **CDK / CloudFormation / Terraform IaC** → Read `references/iac-patterns.md` for production-ready templates
3. **Authentication flow implementation** → Read `references/auth-flows.md` for SDK code and flow selection
4. **Debugging / troubleshooting** → Read `references/troubleshooting.md` for common issues and fixes
5. **Lambda triggers** → Read `references/lambda-triggers.md` for trigger patterns
6. **Security hardening** → Read `references/security.md` for best practices

Read the relevant reference file(s) before generating any code or configuration. Multiple files may apply — for example, a new CDK setup would benefit from both `setup-guide.md` and `iac-patterns.md`.

## Core Concepts (Always Keep in Mind)

### User Pools vs Identity Pools

These are the two main Cognito components and they serve different purposes:

- **User Pool**: A user directory and OIDC identity provider. Handles sign-up, sign-in, MFA, token issuance (ID token, access token, refresh token), and federation with external IdPs. Think of it as "who is this user?"
- **Identity Pool** (Federated Identities): Exchanges tokens (from a user pool, social provider, SAML, or OIDC) for temporary AWS credentials (STS). Think of it as "what AWS resources can this user access?"

A common architecture uses both: User Pool authenticates the user and issues tokens → Identity Pool exchanges those tokens for AWS credentials → User accesses S3, DynamoDB, etc.

### Feature Plans (Pricing Tiers)

As of late 2024, Cognito uses feature plans instead of the old "advanced security" toggle:

- **Lite**: Low-cost, basic auth features. Good for simple apps with fewer MAUs.
- **Essentials** (default for new pools): All latest auth features including access-token customization and managed login.
- **Plus**: Everything in Essentials plus threat protection (adaptive auth, compromised credential detection).

Always ask the user which plan they need, or default to Essentials for new setups.

### Token Types

- **ID Token**: Contains user identity claims (email, name, groups, custom attributes). Use for identity verification on your backend.
- **Access Token**: Contains scopes and authorized actions. Use for API authorization (e.g., API Gateway Cognito Authorizer).
- **Refresh Token**: Long-lived token to obtain new ID/access tokens without re-authentication. Default validity is 30 days.

## Workflow: Building a Cognito Solution

### Step 1: Clarify Requirements

Before writing any code, determine:

- **Auth methods**: Username/password? Email-only? Phone? Social login (Google, Apple, Facebook)? Enterprise SAML/OIDC?
- **MFA**: Required, optional, or off? SMS, TOTP authenticator app, or email?
- **Self-service sign-up**: Enabled or admin-only user creation?
- **Token usage**: Frontend-only (SPA/mobile)? Backend API authorization? Direct AWS resource access?
- **IaC preference**: CDK (TypeScript/Python), CloudFormation, Terraform, or console/CLI?
- **Frontend framework**: React/Amplify, Next.js, Vue, mobile (iOS/Android), or custom?

### Step 2: Design the Architecture

Based on requirements, determine:

- User Pool configuration (sign-in aliases, attributes, password policy, MFA)
- App client(s) — public (no secret, for SPAs/mobile) vs confidential (with secret, for server-side)
- OAuth flows — Authorization Code (with PKCE for public clients), Implicit (legacy, avoid), Client Credentials (M2M)
- Whether an Identity Pool is needed (only if users need direct AWS resource access)
- Lambda triggers needed (pre-sign-up, post-confirmation, pre-token-generation, custom auth, etc.)
- Domain — Cognito-hosted prefix domain or custom domain

### Step 3: Implement

Read the appropriate reference files and generate code. Always:

- Use the latest CDK v2 constructs (`aws-cdk-lib/aws-cognito`) — never CDK v1
- For SDK code, use AWS SDK v3 (`@aws-sdk/client-cognito-identity-provider`) — never v2
- For frontend, prefer Amplify v6 (`aws-amplify`) patterns
- Include proper error handling and token refresh logic
- Set `RemovalPolicy.RETAIN` on user pools in production (data loss prevention)
- Never hardcode secrets — use environment variables or AWS Secrets Manager

### Step 4: Security Review

Before declaring done, verify against `references/security.md`:

- MFA is enabled (at least optional) for production
- Password policy meets requirements (minimum 8 chars, complexity rules)
- Token validity periods are reasonable
- WAF is considered for public-facing auth endpoints
- Least-privilege IAM for any Identity Pool roles
- Client secrets are used for confidential clients
- HTTPS-only callback URLs

## Common Patterns Quick Reference

### Cognito + API Gateway
Use a Cognito User Pool Authorizer on API Gateway. The access token is validated automatically. Scopes in the token control which API methods are accessible.

### Cognito + AppSync
Configure `AMAZON_COGNITO_USER_POOLS` authorization on your GraphQL API. Use `@auth` directives in your schema for fine-grained access control.

### Cognito + S3 (via Identity Pool)
User Pool → Identity Pool → IAM role with S3 permissions scoped to `sub/*` for per-user folders.

### Cognito + Lambda (Custom Auth)
Use `CUSTOM_AUTH` flow with Define, Create, and Verify Auth Challenge triggers for passwordless (magic link, OTP) or multi-step authentication.

### Machine-to-Machine (M2M)
Use Client Credentials grant with a resource server and custom scopes. No user interaction — one app authenticating to another.

## Important Reminders

- User pool attributes marked as required at creation CANNOT be changed later. Plan attributes carefully.
- Custom attributes are always prefixed with `custom:` (e.g., `custom:company`).
- The `sub` attribute is the unique, immutable user identifier. Use it as your primary key, not email or username.
- Email/phone verification is separate from sign-in aliases. Auto-verify what you use for sign-in.
- Cognito has service quotas (e.g., API request rate limits). For high-volume apps, request quota increases proactively.
- Lambda triggers execute synchronously and have a 5-second timeout. Keep them fast.

FILE:references/troubleshooting.md
# Cognito Troubleshooting Guide

Common issues, error messages, and how to fix them.

## Table of Contents
1. [Authentication Errors](#authentication-errors)
2. [Token Issues](#token-issues)
3. [Federation Problems](#federation-problems)
4. [Lambda Trigger Failures](#lambda-trigger-failures)
5. [SDK & Amplify Issues](#sdk-and-amplify-issues)
6. [Quota & Throttling](#quota-and-throttling)
7. [CDK / IaC Deployment Issues](#deployment-issues)

---

## Authentication Errors

### `NotAuthorizedException: Incorrect username or password`

**Causes**:
- Wrong credentials (obvious)
- User exists but is in `FORCE_CHANGE_PASSWORD` status (admin-created, hasn't set own password)
- User pool has case-sensitive usernames and the case doesn't match
- App client doesn't have `ALLOW_USER_SRP_AUTH` enabled

**Fixes**:
- Check user status in Cognito console → Users
- If `FORCE_CHANGE_PASSWORD`: use `AdminSetUserPassword` with `Permanent: true`, or handle the `NEW_PASSWORD_REQUIRED` challenge in your app
- Verify `signInCaseSensitive` setting on the user pool

### `UserNotConfirmedException`

User signed up but hasn't verified their email/phone.

**Fix**: Call `ResendConfirmationCode` to send a new code, or `AdminConfirmSignUp` to confirm manually.

### `UserNotFoundException`

**If `preventUserExistenceErrors` is ON**: You won't see this error — Cognito returns a generic error instead. This is correct behavior.

**If OFF**: The user doesn't exist. Check username spelling and whether the user pool is the right one.

### `InvalidParameterException: USER_SRP_AUTH is not enabled`

The app client doesn't have the SRP auth flow enabled.

**Fix**: Update the app client to include `ALLOW_USER_SRP_AUTH` in `ExplicitAuthFlows`.

### `PasswordResetRequiredException`

Admin called `AdminResetUserPassword`. The user must complete the forgot-password flow before they can sign in.

### `UserLambdaValidationException`

A Lambda trigger threw an error. The error message from the Lambda is included.

**Debug**: Check CloudWatch Logs for the Lambda function. The trigger name is in the error details.

---

## Token Issues

### `Token is expired`

ID and access tokens have a default 1-hour validity.

**Fix**: Implement token refresh logic. With Amplify, `fetchAuthSession()` auto-refreshes. With SDK, call `InitiateAuth` with `REFRESH_TOKEN_AUTH` flow.

### `Invalid token` / Signature verification fails

**Causes**:
- Using the wrong JWKS endpoint (wrong region or user pool ID)
- Token was issued by a different user pool
- Token was tampered with
- Clock skew between your server and Cognito

**Fix**: Verify you're using the correct JWKS URL: `https://cognito-idp.{region}.amazonaws.com/{userPoolId}/.well-known/jwks.json`. Check server clock with NTP.

### `Token use doesn't match`

You're validating an ID token where an access token is expected, or vice versa.

**Fix**: Check the `token_use` claim. ID tokens have `"token_use": "id"`, access tokens have `"token_use": "access"`.

### Refresh token expired

Default refresh token validity is 30 days. After that, the user must re-authenticate.

**Fix**: Increase `RefreshTokenValidity` if needed (max 10 years). Redirect user to sign-in when refresh fails.

---

## Federation Problems

### `Invalid identity provider` / Provider not found

**Causes**:
- The identity provider name in the sign-in request doesn't match what's configured in Cognito
- The provider hasn't been added to the app client's `SupportedIdentityProviders`

**Fix**: Check the provider name exactly matches (case-sensitive). Ensure the provider is listed in the app client's supported providers.

### SAML: `Invalid saml response received`

**Common causes**:
- Metadata is stale (IdP rotated certificates)
- ACS URL in IdP config doesn't match Cognito's endpoint
- Clock skew between IdP and Cognito (SAML assertions have a validity window)
- Attribute mapping mismatch

**Fix**: Re-upload metadata from IdP. Verify ACS URL is `https://{domain}/saml2/idpresponse`. Check IdP logs for the outgoing assertion.

### Social login: Redirect loop or blank page

**Causes**:
- Callback URL in Cognito doesn't match the actual redirect URL
- Scopes mismatch between what's configured in Cognito and the social provider
- Social provider app isn't in "production" mode (e.g., Facebook app in development mode only works for test users)

**Fix**: Verify callback URLs match exactly (including trailing slashes). Check social provider's developer console for app status.

### `Already found an entry for username` (linking accounts)

A user tried to sign in with a social provider but an account with the same email already exists.

**Fix**: Implement account linking in a Pre Sign-Up trigger. When `triggerSource` is `PreSignUp_ExternalProvider`, call `AdminLinkProviderForUser` to link the social identity to the existing account.

---

## Lambda Trigger Failures

### `UserLambdaValidationException` with no useful message

**Debug steps**:
1. Go to CloudWatch Logs → find the log group for your Lambda
2. Check for runtime errors, timeouts, or unhandled exceptions
3. If no logs exist, the Lambda may not have permission to write to CloudWatch

### Lambda times out (5-second limit)

Cognito trigger Lambdas have a hard 5-second timeout.

**Fix**:
- Minimize cold starts: use provisioned concurrency or keep Lambdas warm
- Move slow operations (sending emails, complex DB queries) to async processes (SQS, EventBridge)
- Keep the trigger minimal — return quickly, process asynchronously

### `AccessDeniedException` when trigger calls Cognito API

The Lambda's execution role doesn't have permission to call Cognito APIs.

**Fix**: Add `cognito-idp:Admin*` permissions (scoped to your user pool ARN) to the Lambda's IAM role.

### Trigger not firing

**Causes**:
- Lambda isn't attached to the user pool (check Cognito console → User pool properties → Lambda triggers)
- Cognito doesn't have permission to invoke the Lambda
- The trigger source doesn't match (e.g., `PreSignUp_SignUp` vs `PreSignUp_ExternalProvider`)

**Fix**: Verify the Lambda trigger is configured. Add a resource-based policy on the Lambda allowing `cognito-idp.amazonaws.com` to invoke it.

---

## SDK & Amplify Issues

### Amplify: `No Cognito User Pool configuration`

Amplify isn't configured or the config is missing.

**Fix**: Ensure `Amplify.configure()` is called before any auth operations, typically in your app's entry point (`main.ts`, `App.tsx`, `_app.tsx`).

### Amplify: CORS errors

Cognito's hosted endpoints don't have CORS issues themselves, but your app might.

**Causes**:
- Calling Cognito APIs directly from the browser instead of using the SDK
- API Gateway behind Cognito doesn't have CORS configured
- OAuth redirect issues with different origins

**Fix**: Always use the Amplify SDK (never call Cognito endpoints directly from the browser). Configure CORS on API Gateway.

### SDK v3: `Missing credentials in config`

For admin operations (like `AdminInitiateAuth`), the SDK needs AWS credentials.

**Fix**: Ensure your server has valid AWS credentials (environment variables, IAM role, or credentials file). Client-side code should use Cognito's unauthenticated operations (`SignUp`, `InitiateAuth`) which don't require AWS credentials.

### `CredentialProviderError` in local development

**Fix**: Run `aws configure` or set `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables. For local Lambda development, use `aws-sdk-client-mock` for testing.

---

## Quota & Throttling

### `TooManyRequestsException` / `LimitExceededException`

Cognito has rate limits on API operations.

**Default quotas** (vary by operation and region):
- `UserCreation` (sign-up): 50/sec
- `UserAuthentication` (sign-in): 120/sec
- `UserRead` (get user): 120/sec
- `UserToken` (token refresh): 120/sec
- `UserResourceRead` (list users, etc.): 30/sec

**Fix**:
- Implement exponential backoff with jitter
- Request quota increases via Service Quotas console
- Cache user data instead of re-fetching from Cognito
- Use CloudWatch to identify which operations are being throttled

### Email sending quota

Default Cognito email: 50 emails/day.

**Fix**: Switch to SES. Even SES sandbox is limited to verified addresses — request production access for real usage.

---

## Deployment Issues

### CDK: `User pool already has a domain configured`

You can't have two domains on one user pool.

**Fix**: Delete the existing domain first (`aws cognito-idp delete-user-pool-domain`) or import it into your CDK stack.

### CDK: Circular dependency between user pool and Lambda trigger

**Fix**: Use `addTrigger` method after both resources are created, or break into separate stacks with cross-stack references.

### CloudFormation: User pool replacement (data loss!)

Certain property changes cause CloudFormation to replace the user pool (deleting all users).

**Dangerous changes** (trigger replacement):
- Changing `UsernameAttributes`
- Changing `AliasAttributes`
- Changing schema (adding/removing required attributes)

**Safe changes** (in-place update):
- Password policy
- MFA configuration
- Lambda triggers
- Email configuration
- App client changes

**Protection**: Always set `DeletionPolicy: Retain` and `UpdateReplacePolicy: Retain` on user pool resources.

### Terraform: `SchemaAttribute cannot be modified`

Cognito doesn't allow removing or modifying existing schema attributes.

**Fix**: Add new attributes, don't modify existing ones. If you must change a required attribute, you need a new user pool (and user migration).

### Custom domain: `One or more parameter values are not valid`

**Common cause**: ACM certificate is not in `us-east-1`, or the certificate isn't validated yet.

**Fix**: Create the ACM certificate in `us-east-1` (regardless of where the user pool is). Wait for DNS validation to complete before deploying.

FILE:references/setup-guide.md
# Cognito Setup Guide

Step-by-step guidance for creating Cognito resources from scratch.

## Table of Contents
1. [User Pool Setup](#user-pool-setup)
2. [App Client Configuration](#app-client-configuration)
3. [Domain Setup](#domain-setup)
4. [Identity Pool Setup](#identity-pool-setup)
5. [Federation (Social & Enterprise)](#federation)
6. [MFA Configuration](#mfa-configuration)

---

## User Pool Setup

### Minimum Viable User Pool

Every user pool needs at minimum:
- Sign-in aliases (how users identify themselves)
- A password policy
- At least one app client

**Sign-in alias options** — choose carefully, these can't change after creation:
- `email` — most common for B2C apps
- `phone` — common in regions where phone is primary
- `username` — custom username, often combined with email as a verified contact
- `preferredUsername` — user-changeable display name (not unique)

**Standard attributes** — mark required ones at creation time (immutable decision):
- Common required: `email`, `name`
- Common optional: `phone_number`, `picture`, `address`, `birthdate`
- Only mark as required what you truly need at sign-up time

**Custom attributes** — up to 50, always prefixed `custom:`:
- String, Number, DateTime, Boolean types
- Can be mutable or immutable
- Cannot be made required (only standard attributes can be required)
- Cannot be removed once created (can only add new ones)

### Password Policy

Recommended production defaults:
- Minimum length: 8 (Cognito minimum is 6, but 8+ is best practice)
- Require at least: lowercase, uppercase, digit
- Symbols: optional but recommended
- Temporary password validity: 7 days (for admin-created accounts)

### Account Recovery

Options (pick one or combine):
- `EMAIL_ONLY` — most common, sends code to verified email
- `PHONE_ONLY_WITHOUT_MFA` — sends code to verified phone
- `PHONE_AND_EMAIL` — tries phone first, falls back to email
- `NONE` — no self-service recovery (admin must reset)

### Verification

Auto-verify attributes that are used for sign-in. If email is a sign-in alias, auto-verify email. If phone, auto-verify phone.

**Email verification methods**:
- `CODE` — sends a numeric code, user enters it (default, recommended)
- `LINK` — sends a clickable verification link

**Email sending options**:
- Default (Cognito-managed): Limited to 50 emails/day. Fine for dev/testing.
- Amazon SES: Required for production. Configure a verified SES identity and connect it to the user pool. Allows custom From addresses, higher quotas, and delivery tracking.

---

## App Client Configuration

### Public vs Confidential Clients

**Public client** (no client secret):
- For SPAs (React, Vue, Angular) and mobile apps
- Cannot securely store a client secret
- Use Authorization Code flow with PKCE
- Enable: `ALLOW_USER_SRP_AUTH` for direct sign-in, `ALLOW_REFRESH_TOKEN_AUTH` for token refresh

**Confidential client** (with client secret):
- For server-side apps (Node.js, Python, Java backends)
- Client secret is stored securely on the server
- Can use Authorization Code flow (with or without PKCE) or Client Credentials (M2M)
- Enable: `ALLOW_USER_SRP_AUTH`, `ALLOW_REFRESH_TOKEN_AUTH`

### Auth Flows to Enable

- `ALLOW_USER_SRP_AUTH` — Secure Remote Password, the standard username/password flow. Almost always enabled.
- `ALLOW_REFRESH_TOKEN_AUTH` — Required for token refresh. Almost always enabled.
- `ALLOW_USER_PASSWORD_AUTH` — Sends password in plaintext (over HTTPS). Use only if SRP isn't feasible (e.g., migration triggers).
- `ALLOW_CUSTOM_AUTH` — For Lambda-based custom authentication (passwordless, etc.).
- `ALLOW_ADMIN_USER_PASSWORD_AUTH` — Server-side admin auth. Requires IAM credentials.

### Token Validity

Recommended defaults:
- ID token: 1 hour (range: 5 min to 1 day)
- Access token: 1 hour (range: 5 min to 1 day)
- Refresh token: 30 days (range: 1 hour to 10 years)

For high-security apps, shorten ID/access to 15-30 minutes.

### OAuth Configuration

**Scopes**:
- `openid` — required for OIDC, returns ID token
- `email` — includes email and email_verified in ID token
- `phone` — includes phone_number and phone_number_verified
- `profile` — includes name, family_name, etc.
- `aws.cognito.signin.user.admin` — allows user to manage their own profile via Cognito API
- Custom scopes — defined via resource servers

**Callback URLs**: Where Cognito redirects after login. Must be HTTPS in production (localhost allowed for dev).

**Logout URLs**: Where Cognito redirects after logout.

---

## Domain Setup

Required for hosted UI / managed login and OAuth endpoints.

**Cognito-managed domain**: `https://<prefix>.auth.<region>.amazoncognito.com`
- Quick to set up, no SSL cert needed
- Use for dev/staging

**Custom domain**: `https://auth.yourdomain.com`
- Requires an ACM certificate in us-east-1
- Requires a CNAME DNS record pointing to the Cognito CloudFront distribution
- Use for production (brand consistency)

---

## Identity Pool Setup

Only needed when users require temporary AWS credentials to access AWS services directly (S3, DynamoDB, etc.).

### Configuration Decisions

- **Guest access**: Enable only if unauthenticated users need AWS credentials. Keep permissions extremely minimal.
- **Authentication providers**: Connect your user pool, social providers, SAML providers, or OIDC providers.
- **Role resolution**: Choose between "Use default role" and "Choose role with rules" (for attribute-based access control).

### IAM Roles

**Authenticated role**: Permissions granted to signed-in users.
**Unauthenticated role** (if guest access enabled): Permissions for anonymous users.

Always include trust policy conditions:
- `cognito-identity.amazonaws.com:aud` must match your identity pool ID
- `cognito-identity.amazonaws.com:amr` must be `authenticated` or `unauthenticated`

### Per-User Resource Scoping

Common pattern — scope S3 access to a per-user folder:
```json
{
  "Effect": "Allow",
  "Action": ["s3:GetObject", "s3:PutObject"],
  "Resource": "arn:aws:s3:::my-bucket/sub/*"
}
```

---

## Federation

### Social Identity Providers

**Google**:
1. Create OAuth 2.0 credentials in Google Cloud Console
2. Configure in Cognito: client ID, client secret, authorized scopes (`openid email profile`)
3. Attribute mapping: Google `sub` → Cognito `username`, `email` → `email`, `name` → `name`

**Facebook**:
1. Create a Facebook App in Meta Developer Portal
2. Configure: app ID, app secret, authorized scopes (`public_profile,email`)
3. Attribute mapping: `id` → `username`, `email` → `email`, `name` → `name`

**Apple**:
1. Configure Sign in with Apple in Apple Developer account
2. Need: Services ID, Team ID, Key ID, Private Key
3. Attribute mapping: `sub` → `username`, `email` → `email`, `name` → `name`

### SAML Federation

1. Get metadata XML or URL from your IdP (Okta, Azure AD, ADFS, etc.)
2. Create a SAML identity provider in the user pool
3. Map SAML attributes to Cognito attributes
4. Configure the IdP with Cognito's SP metadata (entity ID and ACS URL from the user pool)

### OIDC Federation

1. Get issuer URL, client ID, and client secret from the OIDC provider
2. Create an OIDC identity provider in the user pool
3. Map OIDC claims to Cognito attributes

---

## MFA Configuration

### Options

- **OFF**: No MFA (not recommended for production)
- **OPTIONAL**: Users can enable MFA but aren't required to
- **REQUIRED**: All users must set up MFA

### MFA Methods

- **SMS**: Sends code via text message. Requires an IAM role for SNS. Vulnerable to SIM swapping — use TOTP when possible.
- **TOTP**: Authenticator app (Google Authenticator, Authy, etc.). More secure than SMS. Recommended as primary MFA method.
- **Email**: Sends code via email. Available on Essentials and Plus plans.

### Adaptive Authentication (Plus plan only)

Cognito evaluates risk for each sign-in attempt and can:
- Allow (low risk)
- Require MFA (medium risk)
- Block (high risk)

Risk factors include: new device, unusual location, impossible travel, compromised credentials.

### Device Tracking

- **Always**: Remember all devices
- **User opt-in**: User chooses to remember a device
- **Off**: No device tracking

Remembered devices can skip MFA on subsequent sign-ins.

FILE:references/lambda-triggers.md
# Cognito Lambda Triggers

Lambda triggers let you customize Cognito's behavior at key points in the authentication lifecycle.

## Table of Contents
1. [Trigger Overview](#trigger-overview)
2. [Pre Sign-Up](#pre-sign-up)
3. [Post Confirmation](#post-confirmation)
4. [Pre Authentication](#pre-authentication)
5. [Post Authentication](#post-authentication)
6. [Pre Token Generation](#pre-token-generation)
7. [Custom Message](#custom-message)
8. [User Migration](#user-migration)
9. [Custom Auth Triggers](#custom-auth-triggers)
10. [CDK Wiring](#cdk-wiring)

---

## Trigger Overview

| Trigger | When It Fires | Common Use Cases |
|---------|--------------|------------------|
| Pre Sign-Up | Before a new user is registered | Auto-confirm users, validate email domains, block disposable emails |
| Post Confirmation | After user confirms their account | Create user record in DynamoDB, send welcome email, add to default group |
| Pre Authentication | Before credentials are validated | Custom validation, rate limiting, block certain users |
| Post Authentication | After successful authentication | Log sign-in events, update last-login timestamp |
| Pre Token Generation | Before tokens are issued | Add custom claims, modify group claims, inject tenant info |
| Custom Message | Before Cognito sends an SMS/email | Customize verification emails, localize messages |
| User Migration | When user signs in but doesn't exist in pool | Migrate users from legacy auth system on-demand |
| Define Auth Challenge | During custom auth flow | Control challenge sequence |
| Create Auth Challenge | During custom auth flow | Generate OTP, magic link |
| Verify Auth Challenge | During custom auth flow | Validate user's challenge response |

**Critical constraints**:
- All triggers have a **5-second timeout**. Keep them fast.
- Triggers run **synchronously** — they block the auth flow until complete.
- The Lambda execution role needs `cognito-idp:*` only if the trigger calls Cognito APIs.

---

## Pre Sign-Up

Fires before Cognito creates the user. You can auto-confirm, auto-verify, or reject the sign-up.

```typescript
export const handler = async (event: any) => {
  const email = event.request.userAttributes.email;

  // Block disposable email domains
  const disposableDomains = ['tempmail.com', 'throwaway.email', 'guerrillamail.com'];
  const domain = email.split('@')[1];
  if (disposableDomains.includes(domain)) {
    throw new Error('Disposable email addresses are not allowed.');
  }

  // Auto-confirm users from your corporate domain
  if (domain === 'yourcompany.com') {
    event.response.autoConfirmUser = true;
    event.response.autoVerifyEmail = true;
  }

  return event;
};
```

### Auto-Confirm and Link Federated Users

When a social login user already has a native account with the same email:

```typescript
export const handler = async (event: any) => {
  // If this is a federated sign-up (external provider), auto-confirm
  if (event.triggerSource === 'PreSignUp_ExternalProvider') {
    event.response.autoConfirmUser = true;
    event.response.autoVerifyEmail = true;
  }
  return event;
};
```

---

## Post Confirmation

Fires after the user confirms their account (or is auto-confirmed). Safe place for side effects.

```typescript
import { DynamoDBClient, PutItemCommand } from '@aws-sdk/client-dynamodb';

const dynamo = new DynamoDBClient({});

export const handler = async (event: any) => {
  // Create user record in DynamoDB
  await dynamo.send(new PutItemCommand({
    TableName: process.env.USERS_TABLE!,
    Item: {
      pk: { S: `USER#event.request.userAttributes.sub` },
      sk: { S: 'PROFILE' },
      email: { S: event.request.userAttributes.email },
      name: { S: event.request.userAttributes.name || '' },
      createdAt: { S: new Date().toISOString() },
    },
    ConditionExpression: 'attribute_not_exists(pk)', // Idempotent
  }));

  return event;
};
```

---

## Pre Authentication

Fires before Cognito validates credentials. Use for custom blocklisting or rate limiting.

```typescript
export const handler = async (event: any) => {
  const email = event.request.userAttributes.email;

  // Block specific users
  const blockedUsers = await getBlockedUsers(); // Your logic
  if (blockedUsers.includes(email)) {
    throw new Error('Your account has been suspended. Contact support.');
  }

  return event;
};
```

---

## Post Authentication

Fires after successful authentication. Good for audit logging.

```typescript
export const handler = async (event: any) => {
  console.log(JSON.stringify({
    event: 'user_sign_in',
    userId: event.request.userAttributes.sub,
    email: event.request.userAttributes.email,
    source: event.triggerSource,
    timestamp: new Date().toISOString(),
  }));

  return event;
};
```

---

## Pre Token Generation

Fires before tokens are created. Add or modify claims in the ID and access tokens.

### V2 Trigger (Essentials/Plus plans — recommended)

```typescript
export const handler = async (event: any) => {
  // Add custom claims to the ID token
  event.response.claimsAndScopeOverrideDetails = {
    idTokenGeneration: {
      claimsToAddOrOverride: {
        'custom:tenant': 'acme-corp',
        'custom:permissions': JSON.stringify(['read', 'write']),
      },
      claimsToSuppress: ['email_verified'], // Remove claims you don't want exposed
    },
    accessTokenGeneration: {
      claimsToAddOrOverride: {
        'custom:role': 'admin',
      },
      scopesToAdd: ['custom-scope'],
      scopesToSuppress: [],
    },
  };

  return event;
};
```

### V1 Trigger (Lite plan)

```typescript
export const handler = async (event: any) => {
  // V1 can only modify ID token claims and group overrides
  event.response.claimsOverrideDetails = {
    claimsToAddOrOverride: {
      'custom:tenant': 'acme-corp',
    },
    groupsToOverride: ['admin', 'users'], // Override cognito:groups claim
  };

  return event;
};
```

---

## Custom Message

Customize the content of emails and SMS messages Cognito sends.

```typescript
export const handler = async (event: any) => {
  const { codeParameter, usernameParameter } = event.request;

  switch (event.triggerSource) {
    case 'CustomMessage_SignUp':
      event.response.emailSubject = 'Welcome! Confirm your account';
      event.response.emailMessage = `
        <h1>Welcome to Our App!</h1>
        <p>Your verification code is: <strong>codeParameter</strong></p>
      `;
      break;

    case 'CustomMessage_ForgotPassword':
      event.response.emailSubject = 'Reset your password';
      event.response.emailMessage = `
        <p>Your password reset code is: <strong>codeParameter</strong></p>
        <p>This code expires in 1 hour.</p>
      `;
      break;

    case 'CustomMessage_ResendCode':
      event.response.emailSubject = 'Your new verification code';
      event.response.emailMessage = `
        <p>Your new code is: <strong>codeParameter</strong></p>
      `;
      break;
  }

  return event;
};
```

---

## User Migration

Migrate users on-demand from a legacy auth system. Fires when a user tries to sign in but doesn't exist in the pool.

```typescript
export const handler = async (event: any) => {
  if (event.triggerSource === 'UserMigration_Authentication') {
    // Validate against your legacy system
    const legacyUser = await legacyAuth(
      event.userName,
      event.request.password
    );

    if (legacyUser) {
      event.response.userAttributes = {
        email: legacyUser.email,
        email_verified: 'true',
        name: legacyUser.name,
        'custom:legacyId': legacyUser.id,
      };
      event.response.finalUserStatus = 'CONFIRMED';
      event.response.messageAction = 'SUPPRESS'; // Don't send welcome email
    } else {
      throw new Error('Bad credentials');
    }
  }

  if (event.triggerSource === 'UserMigration_ForgotPassword') {
    const legacyUser = await legacyLookup(event.userName);
    if (legacyUser) {
      event.response.userAttributes = {
        email: legacyUser.email,
        email_verified: 'true',
      };
      event.response.messageAction = 'SUPPRESS';
    }
  }

  return event;
};
```

---

## CDK Wiring

### Attach Triggers to User Pool

```typescript
import * as lambda from 'aws-cdk-lib/aws-lambda-nodejs';
import * as cognito from 'aws-cdk-lib/aws-cognito';

// Create the Lambda
const preSignUpFn = new lambda.NodejsFunction(this, 'PreSignUp', {
  entry: 'src/triggers/pre-sign-up.ts',
  runtime: lambda.Runtime.NODEJS_20_X,
  timeout: cdk.Duration.seconds(5),
});

const postConfirmFn = new lambda.NodejsFunction(this, 'PostConfirm', {
  entry: 'src/triggers/post-confirmation.ts',
  runtime: lambda.Runtime.NODEJS_20_X,
  timeout: cdk.Duration.seconds(5),
  environment: {
    USERS_TABLE: usersTable.tableName,
  },
});

// Grant DynamoDB access
usersTable.grantWriteData(postConfirmFn);

// Attach to user pool
const userPool = new cognito.UserPool(this, 'UserPool', {
  // ... other config
  lambdaTriggers: {
    preSignUp: preSignUpFn,
    postConfirmation: postConfirmFn,
    preTokenGeneration: preTokenGenFn,
    customMessage: customMessageFn,
  },
});
```

### Pre Token Generation V2 Trigger (CDK)

The V2 trigger requires setting the trigger config explicitly:

```typescript
const cfnUserPool = userPool.node.defaultChild as cognito.CfnUserPool;
cfnUserPool.lambdaConfig = {
  ...cfnUserPool.lambdaConfig,
  preTokenGenerationConfig: {
    lambdaArn: preTokenGenFn.functionArn,
    lambdaVersion: 'V2_0',
  },
};

// Grant Cognito permission to invoke the Lambda
preTokenGenFn.addPermission('CognitoInvoke', {
  principal: new iam.ServicePrincipal('cognito-idp.amazonaws.com'),
  sourceArn: userPool.userPoolArn,
});
```

FILE:references/security.md
# Cognito Security Best Practices

A security checklist and patterns for hardening your Cognito setup.

## Table of Contents
1. [Pre-Launch Checklist](#pre-launch-checklist)
2. [User Pool Security](#user-pool-security)
3. [Identity Pool Security](#identity-pool-security)
4. [Token Security](#token-security)
5. [Network & Infrastructure](#network-and-infrastructure)
6. [Monitoring & Incident Response](#monitoring-and-incident-response)

---

## Pre-Launch Checklist

Run through this before going to production:

- [ ] MFA enabled (at minimum OPTIONAL, ideally REQUIRED for admin users)
- [ ] Password policy: 8+ chars, mixed case, digits required
- [ ] `preventUserExistenceErrors` enabled on all app clients
- [ ] Client secrets set on all confidential (server-side) clients
- [ ] No client secrets on public (SPA/mobile) clients
- [ ] Token validity periods reviewed (default 1hr ID/access, 30d refresh)
- [ ] Callback and logout URLs are HTTPS (no HTTP except localhost for dev)
- [ ] Unused auth flows disabled on each app client
- [ ] User pool has `RemovalPolicy.RETAIN` and deletion protection enabled
- [ ] Identity pool (if used) has minimal IAM permissions
- [ ] Guest access disabled unless explicitly needed
- [ ] WAF web ACL associated with user pool
- [ ] SES configured for email (not default Cognito email)
- [ ] CloudTrail logging enabled for Cognito API calls
- [ ] Lambda triggers have 5-second timeout and proper error handling

---

## User Pool Security

### Prevent User Enumeration

Enable `preventUserExistenceErrors` on every app client. Without this, Cognito returns different error messages for "user not found" vs "wrong password," letting attackers enumerate valid accounts.

### Block Disposable Emails

Use a Pre Sign-Up Lambda trigger to reject sign-ups from disposable email domains. Maintain a blocklist or use a third-party API.

### SMS Pumping Prevention

If SMS verification or MFA is enabled:
- Set up AWS WAF rate limiting on the user pool
- Monitor SNS spending with billing alerts
- Consider requiring email verification first, then phone as a second step
- Use TOTP MFA instead of SMS when possible (more secure and cheaper)

### Attribute Immutability

- Mark sensitive attributes as immutable (e.g., `tenantId`, `role` if set by admins)
- Use Pre Token Generation trigger to inject computed claims rather than trusting user-editable attributes
- Remember: users can modify their own mutable attributes via the `UpdateUserAttributes` API

### Account Takeover Protection (Plus Plan)

Enable adaptive authentication to automatically detect:
- Compromised credentials (checks against known breach databases)
- Unusual sign-in activity (new device, new location, impossible travel)
- Automated sign-in attempts

Configure risk-based responses:
- Low risk → Allow
- Medium risk → Require MFA
- High risk → Block

---

## Identity Pool Security

### Least Privilege IAM Roles

The #1 rule: give identity pool roles the absolute minimum permissions needed.

**Bad** — overly broad:
```json
{
  "Effect": "Allow",
  "Action": "s3:*",
  "Resource": "*"
}
```

**Good** — scoped to user's own data:
```json
{
  "Effect": "Allow",
  "Action": ["s3:GetObject", "s3:PutObject"],
  "Resource": "arn:aws:s3:::my-bucket/sub/*"
}
```

### Trust Policy Conditions

Always include both conditions in the IAM role trust policy:

```json
{
  "Condition": {
    "StringEquals": {
      "cognito-identity.amazonaws.com:aud": "us-east-1:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
    },
    "ForAnyValue:StringLike": {
      "cognito-identity.amazonaws.com:amr": "authenticated"
    }
  }
}
```

This ensures:
- `aud` matches your specific identity pool ID (not any identity pool)
- `amr` restricts to authenticated users only (or unauthenticated for guest role)

### Disable Guest Access

Only enable unauthenticated (guest) access if absolutely required. If enabled:
- Grant read-only access to truly public resources
- Monitor unauthenticated usage patterns
- Set aggressive rate limits

### Use Enhanced Flow (Not Basic)

Enhanced (simplified) flow handles role selection server-side. Basic (classic) flow delegates role selection to the client, which can be manipulated. Always prefer enhanced flow.

---

## Token Security

### Storage

| Platform | ID/Access Tokens | Refresh Token |
|----------|-----------------|---------------|
| Web (SPA) | In-memory (JS variable) | HttpOnly Secure cookie |
| Web (SSR) | HttpOnly Secure cookie | HttpOnly Secure cookie |
| iOS | Keychain | Keychain |
| Android | EncryptedSharedPreferences | EncryptedSharedPreferences |

**Never** store tokens in:
- localStorage (XSS vulnerable)
- sessionStorage (XSS vulnerable)
- Plain cookies without HttpOnly + Secure flags
- URL parameters

### Validation

Always validate tokens server-side:
1. Verify the JWT signature against the JWKS endpoint (`https://cognito-idp.{region}.amazonaws.com/{userPoolId}/.well-known/jwks.json`)
2. Check `exp` claim (expiration)
3. Check `iss` claim matches your user pool
4. Check `aud` (ID token) or `client_id` (access token) matches your app client
5. Check `token_use` is the expected type (`id` or `access`)

Use `aws-jwt-verify` for Node.js — it handles all of this correctly and caches JWKS.

### Token Revocation

- `signOut({ global: true })` in Amplify revokes refresh tokens
- `AdminUserGlobalSignOut` API revokes all sessions for a user
- `RevokeToken` API revokes a specific refresh token
- ID and access tokens remain valid until expiration even after revocation (short validity periods mitigate this)

---

## Network & Infrastructure

### AWS WAF

Associate a WAF web ACL with your user pool to:
- Rate limit authentication requests per IP
- Block known malicious IPs
- Add CAPTCHA for suspicious requests
- Geo-restrict if your app is region-specific

Recommended rules:
- Rate limit: 100-1000 requests per 5 minutes per IP (tune to your traffic)
- AWS Managed Rules: `AWSManagedRulesCommonRuleSet` for common attack patterns
- IP reputation: `AWSManagedRulesAmazonIpReputationList`

### VPC & Private Endpoints

If your backend runs in a VPC and calls Cognito APIs:
- Use VPC endpoints for `cognito-idp` to keep traffic off the public internet
- This doesn't apply to user-facing auth (managed login, OAuth endpoints are always public)

### Custom Domain SSL

When using a custom domain:
- ACM certificate must be in `us-east-1`
- Use TLS 1.2+ (Cognito enforces this)
- Set up a CNAME from your domain to the Cognito CloudFront distribution

---

## Monitoring & Incident Response

### CloudTrail

Enable CloudTrail logging to capture all Cognito API calls:
- `AdminCreateUser`, `AdminDeleteUser` — admin operations
- `InitiateAuth`, `RespondToAuthChallenge` — authentication attempts
- `ForgotPassword`, `ConfirmForgotPassword` — password resets
- `UpdateUserAttributes` — profile changes

### CloudWatch Metrics

Monitor these Cognito metrics:
- `SignUpSuccesses` / `SignUpThrottles`
- `SignInSuccesses` / `SignInThrottles`
- `TokenRefreshSuccesses` / `TokenRefreshThrottles`
- `FederationSuccesses` / `FederationThrottles`

Set alarms for:
- Spike in failed sign-in attempts (brute force)
- Spike in sign-up rate (bot registration)
- Throttling errors (quota exhaustion)

### Advanced Security Logging (Plus Plan)

With the Plus plan, Cognito logs detailed user activity including:
- Risk scores for each authentication attempt
- Device fingerprints
- IP addresses and locations
- Event types (sign-in, sign-up, password change)

Export to S3, CloudWatch Logs, or Data Firehose for analysis.

### Incident Response Playbook

**Compromised user account**:
1. `AdminUserGlobalSignOut` — revoke all sessions
2. `AdminResetUserPassword` — force password reset
3. `AdminSetUserMFAPreference` — enable MFA if not already
4. Review CloudTrail logs for unauthorized actions
5. If Identity Pool is used, the IAM credentials expire automatically (1 hour default)

**Suspected credential stuffing attack**:
1. Check CloudWatch for spike in failed `InitiateAuth` calls
2. Enable WAF rate limiting if not already active
3. Consider temporarily enabling CAPTCHA via WAF
4. Review source IPs in CloudTrail and block via WAF if patterns emerge
5. Enable adaptive authentication (Plus plan) for automatic risk-based responses

**SMS pumping detected**:
1. Monitor SNS spend in AWS Billing
2. Disable SMS as MFA/verification method temporarily
3. Switch to TOTP or email-based MFA
4. Add WAF rules to block traffic from suspicious regions
5. File an AWS Support case for help identifying the attack vector

FILE:references/iac-patterns.md
# Cognito Infrastructure as Code Patterns

Production-ready templates for CDK, CloudFormation, and Terraform.

## Table of Contents
1. [CDK (TypeScript)](#cdk-typescript)
2. [CDK (Python)](#cdk-python)
3. [CloudFormation](#cloudformation)
4. [Terraform](#terraform)
5. [Common Add-ons](#common-add-ons)

---

## CDK (TypeScript)

### Full User Pool + App Client + Domain

```typescript
import * as cdk from 'aws-cdk-lib';
import * as cognito from 'aws-cdk-lib/aws-cognito';
import { Construct } from 'constructs';

interface CognitoStackProps extends cdk.StackProps {
  stage: string;
  callbackUrls: string[];
  logoutUrls: string[];
  domainPrefix: string;
}

export class CognitoStack extends cdk.Stack {
  public readonly userPool: cognito.UserPool;
  public readonly userPoolClient: cognito.UserPoolClient;

  constructor(scope: Construct, id: string, props: CognitoStackProps) {
    super(scope, id, props);

    // User Pool
    this.userPool = new cognito.UserPool(this, 'UserPool', {
      userPoolName: `props.stage-user-pool`,
      selfSignUpEnabled: true,
      signInCaseSensitive: false,

      // Sign-in configuration
      signInAliases: {
        email: true,
      },

      // Auto-verify email since it's a sign-in alias
      autoVerify: {
        email: true,
      },

      // Keep original email/phone until new one is verified
      keepOriginal: {
        email: true,
      },

      // Password requirements
      passwordPolicy: {
        minLength: 8,
        requireLowercase: true,
        requireUppercase: true,
        requireDigits: true,
        requireSymbols: false,
        tempPasswordValidity: cdk.Duration.days(7),
      },

      // MFA
      mfa: cognito.Mfa.OPTIONAL,
      mfaSecondFactor: {
        sms: true,
        otp: true,
      },

      // Account recovery
      accountRecovery: cognito.AccountRecovery.EMAIL_ONLY,

      // Standard attributes
      standardAttributes: {
        email: { required: true, mutable: true },
        fullname: { required: false, mutable: true },
      },

      // Custom attributes
      customAttributes: {
        tenantId: new cognito.StringAttribute({ mutable: false }),
        role: new cognito.StringAttribute({ mutable: true }),
      },

      // Email configuration — switch to SES for production
      email: cognito.UserPoolEmail.withCognito('[email protected]'),

      // Feature plan
      featurePlan: cognito.FeaturePlan.ESSENTIALS,

      // IMPORTANT: Retain user data on stack deletion in production
      removalPolicy: props.stage === 'prod'
        ? cdk.RemovalPolicy.RETAIN
        : cdk.RemovalPolicy.DESTROY,

      // Deletion protection for production
      deletionProtection: props.stage === 'prod',
    });

    // User Pool Domain (for hosted UI / managed login)
    this.userPool.addDomain('CognitoDomain', {
      cognitoDomain: {
        domainPrefix: props.domainPrefix,
      },
    });

    // App Client (public — for SPA/mobile)
    this.userPoolClient = this.userPool.addClient('AppClient', {
      userPoolClientName: `props.stage-app-client`,
      generateSecret: false, // No secret for public clients

      // Auth flows
      authFlows: {
        userSrp: true,
        custom: true,
      },

      // OAuth configuration
      oAuth: {
        flows: {
          authorizationCodeGrant: true,
        },
        scopes: [
          cognito.OAuthScope.OPENID,
          cognito.OAuthScope.EMAIL,
          cognito.OAuthScope.PROFILE,
        ],
        callbackUrls: props.callbackUrls,
        logoutUrls: props.logoutUrls,
      },

      // Token validity
      idTokenValidity: cdk.Duration.hours(1),
      accessTokenValidity: cdk.Duration.hours(1),
      refreshTokenValidity: cdk.Duration.days(30),

      // Prevent user existence errors from leaking info
      preventUserExistenceErrors: true,
    });

    // Outputs
    new cdk.CfnOutput(this, 'UserPoolId', {
      value: this.userPool.userPoolId,
    });
    new cdk.CfnOutput(this, 'UserPoolClientId', {
      value: this.userPoolClient.userPoolClientId,
    });
  }
}
```

### Adding a Confidential (Server-Side) Client

```typescript
const serverClient = userPool.addClient('ServerClient', {
  userPoolClientName: `stage-server-client`,
  generateSecret: true, // Confidential client
  authFlows: {
    userSrp: true,
    adminUserPassword: true,
  },
  oAuth: {
    flows: {
      authorizationCodeGrant: true,
    },
    scopes: [
      cognito.OAuthScope.OPENID,
      cognito.OAuthScope.EMAIL,
    ],
    callbackUrls: ['https://api.yourdomain.com/auth/callback'],
    logoutUrls: ['https://api.yourdomain.com/auth/logout'],
  },
  preventUserExistenceErrors: true,
});
```

### Adding an Identity Pool (CDK v2 L2 Construct)

```typescript
import { IdentityPool, UserPoolAuthenticationProvider } from 'aws-cdk-lib/aws-cognito-identitypool';
import * as iam from 'aws-cdk-lib/aws-iam';
import * as s3 from 'aws-cdk-lib/aws-s3';

const identityPool = new IdentityPool(this, 'IdentityPool', {
  identityPoolName: `stage-identity-pool`,
  allowUnauthenticatedIdentities: false, // No guest access
  authenticationProviders: {
    userPools: [
      new UserPoolAuthenticationProvider({ userPool, userPoolClient }),
    ],
  },
});

// Grant authenticated users access to an S3 bucket (per-user prefix)
const dataBucket = new s3.Bucket(this, 'UserDataBucket');
identityPool.authenticatedRole.addToPrincipalPolicy(
  new iam.PolicyStatement({
    actions: ['s3:GetObject', 's3:PutObject', 's3:DeleteObject'],
    resources: [
      dataBucket.arnForObjects('sub/*'),
    ],
  }),
);
```

### Adding Social Federation (Google Example)

```typescript
const googleProvider = new cognito.UserPoolIdentityProviderGoogle(this, 'Google', {
  userPool,
  clientId: 'your-google-client-id',
  clientSecretValue: cdk.SecretValue.secretsManager('google-client-secret'),
  scopes: ['openid', 'email', 'profile'],
  attributeMapping: {
    email: cognito.ProviderAttribute.GOOGLE_EMAIL,
    fullname: cognito.ProviderAttribute.GOOGLE_NAME,
    profilePicture: cognito.ProviderAttribute.GOOGLE_PICTURE,
  },
});

// Make sure the client depends on the provider
userPoolClient.node.addDependency(googleProvider);
```

### M2M (Client Credentials) Setup

```typescript
// Resource server with custom scopes
const resourceServer = userPool.addResourceServer('ResourceServer', {
  identifier: 'https://api.yourdomain.com',
  scopes: [
    { scopeName: 'read', scopeDescription: 'Read access' },
    { scopeName: 'write', scopeDescription: 'Write access' },
  ],
});

// M2M client
const m2mClient = userPool.addClient('M2MClient', {
  generateSecret: true,
  oAuth: {
    flows: { clientCredentials: true },
    scopes: [
      cognito.OAuthScope.resourceServer(resourceServer, {
        scopeName: 'read',
        scopeDescription: 'Read access',
      }),
    ],
  },
});
```

---

## CDK (Python)

### Full User Pool + App Client

```python
from aws_cdk import (
    Stack, Duration, RemovalPolicy, CfnOutput,
    aws_cognito as cognito,
)
from constructs import Construct


class CognitoStack(Stack):
    def __init__(self, scope: Construct, id: str, stage: str, **kwargs):
        super().__init__(scope, id, **kwargs)

        self.user_pool = cognito.UserPool(
            self, "UserPool",
            user_pool_name=f"{stage}-user-pool",
            self_sign_up_enabled=True,
            sign_in_case_sensitive=False,
            sign_in_aliases=cognito.SignInAliases(email=True),
            auto_verify=cognito.AutoVerifiedAttrs(email=True),
            keep_original=cognito.KeepOriginalAttrs(email=True),
            password_policy=cognito.PasswordPolicy(
                min_length=8,
                require_lowercase=True,
                require_uppercase=True,
                require_digits=True,
                require_symbols=False,
                temp_password_validity=Duration.days(7),
            ),
            mfa=cognito.Mfa.OPTIONAL,
            mfa_second_factor=cognito.MfaSecondFactor(sms=True, otp=True),
            account_recovery=cognito.AccountRecovery.EMAIL_ONLY,
            standard_attributes=cognito.StandardAttributes(
                email=cognito.StandardAttribute(required=True, mutable=True),
                fullname=cognito.StandardAttribute(required=False, mutable=True),
            ),
            feature_plan=cognito.FeaturePlan.ESSENTIALS,
            removal_policy=RemovalPolicy.RETAIN if stage == "prod" else RemovalPolicy.DESTROY,
            deletion_protection=True if stage == "prod" else False,
        )

        self.user_pool_client = self.user_pool.add_client(
            "AppClient",
            user_pool_client_name=f"{stage}-app-client",
            generate_secret=False,
            auth_flows=cognito.AuthFlow(user_srp=True, custom=True),
            o_auth=cognito.OAuthSettings(
                flows=cognito.OAuthFlows(authorization_code_grant=True),
                scopes=[
                    cognito.OAuthScope.OPENID,
                    cognito.OAuthScope.EMAIL,
                    cognito.OAuthScope.PROFILE,
                ],
                callback_urls=["https://yourdomain.com/callback"],
                logout_urls=["https://yourdomain.com/logout"],
            ),
            id_token_validity=Duration.hours(1),
            access_token_validity=Duration.hours(1),
            refresh_token_validity=Duration.days(30),
            prevent_user_existence_errors=True,
        )

        CfnOutput(self, "UserPoolId", value=self.user_pool.user_pool_id)
        CfnOutput(self, "ClientId", value=self.user_pool_client.user_pool_client_id)
```

---

## CloudFormation

### User Pool + Client (YAML)

```yaml
AWSTemplateFormatVersion: '2010-09-09'
Description: Cognito User Pool with App Client

Parameters:
  Stage:
    Type: String
    Default: dev
    AllowedValues: [dev, staging, prod]

Resources:
  UserPool:
    Type: AWS::Cognito::UserPool
    DeletionPolicy: Retain
    UpdateReplacePolicy: Retain
    Properties:
      UserPoolName: !Sub 'Stage-user-pool'
      UsernameAttributes:
        - email
      AutoVerifiedAttributes:
        - email
      UsernameConfiguration:
        CaseSensitive: false
      Policies:
        PasswordPolicy:
          MinimumLength: 8
          RequireLowercase: true
          RequireUppercase: true
          RequireNumbers: true
          RequireSymbols: false
          TemporaryPasswordValidityDays: 7
      MfaConfiguration: OPTIONAL
      EnabledMfas:
        - SOFTWARE_TOKEN_MFA
        - SMS_MFA
      AccountRecoverySetting:
        RecoveryMechanisms:
          - Name: verified_email
            Priority: 1
      Schema:
        - Name: email
          AttributeDataType: String
          Required: true
          Mutable: true
        - Name: name
          AttributeDataType: String
          Required: false
          Mutable: true
        - Name: tenantId
          AttributeDataType: String
          Required: false
          Mutable: false
      UserPoolAddOns:
        AdvancedSecurityMode: ENFORCED  # For Plus plan

  UserPoolClient:
    Type: AWS::Cognito::UserPoolClient
    Properties:
      ClientName: !Sub 'Stage-app-client'
      UserPoolId: !Ref UserPool
      GenerateSecret: false
      ExplicitAuthFlows:
        - ALLOW_USER_SRP_AUTH
        - ALLOW_REFRESH_TOKEN_AUTH
        - ALLOW_CUSTOM_AUTH
      AllowedOAuthFlows:
        - code
      AllowedOAuthScopes:
        - openid
        - email
        - profile
      AllowedOAuthFlowsUserPoolClient: true
      CallbackURLs:
        - https://yourdomain.com/callback
      LogoutURLs:
        - https://yourdomain.com/logout
      SupportedIdentityProviders:
        - COGNITO
      PreventUserExistenceErrors: ENABLED
      IdTokenValidity: 60       # minutes
      AccessTokenValidity: 60   # minutes
      RefreshTokenValidity: 43200  # minutes (30 days)
      TokenValidityUnits:
        IdToken: minutes
        AccessToken: minutes
        RefreshToken: minutes

  UserPoolDomain:
    Type: AWS::Cognito::UserPoolDomain
    Properties:
      Domain: !Sub 'Stage-myapp'
      UserPoolId: !Ref UserPool

Outputs:
  UserPoolId:
    Value: !Ref UserPool
  UserPoolClientId:
    Value: !Ref UserPoolClient
  UserPoolDomain:
    Value: !Sub 'https://Stage-myapp.auth.:Region.amazoncognito.com'
```

---

## Terraform

### User Pool + Client

```hcl
variable "stage" {
  type    = string
  default = "dev"
}

resource "aws_cognito_user_pool" "main" {
  name = "var.stage-user-pool"

  username_attributes      = ["email"]
  auto_verified_attributes = ["email"]

  username_configuration {
    case_sensitive = false
  }

  password_policy {
    minimum_length                   = 8
    require_lowercase                = true
    require_uppercase                = true
    require_numbers                  = true
    require_symbols                  = false
    temporary_password_validity_days = 7
  }

  mfa_configuration = "OPTIONAL"

  software_token_mfa_configuration {
    enabled = true
  }

  account_recovery_setting {
    recovery_mechanism {
      name     = "verified_email"
      priority = 1
    }
  }

  schema {
    name                = "email"
    attribute_data_type = "String"
    required            = true
    mutable             = true
  }

  schema {
    name                = "tenantId"
    attribute_data_type = "String"
    required            = false
    mutable             = false

    string_attribute_constraints {
      min_length = 1
      max_length = 256
    }
  }

  lifecycle {
    prevent_destroy = true  # Production safety
  }

  tags = {
    Environment = var.stage
  }
}

resource "aws_cognito_user_pool_client" "app" {
  name         = "var.stage-app-client"
  user_pool_id = aws_cognito_user_pool.main.id

  generate_secret = false

  explicit_auth_flows = [
    "ALLOW_USER_SRP_AUTH",
    "ALLOW_REFRESH_TOKEN_AUTH",
    "ALLOW_CUSTOM_AUTH",
  ]

  allowed_oauth_flows                  = ["code"]
  allowed_oauth_scopes                 = ["openid", "email", "profile"]
  allowed_oauth_flows_user_pool_client = true
  supported_identity_providers         = ["COGNITO"]

  callback_urls = ["https://yourdomain.com/callback"]
  logout_urls   = ["https://yourdomain.com/logout"]

  prevent_user_existence_errors = "ENABLED"

  id_token_validity      = 60  # minutes
  access_token_validity  = 60  # minutes
  refresh_token_validity = 30  # days

  token_validity_units {
    id_token      = "minutes"
    access_token  = "minutes"
    refresh_token = "days"
  }
}

resource "aws_cognito_user_pool_domain" "main" {
  domain       = "var.stage-myapp"
  user_pool_id = aws_cognito_user_pool.main.id
}

output "user_pool_id" {
  value = aws_cognito_user_pool.main.id
}

output "user_pool_client_id" {
  value = aws_cognito_user_pool_client.app.id
}
```

---

## Common Add-ons

### SES Email Configuration (CDK)

For production, replace Cognito's default email with SES:

```typescript
import * as ses from 'aws-cdk-lib/aws-ses';

const userPool = new cognito.UserPool(this, 'UserPool', {
  // ... other config
  email: cognito.UserPoolEmail.withSES({
    fromEmail: '[email protected]',
    fromName: 'Your App',
    sesRegion: 'us-east-1', // SES must be in a supported region
    sesVerifiedDomain: 'yourdomain.com',
  }),
});
```

### Custom Domain with ACM Certificate (CDK)

```typescript
import * as acm from 'aws-cdk-lib/aws-certificatemanager';

// Certificate must be in us-east-1
const cert = acm.Certificate.fromCertificateArn(
  this, 'Cert',
  'arn:aws:acm:us-east-1:123456789:certificate/abc-123'
);

userPool.addDomain('CustomDomain', {
  customDomain: {
    domainName: 'auth.yourdomain.com',
    certificate: cert,
  },
});
```

### WAF Integration (CDK)

```typescript
import * as wafv2 from 'aws-cdk-lib/aws-wafv2';

const webAcl = new wafv2.CfnWebACL(this, 'CognitoWaf', {
  scope: 'REGIONAL',
  defaultAction: { allow: {} },
  rules: [
    {
      name: 'RateLimit',
      priority: 1,
      action: { block: {} },
      statement: {
        rateBasedStatement: {
          limit: 1000,
          aggregateKeyType: 'IP',
        },
      },
      visibilityConfig: {
        cloudWatchMetricsEnabled: true,
        metricName: 'CognitoRateLimit',
        sampledRequestsEnabled: true,
      },
    },
  ],
  visibilityConfig: {
    cloudWatchMetricsEnabled: true,
    metricName: 'CognitoWaf',
    sampledRequestsEnabled: true,
  },
});

new wafv2.CfnWebACLAssociation(this, 'WafAssociation', {
  resourceArn: userPool.userPoolArn,
  webAclArn: webAcl.attrArn,
});
```

### Cognito + API Gateway Authorizer (CDK)

```typescript
import * as apigateway from 'aws-cdk-lib/aws-apigateway';

const api = new apigateway.RestApi(this, 'Api');

const cognitoAuthorizer = new apigateway.CognitoUserPoolsAuthorizer(this, 'Authorizer', {
  cognitoUserPools: [userPool],
  identitySource: 'method.request.header.Authorization',
});

api.root.addResource('protected').addMethod('GET', integration, {
  authorizer: cognitoAuthorizer,
  authorizationType: apigateway.AuthorizationType.COGNITO,
  authorizationScopes: ['openid'], // Optional: require specific scopes
});
```

FILE:references/auth-flows.md
# Cognito Authentication Flows

SDK code patterns for implementing authentication in your application.

## Table of Contents
1. [Flow Selection Guide](#flow-selection-guide)
2. [Amplify v6 (React / Next.js)](#amplify-v6)
3. [AWS SDK v3 (Node.js Backend)](#aws-sdk-v3-nodejs)
4. [AWS SDK v3 (Python / Boto3)](#boto3-python)
5. [Token Handling](#token-handling)
6. [Custom Auth (Passwordless)](#custom-auth-passwordless)
7. [Machine-to-Machine](#machine-to-machine)

---

## Flow Selection Guide

| Scenario | Flow | Client Type |
|----------|------|-------------|
| SPA or mobile app | Authorization Code + PKCE | Public (no secret) |
| Server-rendered web app | Authorization Code | Confidential (with secret) |
| Direct username/password (SPA) | USER_SRP_AUTH via SDK | Public |
| Admin creates/authenticates users | ADMIN_USER_PASSWORD_AUTH | Server-side only |
| Passwordless (magic link, OTP) | CUSTOM_AUTH with Lambda triggers | Either |
| Service-to-service | Client Credentials | Confidential |
| Migration from legacy auth | USER_PASSWORD_AUTH + migration trigger | Temporary |

**General rules**:
- Always prefer SRP over plaintext password flows
- Always use PKCE for public clients
- Never use Implicit flow (legacy, insecure)
- Always implement token refresh

---

## Amplify v6

### Installation

```bash
npm install aws-amplify
```

### Configuration

```typescript
// amplifyconfiguration.ts
import { Amplify } from 'aws-amplify';

Amplify.configure({
  Auth: {
    Cognito: {
      userPoolId: 'us-east-1_XXXXXXXXX',
      userPoolClientId: 'xxxxxxxxxxxxxxxxxxxxxxxxxx',
      loginWith: {
        oauth: {
          domain: 'your-domain.auth.us-east-1.amazoncognito.com',
          scopes: ['openid', 'email', 'profile'],
          redirectSignIn: ['http://localhost:3000/'],
          redirectSignOut: ['http://localhost:3000/'],
          responseType: 'code',
        },
      },
    },
  },
});
```

### Sign Up

```typescript
import { signUp } from 'aws-amplify/auth';

async function handleSignUp(email: string, password: string) {
  try {
    const { isSignUpComplete, userId, nextStep } = await signUp({
      username: email,
      password,
      options: {
        userAttributes: {
          email,
          name: 'Jane Doe',
        },
      },
    });

    if (nextStep.signUpStep === 'CONFIRM_SIGN_UP') {
      // User needs to enter verification code
      console.log('Verification code sent to:', email);
    }
  } catch (error) {
    console.error('Sign up error:', error);
  }
}
```

### Confirm Sign Up

```typescript
import { confirmSignUp } from 'aws-amplify/auth';

async function handleConfirmation(email: string, code: string) {
  try {
    const { isSignUpComplete } = await confirmSignUp({
      username: email,
      confirmationCode: code,
    });
    console.log('Sign up confirmed:', isSignUpComplete);
  } catch (error) {
    console.error('Confirmation error:', error);
  }
}
```

### Sign In

```typescript
import { signIn } from 'aws-amplify/auth';

async function handleSignIn(email: string, password: string) {
  try {
    const { isSignedIn, nextStep } = await signIn({
      username: email,
      password,
    });

    switch (nextStep.signInStep) {
      case 'DONE':
        console.log('Signed in successfully');
        break;
      case 'CONFIRM_SIGN_IN_WITH_TOTP_CODE':
        // User needs to enter TOTP code
        break;
      case 'CONFIRM_SIGN_IN_WITH_SMS_CODE':
        // User needs to enter SMS code
        break;
      case 'CONFIRM_SIGN_IN_WITH_NEW_PASSWORD_REQUIRED':
        // Admin-created user needs to set password
        break;
    }
  } catch (error) {
    console.error('Sign in error:', error);
  }
}
```

### Social / Federated Sign In

```typescript
import { signInWithRedirect } from 'aws-amplify/auth';

// Google
await signInWithRedirect({ provider: 'Google' });

// Facebook
await signInWithRedirect({ provider: 'Facebook' });

// Apple
await signInWithRedirect({ provider: 'Apple' });

// SAML
await signInWithRedirect({
  provider: { custom: 'YourSAMLProviderName' },
});
```

### Get Current Session / Tokens

```typescript
import { fetchAuthSession, getCurrentUser } from 'aws-amplify/auth';

// Get the current user
const user = await getCurrentUser();
console.log('User ID:', user.userId);
console.log('Username:', user.username);

// Get tokens (automatically refreshes if expired)
const session = await fetchAuthSession();
const idToken = session.tokens?.idToken?.toString();
const accessToken = session.tokens?.accessToken?.toString();

// Get AWS credentials (if Identity Pool is configured)
const credentials = session.credentials;
```

### Sign Out

```typescript
import { signOut } from 'aws-amplify/auth';

// Local sign out
await signOut();

// Global sign out (invalidates all sessions)
await signOut({ global: true });
```

### Password Reset

```typescript
import { resetPassword, confirmResetPassword } from 'aws-amplify/auth';

// Step 1: Initiate reset
const { nextStep } = await resetPassword({ username: email });
// nextStep.resetPasswordStep === 'CONFIRM_RESET_PASSWORD_WITH_CODE'

// Step 2: Confirm with code and new password
await confirmResetPassword({
  username: email,
  confirmationCode: code,
  newPassword: newPassword,
});
```

### MFA Setup

```typescript
import { setUpTOTP, verifyTOTPSetup, updateMFAPreference } from 'aws-amplify/auth';

// Set up TOTP
const totpSetup = await setUpTOTP();
const setupUri = totpSetup.getSetupUri('YourApp', email);
// Display setupUri as QR code for user to scan

// Verify TOTP with code from authenticator app
await verifyTOTPSetup({ code: '123456' });

// Set TOTP as preferred MFA
await updateMFAPreference({ totp: 'PREFERRED' });
```

---

## AWS SDK v3 (Node.js)

### Installation

```bash
npm install @aws-sdk/client-cognito-identity-provider
```

### Admin: Create User

```typescript
import {
  CognitoIdentityProviderClient,
  AdminCreateUserCommand,
} from '@aws-sdk/client-cognito-identity-provider';

const client = new CognitoIdentityProviderClient({ region: 'us-east-1' });

async function createUser(email: string) {
  const command = new AdminCreateUserCommand({
    UserPoolId: process.env.USER_POOL_ID,
    Username: email,
    UserAttributes: [
      { Name: 'email', Value: email },
      { Name: 'email_verified', Value: 'true' },
      { Name: 'name', Value: 'Jane Doe' },
    ],
    DesiredDeliveryMediums: ['EMAIL'],
  });
  return client.send(command);
}
```

### Admin: Authenticate User (Server-Side)

```typescript
import { AdminInitiateAuthCommand } from '@aws-sdk/client-cognito-identity-provider';

async function adminSignIn(email: string, password: string) {
  const command = new AdminInitiateAuthCommand({
    UserPoolId: process.env.USER_POOL_ID,
    ClientId: process.env.CLIENT_ID,
    AuthFlow: 'ADMIN_USER_PASSWORD_AUTH',
    AuthParameters: {
      USERNAME: email,
      PASSWORD: password,
    },
  });
  const response = await client.send(command);
  return response.AuthenticationResult; // { IdToken, AccessToken, RefreshToken }
}
```

### Verify Token (Backend Middleware)

```typescript
import { CognitoJwtVerifier } from 'aws-jwt-verify';

// Create verifier (reuse this — it caches JWKS)
const verifier = CognitoJwtVerifier.create({
  userPoolId: process.env.USER_POOL_ID!,
  tokenUse: 'access', // or 'id'
  clientId: process.env.CLIENT_ID!,
});

async function verifyToken(token: string) {
  try {
    const payload = await verifier.verify(token);
    return payload; // { sub, email, cognito:groups, ... }
  } catch {
    throw new Error('Invalid token');
  }
}
```

Install the JWT verifier: `npm install aws-jwt-verify`

### Admin: Add User to Group

```typescript
import { AdminAddUserToGroupCommand } from '@aws-sdk/client-cognito-identity-provider';

async function addToGroup(username: string, groupName: string) {
  const command = new AdminAddUserToGroupCommand({
    UserPoolId: process.env.USER_POOL_ID,
    Username: username,
    GroupName: groupName,
  });
  return client.send(command);
}
```

### Refresh Tokens

```typescript
import { InitiateAuthCommand } from '@aws-sdk/client-cognito-identity-provider';

async function refreshTokens(refreshToken: string) {
  const command = new InitiateAuthCommand({
    ClientId: process.env.CLIENT_ID,
    AuthFlow: 'REFRESH_TOKEN_AUTH',
    AuthParameters: {
      REFRESH_TOKEN: refreshToken,
    },
  });
  const response = await client.send(command);
  return response.AuthenticationResult; // New IdToken and AccessToken
}
```

---

## Boto3 (Python)

### Admin: Create User

```python
import boto3

client = boto3.client('cognito-idp', region_name='us-east-1')

def create_user(email: str, user_pool_id: str):
    return client.admin_create_user(
        UserPoolId=user_pool_id,
        Username=email,
        UserAttributes=[
            {'Name': 'email', 'Value': email},
            {'Name': 'email_verified', 'Value': 'true'},
            {'Name': 'name', 'Value': 'Jane Doe'},
        ],
        DesiredDeliveryMediums=['EMAIL'],
    )
```

### Admin: Authenticate User

```python
def admin_sign_in(email: str, password: str, user_pool_id: str, client_id: str):
    return client.admin_initiate_auth(
        UserPoolId=user_pool_id,
        ClientId=client_id,
        AuthFlow='ADMIN_USER_PASSWORD_AUTH',
        AuthParameters={
            'USERNAME': email,
            'PASSWORD': password,
        },
    )
```

### User: Sign Up (Self-Service)

```python
def sign_up(email: str, password: str, client_id: str):
    return client.sign_up(
        ClientId=client_id,
        Username=email,
        Password=password,
        UserAttributes=[
            {'Name': 'email', 'Value': email},
            {'Name': 'name', 'Value': 'Jane Doe'},
        ],
    )
```

---

## Token Handling

### Token Structure

**ID Token claims** (JWT):
- `sub` — unique user identifier (UUID)
- `email` — user's email
- `email_verified` — boolean
- `cognito:username` — the username
- `cognito:groups` — array of group names
- `custom:*` — any custom attributes
- `iss` — issuer (user pool URL)
- `aud` — audience (client ID)
- `exp` — expiration timestamp

**Access Token claims**:
- `sub` — same as ID token
- `scope` — OAuth scopes
- `cognito:groups` — group names
- `client_id` — the app client ID
- `token_use` — always "access"

### Best Practices

- Store tokens securely: HttpOnly cookies for web, Keychain/Keystore for mobile
- Never store tokens in localStorage (XSS vulnerable)
- Always validate tokens on your backend — don't trust client-side validation alone
- Use the `aws-jwt-verify` library (not manual JWT parsing) for Node.js
- Check `token_use` claim to ensure you're validating the right token type
- Implement automatic token refresh before expiration

---

## Custom Auth (Passwordless)

### Architecture

Uses three Lambda triggers working together:
1. **Define Auth Challenge**: Decides which challenge to present next
2. **Create Auth Challenge**: Generates the challenge (e.g., sends OTP)
3. **Verify Auth Challenge**: Validates the user's response

### Example: Email OTP Flow

**Define Auth Challenge Lambda**:
```typescript
export const handler = async (event: any) => {
  const session = event.request.session;

  if (session.length === 0) {
    // First attempt — issue custom challenge
    event.response.challengeName = 'CUSTOM_CHALLENGE';
    event.response.issueTokens = false;
    event.response.failAuthentication = false;
  } else if (
    session.length === 1 &&
    session[0].challengeResult === true
  ) {
    // Challenge answered correctly — issue tokens
    event.response.issueTokens = true;
    event.response.failAuthentication = false;
  } else {
    // Wrong answer — fail
    event.response.issueTokens = false;
    event.response.failAuthentication = true;
  }

  return event;
};
```

**Create Auth Challenge Lambda**:
```typescript
import { SESClient, SendEmailCommand } from '@aws-sdk/client-ses';

const ses = new SESClient({});

export const handler = async (event: any) => {
  const otp = Math.floor(100000 + Math.random() * 900000).toString();

  // Send OTP via email
  await ses.send(new SendEmailCommand({
    Destination: { ToAddresses: [event.request.userAttributes.email] },
    Message: {
      Subject: { Data: 'Your verification code' },
      Body: { Text: { Data: `Your code is: otp` } },
    },
    Source: '[email protected]',
  }));

  event.response.publicChallengeParameters = {
    email: event.request.userAttributes.email,
  };
  event.response.privateChallengeParameters = { otp };
  event.response.challengeMetadata = 'EMAIL_OTP';

  return event;
};
```

**Verify Auth Challenge Lambda**:
```typescript
export const handler = async (event: any) => {
  const expected = event.request.privateChallengeParameters.otp;
  const answer = event.request.challengeAnswer;

  event.response.answerCorrect = expected === answer;
  return event;
};
```

---

## Machine-to-Machine

### Getting M2M Tokens

M2M uses the Client Credentials flow — no user interaction involved.

```typescript
async function getM2MToken(
  domain: string,
  clientId: string,
  clientSecret: string,
  scopes: string[]
) {
  const credentials = Buffer.from(`clientId:clientSecret`).toString('base64');

  const response = await fetch(`https://domain/oauth2/token`, {
    method: 'POST',
    headers: {
      'Content-Type': 'application/x-www-form-urlencoded',
      Authorization: `Basic credentials`,
    },
    body: new URLSearchParams({
      grant_type: 'client_credentials',
      scope: scopes.join(' '),
    }),
  });

  const data = await response.json();
  return data.access_token; // Only access token — no ID or refresh token
}
```

### Python M2M

```python
import requests
import base64

def get_m2m_token(domain: str, client_id: str, client_secret: str, scopes: list[str]):
    credentials = base64.b64encode(f"{client_id}:{client_secret}".encode()).decode()
    response = requests.post(
        f"https://{domain}/oauth2/token",
        headers={
            "Content-Type": "application/x-www-form-urlencoded",
            "Authorization": f"Basic {credentials}",
        },
        data={
            "grant_type": "client_credentials",
            "scope": " ".join(scopes),
        },
    )
    return response.json()["access_token"]
```