--- name: prompt-token-counter version: 1.0.10 description: "Count tokens and estimate costs for 300+ LLM models. Primary use: audit OpenClaw workspace token consumption (memory, persona, skills)." trigger: "token count, cost estimate, prompt length, API cost, OpenClaw audit, workspace token usage, memory/persona/skills tokens, context window limit" --- # Prompt Token Counter (toksum) > **First load reminder:** This skill provides the `scripts` CLI (toksum). Use it when the user asks to count tokens, estimate API costs, or **audit OpenClaw component token consumption** (memory, persona, skills). ## Before Installing — Security & Privacy - **What will be read:** The audit workflow reads files under `~/.openclaw/workspace` and `~/.openclaw/skills` (AGENTS.md, SOUL.md, MEMORY.md, SKILL.md, etc.). Those files may contain personal data or secrets. Only install if you accept that access. - **URL fetching:** The CLI can fetch HTTP(S) URLs via `-u`. SKILL.md requires the agent to confirm each URL with the user before fetching. Insist the agent follow that rule; never allow automatic fetching of unknown URLs. - **Source verification:** Source: [https://github.com/Zhaobudaoyuema/prompt-token-counter](https://github.com/Zhaobudaoyuema/prompt-token-counter). Review `scripts/core.py` and `scripts/cli.py` before use. The code performs local file reads and optional HTTP GETs only; no other network calls or data exfiltration. - **Run locally first:** If unsure, run the CLI manually in an isolated environment against safe test files to verify behavior. ## Primary Use: OpenClaw Token Consumption Audit **Goal:** Help users identify which OpenClaw components consume tokens and how much. ### 1. Memory & Persona Files These files are injected into sessions and consume tokens. Search and count them: | File | Purpose | Typical Location | |------|---------|------------------| | `AGENTS.md` | Operating instructions, workflow, priorities | `~/.openclaw/workspace/` | | `SOUL.md` | Persona, tone, values, behavioral guidelines | `~/.openclaw/workspace/` | | `IDENTITY.md` | Name, role, goals, visual description | `~/.openclaw/workspace/` | | `USER.md` | User preferences, communication style | `~/.openclaw/workspace/` | | `MEMORY.md` | Long-term memory, persistent facts | `~/.openclaw/workspace/` | | `TOOLS.md` | Tool quirks, path conventions | `~/.openclaw/workspace/` | | `HEARTBEAT.md` | Periodic maintenance checklist | `~/.openclaw/workspace/` | | `BOOT.md` | Startup ritual (when hooks enabled) | `~/.openclaw/workspace/` | | `memory/YYYY-MM-DD.md` | Daily memory logs | `~/.openclaw/workspace/memory/` | **Workspace path:** Default `~/.openclaw/workspace`; may be overridden in `~/.openclaw/openclaw.json` via `agent.workspace`. ### 2. Skill Files (SKILL.md) Skills are loaded per session. Count each `SKILL.md`: | Location | Scope | |----------|-------| | `~/.openclaw/skills/*/SKILL.md` | OpenClaw managed skills | | `~/.openclaw/workspace/skills/*/SKILL.md` | Workspace-specific skills (override) | ### 3. Audit Workflow 1. **Locate workspace:** Resolve `~/.openclaw/workspace` (or config override). 2. **Collect files:** List all memory/persona files and `SKILL.md` paths above. 3. **Count tokens:** Run `python -m scripts.cli ... -m -c` (batch mode). 4. **Summarize:** Group by category (memory, persona, skills), report total and per-file. **Example audit command (PowerShell):** ```powershell $ws = "$env:USERPROFILE\.openclaw\workspace" python -m scripts.cli -m gpt-4o -c "$ws\AGENTS.md" "$ws\SOUL.md" "$ws\USER.md" "$ws\IDENTITY.md" "$ws\MEMORY.md" "$ws\TOOLS.md" ``` **Example audit (Bash):** ```bash WS=~/.openclaw/workspace python -m scripts.cli -m gpt-4o -c "$WS/AGENTS.md" "$WS/SOUL.md" "$WS/USER.md" "$WS/IDENTITY.md" "$WS/MEMORY.md" "$WS/TOOLS.md" ``` --- ## Project Layout ``` prompt_token_counter/ ├── SKILL.md ├── package.json # npm package (OpenClaw skill) ├── publish_npm.py # Publish to npm; syncs version └── scripts/ # Python package, CLI + examples ├── cli.py # Entry point ├── core.py # TokenCounter, estimate_cost ├── registry/ │ ├── models.py # 300+ models │ └── pricing.py # Pricing data └── examples/ # Script examples ├── count_prompt.py ├── estimate_cost.py ├── batch_compare.py └── benchmark_token_ratio.py ``` Invoke: `python -m scripts.cli` from project root. ### Version Sync (publish_npm.py) When publishing to npm, `publish_npm.py` bumps the patch version and syncs it to: - `package.json` — `version` - `SKILL.md` — frontmatter `version` - `scripts/__init__.py` — `__version__` Run: `python publish_npm.py` (after `npm login`). --- ## Runtime Dependencies - **Python 3** — required - **tiktoken** (optional) — `pip install tiktoken` for exact OpenAI counts --- ## Language Rule **Respond in the user's language.** Match the user's language (e.g. Chinese if they write in Chinese, English if they write in English). --- ## URL Usage — Mandatory Agent Rule **Before using `-u` / `--url` to fetch content from any URL, you MUST:** 1. **Explicitly warn the user** that the CLI will make an outbound HTTP/HTTPS request to the given URL. 2. **Confirm the URL is trusted** — tell the user: "Only use URLs you fully trust. Untrusted URLs may expose your IP, leak data, or be used for SSRF. Do you confirm this URL is safe?" 3. **Prefer alternatives** — if the user can provide the content via `-f` (local file) or inline text, suggest that instead of URL fetch. 4. **Never auto-fetch** — do not invoke `-u` without the user having explicitly provided the URL and acknowledged the risk. **If the user insists on using a URL:** Proceed only after they confirm. State clearly: "I will fetch from [URL] to count tokens. Proceed?" --- ## Model Name — Mandatory Agent Rule **Before invoking the CLI, you MUST have a concrete model name from the user.** 1. **Require explicit model** — `-m` / `--model` is required. Do not guess or assume; the user must provide the exact name (e.g. gpt-4o, claude-3-5-sonnet-20241022). 2. **If unclear, ask** — if the user says "GPT" or "Claude" or "the latest model" without a specific name, ask: "Please specify the exact model name (e.g. gpt-4o, claude-3-5-sonnet-20241022). Run `python -m scripts.cli -l` to list supported models." 3. **Do not auto-pick** — never substitute a model on behalf of the user without their confirmation. 4. **Validate when possible** — if the model name seems ambiguous, offer `-l` output or confirm: "I'll use [model]. Is that correct?" --- ## CLI Usage **Default:** Read from local file(s). No segmentation. Supports multiple file paths for batch execution. ```bash python -m scripts.cli [OPTIONS] [FILE ...] ``` | Option | Short | Description | |--------|-------|-------------| | `--model` | `-m` | Model name (required unless `--list-models`) — **Agent must obtain exact name from user; ask if unclear** | | `--file` | `-f` | Read from file (repeatable) | | `--url` | `-u` | Read from URL (repeatable) — **Agent must warn user before use; only trusted URLs** | | `--list-models` | `-l` | List supported models | | `--cost` | `-c` | Show cost estimate | | `--output-tokens` | | Use output token pricing | | `--currency` | | USD or INR | | `--verbose` | `-v` | Detailed output | ### Examples ```bash # Multiple local files (default batch mode) python -m scripts.cli file1.txt file2.txt -m gpt-4 python -m scripts.cli AGENTS.md SOUL.md MEMORY.md -m gpt-4o -c # Single file with -f python -m scripts.cli -f input.txt -m claude-3-opus -c # Inline text (when arg is not an existing file path) python -m scripts.cli -m gpt-4 "Hello, world!" # List models python -m scripts.cli -l # Run bundled example scripts python scripts/examples/count_prompt.py file1.txt file2.txt -m gpt-4 python scripts/examples/estimate_cost.py "Your text" gpt-4 python scripts/examples/batch_compare.py file1.txt -m gpt-4 claude-3-opus ``` --- ## Python API ```python from scripts import TokenCounter, count_tokens, estimate_cost, get_supported_models tokens = count_tokens("Hello!", "gpt-4") counter = TokenCounter("claude-3-opus") tokens = counter.count_messages([ {"role": "system", "content": "..."}, {"role": "user", "content": "..."} ]) cost = estimate_cost(tokens, "gpt-4", input_tokens=True) ``` --- ## Supported Models 300+ models across 34+ providers: OpenAI, Anthropic, Google, Meta, Mistral, Cohere, xAI, DeepSeek, etc. Use `python -m scripts.cli -l` for full list. - **OpenAI:** exact via tiktoken - **Others:** ~85–95% approximation --- ## Response Output — Agent Guideline **After returning token count or cost estimate results, the agent MUST:** 1. **Include the project link** — e.g. > Source: [prompt-token-counter](https://github.com/Zhaobudaoyuema/prompt-token-counter) 2. **Briefly explain how tokens are calculated** — e.g. > **How tokens are counted:** OpenAI models use tiktoken (exact). Other models use provider-specific formulas calibrated from benchmark data. For CJK-heavy text, the ratio is blended by CJK character ratio so that Chinese gets fewer chars per token. --- ## Common Issues | Issue | Action | |-------|--------| | "tiktoken is required" | `pip install tiktoken` | | UnsupportedModelError | Use `-l` for valid names | | Cost "NA" | Model has no pricing; count still valid | | User provides URL | **Agent must warn:** outbound request, SSRF risk, only trusted URLs; confirm before `-u` | | Model unclear / vague | **Agent must ask:** user to specify exact model name; offer `-l` to list; do not guess | --- ## When to Trigger This Skill Activate this skill when the user: | Trigger | Example phrases | |---------|-----------------| | **Token count** | "How many tokens?", "Count tokens in this prompt", "Token length of X" | | **Cost estimate** | "Estimate API cost", "How much for this text?", "Cost for GPT-4" | | **Prompt size** | "Check prompt length", "Is this too long?", "Context window limit" | | **OpenClaw audit** | "How many tokens does my workspace use?", "Audit OpenClaw memory/persona/skills", "Which components consume tokens?", "Token usage of AGENTS.md / SOUL.md / skills" | | **Model comparison** | "Compare token cost across models", "Which model is cheaper?" | Also trigger when the agent needs to count tokens or estimate cost before/after generating content. --- ## Quick Reference | Item | Command | |------|---------| | Invoke | `python -m scripts.cli` | | List models | `python -m scripts.cli -l` | | Cost | `-c` (input) / `--output-tokens` (output) | | Currency | `--currency USD` or `INR` | FILE:package.json { "name": "prompt-token-counter", "version": "1.0.10", "description": "OpenClaw skill: Count tokens and estimate costs for 300+ LLM models. Audit workspace token consumption (memory, persona, skills).", "keywords": [ "openclaw", "skill", "token", "toksum", "llm" ], "license": "MIT", "repository": { "type": "git", "url": "https://github.com/Zhaobudaoyuema/prompt-token-counter" }, "homepage": "https://github.com/Zhaobudaoyuema/prompt-token-counter", "bugs": { "url": "https://github.com/Zhaobudaoyuema/prompt-token-counter/issues" }, "files": [ "SKILL.md", "publish_npm.py", "scripts" ] } FILE:README.md # prompt-token-counter [中文](README.zh.md) An **OpenClaw skill** for counting tokens and estimating API costs. ## How It Works This project provides **local token counting** without API calls: | Approach | OpenAI | Other models | |----------|--------|--------------| | **Method** | tiktoken (exact) | Formula-based approximation | | **Accuracy** | Exact | ~85–95% typical | For non-OpenAI models, we use **provider-specific formulas** calibrated from benchmark data. When text contains CJK (Chinese/Japanese/Korean), we **blend the ratio** by CJK character ratio so that Chinese-heavy text gets fewer chars-per-token (more tokens per character). ### Benchmark: Main Models (8927 chars, English/mixed) | Model | Chars | Tokens | 1 token ≈ chars | Status | |-------|-------|--------|-----------------|--------| | anthropic/claude-sonnet-4-6 | 8927 | 2763 | 3.23 | ✓ | | anthropic/claude-sonnet-4-5 | 8927 | 2763 | 3.23 | ✓ | | anthropic/claude-opus-4.6 | 8927 | 2763 | 3.23 | ✓ | | openai/gpt-5.2-codex | 8927 | 2459 | 3.63 | ✓ | | google/gemini-3.1-pro-preview | 8927 | 2627 | 3.40 | ✓ | | z-ai/glm-5 | 8927 | 2457 | 3.63 | ✓ | | volcengine/doubao-seed-2-0-pro | 8927 | 2702 | 3.30 | ✓ | | moonshot/kimi-k2.5 | 8927 | 2402 | 3.72 | ✓ | | minimax/MiniMax-M2.5 | 8927 | 2428 | 3.68 | ✓ | | deepseek-v3.2 | 8927 | 2578 | 3.46 | ✓ | ### Benchmark: Main Models (3050 chars, Chinese-mixed 混杂中文) | Model | Chars | Tokens | 1 token ≈ chars | Status | |-------|-------|--------|-----------------|--------| | anthropic/claude-sonnet-4-6 | 3050 | 1913 | 1.59 | ✓ | | anthropic/claude-sonnet-4-5 | 3050 | 1913 | 1.59 | ✓ | | anthropic/claude-opus-4.6 | 3050 | 1913 | 1.59 | ✓ | | openai/gpt-5.2-codex | 3050 | 1564 | 1.95 | ✓ | | google/gemini-3.1-pro-preview | 3050 | 1473 | 2.07 | ✓ | | z-ai/glm-5 | 3050 | 1318 | 2.31 | ✓ | | volcengine/doubao-seed-2-0-pro | 3050 | 1494 | 2.04 | ✓ | | moonshot/kimi-k2.5 | 3050 | 1257 | 2.43 | ✓ | | minimax/MiniMax-M2.5 | 3050 | 1289 | 2.37 | ✓ | | deepseek-v3.2 | 3050 | 1361 | 2.24 | ✓ | *Benchmark data from `scripts/examples/benchmark_token_ratio.py` (API mode).* ## What This Skill Does When loaded, the agent can: | Capability | Use case | |------------|----------| | **Count tokens** | "How many tokens in this prompt?", "Token length of X" | | **Estimate cost** | "How much for this text on GPT-4?", "API cost for Claude" | | **Audit OpenClaw workspace** | "How many tokens does my workspace use?", "Which memory/persona/skills consume tokens?" | | **Compare models** | "Compare token cost across models", "Which model is cheaper?" | ### OpenClaw Token Audit The skill helps identify token consumption of workspace components: - **Memory & persona**: AGENTS.md, SOUL.md, IDENTITY.md, USER.md, MEMORY.md, TOOLS.md, etc. - **Skills**: Each SKILL.md under `~/.openclaw/skills/` or `workspace/skills/` Example audit (batch mode, multiple files): ```bash python -m scripts.cli -m gpt-4o -c AGENTS.md SOUL.md MEMORY.md ``` ## When to Trigger - User asks about token count, prompt length, API cost - User mentions OpenClaw context size or workspace token usage - Agent needs to audit token consumption before/after changes ## Copy and send to OpenClaw **ClawHub (recommended, international)** ```text Please run clawhub install prompt-token-counter to install this skill, and help me count tokens and estimate API costs. ``` **npm** ```text Please run npm i prompt-token-counter to install this skill, and help me count tokens and estimate API costs. ``` ## Quick Reference ```bash python -m scripts.cli -m gpt-4 "Hello, world!" python -m scripts.cli -f input.txt -m claude-3-opus -c python -m scripts.cli -l # list 300+ models ``` ## Benchmark Script Run `scripts/examples/benchmark_token_ratio.py` to test token ratios across models: - **API mode** (default): Uses model API to get exact `prompt_tokens`. Set `API_KEY` and `BASE_URL` in the script. - **Local mode** (`--local`): Uses this project's TokenCounter (no API). Good for quick comparison. ```bash python scripts/examples/benchmark_token_ratio.py # API mode python scripts/examples/benchmark_token_ratio.py --local # local approximation ``` MIT FILE:README.zh.md # prompt-token-counter [English](README.md) 一个 **OpenClaw skill**，用于统计 token 数量并估算 API 成本。 ## 实现原理本项目提供**本地 token 计数**，无需调用 API： | 方式 | OpenAI | 其他模型 | |------|--------|----------| | **方法** | tiktoken（精确） | 公式近似 | | **精度** | 精确 | 约 85–95% | 对非 OpenAI 模型，使用**按 provider 校准的公式**。当文本包含 CJK（中/日/韩）时，会根据 CJK 字符占比**混合比例**，使中文为主的文本获得更低的 chars/token（每字符更多 token）。 ### 基准测试：主力模型（8927 字符，英文/混合） | 模型 | 字符数 | Token 数 | 1 token ≈ 多少字符 | 状态 | |------|--------|----------|-------------------|------| | anthropic/claude-sonnet-4-6 | 8927 | 2763 | 3.23 | ✓ | | anthropic/claude-sonnet-4-5 | 8927 | 2763 | 3.23 | ✓ | | anthropic/claude-opus-4.6 | 8927 | 2763 | 3.23 | ✓ | | openai/gpt-5.2-codex | 8927 | 2459 | 3.63 | ✓ | | google/gemini-3.1-pro-preview | 8927 | 2627 | 3.40 | ✓ | | z-ai/glm-5 | 8927 | 2457 | 3.63 | ✓ | | volcengine/doubao-seed-2-0-pro | 8927 | 2702 | 3.30 | ✓ | | moonshot/kimi-k2.5 | 8927 | 2402 | 3.72 | ✓ | | minimax/MiniMax-M2.5 | 8927 | 2428 | 3.68 | ✓ | | deepseek-v3.2 | 8927 | 2578 | 3.46 | ✓ | ### 基准测试：主力模型（3050 字符，混杂中文） | 模型 | 字符数 | Token 数 | 1 token ≈ 多少字符 | 状态 | |------|--------|----------|-------------------|------| | anthropic/claude-sonnet-4-6 | 3050 | 1913 | 1.59 | ✓ | | anthropic/claude-sonnet-4-5 | 3050 | 1913 | 1.59 | ✓ | | anthropic/claude-opus-4.6 | 3050 | 1913 | 1.59 | ✓ | | openai/gpt-5.2-codex | 3050 | 1564 | 1.95 | ✓ | | google/gemini-3.1-pro-preview | 3050 | 1473 | 2.07 | ✓ | | z-ai/glm-5 | 3050 | 1318 | 2.31 | ✓ | | volcengine/doubao-seed-2-0-pro | 3050 | 1494 | 2.04 | ✓ | | moonshot/kimi-k2.5 | 3050 | 1257 | 2.43 | ✓ | | minimax/MiniMax-M2.5 | 3050 | 1289 | 2.37 | ✓ | | deepseek-v3.2 | 3050 | 1361 | 2.24 | ✓ | *基准数据来自 `scripts/examples/benchmark_token_ratio.py`（API 模式）。* ## 这个 Skill 能做什么加载后，Agent 可以： | 能力 | 使用场景 | |------|----------| | **统计 token** | 「这段 prompt 有多少 token？」、「X 的 token 长度」 | | **估算成本** | 「这段文字用 GPT-4 要多少钱？」、「Claude 的 API 成本」 | | **审计 OpenClaw 工作区** | 「我的工作区用了多少 token？」、「哪些 memory/persona/skills 消耗 token？」 | | **对比模型** | 「对比不同模型的 token 成本」、「哪个模型更便宜？」 | ### OpenClaw Token 审计该 skill 帮助识别工作区各组件的 token 消耗： - **Memory 与 persona**：AGENTS.md、SOUL.md、IDENTITY.md、USER.md、MEMORY.md、TOOLS.md 等 - **Skills**：`~/.openclaw/skills/` 或 `workspace/skills/` 下的每个 SKILL.md 审计示例（批量多文件）： ```bash python -m scripts.cli -m gpt-4o -c AGENTS.md SOUL.md MEMORY.md ``` ## 何时触发 - 用户询问 token 数量、prompt 长度、API 成本 - 用户提到 OpenClaw 上下文大小或工作区 token 使用 - Agent 需要在变更前后审计 token 消耗 ## 复制发送给 OpenClaw **ClawHub（推荐，国际）** ```text Please run clawhub install prompt-token-counter to install this skill, and help me count tokens and estimate API costs. ``` **npm** ```text Please run npm i prompt-token-counter to install this skill, and help me count tokens and estimate API costs. ``` ## 快速参考 ```bash python -m scripts.cli -m gpt-4 "Hello, world!" python -m scripts.cli -f input.txt -m claude-3-opus -c python -m scripts.cli -l # 列出 300+ 模型 ``` ## 基准测试脚本运行 `scripts/examples/benchmark_token_ratio.py` 可批量测试各模型的 token 比例： - **API 模式**（默认）：调用模型 API 获取精确 `prompt_tokens`。需在脚本中设置 `API_KEY` 和 `BASE_URL`。 - **本地模式**（`--local`）：使用本项目的 TokenCounter 近似计算，无需 API。 ```bash python scripts/examples/benchmark_token_ratio.py # API 模式 python scripts/examples/benchmark_token_ratio.py --local # 本地近似 ``` MIT FILE:scripts/cli.py """ Command-line interface for scripts. This module provides a comprehensive command-line interface for the scripts library, allowing users to count tokens and estimate costs for various LLM models directly from the terminal. The CLI supports: - Token counting for text input or files - Cost estimation with detailed breakdowns - Listing all supported models by provider - Verbose output with detailed information - Support for both input and output token pricing Examples: Basic token counting: .. code-block:: bash scripts "Hello, world!" gpt-4 scripts --file input.txt claude-3-opus-20240229 Cost estimation: .. code-block:: bash scripts --cost "Your text here" gpt-4 scripts --cost --output-tokens "Response text" gpt-4 List supported models: .. code-block:: bash scripts --list-models Verbose output: .. code-block:: bash scripts --verbose --cost --file large_document.txt gpt-4 Functions: main: Main CLI entry point that handles argument parsing and execution list_models: Display all supported models organized by provider The CLI provides comprehensive error handling and user-friendly output formatting for both simple token counting and detailed cost analysis workflows. """ import argparse import sys import urllib.parse import urllib.request from pathlib import Path from typing import List, Optional, Tuple from .core import TokenCounter, get_supported_models, estimate_cost from .exceptions import UnsupportedModelError, TokenizationError def _read_file(path: str) -> str: with open(path, "r", encoding="utf-8") as file_handle: return file_handle.read() def _read_url(url: str) -> str: parsed = urllib.parse.urlparse(url) if parsed.scheme not in {"http", "https"}: raise ValueError("URL must start with http:// or https://") with urllib.request.urlopen(url, timeout=10) as response: return response.read().decode("utf-8", errors="replace") def _collect_inputs(argv: List[str]) -> List[Tuple[str, str, Optional[str]]]: """Collect inputs. Returns list of (source, value, label) where label is path for display.""" items: List[Tuple[str, str, Optional[str]]] = [] i = 0 while i < len(argv): arg = argv[i] if arg in {"--file", "-f"}: if i + 1 >= len(argv): raise ValueError("--file requires a path") path = argv[i + 1] items.append(("file", path, path)) i += 2 continue if arg in {"--url", "-u"}: if i + 1 >= len(argv): raise ValueError("--url requires a URL") items.append(("url", argv[i + 1], argv[i + 1])) i += 2 continue if arg in {"--model", "-m", "--currency"}: i += 2 continue if arg in {"--cost", "-c", "--output-tokens", "--verbose", "-v", "--list-models", "-l"}: i += 1 continue if arg.startswith("-"): i += 1 continue # Positional: treat as file path if it exists, else inline text p = Path(arg) if p.is_file(): items.append(("file", arg, arg)) else: items.append(("text", arg, None)) i += 1 return items def _format_cost(cost: float, currency: str) -> str: if cost <= 0: return "NA" symbol = "₹" if currency.upper() == "INR" else "$" precision = 2 if currency.upper() == "INR" else 6 return f"{symbol}{cost:.{precision}f}" def main() -> None: """ Main CLI entry point. Parses command-line arguments and executes the appropriate scripts functionality. Supports token counting, cost estimation, model listing, and file input processing. The function handles: - Argument parsing and validation - Text input from command line or file - Token counting for specified models - Cost estimation with input/output token differentiation - Model listing with provider organization - Comprehensive error handling and user feedback - Verbose output formatting Command-line Arguments: text (str, optional): Text to count tokens for model (str, optional): Model name (required unless using --list-models) --file, -f (str): Read text from file instead of command line --list-models, -l: List all supported models by provider --cost, -c: Show cost estimation along with token count --output-tokens: Calculate cost for output tokens instead of input --verbose, -v: Show detailed output with additional information Exit Codes: 0: Success 1: Error (unsupported model, file not found, tokenization failure, etc.) Raises: SystemExit: On error conditions or user interruption Examples: Basic usage: .. code-block:: bash scripts "Hello, world!" gpt-4 scripts --file document.txt claude-3-opus-20240229 With cost estimation: .. code-block:: bash scripts --cost --verbose "Long text content" gpt-4 scripts --cost --output-tokens "Response text" gpt-4 List models: .. code-block:: bash scripts --list-models """ parser = argparse.ArgumentParser( description="Count tokens for various LLM models", formatter_class=argparse.RawDescriptionHelpFormatter, epilog=""" Examples: scripts file1.txt file2.txt --model gpt-4 scripts --file input.txt --model claude-3-opus-20240229 scripts --cost --model gpt-4 AGENTS.md SOUL.md MEMORY.md scripts --list-models """ ) parser.add_argument( "paths", nargs="*", metavar="FILE", help="Local file path(s) to count tokens (default input mode). Multiple paths supported for batch." ) parser.add_argument( "--model", "-m", help="Model name (e.g., gpt-4, claude-3-opus-20240229)" ) parser.add_argument( "--file", "-f", action="append", help="Read text from file (can be used multiple times)" ) parser.add_argument( "--url", "-u", action="append", help="Read text from URL (http/https, can be used multiple times)" ) parser.add_argument( "--list-models", "-l", action="store_true", help="List all supported models" ) parser.add_argument( "--cost", "-c", action="store_true", help="Show cost estimation along with token count" ) parser.add_argument( "--currency", default="USD", help="Currency for cost output (USD or INR)" ) parser.add_argument( "--output-tokens", action="store_true", help="Calculate cost for output tokens instead of input tokens" ) parser.add_argument( "--verbose", "-v", action="store_true", help="Show verbose output" ) args = None try: args = parser.parse_args() if args.list_models: list_models() return if not args.model: parser.error("Model name is required unless using --list-models") inputs = _collect_inputs(sys.argv[1:]) # Also add positional paths from argparse (in case -m was before paths) for p in (args.paths or []): if Path(p).is_file(): if not any(src == "file" and val == p for src, val, _ in inputs): inputs.append(("file", p, p)) else: if not any(src == "text" and val == p for src, val, _ in inputs): inputs.append(("text", p, None)) if not inputs: parser.error("Provide local file path(s), --file, or --url inputs") items: List[Tuple[str, str, Optional[str]]] = [] for source, value, label in inputs: if source == "text": items.append((source, value, label)) continue try: if source == "file": content = _read_file(value) items.append((source, content, value)) elif source == "url": content = _read_url(value) items.append((source, content, value)) else: continue except FileNotFoundError: print(f"Error: File '{value}' not found", file=sys.stderr) sys.exit(1) except Exception as e: print(f"Error reading {source} '{value}': {e}", file=sys.stderr) sys.exit(1) try: counter = TokenCounter(args.model) for index, (source, content, label) in enumerate(items, start=1): token_count = counter.count(content) prefix = f"{label}: " if label else f"{index}) " if args.cost: cost = estimate_cost( token_count, args.model, input_tokens=not args.output_tokens, currency=args.currency, ) cost_display = _format_cost(cost, args.currency) print(f"{prefix}tokens={token_count} cost={cost_display}") else: print(f"{prefix}tokens={token_count}") except UnsupportedModelError as e: print(f"Error: {e}", file=sys.stderr) if args.verbose: print("\nUse --list-models to see supported models", file=sys.stderr) sys.exit(1) except TokenizationError as e: print(f"Error: {e}", file=sys.stderr) sys.exit(1) except KeyboardInterrupt: print("\nInterrupted by user", file=sys.stderr) sys.exit(1) except Exception as e: print(f"Unexpected error: {e}", file=sys.stderr) if args is not None and args.verbose: import traceback traceback.print_exc() sys.exit(1) def list_models() -> None: """ List all supported models organized by provider. Displays a comprehensive list of all supported models grouped by their respective providers (OpenAI, Anthropic, Google, Meta, etc.). The output includes model counts per provider and a total count across all providers. The function: - Retrieves all supported models using get_supported_models() - Groups models by provider with clear section headers - Sorts models alphabetically within each provider - Shows model counts for each provider and overall total - Formats output for easy readability Output Format: .. code-block:: text Supported models: ================================================== OPENAI (25 models): ------------------------------ gpt-3.5-turbo gpt-4 gpt-4o ... ANTHROPIC (12 models): ------------------------------ claude-3-haiku-20240307 claude-3-opus-20240229 ... Total: 200+ models Note: This function is typically called when the --list-models CLI flag is used. It provides users with a complete overview of available models for token counting and cost estimation. """ models = get_supported_models() print("Supported models:") print("=" * 50) for provider, model_list in models.items(): print(f"\n{provider.upper()} ({len(model_list)} models):") print("-" * 30) for model in sorted(model_list): print(f" {model}") print(f"\nTotal: {sum(len(models) for models in models.values())} models") if __name__ == "__main__": main() FILE:scripts/core.py """ Core functionality for token counting across different LLM providers. This module contains the main TokenCounter class and supporting functions that provide token counting capabilities for 200+ Large Language Models from 25+ providers. The module implements: - **Precise tokenization** for OpenAI models using the tiktoken library - **Intelligent approximation algorithms** for all other providers - **Provider detection** with case-insensitive model name matching - **Message format support** for chat-based interactions - **Comprehensive error handling** with detailed error messages - **Cost estimation** for supported models with USD/INR currency support Key Components: - :class:`TokenCounter`: Main class for token counting operations - :func:`count_tokens`: Convenience function for quick token counting - :func:`get_supported_models`: Returns all supported models by provider - :func:`estimate_cost`: Calculates estimated costs for token usage Provider Support: The module supports models from major providers including: - **OpenAI**: GPT-4, GPT-3.5, GPT-4o, O1 models, embeddings - **Anthropic**: Claude 3/3.5 (Opus, Sonnet, Haiku), Claude 2, Instant - **Google**: Gemini Pro/Flash, Gemini 1.5/2.0, PaLM models - **Meta**: LLaMA 2/3/3.1/3.2/3.3 in various sizes - **Mistral**: Mistral 7B, Mixtral, Mistral Large variants - **Cohere**: Command, Command-R, Command-R+ models - **xAI**: Grok 1/1.5/2 and beta models - **Chinese providers**: Alibaba Qwen, Baidu ERNIE, Huawei PanGu, Tsinghua ChatGLM - **Code-specialized**: DeepSeek Coder, Replit Code, BigCode StarCoder - **Open source**: EleutherAI, Stability AI, TII Falcon, RWKV - **Enterprise**: Databricks DBRX, Microsoft Phi, Amazon Titan, IBM Granite Tokenization Approach: - **OpenAI models**: Uses official tiktoken encodings (cl100k_base, p50k_base, r50k_base) - **Other providers**: Intelligent approximation based on: - Character count analysis - Whitespace and punctuation detection - Provider-specific adjustment factors - Language-optimized calculations (Chinese, Russian, etc.) The approximation algorithms are calibrated to provide reasonable accuracy for: - Cost estimation and budgeting - Rate limit planning - Content length assessment - Comparative analysis across providers Note: For production applications requiring exact token counts, use OpenAI models with tiktoken. For other providers, the approximations are suitable for planning and estimation purposes. """ import re from typing import Dict, List, Optional, Union, TYPE_CHECKING, Any if TYPE_CHECKING: import tiktoken from anthropic import Anthropic else: try: import tiktoken except ImportError: tiktoken = None try: from anthropic import Anthropic except ImportError: Anthropic = None from .exceptions import UnsupportedModelError, TokenizationError from .registry.models import ( ModelInfo, get_all_supported_models, get_model_info, get_supported_models_by_provider, normalize_model_name, ) from .registry.pricing import USD_TO_INR, get_pricing_rate DEFAULT_FORMULA = (4.0, 0.3) FORMULAS = { "anthropic": (4.0, 0.3), "google": (3.8, 0.25), "meta": (3.5, 0.2), "mistral": (3.7, 0.25), "cohere": (4.2, 0.3), "perplexity": (3.6, 0.2), "huggingface": (4.0, 0.25), "ai21": (3.8, 0.25), "together": (3.9, 0.25), "xai": (3.8, 0.25), "alibaba": (3.2, 0.2), "baidu": (3.3, 0.2), "huawei": (3.4, 0.2), "yandex": (3.6, 0.2), "stability": (3.8, 0.25), "tii": (3.7, 0.25), "eleutherai": (3.6, 0.2), "mosaicml": (3.7, 0.25), "replit": (3.5, 0.2), "minimax": (3.3, 0.2), "aleph_alpha": (3.9, 0.25), "deepseek": (3.6, 0.2), "tsinghua": (3.2, 0.2), "rwkv": (3.8, 0.25), "community": (3.6, 0.2), "microsoft": (3.7, 0.25), "amazon": (3.9, 0.25), "nvidia": (3.6, 0.2), "ibm": (3.8, 0.25), "salesforce": (3.5, 0.2), "bigcode": (3.4, 0.2), "databricks": (4.0, 0.25), "voyage": (3.8, 0.25), "volcengine": (3.3, 0.2), "moonshot": (3.7, 0.2), "glm5": (3.63, 0.15), "minimax_m25": (3.68, 0.15), } # Chars per token for CJK-heavy text (3050 chars benchmark). Used when cjk_ratio > 0.2. CJK_RATIO = { "anthropic": 1.59, "google": 2.07, "volcengine": 2.04, "moonshot": 2.43, "glm5": 2.31, "minimax_m25": 2.37, "minimax": 2.37, "deepseek": 2.24, } DEFAULT_CJK_RATIO = 2.0 def _cjk_ratio(text: str) -> float: """Return fraction of CJK characters (0..1). CJK chars tokenize ~1 char/token vs ~4 for English.""" if not text: return 0.0 cjk = sum(1 for c in text if "\u4e00" <= c <= "\u9fff" or "\u3400" <= c <= "\u4dbf" or "\u3000" <= c <= "\u303f") return cjk / len(text) class TokenCounter: """ A comprehensive token counter for various Large Language Model (LLM) providers. This class provides functionality to count tokens for 200+ different LLMs from 25+ providers, including OpenAI, Anthropic, Google, Meta, Mistral, and many others. It supports both individual text strings and lists of messages (for chat-like interactions). The token counting is precise for OpenAI models using the official tiktoken library, and provides reasonable approximations for other providers using intelligent algorithms calibrated for each provider's tokenization characteristics. Attributes: model (str): The model name (converted to lowercase) provider (str): The detected provider name tokenizer (Optional[Any]): The tokenizer instance (tiktoken for OpenAI, None for others) Supported Providers: - **OpenAI**: GPT-4, GPT-3.5, GPT-4o, O1 models, embeddings (25+ models) - **Anthropic**: Claude 3/3.5 (Opus, Sonnet, Haiku), Claude 2, Instant (12+ models) - **Google**: Gemini Pro/Flash, Gemini 1.5/2.0, PaLM (10+ models) - **Meta**: LLaMA 2/3/3.1/3.2/3.3 in various sizes (15+ models) - **Mistral**: Mistral 7B, Mixtral, Mistral Large variants (10+ models) - **Cohere**: Command, Command-R, Command-R+ (8+ models) - **xAI**: Grok 1/1.5/2 and beta models (4+ models) - **Alibaba**: Qwen 1.5/2.0/2.5 and vision models (20+ models) - **Baidu**: ERNIE 3.0/3.5/4.0 and variants (8+ models) - **Huawei**: PanGu Alpha and Coder models (5+ models) - **Yandex**: YaLM and YaGPT models (4+ models) - **DeepSeek**: Coder, VL, and LLM models (8+ models) - **Tsinghua**: ChatGLM and GLM models (5+ models) - **And 15+ more providers with specialized models** Examples: Basic usage: .. code-block:: python # Count tokens for a single text string counter = TokenCounter("gpt-4") token_count = counter.count("This is a test string.") print(f"Token count: {token_count}") Chat message format: .. code-block:: python # Count tokens for a list of messages (chat format) messages = [ {"role": "user", "content": "Hello!"}, {"role": "assistant", "content": "How can I help you?"}, ] token_count = counter.count_messages(messages) print(f"Token count (messages): {token_count}") Different providers: .. code-block:: python # Compare token counts across providers models = ["gpt-4", "claude-3-opus", "gemini-pro", "llama-3-70b"] text = "Compare tokenization across different models." for model in models: counter = TokenCounter(model) tokens = counter.count(text) print(f"{model}: {tokens} tokens") Cost estimation: .. code-block:: python from scripts.core import estimate_cost counter = TokenCounter("gpt-4") tokens = counter.count("Your text here") cost = estimate_cost(tokens, "gpt-4", input_tokens=True) print(f"Estimated cost: ${cost:.4f}") Tokenization Accuracy: - **OpenAI models**: Exact token counts using official tiktoken encodings - **Other providers**: Approximations with typical accuracy of ±10-20% - **Approximation factors**: Calibrated per provider based on tokenization patterns - **Language optimization**: Adjusted for Chinese, Russian, and other languages Note: For production applications requiring exact token counts, use OpenAI models. For other providers, approximations are suitable for cost estimation, rate limit planning, and comparative analysis. Raises: UnsupportedModelError: If the specified model is not supported TokenizationError: If tokenization fails or required dependencies are missing """ def __init__(self, model: str): """ Initialize the TokenCounter with a specific model. Sets up the appropriate tokenizer based on the model's provider. For OpenAI models, initializes the tiktoken tokenizer with the correct encoding. For other providers, sets up approximation-based token counting. Args: model (str): The model name (e.g., 'gpt-4', 'claude-3-opus-20240229', 'gemini-pro'). Model names are case-insensitive and will be converted to lowercase. Raises: UnsupportedModelError: If the model is not supported. The exception includes a list of all supported models for reference. TokenizationError: If required dependencies are missing (e.g., tiktoken for OpenAI models) or if tokenizer initialization fails. Examples: .. code-block:: python # OpenAI model (requires tiktoken) counter = TokenCounter("gpt-4") # Anthropic model (uses approximation) counter = TokenCounter("claude-3-opus-20240229") # Case-insensitive model names counter = TokenCounter("GPT-4") # Same as "gpt-4" # Google model counter = TokenCounter("gemini-pro") # Meta model counter = TokenCounter("llama-3-70b") Note: The constructor automatically detects the provider based on the model name and sets up the appropriate tokenization method. OpenAI models use precise tiktoken-based counting, while other providers use calibrated approximations. """ self.tokenizer: Optional[Any] = None self.model = normalize_model_name(model) self.model_info = self._get_model_info() self.provider = self.model_info.provider self._setup_tokenizer() def _get_model_info(self) -> ModelInfo: model_info = get_model_info(self.model) if not model_info: raise UnsupportedModelError(self.model, get_all_supported_models()) return model_info def _detect_provider(self) -> str: return self.model_info.provider def _setup_tokenizer(self) -> None: """ Setup the appropriate tokenizer for the model based on its provider. For OpenAI models, initializes the tiktoken tokenizer with the correct encoding (cl100k_base, p50k_base, or r50k_base). For all other providers, sets the tokenizer to None to indicate approximation-based counting will be used. OpenAI Encodings: - **cl100k_base**: GPT-4, GPT-3.5-turbo, GPT-4o, embeddings (most models) - **p50k_base**: text-davinci-003, text-davinci-002 (legacy completion models) - **r50k_base**: GPT-3, davinci, curie, babbage, ada (oldest models) Raises: TokenizationError: If tiktoken is not installed for OpenAI models, or if the tokenizer fails to initialize for any reason. Examples: The tokenizer setup is automatic and transparent: .. code-block:: python # OpenAI model - sets up tiktoken with cl100k_base encoding counter = TokenCounter("gpt-4") # Anthropic model - sets tokenizer to None for approximation counter = TokenCounter("claude-3-opus") Note: This method is called automatically during initialization. Users should not call this method directly. The tokenizer instance is stored in self.tokenizer and used by the count() method. Dependencies: - **tiktoken**: Required for OpenAI models. Install with: ``pip install tiktoken`` - **No dependencies**: Required for other providers (uses built-in approximation) """ if self.provider == "openai": if tiktoken is None: raise TokenizationError( "tiktoken is required for OpenAI models. Install with: pip install tiktoken", model=self.model ) encoding_name = self.model_info.encoding or "cl100k_base" try: self.tokenizer = tiktoken.get_encoding(encoding_name) except Exception as e: raise TokenizationError(f"Failed to load tokenizer: {str(e)}", model=self.model) else: # For all other providers, we'll use approximation since they don't provide public tokenizers self.tokenizer = None def count(self, text: str) -> int: """ Count tokens in the given text. Performs token counting using the appropriate method for the model's provider. For OpenAI models, uses precise tiktoken-based counting. For other providers, uses intelligent approximation algorithms calibrated for each provider. Args: text (str): The text to count tokens for. Must be a string. Returns: int: The number of tokens in the text. Returns 0 for empty strings. Raises: TokenizationError: If tokenization fails, input is invalid, or required dependencies are missing. Includes detailed error context with model name and text preview. Input Validation: The method performs comprehensive input validation: - **None check**: Rejects None input with clear error message - **Type check**: Ensures input is a string, not int/float/list/dict/etc. - **Empty string**: Returns 0 for empty strings (valid case) Tokenization Methods: - **OpenAI models**: Uses tiktoken.encode() for exact token counts - **Other providers**: Uses _approximate_tokens() with provider-specific calibration Provider-Specific Accuracy: - **OpenAI**: 100% accurate (official tokenizer) - **Anthropic**: ~90-95% accurate (well-calibrated approximation) - **Google**: ~85-90% accurate (Gemini-optimized approximation) - **Meta**: ~85-90% accurate (LLaMA-optimized approximation) - **Chinese models**: ~80-90% accurate (character-optimized for Chinese) - **Code models**: ~85-95% accurate (code-pattern optimized) - **Other providers**: ~80-90% accurate (general approximation) Examples: Basic usage: .. code-block:: python counter = TokenCounter("gpt-4") # Simple text tokens = counter.count("Hello, world!") print(f"Tokens: {tokens}") # Exact count for OpenAI # Empty string tokens = counter.count("") print(f"Tokens: {tokens}") # Always returns 0 # Longer text text = "This is a longer text that will be tokenized." tokens = counter.count(text) print(f"Tokens: {tokens}") Comparing providers: .. code-block:: python text = "Compare tokenization across different models." models = ["gpt-4", "claude-3-opus", "gemini-pro"] for model in models: counter = TokenCounter(model) tokens = counter.count(text) print(f"{model}: {tokens} tokens") Error handling: .. code-block:: python try: counter = TokenCounter("gpt-4") tokens = counter.count("Valid text") except TokenizationError as e: print(f"Tokenization failed: {e}") Performance: - **OpenAI models**: Fast (native tiktoken performance) - **Other providers**: Very fast (lightweight approximation algorithms) - **Typical speed**: 10,000+ texts per second for approximation methods Note: For production applications requiring exact token counts, use OpenAI models. For cost estimation, rate limiting, and comparative analysis, approximations provide sufficient accuracy with much better performance. """ # Comprehensive input validation if text is None: raise TokenizationError("Input cannot be None", model=self.model) if not isinstance(text, str): # Handle common invalid types explicitly if isinstance(text, (int, float, bool)): raise TokenizationError(f"Input must be a string, got {type(text).__name__}", model=self.model) elif isinstance(text, (list, tuple, dict, set)): raise TokenizationError(f"Input must be a string, got {type(text).__name__}", model=self.model) else: raise TokenizationError("Input must be a string", model=self.model) # Handle empty string case if not text: return 0 try: if self.provider == "openai": if self.tokenizer is None: raise TokenizationError("Tokenizer not initialized", model=self.model) if not text: return 0 return len(self.tokenizer.encode(text)) else: # Use approximation for all other providers return self._approximate_tokens(text) except TokenizationError: # Re-raise TokenizationError as-is raise except Exception as e: raise TokenizationError(str(e), model=self.model, text_preview=text) def _approximate_tokens(self, text: str) -> int: """ Approximate token count for non-OpenAI models. Uses intelligent approximation algorithms calibrated for each provider's tokenization characteristics. The approximation considers character count, whitespace patterns, punctuation density, and provider-specific factors. Args: text (str): The text to approximate tokens for. Must be a string. Returns: int: Approximated number of tokens. Minimum return value is 1 for non-empty text. Raises: TokenizationError: If text processing fails or input is invalid. Algorithm Components: The approximation algorithm analyzes several text characteristics: 1. **Character count**: Base measurement for token estimation 2. **Whitespace analysis**: Spaces and newlines often become separate tokens 3. **Punctuation analysis**: Special characters frequently tokenize separately 4. **Provider calibration**: Adjustment factors based on tokenizer characteristics Provider-Specific Calibrations: Each provider has calibrated ratios based on empirical analysis: **Major Providers:** - **Anthropic**: ~4 chars/token (Claude guidance), +30% punctuation adjustment - **Google**: ~3.8 chars/token (Gemini-optimized), +25% adjustment - **Meta**: ~3.5 chars/token (LLaMA-optimized), +20% adjustment - **Mistral**: ~3.7 chars/token (GPT-similar), +25% adjustment - **Cohere**: ~4.2 chars/token (conservative), +30% adjustment **Regional/Language-Optimized:** - **Alibaba/Qwen**: ~3.2 chars/token (Chinese-optimized) - **Baidu/ERNIE**: ~3.3 chars/token (Chinese-optimized) - **Huawei/PanGu**: ~3.4 chars/token (Chinese-optimized) - **Yandex/YaLM**: ~3.6 chars/token (Russian-optimized) - **Tsinghua/ChatGLM**: ~3.2 chars/token (Chinese-optimized) **Code-Specialized:** - **DeepSeek Coder**: ~3.6 chars/token (code-optimized) - **Replit Code**: ~3.5 chars/token (code-optimized) - **BigCode StarCoder**: ~3.4 chars/token (code-optimized) - **Salesforce CodeGen**: ~3.5 chars/token (code-optimized) Accuracy Expectations: - **Well-calibrated providers**: ±10-15% accuracy - **Language-optimized**: ±15-20% for target languages - **General approximation**: ±20-25% accuracy - **Code models**: ±10-20% for code content Examples: This method is called automatically by count() for non-OpenAI models: .. code-block:: python # Automatic approximation for Anthropic counter = TokenCounter("claude-3-opus") tokens = counter.count("Hello, world!") # Uses _approximate_tokens() # Different providers give different approximations text = "This is a test sentence with punctuation!" anthropic_counter = TokenCounter("claude-3-opus") google_counter = TokenCounter("gemini-pro") meta_counter = TokenCounter("llama-3-70b") print(f"Anthropic: {anthropic_counter.count(text)} tokens") print(f"Google: {google_counter.count(text)} tokens") print(f"Meta: {meta_counter.count(text)} tokens") Performance: - **Speed**: Very fast, 10,000+ approximations per second - **Memory**: Minimal memory usage, no model loading required - **Dependencies**: No external dependencies required Note: This method should not be called directly. Use the count() method instead, which automatically selects the appropriate tokenization method based on the model's provider. """ if not isinstance(text, str): raise TokenizationError(f"Input must be a string, got {type(text).__name__}", model=self.model) if not text: return 0 try: # Basic character-based approximation char_count = len(text) # Adjust for whitespace (spaces and newlines are often separate tokens) whitespace_count = len(re.findall(r'\s+', text)) # Adjust for punctuation (often separate tokens) punctuation_count = len(re.findall(r'[^\w\s]', text)) except Exception as e: raise TokenizationError(f"Failed to process text: {str(e)}", model=self.model, text_preview=text) ratio, adjustment_factor = FORMULAS.get(self.model_info.formula or self.provider, DEFAULT_FORMULA) formula_key = self.model_info.formula or self.provider cjk = _cjk_ratio(text) if cjk > 0.2: cjk_chars_per_token = CJK_RATIO.get(formula_key) or CJK_RATIO.get(self.provider) or DEFAULT_CJK_RATIO ratio = ratio * (1 - cjk) + cjk_chars_per_token * cjk base_tokens = char_count / ratio adjustment = (whitespace_count + punctuation_count) * adjustment_factor return max(1, int(base_tokens + adjustment)) def count_messages(self, messages: List[Dict[str, str]]) -> int: """ Count tokens for a list of messages in chat format. Processes a list of message dictionaries (typical chat/conversation format) and returns the total token count including any formatting overhead. This method is essential for chat-based applications and conversation analysis. Args: messages (List[Dict[str, str]]): List of message dictionaries. Each message must contain 'role' and 'content' keys. Expected format: .. code-block:: python [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Hello!"}, {"role": "assistant", "content": "Hi there!"} ] Returns: int: Total token count for all messages including formatting overhead. Raises: TokenizationError: If messages format is invalid, contains non-string content, or if tokenization of individual messages fails. Includes detailed error context with message index and content preview. Message Format Validation: The method performs comprehensive validation: - **Input type**: Must be a list, not string/dict/int/etc. - **Message structure**: Each message must be a dictionary - **Required keys**: Each message must have 'role' and 'content' keys - **Content type**: Message content must be a string, not None/int/list/etc. - **Role type**: Message role must be a string if present Formatting Overhead: Different providers handle message formatting differently: - **OpenAI**: Minimal overhead (~1 token per role) - **Anthropic**: No additional formatting overhead - **Other providers**: No additional overhead assumed Common Message Roles: - **system**: System instructions or context - **user**: User input or questions - **assistant**: AI assistant responses - **function**: Function call results (some providers) Examples: Basic chat conversation: .. code-block:: python counter = TokenCounter("gpt-4") messages = [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "What is the capital of France?"}, {"role": "assistant", "content": "The capital of France is Paris."} ] total_tokens = counter.count_messages(messages) print(f"Total conversation tokens: {total_tokens}") Comparing individual vs. message counting: .. code-block:: python counter = TokenCounter("gpt-4") # Count individual messages individual_total = 0 for msg in messages: tokens = counter.count(msg["content"]) individual_total += tokens print(f"{msg['role']}: {tokens} tokens") # Count as message format (includes formatting overhead) message_total = counter.count_messages(messages) print(f"Individual sum: {individual_total}") print(f"Message format: {message_total}") print(f"Formatting overhead: {message_total - individual_total}") Error handling: .. code-block:: python try: counter = TokenCounter("gpt-4") # Invalid format - missing content invalid_messages = [{"role": "user"}] tokens = counter.count_messages(invalid_messages) except TokenizationError as e: print(f"Message format error: {e}") Multi-provider comparison: .. code-block:: python messages = [ {"role": "user", "content": "Hello!"}, {"role": "assistant", "content": "Hi there! How can I help?"} ] models = ["gpt-4", "claude-3-opus", "gemini-pro"] for model in models: counter = TokenCounter(model) tokens = counter.count_messages(messages) print(f"{model}: {tokens} tokens") Performance: - **Speed**: Processes thousands of message lists per second - **Memory**: Minimal additional memory overhead - **Scalability**: Handles conversations with hundreds of messages Use Cases: - **Chat applications**: Calculate conversation costs - **API rate limiting**: Plan request sizes for chat endpoints - **Conversation analysis**: Analyze dialogue token patterns - **Cost estimation**: Budget for chat-based AI applications - **Content moderation**: Assess conversation length and complexity Note: This method is specifically designed for chat/conversation formats. For simple text token counting, use the count() method instead. """ # Comprehensive input validation if messages is None: raise TokenizationError("Messages cannot be None", model=self.model) if not isinstance(messages, list): # Handle common invalid types explicitly if isinstance(messages, str): raise TokenizationError("Messages must be a list, got string", model=self.model) elif isinstance(messages, (int, float, bool)): raise TokenizationError(f"Messages must be a list, got {type(messages).__name__}", model=self.model) elif isinstance(messages, (dict, tuple, set)): raise TokenizationError(f"Messages must be a list, got {type(messages).__name__}", model=self.model) else: raise TokenizationError("Messages must be a list", model=self.model) total_tokens = 0 for i, message in enumerate(messages): if not isinstance(message, dict): raise TokenizationError(f"Message at index {i} must be a dict, got {type(message).__name__}", model=self.model) if 'role' not in message: raise TokenizationError(f"Message at index {i} must have 'role' key", model=self.model) if 'content' not in message: raise TokenizationError(f"Message at index {i} must have 'content' key", model=self.model) # Validate content is a string if not isinstance(message['content'], str): if message['content'] is None: raise TokenizationError(f"Message content at index {i} cannot be None", model=self.model) else: raise TokenizationError(f"Message content at index {i} must be a string, got {type(message['content']).__name__}", model=self.model) # Validate role if present if 'role' in message and not isinstance(message['role'], str): if message['role'] is None: raise TokenizationError(f"Message role at index {i} cannot be None", model=self.model) else: raise TokenizationError(f"Message role at index {i} must be a string, got {type(message['role']).__name__}", model=self.model) # Count content tokens try: content_tokens = self.count(message['content']) total_tokens += content_tokens except TokenizationError: # Re-raise TokenizationError as-is raise except Exception as e: raise TokenizationError(f"Failed to count tokens for message at index {i}: {str(e)}", model=self.model) # Add overhead for message formatting (extremely minimal overhead) if self.provider == "openai": # OpenAI adds minimal formatting overhead if 'role' in message: total_tokens += 1 # Role is typically 1 token elif self.provider == "anthropic": # Claude has no additional formatting overhead beyond content pass else: # Other providers have no additional overhead pass # No additional final message overhead return total_tokens def count_tokens(text: str, model: str) -> int: """ Convenience function to count tokens for a given text and model. This is a simplified interface that creates a TokenCounter instance and performs token counting in a single function call. Ideal for one-off token counting operations without needing to manage TokenCounter instances. Args: text (str): The text to count tokens for. Must be a string. model (str): The model name (e.g., 'gpt-4', 'claude-3-opus-20240229'). Model names are case-insensitive. Returns: int: The number of tokens in the text. Raises: UnsupportedModelError: If the specified model is not supported. TokenizationError: If tokenization fails or input is invalid. Examples: Basic usage: .. code-block:: python from scripts import count_tokens # OpenAI model tokens = count_tokens("Hello, world!", "gpt-4") print(f"GPT-4 tokens: {tokens}") # Anthropic model tokens = count_tokens("Hello, world!", "claude-3-opus") print(f"Claude tokens: {tokens}") # Case-insensitive model names tokens = count_tokens("Hello, world!", "GPT-4") # Same as "gpt-4" Comparing models: .. code-block:: python text = "This is a sample text for comparison." models = ["gpt-4", "gpt-3.5-turbo", "claude-3-opus", "gemini-pro"] for model in models: tokens = count_tokens(text, model) print(f"{model}: {tokens} tokens") Error handling: .. code-block:: python try: tokens = count_tokens("Hello!", "unsupported-model") except UnsupportedModelError as e: print(f"Model not supported: {e}") except TokenizationError as e: print(f"Tokenization failed: {e}") Performance: This function creates a new TokenCounter instance for each call. For multiple operations with the same model, consider using TokenCounter directly for better performance: .. code-block:: python # Less efficient for multiple calls for text in texts: tokens = count_tokens(text, "gpt-4") # More efficient for multiple calls counter = TokenCounter("gpt-4") for text in texts: tokens = counter.count(text) Note: This function is equivalent to: .. code-block:: python counter = TokenCounter(model) return counter.count(text) """ counter = TokenCounter(model) return counter.count(text) def get_supported_models() -> Dict[str, List[str]]: """ Get a comprehensive dictionary of supported models organized by provider. Returns all 200+ supported models grouped by their respective providers, making it easy to discover available models and understand the scope of scripts's capabilities. Returns: Dict[str, List[str]]: Dictionary with provider names as keys and lists of model names as values. Providers include: - **openai**: GPT-4, GPT-3.5, GPT-4o, O1, embeddings (25+ models) - **anthropic**: Claude 3/3.5, Claude 2, Instant (12+ models) - **google**: Gemini Pro/Flash, Gemini 1.5/2.0, PaLM (10+ models) - **meta**: LLaMA 2/3/3.1/3.2/3.3 variants (15+ models) - **mistral**: Mistral 7B, Mixtral, Large variants (10+ models) - **cohere**: Command, Command-R, Command-R+ (8+ models) - **xai**: Grok 1/1.5/2 and beta models (4+ models) - **alibaba**: Qwen 1.5/2.0/2.5 and vision models (20+ models) - **baidu**: ERNIE 3.0/3.5/4.0 variants (8+ models) - **huawei**: PanGu Alpha and Coder models (5+ models) - **yandex**: YaLM and YaGPT models (4+ models) - **deepseek**: Coder, VL, and LLM models (8+ models) - **tsinghua**: ChatGLM and GLM models (5+ models) - **databricks**: DBRX and Dolly models (6+ models) - **voyage**: Voyage embedding models (6+ models) - **And 10+ more providers** Examples: Basic usage: .. code-block:: python from scripts import get_supported_models models = get_supported_models() # List all providers print("Supported providers:") for provider in models.keys(): print(f" {provider}") Explore specific providers: .. code-block:: python models = get_supported_models() # OpenAI models print("OpenAI models:") for model in models["openai"]: print(f" {model}") # Anthropic models print("\\nAnthropic models:") for model in models["anthropic"]: print(f" {model}") Count models by provider: .. code-block:: python models = get_supported_models() print("Model counts by provider:") total_models = 0 for provider, model_list in models.items(): count = len(model_list) total_models += count print(f" {provider}: {count} models") print(f"\\nTotal: {total_models} models") Find models by pattern: .. code-block:: python models = get_supported_models() # Find all GPT-4 variants gpt4_models = [] for model in models["openai"]: if "gpt-4" in model: gpt4_models.append(model) print("GPT-4 variants:") for model in gpt4_models: print(f" {model}") Validate model support: .. code-block:: python models = get_supported_models() def is_model_supported(model_name): model_lower = model_name.lower() for provider_models in models.values(): if model_lower in [m.lower() for m in provider_models]: return True return False # Check if models are supported test_models = ["gpt-4", "claude-3-opus", "unknown-model"] for model in test_models: supported = is_model_supported(model) print(f"{model}: {'✓' if supported else '✗'}") Integration with TokenCounter: .. code-block:: python from scripts import TokenCounter, get_supported_models models = get_supported_models() text = "Test tokenization across providers." # Test a few models from each major provider test_models = { "openai": models["openai"][0], # First OpenAI model "anthropic": models["anthropic"][0], # First Anthropic model "google": models["google"][0], # First Google model "meta": models["meta"][0] # First Meta model } for provider, model in test_models.items(): counter = TokenCounter(model) tokens = counter.count(text) print(f"{provider} ({model}): {tokens} tokens") Provider Categories: The returned dictionary includes models from these categories: **Major Cloud Providers:** - OpenAI, Anthropic, Google, Microsoft, Amazon **AI-First Companies:** - Mistral, Cohere, xAI, Perplexity, AI21 **Regional/Language-Specific:** - Alibaba (Chinese), Baidu (Chinese), Huawei (Chinese) - Yandex (Russian), Tsinghua (Chinese) **Open Source/Research:** - EleutherAI, Stability AI, TII, RWKV, Community models **Enterprise/Specialized:** - Databricks, Voyage, DeepSeek, BigCode, Replit - Nvidia, IBM, Salesforce Note: The model lists are comprehensive but may not include every variant or the very latest models. The library is regularly updated to include new models as they become available. See Also: - :class:`TokenCounter`: For creating token counters with specific models - :func:`count_tokens`: For quick token counting with model validation - :exc:`UnsupportedModelError`: Exception raised for unsupported models """ return get_supported_models_by_provider() def estimate_cost( token_count: int, model: str, input_tokens: bool = True, currency: str = "USD" ) -> float: """ Estimate the cost for a given number of tokens and model. Calculates estimated costs based on current pricing for supported models. Supports both input and output token pricing, as many models have different rates for input vs. output tokens. Provides costs in USD or INR currency. Args: token_count (int): Number of tokens to estimate cost for. Must be non-negative. model (str): Model name (e.g., "gpt-4", "gpt-4o", "claude-3-opus-20240229"). Model names are case-insensitive. input_tokens (bool, optional): True for input token pricing, False for output token pricing. Defaults to True. Many models charge more for output tokens than input tokens. currency (str, optional): Currency code ("USD" or "INR"). Defaults to "USD". Uses current conversion rate for INR. Returns: float: Estimated cost in the specified currency. Returns 0.0 if the model is not in the pricing database or if pricing is not available. Pricing Coverage: The function includes pricing for major models: **OpenAI Models:** - GPT-4: $0.03/$0.06 per 1K tokens (input/output) - GPT-4 Turbo: $0.01/$0.03 per 1K tokens - GPT-4o: $0.005/$0.015 per 1K tokens - GPT-4o Mini: $0.00015/$0.0006 per 1K tokens - GPT-3.5 Turbo: $0.001/$0.002 per 1K tokens **Anthropic Models:** - Claude-3 Opus: $0.015/$0.075 per 1K tokens - Claude-3 Sonnet: $0.003/$0.015 per 1K tokens - Claude-3 Haiku: $0.00025/$0.00125 per 1K tokens - Claude-3.5 Sonnet: $0.003/$0.015 per 1K tokens - Claude-3.5 Haiku: $0.001/$0.005 per 1K tokens **Databricks Models:** - DBRX Instruct: $0.001/$0.002 per 1K tokens - Dolly models: $0.001/$0.002 per 1K tokens **Voyage AI Models:** - All Voyage models: $0.0001/$0.0001 per 1K tokens Examples: Basic cost estimation: .. code-block:: python from scripts import count_tokens, estimate_cost text = "This is a sample text for cost estimation." model = "gpt-4" # Count tokens and estimate cost tokens = count_tokens(text, model) input_cost = estimate_cost(tokens, model, input_tokens=True) output_cost = estimate_cost(tokens, model, input_tokens=False) print(f"Text: '{text}'") print(f"Tokens: {tokens}") print(f"Input cost: ${input_cost:.4f}") print(f"Output cost: ${output_cost:.4f}") Compare costs across models: .. code-block:: python text = "Compare costs across different models." * 100 # Longer text models = ["gpt-4", "gpt-4o", "gpt-3.5-turbo", "claude-3-opus", "claude-3-haiku"] print(f"Text length: {len(text)} characters") print("\\nCost comparison:") for model in models: try: tokens = count_tokens(text, model) input_cost = estimate_cost(tokens, model, input_tokens=True) output_cost = estimate_cost(tokens, model, input_tokens=False) print(f"{model}:") print(f" Tokens: {tokens}") print(f" Input: ${input_cost:.4f}") print(f" Output: ${output_cost:.4f}") except Exception as e: print(f"{model}: Error - {e}") Currency conversion: .. code-block:: python tokens = 1000 model = "gpt-4" # USD pricing cost_usd = estimate_cost(tokens, model, currency="USD") print(f"Cost in USD: ${cost_usd:.4f}") # INR pricing cost_inr = estimate_cost(tokens, model, currency="INR") print(f"Cost in INR: ₹{cost_inr:.2f}") Batch cost estimation: .. code-block:: python texts = [ "Short text", "Medium length text with more content", "Much longer text that will cost more to process" * 10 ] model = "gpt-4o" total_cost = 0 print("Individual text costs:") for i, text in enumerate(texts, 1): tokens = count_tokens(text, model) cost = estimate_cost(tokens, model) total_cost += cost print(f"Text {i}: {tokens} tokens, ${cost:.4f}") print(f"\\nTotal estimated cost: ${total_cost:.4f}") Chat conversation costing: .. code-block:: python from scripts import TokenCounter messages = [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Explain quantum computing."}, {"role": "assistant", "content": "Quantum computing is a revolutionary..."} ] counter = TokenCounter("gpt-4") total_tokens = counter.count_messages(messages) # Estimate costs for the conversation input_cost = estimate_cost(total_tokens, "gpt-4", input_tokens=True) output_cost = estimate_cost(total_tokens, "gpt-4", input_tokens=False) print(f"Conversation tokens: {total_tokens}") print(f"If all input: ${input_cost:.4f}") print(f"If all output: ${output_cost:.4f}") Currency Conversion: - **USD to INR rate**: 83.0 (as of July 2025) - **Rate updates**: The conversion rate is periodically updated - **Precision**: INR costs are calculated from USD base prices Limitations: - **Pricing accuracy**: Based on publicly available pricing, may not reflect current rates or enterprise discounts - **Model coverage**: Only includes models with known pricing - **Rate changes**: Pricing may change without notice - **Approximation**: For non-OpenAI models, token counts are approximated Note: This function provides cost estimates for planning and budgeting purposes. Actual costs may vary based on current pricing, volume discounts, and exact tokenization. Always verify current pricing with the model provider for production applications. See Also: - :func:`count_tokens`: For getting token counts to use with this function - :class:`TokenCounter`: For more complex token counting scenarios - :func:`get_supported_models`: For checking which models are available """ model = normalize_model_name(model) model_info = get_model_info(model) pricing_key = model_info.pricing_id if model_info else model rate = get_pricing_rate(pricing_key, input_tokens) if rate is None: return 0.0 cost_usd: float = (token_count / 1000) * rate return cost_usd * USD_TO_INR if currency.upper() == "INR" else cost_usd FILE:scripts/examples/batch_compare.py #!/usr/bin/env python3 """ Compare token counts across multiple models for local file(s). Run from project root: python scripts/examples/batch_compare.py file1.txt file2.txt python scripts/examples/batch_compare.py -f input.txt -m gpt-4 claude-3-opus """ import argparse import sys from pathlib import Path # Ensure project root is on path sys.path.insert(0, str(Path(__file__).resolve().parents[2])) from scripts.core import TokenCounter from scripts.exceptions import UnsupportedModelError, TokenizationError def main() -> None: parser = argparse.ArgumentParser( description="Compare token counts across models for local file(s)" ) parser.add_argument( "paths", nargs="*", metavar="FILE", help="Local file path(s) to compare", ) parser.add_argument( "-f", "--file", action="append", dest="files", help="Read from file (repeatable)", ) parser.add_argument( "-m", "--models", nargs="*", default=["gpt-4", "claude-3-opus", "gemini-pro"], help="Model names (default: gpt-4 claude-3-opus gemini-pro)", ) args = parser.parse_args() paths = list(args.paths or []) if args.files: paths.extend(args.files) if not paths: parser.error("Provide at least one local file path") models = args.models or ["gpt-4", "claude-3-opus", "gemini-pro"] print(f"Files: {paths}") print(f"Models: {models}") print("---") for path_str in paths: path = Path(path_str) if not path.exists(): print(f"Error: File '{path}' not found", file=sys.stderr) sys.exit(1) text = path.read_text(encoding="utf-8") print(f"\n{path_str}:") for model in models: try: tokens = TokenCounter(model).count(text) print(f" {model}: tokens={tokens}") except (UnsupportedModelError, TokenizationError) as e: print(f" {model}: Error - {e}") if __name__ == "__main__": main() FILE:scripts/examples/benchmark_token_ratio.py #!/usr/bin/env python3 """ 批量测试多个模型的 token 与文字比例。 - API 模式：通过请求模型 API 获取 token 数（需配置 API_KEY、BASE_URL） - 本地模式（--local）：使用本项目的 TokenCounter 近似计算，无需 API """ import argparse import asyncio import sys from datetime import datetime from pathlib import Path from typing import Optional # 测试文本：包含中文、英文、符号（使用 r 前缀避免 \U \u 等被当作转义） SAMPLE_TEXT = r""" --- name: prompt-token-counter description: "Count tokens and estimate costs for 300+ LLM models. Primary use: audit OpenClaw workspace token consumption (memory, persona, skills)." trigger: "token count, cost estimate, prompt length, API cost, OpenClaw audit, workspace token usage, memory/persona/skills tokens, context window limit" --- # Prompt Token Counter (toksum) > **First load reminder:** This skill provides the `scripts` CLI (toksum). Use it when the user asks to count tokens, estimate API costs, or **audit OpenClaw component token consumption** (memory, persona, skills). ## Primary Use: OpenClaw Token Consumption Audit **Goal:** Help users identify which OpenClaw components consume tokens and how much. ### 1. Memory & Persona Files These files are injected into sessions and consume tokens. Search and count them: | File | Purpose | Typical Location | |------|---------|------------------| | `AGENTS.md` | Operating instructions, workflow, priorities | `~/.openclaw/workspace/` | | `SOUL.md` | Persona, tone, values, behavioral guidelines | `~/.openclaw/workspace/` | | `IDENTITY.md` | Name, role, goals, visual description | `~/.openclaw/workspace/` | | `USER.md` | User preferences, communication style | `~/.openclaw/workspace/` | | `MEMORY.md` | Long-term memory, persistent facts | `~/.openclaw/workspace/` | | `TOOLS.md` | Tool quirks, path conventions | `~/.openclaw/workspace/` | | `HEARTBEAT.md` | Periodic maintenance checklist | `~/.openclaw/workspace/` | | `BOOT.md` | Startup ritual (when hooks enabled) | `~/.openclaw/workspace/` | | `memory/YYYY-MM-DD.md` | Daily memory logs | `~/.openclaw/workspace/memory/` | **Workspace path:** Default `~/.openclaw/workspace`; may be overridden in `~/.openclaw/openclaw.json` via `agent.workspace`. ### 2. Skill Files (SKILL.md) Skills are loaded per session. Count each `SKILL.md`: | Location | Scope | |----------|-------| | `~/.openclaw/skills/*/SKILL.md` | OpenClaw managed skills | | `~/.openclaw/workspace/skills/*/SKILL.md` | Workspace-specific skills (override) | ### 3. Audit Workflow 1. **Locate workspace:** Resolve `~/.openclaw/workspace` (or config override). 2. **Collect files:** List all memory/persona files and `SKILL.md` paths above. 3. **Count tokens:** For each file, run `python -m scripts.cli -f -m -c`. 4. **Summarize:** Group by category (memory, persona, skills), report total and per-file. **Example audit command (PowerShell):** ```powershell $ws = "$env:USERPROFILE\.openclaw\workspace" python -m scripts.cli -m gpt-4o -c -f "$ws\AGENTS.md" -f "$ws\SOUL.md" -f "$ws\USER.md" -f "$ws\IDENTITY.md" -f "$ws\MEMORY.md" -f "$ws\TOOLS.md" ``` **Example audit (Bash):** ```bash WS=~/.openclaw/workspace python -m scripts.cli -m gpt-4o -c -f "$WS/AGENTS.md" -f "$WS/SOUL.md" -f "$WS/USER.md" -f "$WS/IDENTITY.md" -f "$WS/MEMORY.md" -f "$WS/TOOLS.md" ``` --- ## Project Layout ``` prompt_token_counter/ ├── SKILL.md ├── package.json # npm package (OpenClaw skill) ├── publish_npm.py # Publish to npm; syncs version └── scripts/ # Python package, CLI + examples ├── cli.py # Entry point ├── core.py # TokenCounter, estimate_cost ├── registry/ │ ├── models.py # 300+ models │ └── pricing.py # Pricing data └── examples/ # Script examples ├── count_prompt.py ├── estimate_cost.py ├── batch_compare.py └── benchmark_token_ratio.py ``` Invoke: `python -m scripts.cli` from project root. ### Version Sync (publish_npm.py) When publishing to npm, `publish_npm.py` bumps the patch version and syncs it to: - `package.json` — `version` - `SKILL.md` — frontmatter `version` - `scripts/__init__.py` — `__version__` Run: `python publish_npm.py` (after `npm login`). --- ## Runtime Dependencies - **Python 3** — required - **tiktoken** (optional) — `pip install tiktoken` for exact OpenAI counts --- ## Language Rule **Respond in the user's language.** Match the user's language (e.g. Chinese if they write in Chinese, English if they write in English). --- ## URL Usage — Mandatory Agent Rule **Before using `-u` / `--url` to fetch content from any URL, you MUST:** 1. **Explicitly warn the user** that the CLI will make an outbound HTTP/HTTPS request to the given URL. 2. **Confirm the URL is trusted** — tell the user: "Only use URLs you fully trust. Untrusted URLs may expose your IP, leak data, or be used for SSRF. Do you confirm this URL is safe?" 3. **Prefer alternatives** — if the user can provide the content via `-f` (local file) or inline text, suggest that instead of URL fetch. 4. **Never auto-fetch** — do not invoke `-u` without the user having explicitly provided the URL and acknowledged the risk. **If the user insists on using a URL:** Proceed only after they confirm. State clearly: "I will fetch from [URL] to count tokens. Proceed?" --- ## Model Name — Mandatory Agent Rule **Before invoking the CLI, you MUST have a concrete model name from the user.** 1. **Require explicit model** — `-m` / `--model` is required. Do not guess or assume; the user must provide the exact name (e.g. gpt-4o, claude-3-5-sonnet-20241022). 2. **If unclear, ask** — if the user says "GPT" or "Claude" or "the latest model" without a specific name, ask: "Please specify the exact model name (e.g. gpt-4o, claude-3-5-sonnet-20241022). Run `python -m scripts.cli -l` to list supported models." 3. **Do not auto-pick** — never substitute a model on behalf of the user without their confirmation. 4. **Validate when possible** — if the model name seems ambiguous, offer `-l` output or confirm: "I'll use [model]. Is that correct?" --- ## CLI Usage ```bash python -m scripts.cli [OPTIONS] [TEXT ...] ``` | Option | Short | Description | |--------|-------|-------------| | `--model` | `-m` | Model name (required unless `--list-models`) — **Agent must obtain exact name from user; ask if unclear** | | `--file` | `-f` | Read from file (repeatable) | | `--url` | `-u` | Read from URL (repeatable) — **Agent must warn user before use; only trusted URLs** | | `--list-models` | `-l` | List supported models | | `--cost` | `-c` | Show cost estimate | | `--output-tokens` | | Use output token pricing | | `--currency` | | USD or INR | | `--verbose` | `-v` | Detailed output | ### Examples ```bash # Inline text python -m scripts.cli -m gpt-4 "Hello, world!" # File with cost python -m scripts.cli -f input.txt -m claude-3-opus -c # Multiple files (OpenClaw audit) python -m scripts.cli -v -c -f AGENTS.md -f SOUL.md -f MEMORY.md -m gpt-4o # List models python -m scripts.cli -l # Run bundled example scripts python scripts/examples/count_prompt.py "Hello, world!" gpt-4 python scripts/examples/estimate_cost.py "Your text" gpt-4 python scripts/examples/batch_compare.py "Compare text" gpt-4 claude-3-opus ``` --- ## Python API ```python from scripts import TokenCounter, count_tokens, estimate_cost, get_supported_models tokens = count_tokens("Hello!", "gpt-4") counter = TokenCounter("claude-3-opus") tokens = counter.count_messages([ {"role": "system", "content": "..."}, {"role": "user", "content": "..."} ]) cost = estimate_cost(tokens, "gpt-4", input_tokens=True) ``` --- ## Supported Models 300+ models across 34+ providers: OpenAI, Anthropic, Google, Meta, Mistral, Cohere, xAI, DeepSeek, etc. Use `python -m scripts.cli -l` for full list. - **OpenAI:** exact via tiktoken - **Others:** ~85–95% approximation --- ## Common Issues | Issue | Action | |-------|--------| | "tiktoken is required" | `pip install tiktoken` | | UnsupportedModelError | Use `-l` for valid names | | Cost "NA" | Model has no pricing; count still valid | | User provides URL | **Agent must warn:** outbound request, SSRF risk, only trusted URLs; confirm before `-u` | | Model unclear / vague | **Agent must ask:** user to specify exact model name; offer `-l` to list; do not guess | --- ## When to Trigger This Skill Activate this skill when the user: | Trigger | Example phrases | |---------|-----------------| | **Token count** | "How many tokens?", "Count tokens in this prompt", "Token length of X" | | **Cost estimate** | "Estimate API cost", "How much for this text?", "Cost for GPT-4" | | **Prompt size** | "Check prompt length", "Is this too long?", "Context window limit" | | **OpenClaw audit** | "How many tokens does my workspace use?", "Audit OpenClaw memory/persona/skills", "Which components consume tokens?", "Token usage of AGENTS.md / SOUL.md / skills" | | **Model comparison** | "Compare token cost across models", "Which model is cheaper?" | Also trigger when the agent needs to count tokens or estimate cost before/after generating content. --- ## Quick Reference | Item | Command | |------|---------| | Invoke | `python -m scripts.cli` | | List models | `python -m scripts.cli -l` | | Cost | `-c` (input) / `--output-tokens` (output) | | Currency | `--currency USD` or `INR` | """ # 待测试的模型列表（可按需修改） MODELS = [ "anthropic/claude-sonnet-4-6", "anthropic/claude-sonnet-4-5", "anthropic/claude-opus-4.6", "openai/gpt-5.2-codex", "google/gemini-3.1-pro-preview", "z-ai/glm-5", "volcengine/doubao-seed-2-0-pro", "moonshot/kimi-k2.5", "minimax/MiniMax-M2.5", "deepseek-v3.2", ] # API 配置（与 skill_executor 保持一致，API 模式时使用） API_KEY = "" BASE_URL = "" def get_token_count_local(model: str, text: str) -> dict: """使用本项目的 TokenCounter 本地近似计算 token 数。""" try: from scripts.core import TokenCounter counter = TokenCounter(model) token_count = counter.count(text) except Exception as e: return { "model": model, "success": False, "error": str(e), "char_count": len(text), "token_count": None, "ratio": None, } char_count = len(text) ratio = char_count / token_count if token_count > 0 else 0 return { "model": model, "success": True, "char_count": char_count, "token_count": token_count, "ratio": round(ratio, 4), "ratio_desc": f"1 token ≈ {ratio:.4f} 字符", } async def get_token_count_api(model: str, text: str) -> dict: """ 请求模型 API，获取输入文本的 token 数。使用 chat completion，从 response.usage.prompt_tokens 获取。依赖: pip install openai """ try: import openai client = openai.AsyncOpenAI(api_key=API_KEY, base_url=BASE_URL) messages = [{"role": "user", "content": text}] response = await client.chat.completions.create( model=model, messages=messages, max_tokens=8 * 1024, # 部分 API 要求 >= 16，只关心 prompt_tokens temperature=0, ) usage = getattr(response, "usage", None) if usage is None: return { "model": model, "success": False, "error": "API 响应中无 usage 字段", "char_count": len(text), "token_count": None, "ratio": None, } prompt_tokens = getattr(usage, "prompt_tokens", None) if prompt_tokens is None: prompt_tokens = getattr(usage, "total_tokens", None) if prompt_tokens is None: return { "model": model, "success": False, "error": "usage 中无 prompt_tokens/total_tokens", "char_count": len(text), "token_count": None, "ratio": None, } char_count = len(text) ratio = char_count / prompt_tokens if prompt_tokens > 0 else 0 return { "model": model, "success": True, "char_count": char_count, "token_count": prompt_tokens, "ratio": round(ratio, 4), "ratio_desc": f"1 token ≈ {ratio:.4f} 字符", } except Exception as e: return { "model": model, "success": False, "error": str(e), "char_count": len(text), "token_count": None, "ratio": None, } async def batch_test_api(models: list[str], text: str) -> list[dict]: """批量异步请求所有模型（API 模式）。""" tasks = [get_token_count_api(m, text) for m in models] results = await asyncio.gather(*tasks, return_exceptions=True) out = [] for i, r in enumerate(results): if isinstance(r, Exception): out.append( { "model": models[i], "success": False, "error": str(r), "char_count": len(text), "token_count": None, "ratio": None, } ) else: out.append(r) return out def batch_test_local(models: list[str], text: str) -> list[dict]: """批量测试所有模型（本地模式）。""" return [get_token_count_local(m, text) for m in models] def write_report(results: list[dict], text: str, output_path: Path, mode: str) -> None: """将测试结果写入 Markdown 文档。""" mode_label = "API" if mode == "api" else "本地近似" lines = [ "# 模型 Token 与文字比例测试报告", "", f"**生成时间**: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}", f"**测试字符数**: {len(text)}", f"**模式**: {mode_label}", "", "| 模型 | 字符数 | Token 数 | 1 token ≈ 多少字符 | 状态 |", "|------|--------|----------|-------------------|------|", ] for r in results: model = r.get("model", "-") char_count = r.get("char_count") or "-" token_count = r.get("token_count") if r.get("token_count") is not None else "-" ratio = r.get("ratio") ratio_str = f"{ratio:.4f}" if isinstance(ratio, (int, float)) else "-" status = "✓" if r.get("success") else f"✗ {r.get('error', '')}" lines.append(f"| {model} | {char_count} | {token_count} | {ratio_str} | {status} |") lines.extend( [ "", "## 说明", "", "- **1 token ≈ 多少字符**：字符数 / token 数，数值越大表示 1 个 token 能表示更多文字", "- 中文通常比英文消耗更多 token", "- 不同模型使用不同 tokenizer，比例会有差异", "", ] ) output_path.write_text("\n".join(lines), encoding="utf-8") print(f"报告已保存: {output_path}") def main( text: Optional[str] = None, text_file: Optional[Path] = None, models: Optional[list[str]] = None, output_path: Optional[Path] = None, use_local: bool = False, ) -> list[dict]: """ 主流程：批量测试并输出报告。 Args: text: 测试文本，默认使用 SAMPLE_TEXT text_file: 从文件读取测试文本（覆盖 text） models: 模型列表，默认使用 MODELS output_path: 输出文件路径，默认 token_ratio_report.md use_local: True 使用本地 TokenCounter，False 使用 API """ if text_file and text_file.exists(): text = text_file.read_text(encoding="utf-8") elif text is None: text = SAMPLE_TEXT models = models or MODELS if output_path is None: output_path = Path(__file__).parent / "token_ratio_report.md" mode = "local" if use_local else "api" if use_local: print(f"本地模式：测试 {len(models)} 个模型，文本长度 {len(text)} 字符...") results = batch_test_local(models, text) else: try: import openai # noqa: F401 except ImportError: print("API 模式需要 openai: pip install openai") print("或使用 --local 进行本地近似测试") return [] if not API_KEY or not BASE_URL: print("API 模式需在脚本中设置 API_KEY 和 BASE_URL") print("或使用 --local 进行本地近似测试") return [] print(f"API 模式：测试 {len(models)} 个模型，文本长度 {len(text)} 字符...") results = asyncio.run(batch_test_api(models, text)) for r in results: if r.get("success"): print(f" {r['model']}: {r['token_count']} tokens, 1 token ≈ {r['ratio']:.4f} 字符") else: print(f" {r['model']}: 失败 - {r.get('error', '')}") write_report(results, text, output_path, mode) return results if __name__ == "__main__": parser = argparse.ArgumentParser(description="批量测试模型 token 与文字比例") parser.add_argument("--local", action="store_true", help="使用本地 TokenCounter 近似（无需 API）") parser.add_argument("--text-file", "-f", type=Path, help="从文件读取测试文本") parser.add_argument("--output", "-o", type=Path, help="输出报告路径") args = parser.parse_args() # 确保从项目根目录运行时可导入 scripts script_dir = Path(__file__).resolve().parent project_root = script_dir.parent.parent if str(project_root) not in sys.path: sys.path.insert(0, str(project_root)) main(text_file=args.text_file, output_path=args.output, use_local=args.local) FILE:scripts/examples/count_prompt.py #!/usr/bin/env python3 """ Count tokens for local file(s). Default: read from local files, batch mode. Run from project root: python scripts/examples/count_prompt.py file1.txt file2.txt -m gpt-4 python scripts/examples/count_prompt.py -f input.txt -m gpt-4 """ import argparse import sys from pathlib import Path # Ensure project root is on path sys.path.insert(0, str(Path(__file__).resolve().parents[2])) from scripts.core import TokenCounter from scripts.exceptions import UnsupportedModelError, TokenizationError def main() -> None: parser = argparse.ArgumentParser( description="Count tokens for local file(s), batch mode" ) parser.add_argument( "paths", nargs="*", metavar="FILE", help="Local file path(s) to count (default input)", ) parser.add_argument( "-f", "--file", action="append", dest="files", help="Read from file (repeatable)", ) parser.add_argument( "-m", "--model", default="gpt-4", help="Model name (default: gpt-4)", ) args = parser.parse_args() paths = list(args.paths or []) if args.files: paths.extend(args.files) if not paths: parser.error("Provide at least one local file path") model = args.model print(f"Model: {model}") print("---") try: counter = TokenCounter(model) for p in paths: path = Path(p) if not path.exists(): print(f"Error: File '{path}' not found", file=sys.stderr) sys.exit(1) text = path.read_text(encoding="utf-8") tokens = counter.count(text) print(f"{p}: tokens={tokens}") except UnsupportedModelError as e: print(f"Error: {e}", file=sys.stderr) sys.exit(1) except TokenizationError as e: print(f"Error: {e}", file=sys.stderr) sys.exit(1) if __name__ == "__main__": main() FILE:scripts/examples/estimate_cost.py #!/usr/bin/env python3 """ Estimate API cost for a prompt (text or file). Run from project root: python scripts/examples/estimate_cost.py [text_or_file] [model] Or: python scripts/examples/estimate_cost.py input.txt gpt-4 """ import argparse import sys from pathlib import Path # Ensure project root is on path sys.path.insert(0, str(Path(__file__).resolve().parents[2])) from scripts.core import TokenCounter, estimate_cost from scripts.exceptions import UnsupportedModelError, TokenizationError def main() -> None: parser = argparse.ArgumentParser( description="Estimate API cost for a prompt (text or file)" ) parser.add_argument( "input", nargs="?", default="Your prompt text here", help="Text to estimate, or path to file", ) parser.add_argument( "model", nargs="?", default="gpt-4", help="Model name (default: gpt-4)", ) parser.add_argument( "-m", "--model", dest="model_opt", help="Model name (alternative to positional)", ) args = parser.parse_args() model = args.model_opt or args.model if Path(args.input).is_file(): path = Path(args.input) if not path.exists(): print(f"Error: File '{path}' not found", file=sys.stderr) sys.exit(1) text = path.read_text(encoding="utf-8") print(f"Estimating cost for file: {path}") else: text = args.input print(f'Estimating cost for text: "{text[:50]}{"..." if len(text) > 50 else ""}"') try: tokens = TokenCounter(model).count(text) cost = estimate_cost(tokens, model, input_tokens=True) symbol = "$" print(f"tokens={tokens} cost={symbol}{cost:.6f}") except UnsupportedModelError as e: print(f"Error: {e}", file=sys.stderr) sys.exit(1) except TokenizationError as e: print(f"Error: {e}", file=sys.stderr) sys.exit(1) if __name__ == "__main__": main() FILE:scripts/exceptions.py """ Custom exceptions for the scripts library. This module defines a hierarchy of custom exceptions that provide detailed error information for various failure modes in token counting operations. The exceptions are designed to give users clear, actionable error messages with relevant context. Exception Hierarchy: .. code-block:: text ToksumError (base) ├── UnsupportedModelError ├── ModelNotFoundError ├── TokenizationError │ ├── InvalidTokenError │ └── EmptyTextError The exception hierarchy allows for both specific error handling and general error catching at different levels of granularity. Usage Examples: Specific exception handling: .. code-block:: python from scripts import TokenCounter from scripts.exceptions import UnsupportedModelError, TokenizationError try: counter = TokenCounter("unknown-model") tokens = counter.count("Hello, world!") except UnsupportedModelError as e: print(f"Model not supported: {e.model}") print(f"Available models: {e.supported_models}") except TokenizationError as e: print(f"Tokenization failed: {e}") if e.model: print(f"Model: {e.model}") if e.text_preview: print(f"Text preview: {e.text_preview}") General error handling: .. code-block:: python from scripts import count_tokens from scripts.exceptions import ToksumError try: tokens = count_tokens("Hello!", "gpt-4") except ToksumError as e: print(f"Toksum error: {e}") except Exception as e: print(f"Unexpected error: {e}") Error Context: All exceptions include relevant context information: - **Model information**: Which model caused the error - **Text previews**: Sample of problematic text (truncated for privacy) - **Supported alternatives**: List of valid options when applicable - **Detailed messages**: Clear descriptions of what went wrong The exceptions are designed to be both human-readable and machine-parseable, making them suitable for both debugging and automated error handling. """ from typing import List, Optional class ToksumError(Exception): """ Base exception class for all scripts library errors. This is the root exception class that all other scripts exceptions inherit from. It allows for catching any scripts-related error with a single exception type while still providing the ability to catch specific error types when needed. This exception should generally not be raised directly. Instead, use one of the more specific exception subclasses that provide additional context and error-specific information. Examples: Catching all scripts errors: .. code-block:: python from scripts import count_tokens from scripts.exceptions import ToksumError try: tokens = count_tokens("Hello!", "some-model") except ToksumError as e: print(f"Toksum error occurred: {e}") # Handle any scripts-related error except Exception as e: print(f"Unexpected error: {e}") # Handle non-scripts errors Catching specific errors with fallback: .. code-block:: python from scripts import TokenCounter from scripts.exceptions import UnsupportedModelError, TokenizationError, ToksumError try: counter = TokenCounter("gpt-4") tokens = counter.count("Hello!") except UnsupportedModelError: print("Model not supported") except TokenizationError: print("Tokenization failed") except ToksumError: print("Other scripts error") except Exception: print("Non-scripts error") Note: This base class provides a common interface for all scripts exceptions but does not add any additional functionality beyond the standard Python Exception class. """ class UnsupportedModelError(ToksumError): """ Raised when an unsupported model is specified. This exception is raised when attempting to create a TokenCounter or use count_tokens() with a model name that is not recognized by the library. The exception includes the invalid model name and optionally a list of supported models for reference. Attributes: model (str): The unsupported model name that was specified supported_models (List[str]): List of supported model names (optional) The exception message automatically includes the unsupported model name and, if provided, a formatted list of supported alternatives. Examples: Basic usage with model validation: .. code-block:: python from scripts import TokenCounter from scripts.exceptions import UnsupportedModelError try: counter = TokenCounter("invalid-model-name") except UnsupportedModelError as e: print(f"Model '{e.model}' is not supported") if e.supported_models: print("Supported models include:") for model in e.supported_models[:5]: # Show first 5 print(f" - {model}") Handling with model suggestions: .. code-block:: python from scripts import get_supported_models from scripts.exceptions import UnsupportedModelError def safe_create_counter(model_name): try: return TokenCounter(model_name) except UnsupportedModelError as e: print(f"Error: {e}") # Suggest similar models all_models = get_supported_models() suggestions = [] for provider_models in all_models.values(): for model in provider_models: if model_name.lower() in model.lower(): suggestions.append(model) if suggestions: print("Did you mean one of these?") for suggestion in suggestions[:3]: print(f" - {suggestion}") return None Programmatic error handling: .. code-block:: python def validate_model(model_name): try: TokenCounter(model_name) return True except UnsupportedModelError: return False # Test multiple models test_models = ["gpt-4", "invalid-model", "claude-3-opus"] for model in test_models: if validate_model(model): print(f"✓ {model} is supported") else: print(f"✗ {model} is not supported") Common Causes: - **Typos in model names**: "gpt4" instead of "gpt-4" - **Incorrect casing**: "GPT-4" vs "gpt-4" (though scripts is case-insensitive) - **Outdated model names**: Using deprecated or renamed models - **Provider confusion**: Using model names from unsupported providers - **Version specificity**: Using overly specific version numbers Resolution: 1. **Check spelling**: Verify the model name is spelled correctly 2. **Use get_supported_models()**: Get the complete list of supported models 3. **Check provider**: Ensure the model provider is supported 4. **Update library**: Newer models may require a library update See Also: - :func:`get_supported_models`: Get all supported models by provider - :class:`TokenCounter`: Main class that raises this exception - :func:`count_tokens`: Convenience function that may raise this exception """ def __init__(self, model: str, supported_models: Optional[List[str]] = None): """ Initialize UnsupportedModelError with model name and optional supported models list. Args: model (str): The unsupported model name that was specified supported_models (Optional[List[str]]): List of supported model names for user reference. If provided, will be included in error message. """ self.model = model self.supported_models = supported_models or [] if self.supported_models: message = f"Model '{model}' is not supported. Supported models: {', '.join(self.supported_models)}" else: message = f"Model '{model}' is not supported." super().__init__(message) class ModelNotFoundError(ToksumError): """ Raised when a specified model is not found. This exception is raised in scenarios where a model name is recognized as potentially valid but cannot be located or accessed. This is distinct from UnsupportedModelError, which is raised for completely unrecognized model names. This exception is typically used for cases where: - A model exists but is temporarily unavailable - A model requires special access or authentication - A model has been deprecated or removed - There are network or service issues preventing model access Attributes: model (str): The model name that could not be found Examples: Basic error handling: .. code-block:: python from scripts.exceptions import ModelNotFoundError try: # Some operation that might fail to find a model result = some_model_operation("gpt-4") except ModelNotFoundError as e: print(f"Model not found: {e}") # Maybe try a fallback model or retry later Distinguishing from UnsupportedModelError: .. code-block:: python from scripts import TokenCounter from scripts.exceptions import UnsupportedModelError, ModelNotFoundError try: counter = TokenCounter("some-model") except UnsupportedModelError: print("Model is not supported by scripts") except ModelNotFoundError: print("Model is supported but cannot be found/accessed") Note: This exception is less commonly used in the current scripts implementation but is available for future extensions and specific use cases where model availability needs to be distinguished from model support. See Also: - :exc:`UnsupportedModelError`: For completely unsupported models - :class:`TokenCounter`: May raise this exception in certain scenarios """ def __init__(self, model: str): """ Initialize ModelNotFoundError with the model name. Args: model (str): The model name that could not be found """ super().__init__(f"Model '{model}' not found.") class TokenizationError(ToksumError): """ Raised when tokenization fails for any reason. This is the primary exception for tokenization-related failures. It provides detailed context about what went wrong, including the model being used and a preview of the problematic text (truncated for privacy and readability). This exception covers a wide range of tokenization failures: - Invalid input types (non-string input) - Missing dependencies (e.g., tiktoken for OpenAI models) - Tokenizer initialization failures - Text processing errors - Encoding/decoding issues Attributes: model (Optional[str]): The model name being used when the error occurred text_preview (Optional[str]): A preview of the problematic text (truncated) The exception message includes the base error description and automatically appends model and text preview information when available. Examples: Basic error handling: .. code-block:: python from scripts import TokenCounter from scripts.exceptions import TokenizationError try: counter = TokenCounter("gpt-4") tokens = counter.count(123) # Invalid input type except TokenizationError as e: print(f"Tokenization failed: {e}") if e.model: print(f"Model: {e.model}") if e.text_preview: print(f"Text preview: {e.text_preview}") Handling missing dependencies: .. code-block:: python try: counter = TokenCounter("gpt-4") # Requires tiktoken tokens = counter.count("Hello!") except TokenizationError as e: if "tiktoken" in str(e): print("Please install tiktoken: pip install tiktoken") else: print(f"Tokenization error: {e}") Robust error handling with fallbacks: .. code-block:: python def safe_count_tokens(text, model, fallback_model=None): try: counter = TokenCounter(model) return counter.count(text) except TokenizationError as e: print(f"Primary model failed: {e}") if fallback_model: try: print(f"Trying fallback model: {fallback_model}") counter = TokenCounter(fallback_model) return counter.count(text) except TokenizationError: print("Fallback model also failed") return None Input validation with detailed errors: .. code-block:: python def validate_and_count(text, model): try: if not isinstance(text, str): raise TokenizationError( f"Input must be string, got {type(text).__name__}", model=model ) counter = TokenCounter(model) return counter.count(text) except TokenizationError as e: print(f"Validation failed: {e}") return None Text Preview Handling: The text_preview attribute contains a truncated version of the problematic text to help with debugging while protecting privacy: - **Truncation**: Long text is truncated to ~50 characters + "..." - **Privacy**: Sensitive information is not fully exposed in error messages - **Debugging**: Provides enough context to identify the problematic content Common Causes: - **Wrong input type**: Passing int, list, dict instead of string - **None input**: Passing None instead of a string - **Missing dependencies**: tiktoken not installed for OpenAI models - **Encoding issues**: Text with problematic character encodings - **Memory issues**: Extremely large text causing processing failures Resolution: 1. **Check input type**: Ensure text is a string 2. **Install dependencies**: Install tiktoken for OpenAI models 3. **Validate text**: Check for encoding issues or special characters 4. **Check text size**: Very large texts may cause issues 5. **Try different model**: Some models may handle edge cases better See Also: - :exc:`InvalidTokenError`: For specific token-related errors - :exc:`EmptyTextError`: For empty text specific errors - :class:`TokenCounter`: Main class that raises this exception """ def __init__(self, message: str, model: Optional[str] = None, text_preview: Optional[str] = None): """ Initialize TokenizationError with detailed context. Args: message (str): The base error message describing what went wrong model (Optional[str]): The model name being used when error occurred text_preview (Optional[str]): Preview of the problematic text. Will be truncated if longer than 50 characters. """ self.model = model self.text_preview = text_preview full_message = f"Tokenization failed: {message}" if model: full_message += f" (model: {model})" if text_preview: preview = text_preview[:50] + "..." if len(text_preview) > 50 else text_preview full_message += f" (text preview: '{preview}')" super().__init__(full_message) class InvalidTokenError(TokenizationError): """ Raised when an invalid token is encountered during tokenization. This exception is a specialized form of TokenizationError that specifically handles cases where individual tokens are invalid or problematic. It provides additional context about the specific token that caused the issue. This exception might be raised when: - A token contains invalid characters or sequences - A token exceeds maximum length limits - A token has encoding issues - A token conflicts with model-specific restrictions Attributes: token (str): The specific invalid token that caused the error model (Optional[str]): The model name (inherited from TokenizationError) text_preview (Optional[str]): Preview of the text (inherited from TokenizationError) Examples: Handling invalid token errors: .. code-block:: python from scripts import TokenCounter from scripts.exceptions import InvalidTokenError, TokenizationError try: counter = TokenCounter("gpt-4") tokens = counter.count("Some text with problematic content") except InvalidTokenError as e: print(f"Invalid token encountered: '{e.token}'") print(f"Error details: {e}") # Maybe try preprocessing the text to remove problematic tokens except TokenizationError as e: print(f"General tokenization error: {e}") Token-specific error handling: .. code-block:: python def safe_tokenize_with_cleanup(text, model): try: counter = TokenCounter(model) return counter.count(text) except InvalidTokenError as e: print(f"Removing problematic token: {e.token}") # Remove or replace the problematic token cleaned_text = text.replace(e.token, "") return counter.count(cleaned_text) except TokenizationError: print("Could not tokenize even after cleanup") return None Note: This exception is not commonly raised in the current scripts implementation but is available for future enhancements and edge cases where specific token validation is needed. See Also: - :exc:`TokenizationError`: Parent class for general tokenization errors - :exc:`EmptyTextError`: For empty text specific cases - :class:`TokenCounter`: May raise this exception in specific scenarios """ def __init__(self, token: str, message: str, model: Optional[str] = None, text_preview: Optional[str] = None): """ Initialize InvalidTokenError with token and context information. Args: token (str): The specific invalid token that caused the error message (str): Description of why the token is invalid model (Optional[str]): The model name being used text_preview (Optional[str]): Preview of the problematic text """ full_message = f"Invalid token '{token}': {message}" super().__init__(full_message, model, text_preview) class EmptyTextError(TokenizationError): """ Raised when attempting to tokenize empty text in contexts where it's not allowed. This exception is a specialized form of TokenizationError for cases where empty text is specifically problematic. Note that in most scripts operations, empty text is handled gracefully and returns 0 tokens. This exception is reserved for contexts where empty text indicates a logical error. This exception might be raised when: - Empty text is passed to functions that require non-empty content - Batch operations encounter unexpected empty strings - Validation functions detect empty content where it shouldn't be - Message content is empty in chat format validation Attributes: model (Optional[str]): The model name (inherited from TokenizationError) Examples: Handling empty text validation: .. code-block:: python from scripts.exceptions import EmptyTextError, TokenizationError def validate_content(text, model): try: if not text.strip(): # Check for empty or whitespace-only raise EmptyTextError(model=model) counter = TokenCounter(model) return counter.count(text) except EmptyTextError as e: print("Error: Content cannot be empty") return None except TokenizationError as e: print(f"Tokenization error: {e}") return None Batch processing with empty text handling: .. code-block:: python def process_text_batch(texts, model): results = [] counter = TokenCounter(model) for i, text in enumerate(texts): try: if not text: raise EmptyTextError(model=model) tokens = counter.count(text) results.append(tokens) except EmptyTextError: print(f"Skipping empty text at index {i}") results.append(0) # or None, depending on requirements except TokenizationError as e: print(f"Error processing text {i}: {e}") results.append(None) return results Message validation: .. code-block:: python def validate_messages(messages): for i, msg in enumerate(messages): if not msg.get("content", "").strip(): raise EmptyTextError( f"Message {i} has empty content" ) Normal Empty Text Handling: In most scripts operations, empty text is handled normally: .. code-block:: python counter = TokenCounter("gpt-4") tokens = counter.count("") # Returns 0, no exception messages = [ {"role": "user", "content": ""}, # This might raise EmptyTextError {"role": "assistant", "content": "Hello!"} ] Use Cases: - **Content validation**: Ensuring required fields are not empty - **Batch processing**: Identifying problematic entries in datasets - **API validation**: Checking inputs before expensive operations - **Data quality**: Ensuring content meets minimum requirements Note: This exception is used sparingly in scripts. Most empty text cases are handled gracefully by returning 0 tokens. Use this exception only when empty text represents a logical error in your application. See Also: - :exc:`TokenizationError`: Parent class for general tokenization errors - :exc:`InvalidTokenError`: For specific token validation issues - :meth:`TokenCounter.count`: Handles empty text gracefully (returns 0) - :meth:`TokenCounter.count_messages`: May raise this for empty message content """ def __init__(self, model: Optional[str] = None): """ Initialize EmptyTextError with optional model context. Args: model (Optional[str]): The model name being used when the error occurred """ super().__init__("Cannot tokenize empty text.", model) FILE:scripts/registry/models.py """Model registry for scripts. Defines a canonical model registry with metadata needed for provider detection, OpenAI encoding selection, approximate token formulas, and pricing lookup. """ from __future__ import annotations import difflib import re from dataclasses import dataclass from typing import Dict, Iterable, List, Optional @dataclass(frozen=True) class ModelInfo: name: str provider: str encoding: Optional[str] = None formula: Optional[str] = None pricing_id: Optional[str] = None aliases: Optional[List[str]] = None def normalize_model_name(model: str) -> str: return model.strip().lower() MODELS: Dict[str, ModelInfo] = { # OpenAI "gpt-4": ModelInfo("gpt-4", "openai", encoding="cl100k_base", pricing_id="gpt-4"), "gpt-4-0314": ModelInfo("gpt-4-0314", "openai", encoding="cl100k_base", pricing_id="gpt-4"), "gpt-4-0613": ModelInfo("gpt-4-0613", "openai", encoding="cl100k_base", pricing_id="gpt-4"), "gpt-4-32k": ModelInfo("gpt-4-32k", "openai", encoding="cl100k_base", pricing_id="gpt-4-32k"), "gpt-4-32k-0314": ModelInfo("gpt-4-32k-0314", "openai", encoding="cl100k_base", pricing_id="gpt-4-32k"), "gpt-4-32k-0613": ModelInfo("gpt-4-32k-0613", "openai", encoding="cl100k_base", pricing_id="gpt-4-32k"), "gpt-4-1106-preview": ModelInfo("gpt-4-1106-preview", "openai", encoding="cl100k_base"), "gpt-4-0125-preview": ModelInfo("gpt-4-0125-preview", "openai", encoding="cl100k_base"), "gpt-4-turbo-preview": ModelInfo("gpt-4-turbo-preview", "openai", encoding="cl100k_base"), "gpt-4-vision-preview": ModelInfo("gpt-4-vision-preview", "openai", encoding="cl100k_base"), "gpt-4-turbo": ModelInfo("gpt-4-turbo", "openai", encoding="cl100k_base", pricing_id="gpt-4-turbo"), "gpt-4-turbo-2024-04-09": ModelInfo("gpt-4-turbo-2024-04-09", "openai", encoding="cl100k_base", pricing_id="gpt-4-turbo"), "gpt-4o": ModelInfo("gpt-4o", "openai", encoding="cl100k_base", pricing_id="gpt-4o"), "gpt-4o-2024-05-13": ModelInfo("gpt-4o-2024-05-13", "openai", encoding="cl100k_base", pricing_id="gpt-4o"), "gpt-4o-mini": ModelInfo("gpt-4o-mini", "openai", encoding="cl100k_base", pricing_id="gpt-4o-mini"), "gpt-4o-mini-2024-07-18": ModelInfo("gpt-4o-mini-2024-07-18", "openai", encoding="cl100k_base", pricing_id="gpt-4o-mini"), "gpt-4o-2024-08-06": ModelInfo("gpt-4o-2024-08-06", "openai", encoding="cl100k_base", pricing_id="gpt-4o"), "gpt-4o-2024-11-20": ModelInfo("gpt-4o-2024-11-20", "openai", encoding="cl100k_base", pricing_id="gpt-4o"), "gpt-4-1106-vision-preview": ModelInfo("gpt-4-1106-vision-preview", "openai", encoding="cl100k_base"), "gpt-3.5-turbo": ModelInfo("gpt-3.5-turbo", "openai", encoding="cl100k_base", pricing_id="gpt-3.5-turbo"), "gpt-3.5-turbo-0301": ModelInfo("gpt-3.5-turbo-0301", "openai", encoding="cl100k_base", pricing_id="gpt-3.5-turbo"), "gpt-3.5-turbo-0613": ModelInfo("gpt-3.5-turbo-0613", "openai", encoding="cl100k_base", pricing_id="gpt-3.5-turbo"), "gpt-3.5-turbo-1106": ModelInfo("gpt-3.5-turbo-1106", "openai", encoding="cl100k_base", pricing_id="gpt-3.5-turbo"), "gpt-3.5-turbo-0125": ModelInfo("gpt-3.5-turbo-0125", "openai", encoding="cl100k_base", pricing_id="gpt-3.5-turbo"), "gpt-3.5-turbo-16k": ModelInfo("gpt-3.5-turbo-16k", "openai", encoding="cl100k_base", pricing_id="gpt-3.5-turbo-16k"), "gpt-3.5-turbo-16k-0613": ModelInfo("gpt-3.5-turbo-16k-0613", "openai", encoding="cl100k_base", pricing_id="gpt-3.5-turbo-16k"), "gpt-3.5-turbo-instruct": ModelInfo("gpt-3.5-turbo-instruct", "openai", encoding="cl100k_base"), "text-davinci-003": ModelInfo("text-davinci-003", "openai", encoding="p50k_base"), "text-davinci-002": ModelInfo("text-davinci-002", "openai", encoding="p50k_base"), "text-curie-001": ModelInfo("text-curie-001", "openai", encoding="r50k_base"), "text-babbage-001": ModelInfo("text-babbage-001", "openai", encoding="r50k_base"), "text-ada-001": ModelInfo("text-ada-001", "openai", encoding="r50k_base"), "davinci": ModelInfo("davinci", "openai", encoding="r50k_base"), "curie": ModelInfo("curie", "openai", encoding="r50k_base"), "babbage": ModelInfo("babbage", "openai", encoding="r50k_base"), "ada": ModelInfo("ada", "openai", encoding="r50k_base"), "gpt-3": ModelInfo("gpt-3", "openai", encoding="r50k_base"), "text-embedding-ada-002": ModelInfo("text-embedding-ada-002", "openai", encoding="cl100k_base"), "text-embedding-3-small": ModelInfo("text-embedding-3-small", "openai", encoding="cl100k_base"), "text-embedding-3-large": ModelInfo("text-embedding-3-large", "openai", encoding="cl100k_base"), "gpt-4-base": ModelInfo("gpt-4-base", "openai", encoding="cl100k_base"), "gpt-3.5-turbo-instruct-0914": ModelInfo("gpt-3.5-turbo-instruct-0914", "openai", encoding="cl100k_base"), "o1-preview": ModelInfo("o1-preview", "openai", encoding="cl100k_base"), "o1-mini": ModelInfo("o1-mini", "openai", encoding="cl100k_base"), "o1-preview-2024-09-12": ModelInfo("o1-preview-2024-09-12", "openai", encoding="cl100k_base"), "o1-mini-2024-09-12": ModelInfo("o1-mini-2024-09-12", "openai", encoding="cl100k_base"), "gpt-5.2-codex": ModelInfo("gpt-5.2-codex", "openai", encoding="cl100k_base"), "gpt-4-vision": ModelInfo("gpt-4-vision", "openai", encoding="cl100k_base"), "gpt-4-vision-preview-0409": ModelInfo("gpt-4-vision-preview-0409", "openai", encoding="cl100k_base"), "gpt-4-vision-preview-1106": ModelInfo("gpt-4-vision-preview-1106", "openai", encoding="cl100k_base"), "text-similarity-ada-001": ModelInfo("text-similarity-ada-001", "openai", encoding="r50k_base"), "text-similarity-babbage-001": ModelInfo("text-similarity-babbage-001", "openai", encoding="r50k_base"), "text-similarity-curie-001": ModelInfo("text-similarity-curie-001", "openai", encoding="r50k_base"), "text-similarity-davinci-001": ModelInfo("text-similarity-davinci-001", "openai", encoding="r50k_base"), # Anthropic "claude-3-opus-20240229": ModelInfo("claude-3-opus-20240229", "anthropic", formula="anthropic", pricing_id="claude-3-opus-20240229"), "claude-3-sonnet-20240229": ModelInfo("claude-3-sonnet-20240229", "anthropic", formula="anthropic", pricing_id="claude-3-sonnet-20240229"), "claude-3-haiku-20240307": ModelInfo("claude-3-haiku-20240307", "anthropic", formula="anthropic", pricing_id="claude-3-haiku-20240307"), "claude-3.5-sonnet-20240620": ModelInfo("claude-3.5-sonnet-20240620", "anthropic", formula="anthropic", pricing_id="claude-3.5-sonnet-20240620"), "claude-3.5-sonnet-20241022": ModelInfo("claude-3.5-sonnet-20241022", "anthropic", formula="anthropic", pricing_id="claude-3.5-sonnet-20241022"), "claude-3.5-haiku-20241022": ModelInfo("claude-3.5-haiku-20241022", "anthropic", formula="anthropic", pricing_id="claude-3.5-haiku-20241022"), "claude-3-5-sonnet-20240620": ModelInfo("claude-3-5-sonnet-20240620", "anthropic", formula="anthropic", pricing_id="claude-3.5-sonnet-20240620"), "claude-3-opus": ModelInfo("claude-3-opus", "anthropic", formula="anthropic", pricing_id="claude-3-opus-20240229"), "claude-3-sonnet": ModelInfo("claude-3-sonnet", "anthropic", formula="anthropic", pricing_id="claude-3-sonnet-20240229"), "claude-3-haiku": ModelInfo("claude-3-haiku", "anthropic", formula="anthropic", pricing_id="claude-3-haiku-20240307"), "claude-2.1": ModelInfo("claude-2.1", "anthropic", formula="anthropic"), "claude-2.0": ModelInfo("claude-2.0", "anthropic", formula="anthropic"), "claude-instant-1.2": ModelInfo("claude-instant-1.2", "anthropic", formula="anthropic"), "claude-instant-1.1": ModelInfo("claude-instant-1.1", "anthropic", formula="anthropic"), "claude-instant-1.0": ModelInfo("claude-instant-1.0", "anthropic", formula="anthropic"), "claude-instant": ModelInfo("claude-instant", "anthropic", formula="anthropic"), "claude-1": ModelInfo("claude-1", "anthropic", formula="anthropic"), "claude-1.3": ModelInfo("claude-1.3", "anthropic", formula="anthropic"), "claude-1.3-100k": ModelInfo("claude-1.3-100k", "anthropic", formula="anthropic"), "claude-3-5-haiku-20241022": ModelInfo("claude-3-5-haiku-20241022", "anthropic", formula="anthropic", pricing_id="claude-3.5-haiku-20241022"), "claude-3-5-sonnet-20241022": ModelInfo("claude-3-5-sonnet-20241022", "anthropic", formula="anthropic"), "claude-3.5-sonnet-computer-use": ModelInfo("claude-3.5-sonnet-computer-use", "anthropic", formula="anthropic"), "claude-2.1-200k": ModelInfo("claude-2.1-200k", "anthropic", formula="anthropic"), "claude-2.1-100k": ModelInfo("claude-2.1-100k", "anthropic", formula="anthropic"), "claude-instant-2": ModelInfo("claude-instant-2", "anthropic", formula="anthropic"), "claude-instant-2.0": ModelInfo("claude-instant-2.0", "anthropic", formula="anthropic"), "claude-3-opus-latest": ModelInfo("claude-3-opus-latest", "anthropic", formula="anthropic", pricing_id="claude-3-opus-20240229"), "claude-3-sonnet-latest": ModelInfo("claude-3-sonnet-latest", "anthropic", formula="anthropic", pricing_id="claude-3-sonnet-20240229"), "claude-sonnet-4-6": ModelInfo("claude-sonnet-4-6", "anthropic", formula="anthropic"), "claude-sonnet-4-5": ModelInfo("claude-sonnet-4-5", "anthropic", formula="anthropic"), "claude-opus-4.6": ModelInfo("claude-opus-4.6", "anthropic", formula="anthropic"), # Google "gemini-pro": ModelInfo("gemini-pro", "google", formula="google"), "gemini-pro-vision": ModelInfo("gemini-pro-vision", "google", formula="google"), "gemini-1.5-pro": ModelInfo("gemini-1.5-pro", "google", formula="google"), "gemini-1.5-flash": ModelInfo("gemini-1.5-flash", "google", formula="google"), "gemini-1.5-pro-latest": ModelInfo("gemini-1.5-pro-latest", "google", formula="google"), "gemini-1.5-flash-latest": ModelInfo("gemini-1.5-flash-latest", "google", formula="google"), "gemini-1.0-pro": ModelInfo("gemini-1.0-pro", "google", formula="google"), "gemini-1.0-pro-vision": ModelInfo("gemini-1.0-pro-vision", "google", formula="google"), "gemini-ultra": ModelInfo("gemini-ultra", "google", formula="google"), "gemini-2.0-flash-exp": ModelInfo("gemini-2.0-flash-exp", "google", formula="google"), "gemini-2.0-flash": ModelInfo("gemini-2.0-flash", "google", formula="google"), "gemini-exp-1206": ModelInfo("gemini-exp-1206", "google", formula="google"), "gemini-exp-1121": ModelInfo("gemini-exp-1121", "google", formula="google"), "palm-2": ModelInfo("palm-2", "google", formula="google"), "palm-2-chat": ModelInfo("palm-2-chat", "google", formula="google"), "palm-2-codechat": ModelInfo("palm-2-codechat", "google", formula="google"), "gemini-1.0-pro-001": ModelInfo("gemini-1.0-pro-001", "google", formula="google"), "gemini-1.0-pro-latest": ModelInfo("gemini-1.0-pro-latest", "google", formula="google"), "gemini-1.0-pro-vision-latest": ModelInfo("gemini-1.0-pro-vision-latest", "google", formula="google"), "gemini-3.1-pro-preview": ModelInfo("gemini-3.1-pro-preview", "google", formula="google"), # Meta "llama-2-7b": ModelInfo("llama-2-7b", "meta", formula="meta"), "llama-2-13b": ModelInfo("llama-2-13b", "meta", formula="meta"), "llama-2-70b": ModelInfo("llama-2-70b", "meta", formula="meta"), "llama-3-8b": ModelInfo("llama-3-8b", "meta", formula="meta"), "llama-3-70b": ModelInfo("llama-3-70b", "meta", formula="meta"), "llama-3.1-8b": ModelInfo("llama-3.1-8b", "meta", formula="meta"), "llama-3.1-70b": ModelInfo("llama-3.1-70b", "meta", formula="meta"), "llama-3.1-405b": ModelInfo("llama-3.1-405b", "meta", formula="meta"), "llama-3.2-1b": ModelInfo("llama-3.2-1b", "meta", formula="meta"), "llama-3.2-3b": ModelInfo("llama-3.2-3b", "meta", formula="meta"), "llama-3.3-70b": ModelInfo("llama-3.3-70b", "meta", formula="meta"), "llama-3.3-70b-instruct": ModelInfo("llama-3.3-70b-instruct", "meta", formula="meta"), "llama-2-7b-chat": ModelInfo("llama-2-7b-chat", "meta", formula="meta"), "llama-2-13b-chat": ModelInfo("llama-2-13b-chat", "meta", formula="meta"), "llama-2-70b-chat": ModelInfo("llama-2-70b-chat", "meta", formula="meta"), "llama-2-7b-chat-hf": ModelInfo("llama-2-7b-chat-hf", "meta", formula="meta"), "llama-2-13b-chat-hf": ModelInfo("llama-2-13b-chat-hf", "meta", formula="meta"), "llama-2-70b-chat-hf": ModelInfo("llama-2-70b-chat-hf", "meta", formula="meta"), "llama-3-8b-instruct": ModelInfo("llama-3-8b-instruct", "meta", formula="meta"), "llama-3-70b-instruct": ModelInfo("llama-3-70b-instruct", "meta", formula="meta"), "llama-3.1-8b-instruct": ModelInfo("llama-3.1-8b-instruct", "meta", formula="meta"), "llama-3.1-70b-instruct": ModelInfo("llama-3.1-70b-instruct", "meta", formula="meta"), "llama-3.1-405b-instruct": ModelInfo("llama-3.1-405b-instruct", "meta", formula="meta"), "llama-3.2-1b-instruct": ModelInfo("llama-3.2-1b-instruct", "meta", formula="meta"), "llama-3.2-3b-instruct": ModelInfo("llama-3.2-3b-instruct", "meta", formula="meta"), # Mistral "mistral-7b": ModelInfo("mistral-7b", "mistral", formula="mistral"), "mistral-8x7b": ModelInfo("mistral-8x7b", "mistral", formula="mistral"), "mistral-large": ModelInfo("mistral-large", "mistral", formula="mistral"), "mistral-medium": ModelInfo("mistral-medium", "mistral", formula="mistral"), "mistral-small": ModelInfo("mistral-small", "mistral", formula="mistral"), "mistral-tiny": ModelInfo("mistral-tiny", "mistral", formula="mistral"), "mixtral-8x7b": ModelInfo("mixtral-8x7b", "mistral", formula="mistral"), "mixtral-8x22b": ModelInfo("mixtral-8x22b", "mistral", formula="mistral"), "mistral-large-2": ModelInfo("mistral-large-2", "mistral", formula="mistral"), "mistral-large-2407": ModelInfo("mistral-large-2407", "mistral", formula="mistral"), "mistral-7b-instruct": ModelInfo("mistral-7b-instruct", "mistral", formula="mistral"), "mistral-7b-instruct-v0.1": ModelInfo("mistral-7b-instruct-v0.1", "mistral", formula="mistral"), "mistral-7b-instruct-v0.2": ModelInfo("mistral-7b-instruct-v0.2", "mistral", formula="mistral"), "mistral-7b-instruct-v0.3": ModelInfo("mistral-7b-instruct-v0.3", "mistral", formula="mistral"), "mixtral-8x7b-instruct": ModelInfo("mixtral-8x7b-instruct", "mistral", formula="mistral"), "mixtral-8x22b-instruct": ModelInfo("mixtral-8x22b-instruct", "mistral", formula="mistral"), # Cohere "command": ModelInfo("command", "cohere", formula="cohere"), "command-light": ModelInfo("command-light", "cohere", formula="cohere"), "command-nightly": ModelInfo("command-nightly", "cohere", formula="cohere"), "command-r": ModelInfo("command-r", "cohere", formula="cohere"), "command-r-plus": ModelInfo("command-r-plus", "cohere", formula="cohere"), "command-r-08-2024": ModelInfo("command-r-08-2024", "cohere", formula="cohere"), "command-r-plus-08-2024": ModelInfo("command-r-plus-08-2024", "cohere", formula="cohere"), "command-r-plus-04-2024": ModelInfo("command-r-plus-04-2024", "cohere", formula="cohere"), # Perplexity "pplx-7b-online": ModelInfo("pplx-7b-online", "perplexity", formula="perplexity"), "pplx-70b-online": ModelInfo("pplx-70b-online", "perplexity", formula="perplexity"), "pplx-7b-chat": ModelInfo("pplx-7b-chat", "perplexity", formula="perplexity"), "pplx-70b-chat": ModelInfo("pplx-70b-chat", "perplexity", formula="perplexity"), "codellama-34b-instruct": ModelInfo("codellama-34b-instruct", "perplexity", formula="perplexity"), # HuggingFace "microsoft/DialoGPT-medium": ModelInfo("microsoft/DialoGPT-medium", "huggingface", formula="huggingface"), "microsoft/DialoGPT-large": ModelInfo("microsoft/DialoGPT-large", "huggingface", formula="huggingface"), "facebook/blenderbot-400M-distill": ModelInfo("facebook/blenderbot-400M-distill", "huggingface", formula="huggingface"), "facebook/blenderbot-1B-distill": ModelInfo("facebook/blenderbot-1B-distill", "huggingface", formula="huggingface"), "facebook/blenderbot-3B": ModelInfo("facebook/blenderbot-3B", "huggingface", formula="huggingface"), # AI21 "j2-light": ModelInfo("j2-light", "ai21", formula="ai21"), "j2-mid": ModelInfo("j2-mid", "ai21", formula="ai21"), "j2-ultra": ModelInfo("j2-ultra", "ai21", formula="ai21"), "j2-jumbo-instruct": ModelInfo("j2-jumbo-instruct", "ai21", formula="ai21"), # Together "togethercomputer/RedPajama-INCITE-Chat-3B-v1": ModelInfo("togethercomputer/RedPajama-INCITE-Chat-3B-v1", "together", formula="together"), "togethercomputer/RedPajama-INCITE-Chat-7B-v1": ModelInfo("togethercomputer/RedPajama-INCITE-Chat-7B-v1", "together", formula="together"), "NousResearch/Nous-Hermes-Llama2-13b": ModelInfo("NousResearch/Nous-Hermes-Llama2-13b", "together", formula="together"), # xAI "grok-1": ModelInfo("grok-1", "xai", formula="xai"), "grok-1.5": ModelInfo("grok-1.5", "xai", formula="xai"), "grok-2": ModelInfo("grok-2", "xai", formula="xai"), "grok-beta": ModelInfo("grok-beta", "xai", formula="xai"), # Alibaba "qwen-1.5-0.5b": ModelInfo("qwen-1.5-0.5b", "alibaba", formula="alibaba"), "qwen-1.5-1.8b": ModelInfo("qwen-1.5-1.8b", "alibaba", formula="alibaba"), "qwen-1.5-4b": ModelInfo("qwen-1.5-4b", "alibaba", formula="alibaba"), "qwen-1.5-7b": ModelInfo("qwen-1.5-7b", "alibaba", formula="alibaba"), "qwen-1.5-14b": ModelInfo("qwen-1.5-14b", "alibaba", formula="alibaba"), "qwen-1.5-32b": ModelInfo("qwen-1.5-32b", "alibaba", formula="alibaba"), "qwen-1.5-72b": ModelInfo("qwen-1.5-72b", "alibaba", formula="alibaba"), "qwen-1.5-110b": ModelInfo("qwen-1.5-110b", "alibaba", formula="alibaba"), "qwen-2-0.5b": ModelInfo("qwen-2-0.5b", "alibaba", formula="alibaba"), "qwen-2-1.5b": ModelInfo("qwen-2-1.5b", "alibaba", formula="alibaba"), "qwen-2-7b": ModelInfo("qwen-2-7b", "alibaba", formula="alibaba"), "qwen-2-57b": ModelInfo("qwen-2-57b", "alibaba", formula="alibaba"), "qwen-2-72b": ModelInfo("qwen-2-72b", "alibaba", formula="alibaba"), "qwen-vl": ModelInfo("qwen-vl", "alibaba", formula="alibaba"), "qwen-vl-chat": ModelInfo("qwen-vl-chat", "alibaba", formula="alibaba"), "qwen-vl-plus": ModelInfo("qwen-vl-plus", "alibaba", formula="alibaba"), "qwen-2.5-72b": ModelInfo("qwen-2.5-72b", "alibaba", formula="alibaba"), "qwen-2.5-32b": ModelInfo("qwen-2.5-32b", "alibaba", formula="alibaba"), "qwen-2.5-14b": ModelInfo("qwen-2.5-14b", "alibaba", formula="alibaba"), "qwen-2.5-7b": ModelInfo("qwen-2.5-7b", "alibaba", formula="alibaba"), # Baidu "ernie-4.0": ModelInfo("ernie-4.0", "baidu", formula="baidu"), "ernie-3.5": ModelInfo("ernie-3.5", "baidu", formula="baidu"), "ernie-3.0": ModelInfo("ernie-3.0", "baidu", formula="baidu"), "ernie-speed": ModelInfo("ernie-speed", "baidu", formula="baidu"), "ernie-lite": ModelInfo("ernie-lite", "baidu", formula="baidu"), "ernie-tiny": ModelInfo("ernie-tiny", "baidu", formula="baidu"), "ernie-bot": ModelInfo("ernie-bot", "baidu", formula="baidu"), "ernie-bot-4": ModelInfo("ernie-bot-4", "baidu", formula="baidu"), # Huawei "pangu-alpha-2.6b": ModelInfo("pangu-alpha-2.6b", "huawei", formula="huawei"), "pangu-alpha-13b": ModelInfo("pangu-alpha-13b", "huawei", formula="huawei"), "pangu-alpha-200b": ModelInfo("pangu-alpha-200b", "huawei", formula="huawei"), "pangu-coder": ModelInfo("pangu-coder", "huawei", formula="huawei"), "pangu-coder-15b": ModelInfo("pangu-coder-15b", "huawei", formula="huawei"), # Yandex "yalm-100b": ModelInfo("yalm-100b", "yandex", formula="yandex"), "yalm-200b": ModelInfo("yalm-200b", "yandex", formula="yandex"), "yagpt": ModelInfo("yagpt", "yandex", formula="yandex"), "yagpt-2": ModelInfo("yagpt-2", "yandex", formula="yandex"), # Stability "stablelm-alpha-3b": ModelInfo("stablelm-alpha-3b", "stability", formula="stability"), "stablelm-alpha-7b": ModelInfo("stablelm-alpha-7b", "stability", formula="stability"), "stablelm-base-alpha-3b": ModelInfo("stablelm-base-alpha-3b", "stability", formula="stability"), "stablelm-base-alpha-7b": ModelInfo("stablelm-base-alpha-7b", "stability", formula="stability"), "stablelm-tuned-alpha-3b": ModelInfo("stablelm-tuned-alpha-3b", "stability", formula="stability"), "stablelm-tuned-alpha-7b": ModelInfo("stablelm-tuned-alpha-7b", "stability", formula="stability"), "stablelm-zephyr-3b": ModelInfo("stablelm-zephyr-3b", "stability", formula="stability"), # TII "falcon-7b": ModelInfo("falcon-7b", "tii", formula="tii"), "falcon-7b-instruct": ModelInfo("falcon-7b-instruct", "tii", formula="tii"), "falcon-40b": ModelInfo("falcon-40b", "tii", formula="tii"), "falcon-40b-instruct": ModelInfo("falcon-40b-instruct", "tii", formula="tii"), "falcon-180b": ModelInfo("falcon-180b", "tii", formula="tii"), "falcon-180b-chat": ModelInfo("falcon-180b-chat", "tii", formula="tii"), # EleutherAI "gpt-neo-125m": ModelInfo("gpt-neo-125m", "eleutherai", formula="eleutherai"), "gpt-neo-1.3b": ModelInfo("gpt-neo-1.3b", "eleutherai", formula="eleutherai"), "gpt-neo-2.7b": ModelInfo("gpt-neo-2.7b", "eleutherai", formula="eleutherai"), "gpt-neox-20b": ModelInfo("gpt-neox-20b", "eleutherai", formula="eleutherai"), "pythia-70m": ModelInfo("pythia-70m", "eleutherai", formula="eleutherai"), "pythia-160m": ModelInfo("pythia-160m", "eleutherai", formula="eleutherai"), "pythia-410m": ModelInfo("pythia-410m", "eleutherai", formula="eleutherai"), "pythia-1b": ModelInfo("pythia-1b", "eleutherai", formula="eleutherai"), "pythia-1.4b": ModelInfo("pythia-1.4b", "eleutherai", formula="eleutherai"), "pythia-2.8b": ModelInfo("pythia-2.8b", "eleutherai", formula="eleutherai"), "pythia-6.9b": ModelInfo("pythia-6.9b", "eleutherai", formula="eleutherai"), "pythia-12b": ModelInfo("pythia-12b", "eleutherai", formula="eleutherai"), # MosaicML "mpt-7b": ModelInfo("mpt-7b", "mosaicml", formula="mosaicml"), "mpt-7b-chat": ModelInfo("mpt-7b-chat", "mosaicml", formula="mosaicml"), "mpt-7b-instruct": ModelInfo("mpt-7b-instruct", "mosaicml", formula="mosaicml"), "mpt-30b": ModelInfo("mpt-30b", "mosaicml", formula="mosaicml"), "mpt-30b-chat": ModelInfo("mpt-30b-chat", "mosaicml", formula="mosaicml"), "mpt-30b-instruct": ModelInfo("mpt-30b-instruct", "mosaicml", formula="mosaicml"), # Replit "replit-code-v1-3b": ModelInfo("replit-code-v1-3b", "replit", formula="replit"), "replit-code-v1.5-3b": ModelInfo("replit-code-v1.5-3b", "replit", formula="replit"), "replit-code-v2-3b": ModelInfo("replit-code-v2-3b", "replit", formula="replit"), # MiniMax "abab5.5-chat": ModelInfo("abab5.5-chat", "minimax", formula="minimax"), "abab5.5s-chat": ModelInfo("abab5.5s-chat", "minimax", formula="minimax"), "abab6-chat": ModelInfo("abab6-chat", "minimax", formula="minimax"), "abab6.5-chat": ModelInfo("abab6.5-chat", "minimax", formula="minimax"), "abab6.5s-chat": ModelInfo("abab6.5s-chat", "minimax", formula="minimax"), "minimax-m2.5": ModelInfo("minimax-m2.5", "minimax", formula="minimax_m25"), # Aleph Alpha "luminous-base": ModelInfo("luminous-base", "aleph_alpha", formula="aleph_alpha"), "luminous-extended": ModelInfo("luminous-extended", "aleph_alpha", formula="aleph_alpha"), "luminous-supreme": ModelInfo("luminous-supreme", "aleph_alpha", formula="aleph_alpha"), "luminous-supreme-control": ModelInfo("luminous-supreme-control", "aleph_alpha", formula="aleph_alpha"), # DeepSeek "deepseek-coder-1.3b": ModelInfo("deepseek-coder-1.3b", "deepseek", formula="deepseek"), "deepseek-coder-6.7b": ModelInfo("deepseek-coder-6.7b", "deepseek", formula="deepseek"), "deepseek-coder-33b": ModelInfo("deepseek-coder-33b", "deepseek", formula="deepseek"), "deepseek-coder-instruct": ModelInfo("deepseek-coder-instruct", "deepseek", formula="deepseek"), "deepseek-vl-1.3b": ModelInfo("deepseek-vl-1.3b", "deepseek", formula="deepseek"), "deepseek-vl-7b": ModelInfo("deepseek-vl-7b", "deepseek", formula="deepseek"), "deepseek-llm-7b": ModelInfo("deepseek-llm-7b", "deepseek", formula="deepseek"), "deepseek-llm-67b": ModelInfo("deepseek-llm-67b", "deepseek", formula="deepseek"), "deepseek-v3": ModelInfo("deepseek-v3", "deepseek", formula="deepseek"), "deepseek-v3-base": ModelInfo("deepseek-v3-base", "deepseek", formula="deepseek"), "deepseek-v3.2": ModelInfo("deepseek-v3.2", "deepseek", formula="deepseek"), # Tsinghua "chatglm-6b": ModelInfo("chatglm-6b", "tsinghua", formula="tsinghua"), "chatglm2-6b": ModelInfo("chatglm2-6b", "tsinghua", formula="tsinghua"), "chatglm3-6b": ModelInfo("chatglm3-6b", "tsinghua", formula="tsinghua"), "glm-4": ModelInfo("glm-4", "tsinghua", formula="tsinghua"), "glm-4v": ModelInfo("glm-4v", "tsinghua", formula="tsinghua"), "glm-5": ModelInfo("glm-5", "tsinghua", formula="glm5"), # Volcengine (Doubao) "doubao-seed-2-0-pro": ModelInfo("doubao-seed-2-0-pro", "volcengine", formula="volcengine"), # Moonshot (Kimi) "kimi-k2.5": ModelInfo("kimi-k2.5", "moonshot", formula="moonshot"), # RWKV "rwkv-4-169m": ModelInfo("rwkv-4-169m", "rwkv", formula="rwkv"), "rwkv-4-430m": ModelInfo("rwkv-4-430m", "rwkv", formula="rwkv"), "rwkv-4-1b5": ModelInfo("rwkv-4-1b5", "rwkv", formula="rwkv"), "rwkv-4-3b": ModelInfo("rwkv-4-3b", "rwkv", formula="rwkv"), "rwkv-4-7b": ModelInfo("rwkv-4-7b", "rwkv", formula="rwkv"), "rwkv-4-14b": ModelInfo("rwkv-4-14b", "rwkv", formula="rwkv"), "rwkv-5-world": ModelInfo("rwkv-5-world", "rwkv", formula="rwkv"), # Community "vicuna-7b": ModelInfo("vicuna-7b", "community", formula="community"), "vicuna-13b": ModelInfo("vicuna-13b", "community", formula="community"), "vicuna-33b": ModelInfo("vicuna-33b", "community", formula="community"), "alpaca-7b": ModelInfo("alpaca-7b", "community", formula="community"), "alpaca-13b": ModelInfo("alpaca-13b", "community", formula="community"), "wizardlm-7b": ModelInfo("wizardlm-7b", "community", formula="community"), "wizardlm-13b": ModelInfo("wizardlm-13b", "community", formula="community"), "wizardlm-30b": ModelInfo("wizardlm-30b", "community", formula="community"), "orca-mini-3b": ModelInfo("orca-mini-3b", "community", formula="community"), "orca-mini-7b": ModelInfo("orca-mini-7b", "community", formula="community"), "orca-mini-13b": ModelInfo("orca-mini-13b", "community", formula="community"), "zephyr-7b-alpha": ModelInfo("zephyr-7b-alpha", "community", formula="community"), "zephyr-7b-beta": ModelInfo("zephyr-7b-beta", "community", formula="community"), # Microsoft "phi-3-mini": ModelInfo("phi-3-mini", "microsoft", formula="microsoft"), "phi-3-small": ModelInfo("phi-3-small", "microsoft", formula="microsoft"), "phi-3-medium": ModelInfo("phi-3-medium", "microsoft", formula="microsoft"), "phi-3.5-mini": ModelInfo("phi-3.5-mini", "microsoft", formula="microsoft"), # Amazon "titan-text-express": ModelInfo("titan-text-express", "amazon", formula="amazon"), "titan-text-lite": ModelInfo("titan-text-lite", "amazon", formula="amazon"), "titan-embed-text": ModelInfo("titan-embed-text", "amazon", formula="amazon"), # Nvidia "nemotron-4-340b": ModelInfo("nemotron-4-340b", "nvidia", formula="nvidia"), "nemotron-3-8b": ModelInfo("nemotron-3-8b", "nvidia", formula="nvidia"), # IBM "granite-13b-chat": ModelInfo("granite-13b-chat", "ibm", formula="ibm"), "granite-13b-instruct": ModelInfo("granite-13b-instruct", "ibm", formula="ibm"), "granite-20b-code": ModelInfo("granite-20b-code", "ibm", formula="ibm"), # Salesforce "codegen-16b": ModelInfo("codegen-16b", "salesforce", formula="salesforce"), "codegen-6b": ModelInfo("codegen-6b", "salesforce", formula="salesforce"), "codegen-2b": ModelInfo("codegen-2b", "salesforce", formula="salesforce"), # BigCode "starcoder": ModelInfo("starcoder", "bigcode", formula="bigcode"), "starcoder2-15b": ModelInfo("starcoder2-15b", "bigcode", formula="bigcode"), "starcoderbase": ModelInfo("starcoderbase", "bigcode", formula="bigcode"), "starcoder2-3b": ModelInfo("starcoder2-3b", "bigcode", formula="bigcode"), "starcoder2-7b": ModelInfo("starcoder2-7b", "bigcode", formula="bigcode"), "starcoder-plus": ModelInfo("starcoder-plus", "bigcode", formula="bigcode"), "starcoderbase-1b": ModelInfo("starcoderbase-1b", "bigcode", formula="bigcode"), "starcoderbase-3b": ModelInfo("starcoderbase-3b", "bigcode", formula="bigcode"), "starcoderbase-7b": ModelInfo("starcoderbase-7b", "bigcode", formula="bigcode"), # Databricks "dbrx": ModelInfo("dbrx", "databricks", formula="databricks"), "dbrx-instruct": ModelInfo("dbrx-instruct", "databricks", formula="databricks", pricing_id="dbrx-instruct"), "dbrx-base": ModelInfo("dbrx-base", "databricks", formula="databricks", pricing_id="dbrx-base"), "dolly-v2-12b": ModelInfo("dolly-v2-12b", "databricks", formula="databricks", pricing_id="dolly-v2-12b"), "dolly-v2-7b": ModelInfo("dolly-v2-7b", "databricks", formula="databricks", pricing_id="dolly-v2-7b"), "dolly-v2-3b": ModelInfo("dolly-v2-3b", "databricks", formula="databricks", pricing_id="dolly-v2-3b"), # Voyage "voyage-2": ModelInfo("voyage-2", "voyage", formula="voyage", pricing_id="voyage-2"), "voyage-large-2": ModelInfo("voyage-large-2", "voyage", formula="voyage", pricing_id="voyage-large-2"), "voyage-code-2": ModelInfo("voyage-code-2", "voyage", formula="voyage", pricing_id="voyage-code-2"), "voyage-finance-2": ModelInfo("voyage-finance-2", "voyage", formula="voyage", pricing_id="voyage-finance-2"), "voyage-law-2": ModelInfo("voyage-law-2", "voyage", formula="voyage", pricing_id="voyage-law-2"), "voyage-multilingual-2": ModelInfo("voyage-multilingual-2", "voyage", formula="voyage", pricing_id="voyage-multilingual-2"), } def _build_alias_map(models: Iterable[ModelInfo]) -> Dict[str, str]: alias_map: Dict[str, str] = {} for model in models: if model.aliases: for alias in model.aliases: alias_map[normalize_model_name(alias)] = model.name return alias_map ALIASES: Dict[str, str] = _build_alias_map(MODELS.values()) MODEL_LOOKUP: Dict[str, ModelInfo] = {normalize_model_name(k): v for k, v in MODELS.items()} MODEL_LOOKUP.update({alias: MODELS[target] for alias, target in ALIASES.items()}) # All normalized model names for fuzzy matching _ALL_NORMALIZED_NAMES: List[str] = list(MODEL_LOOKUP.keys()) def _extract_model_part(name: str) -> str: """Extract model part from 'provider/model' or 'provider:model' format.""" name = name.strip() for sep in ("/", ":"): if sep in name: return name.split(sep, 1)[-1].strip() return name def _fuzzy_match_model(model_name: str) -> Optional[ModelInfo]: """Fuzzy match model name when exact lookup fails.""" normalized = normalize_model_name(model_name) model_part = normalize_model_name(_extract_model_part(model_name)) # 1. Try model part only (e.g. anthropic/claude-sonnet-4-6 -> claude-sonnet-4-6) if model_part != normalized: info = MODEL_LOOKUP.get(model_part) if info: return info # 2. Substring match: input contains a known model, or known model contains input for key, info in MODEL_LOOKUP.items(): if model_part in key or key in model_part: return info # 3. difflib fuzzy match (cutoff 0.5 = 50% similarity) candidates = difflib.get_close_matches(model_part, _ALL_NORMALIZED_NAMES, n=1, cutoff=0.5) if candidates: return MODEL_LOOKUP.get(candidates[0]) # 4. Relaxed: match key parts (e.g. claude-sonnet-4 matches claude-3.5-sonnet) model_tokens = set(re.split(r"[-_.]", model_part)) - {""} for key in _ALL_NORMALIZED_NAMES: key_tokens = set(re.split(r"[-_.]", key)) - {""} overlap = len(model_tokens & key_tokens) / max(len(model_tokens), 1) if overlap >= 0.6: return MODEL_LOOKUP.get(key) return None def get_model_info(model_name: str) -> Optional[ModelInfo]: info = MODEL_LOOKUP.get(normalize_model_name(model_name)) if info: return info return _fuzzy_match_model(model_name) def get_all_supported_models() -> List[str]: return sorted(set(MODELS.keys())) def get_supported_models_by_provider() -> Dict[str, List[str]]: models_by_provider: Dict[str, List[str]] = {} for model in MODELS.values(): models_by_provider.setdefault(model.provider, []).append(model.name) for provider, models in models_by_provider.items(): models_by_provider[provider] = sorted(set(models)) return dict(sorted(models_by_provider.items())) FILE:scripts/registry/pricing.py """Pricing registry for scripts.""" from typing import Optional USD_TO_INR = 83.0 # Conversion rate as of July 10 2025 PRICING = { "gpt-4": {"input": 0.03, "output": 0.06}, "gpt-4-32k": {"input": 0.06, "output": 0.12}, "dbrx-instruct": {"input": 0.001, "output": 0.002}, "dbrx-base": {"input": 0.001, "output": 0.002}, "dolly-v2-12b": {"input": 0.001, "output": 0.002}, "dolly-v2-7b": {"input": 0.001, "output": 0.002}, "dolly-v2-3b": {"input": 0.001, "output": 0.002}, "voyage-2": {"input": 0.0001, "output": 0.0001}, "voyage-large-2": {"input": 0.0001, "output": 0.0001}, "voyage-code-2": {"input": 0.0001, "output": 0.0001}, "voyage-finance-2": {"input": 0.0001, "output": 0.0001}, "voyage-law-2": {"input": 0.0001, "output": 0.0001}, "voyage-multilingual-2": {"input": 0.0001, "output": 0.0001}, "gpt-4-turbo": {"input": 0.01, "output": 0.03}, "gpt-4o": {"input": 0.005, "output": 0.015}, "gpt-4o-mini": {"input": 0.00015, "output": 0.0006}, "gpt-3.5-turbo": {"input": 0.001, "output": 0.002}, "gpt-3.5-turbo-16k": {"input": 0.003, "output": 0.004}, "claude-3-opus-20240229": {"input": 0.015, "output": 0.075}, "claude-3-sonnet-20240229": {"input": 0.003, "output": 0.015}, "claude-3-haiku-20240307": {"input": 0.00025, "output": 0.00125}, "claude-3.5-sonnet-20240620": {"input": 0.003, "output": 0.015}, "claude-3.5-sonnet-20241022": {"input": 0.003, "output": 0.015}, "claude-3.5-haiku-20241022": {"input": 0.001, "output": 0.005}, "claude-3-5-sonnet-20240620": {"input": 0.003, "output": 0.015}, } def get_pricing_rate(model: str, input_tokens: bool) -> Optional[float]: pricing = PRICING.get(model) if not pricing: return None return pricing["input" if input_tokens else "output"] FILE:scripts/registry/__init__.py """Registry package for model metadata, pricing, and aliases.""" from .models import ( # noqa: F401 ModelInfo, get_all_supported_models, get_model_info, get_supported_models_by_provider, normalize_model_name, ) from .pricing import USD_TO_INR, get_pricing_rate # noqa: F401 FILE:scripts/__init__.py """ scripts CLI package. This package powers the scripts command-line interface for counting tokens and estimating costs across a large catalog of LLM models. """ __version__ = "1.0.10" __author__ = "Raja CSP Raman" __email__ = "raja.csp@gmail.com" FILE:tests/test_main_models.py """ Tests for main models with benchmark data. Uses fuzzy model name matching (e.g. anthropic/claude-sonnet-4-6 -> claude-sonnet-4-6). Benchmark: 8927 chars -> expected token counts per model. Tolerance: ±30% for approximation-based models (formula-based estimates vary with text). OpenAI (tiktoken): skip when tiktoken not installed. """ import pytest from scripts.core import TokenCounter, count_tokens from scripts.registry.models import get_model_info as registry_get_model_info try: import tiktoken TIKTOKEN_AVAILABLE = True except ImportError: TIKTOKEN_AVAILABLE = False # Benchmark: 8927 chars, expected tokens per model # 1 token ≈ chars/token from user's table MAIN_MODELS_BENCHMARK = [ ("anthropic/claude-sonnet-4-6", 8927, 2763), ("anthropic/claude-sonnet-4-5", 8927, 2763), ("anthropic/claude-opus-4.6", 8927, 2763), ("openai/gpt-5.2-codex", 8927, 2459), ("google/gemini-3.1-pro-preview", 8927, 2627), ("z-ai/glm-5", 8927, 2457), ("volcengine/doubao-seed-2-0-pro", 8927, 2702), ("moonshot/kimi-k2.5", 8927, 2402), ("minimax/MiniMax-M2.5", 8927, 2428), ("deepseek-v3.2", 8927, 2578), ] # Also test canonical names (no provider prefix) CANONICAL_BENCHMARK = [ ("claude-sonnet-4-6", 8927, 2763), ("gpt-5.2-codex", 8927, 2459), # OpenAI, requires tiktoken ("glm-5", 8927, 2457), ("deepseek-v3.2", 8927, 2578), ] # OpenAI models that need tiktoken OPENAI_MODELS = {"gpt-5.2-codex", "openai/gpt-5.2-codex"} # Benchmark: 3050 chars, Chinese-mixed content (混杂中文) # 1 token ≈ 1.5–2.4 chars (CJK tokenizes to fewer chars/token) MAIN_MODELS_BENCHMARK_ZH = [ ("anthropic/claude-sonnet-4-6", 3050, 1913), ("anthropic/claude-sonnet-4-5", 3050, 1913), ("anthropic/claude-opus-4.6", 3050, 1913), ("openai/gpt-5.2-codex", 3050, 1564), ("google/gemini-3.1-pro-preview", 3050, 1473), ("z-ai/glm-5", 3050, 1318), ("volcengine/doubao-seed-2-0-pro", 3050, 1494), ("moonshot/kimi-k2.5", 3050, 1257), ("minimax/MiniMax-M2.5", 3050, 1289), ("deepseek-v3.2", 3050, 1361), ] def _make_test_text(length: int) -> str: """Generate test text of given length (mixed content for realistic tokenization).""" base = ( "Token counting benchmark. 这是一段混合中英文的测试文本。" "Code: def hello(): return 42\n" "Lorem ipsum dolor sit amet, consectetur adipiscing elit. " ) if length <= len(base): return base[:length] repeat = (length // len(base)) + 1 return (base * repeat)[:length] def _make_chinese_mixed_text(length: int) -> str: """Generate Chinese-heavy mixed text (混杂中文) for CJK tokenization benchmark. ~90% CJK to match user benchmark (3050 chars -> ~1.6-2.4 chars/token).""" base = ( "大语言模型token计数基准测试。这是一段混杂了中文与英文的测试文本，" "用于验证不同模型对CJK字符的tokenization表现。" "大语言模型在处理中文时通常每个汉字约一至两个token。" "测试数据包含中英混合内容。" ) if length <= len(base): return base[:length] repeat = (length // len(base)) + 1 return (base * repeat)[:length] @pytest.fixture def benchmark_text(): return _make_test_text(8927) @pytest.fixture def benchmark_text_zh(): return _make_chinese_mixed_text(3050) class TestFuzzyModelMatching: """Test fuzzy model name matching.""" @pytest.mark.parametrize( "input_name,expected_canonical", [ ("anthropic/claude-sonnet-4-6", "claude-sonnet-4-6"), ("openai/gpt-5.2-codex", "gpt-5.2-codex"), ("z-ai/glm-5", "glm-5"), ("minimax/MiniMax-M2.5", "minimax-m2.5"), ("deepseek-v3.2", "deepseek-v3.2"), ], ) def test_fuzzy_match_resolves(self, input_name, expected_canonical): info = registry_get_model_info(input_name) assert info is not None, f"Model '{input_name}' should resolve via fuzzy match" assert info.name.lower() == expected_canonical.lower() def test_exact_match_unchanged(self): assert registry_get_model_info("gpt-4") is not None assert registry_get_model_info("claude-3-opus") is not None class TestMainModelsTokenCount: """Test token counts for main models against benchmark data.""" @pytest.mark.parametrize("model_name,char_count,expected_tokens", MAIN_MODELS_BENCHMARK) def test_main_models_with_fuzzy_name(self, benchmark_text, model_name, char_count, expected_tokens): if model_name in OPENAI_MODELS and not TIKTOKEN_AVAILABLE: pytest.skip("tiktoken required for OpenAI models") assert len(benchmark_text) == char_count counter = TokenCounter(model_name) tokens = counter.count(benchmark_text) # ±30% tolerance for approximation-based models tolerance = 0.30 low = int(expected_tokens * (1 - tolerance)) high = int(expected_tokens * (1 + tolerance)) assert low <= tokens <= high, ( f"{model_name}: got {tokens} tokens, expected ~{expected_tokens} (range {low}-{high})" ) @pytest.mark.parametrize("model_name,char_count,expected_tokens", CANONICAL_BENCHMARK) def test_canonical_names(self, benchmark_text, model_name, char_count, expected_tokens): if model_name in OPENAI_MODELS and not TIKTOKEN_AVAILABLE: pytest.skip("tiktoken required for OpenAI models") assert len(benchmark_text) == char_count tokens = count_tokens(benchmark_text, model_name) tolerance = 0.30 low = int(expected_tokens * (1 - tolerance)) high = int(expected_tokens * (1 + tolerance)) assert low <= tokens <= high, ( f"{model_name}: got {tokens} tokens, expected ~{expected_tokens} (range {low}-{high})" ) @pytest.mark.parametrize("model_name,char_count,expected_tokens", MAIN_MODELS_BENCHMARK_ZH) def test_main_models_chinese_mixed(self, benchmark_text_zh, model_name, char_count, expected_tokens): """Test token counts for Chinese-mixed text (混杂中文). CJK chars ≈ 1–2 chars/token.""" if model_name in OPENAI_MODELS and not TIKTOKEN_AVAILABLE: pytest.skip("tiktoken required for OpenAI models") assert len(benchmark_text_zh) == char_count counter = TokenCounter(model_name) tokens = counter.count(benchmark_text_zh) tolerance = 0.30 low = int(expected_tokens * (1 - tolerance)) high = int(expected_tokens * (1 + tolerance)) assert low <= tokens <= high, ( f"{model_name} (ZH): got {tokens} tokens, expected ~{expected_tokens} (range {low}-{high})" ) FILE:tests/__init__.py """Tests for prompt_token_counter."""

Prompt Token Counter

Comments (0)