--- name: "skill-tester" description: "Skill Tester" --- # Skill Tester --- **Name**: skill-tester **Tier**: POWERFUL **Category**: Engineering Quality Assurance **Dependencies**: None (Python Standard Library Only) **Author**: Claude Skills Engineering Team **Version**: 1.0.0 **Last Updated**: 2026-02-16 --- ## Description The Skill Tester is a comprehensive meta-skill designed to validate, test, and score the quality of skills within the claude-skills ecosystem. This powerful quality assurance tool ensures that all skills meet the rigorous standards required for BASIC, STANDARD, and POWERFUL tier classifications through automated validation, testing, and scoring mechanisms. As the gatekeeping system for skill quality, this meta-skill provides three core capabilities: 1. **Structure Validation** - Ensures skills conform to required directory structures, file formats, and documentation standards 2. **Script Testing** - Validates Python scripts for syntax, imports, functionality, and output format compliance 3. **Quality Scoring** - Provides comprehensive quality assessment across multiple dimensions with letter grades and improvement recommendations This skill is essential for maintaining ecosystem consistency, enabling automated CI/CD integration, and supporting both manual and automated quality assurance workflows. It serves as the foundation for pre-commit hooks, pull request validation, and continuous integration processes that maintain the high-quality standards of the claude-skills repository. ## Core Features ### Comprehensive Skill Validation - **Structure Compliance**: Validates directory structure, required files (SKILL.md, README.md, scripts/, references/, assets/, expected_outputs/) - **Documentation Standards**: Checks SKILL.md frontmatter, section completeness, minimum line counts per tier - **File Format Validation**: Ensures proper Markdown formatting, YAML frontmatter syntax, and file naming conventions ### Advanced Script Testing - **Syntax Validation**: Compiles Python scripts to detect syntax errors before execution - **Import Analysis**: Enforces standard library only policy, identifies external dependencies - **Runtime Testing**: Executes scripts with sample data, validates argparse implementation, tests --help functionality - **Output Format Compliance**: Verifies dual output support (JSON + human-readable), proper error handling ### Multi-Dimensional Quality Scoring - **Documentation Quality (25%)**: SKILL.md depth and completeness, README clarity, reference documentation quality - **Code Quality (25%)**: Script complexity, error handling robustness, output format consistency, maintainability - **Completeness (25%)**: Required directory presence, sample data adequacy, expected output verification - **Usability (25%)**: Example clarity, argparse help text quality, installation simplicity, user experience ### Tier Classification System Automatically classifies skills based on complexity and functionality: #### BASIC Tier Requirements - Minimum 100 lines in SKILL.md - At least 1 Python script (100-300 LOC) - Basic argparse implementation - Simple input/output handling - Essential documentation coverage #### STANDARD Tier Requirements - Minimum 200 lines in SKILL.md - 1-2 Python scripts (300-500 LOC each) - Advanced argparse with subcommands - JSON + text output formats - Comprehensive examples and references - Error handling and edge case management #### POWERFUL Tier Requirements - Minimum 300 lines in SKILL.md - 2-3 Python scripts (500-800 LOC each) - Complex argparse with multiple modes - Sophisticated output formatting and validation - Extensive documentation and reference materials - Advanced error handling and recovery mechanisms - CI/CD integration capabilities ## Architecture & Design ### Modular Design Philosophy The skill-tester follows a modular architecture where each component serves a specific validation purpose: - **skill_validator.py**: Core structural and documentation validation engine - **script_tester.py**: Runtime testing and execution validation framework - **quality_scorer.py**: Multi-dimensional quality assessment and scoring system ### Standards Enforcement All validation is performed against well-defined standards documented in the references/ directory: - **Skill Structure Specification**: Defines mandatory and optional components - **Tier Requirements Matrix**: Detailed requirements for each skill tier - **Quality Scoring Rubric**: Comprehensive scoring methodology and weightings ### Integration Capabilities Designed for seamless integration into existing development workflows: - **Pre-commit Hooks**: Prevents substandard skills from being committed - **CI/CD Pipelines**: Automated quality gates in pull request workflows - **Manual Validation**: Interactive command-line tools for development-time validation - **Batch Processing**: Bulk validation and scoring of existing skill repositories ## Implementation Details ### skill_validator.py Core Functions ```python # Primary validation workflow validate_skill_structure() -> ValidationReport check_skill_md_compliance() -> DocumentationReport validate_python_scripts() -> ScriptReport generate_compliance_score() -> float ``` Key validation checks include: - SKILL.md frontmatter parsing and validation - Required section presence (Description, Features, Usage, etc.) - Minimum line count enforcement per tier - Python script argparse implementation verification - Standard library import enforcement - Directory structure compliance - README.md quality assessment ### script_tester.py Testing Framework ```python # Core testing functions syntax_validation() -> SyntaxReport import_validation() -> ImportReport runtime_testing() -> RuntimeReport output_format_validation() -> OutputReport ``` Testing capabilities encompass: - Python AST-based syntax validation - Import statement analysis and external dependency detection - Controlled script execution with timeout protection - Argparse --help functionality verification - Sample data processing and output validation - Expected output comparison and difference reporting ### quality_scorer.py Scoring System ```python # Multi-dimensional scoring score_documentation() -> float # 25% weight score_code_quality() -> float # 25% weight score_completeness() -> float # 25% weight score_usability() -> float # 25% weight calculate_overall_grade() -> str # A-F grade ``` Scoring dimensions include: - **Documentation**: Completeness, clarity, examples, reference quality - **Code Quality**: Complexity, maintainability, error handling, output consistency - **Completeness**: Required files, sample data, expected outputs, test coverage - **Usability**: Help text quality, example clarity, installation simplicity ## Usage Scenarios ### Development Workflow Integration ```bash # Pre-commit hook validation skill_validator.py path/to/skill --tier POWERFUL --json # Comprehensive skill testing script_tester.py path/to/skill --timeout 30 --sample-data # Quality assessment and scoring quality_scorer.py path/to/skill --detailed --recommendations ``` ### CI/CD Pipeline Integration ```yaml # GitHub Actions workflow example - name: "validate-skill-quality" run: | python skill_validator.py engineering/${{ matrix.skill }} --json | tee validation.json python script_tester.py engineering/${{ matrix.skill }} | tee testing.json python quality_scorer.py engineering/${{ matrix.skill }} --json | tee scoring.json ``` ### Batch Repository Analysis ```bash # Validate all skills in repository find engineering/ -type d -maxdepth 1 | xargs -I {} skill_validator.py {} # Generate repository quality report quality_scorer.py engineering/ --batch --output-format json > repo_quality.json ``` ## Output Formats & Reporting ### Dual Output Support All tools provide both human-readable and machine-parseable output: #### Human-Readable Format ``` === SKILL VALIDATION REPORT === Skill: engineering/example-skill Tier: STANDARD Overall Score: 85/100 (B) Structure Validation: ✓ PASS ├─ SKILL.md: ✓ EXISTS (247 lines) ├─ README.md: ✓ EXISTS ├─ scripts/: ✓ EXISTS (2 files) └─ references/: ⚠ MISSING (recommended) Documentation Quality: 22/25 (88%) Code Quality: 20/25 (80%) Completeness: 18/25 (72%) Usability: 21/25 (84%) Recommendations: • Add references/ directory with documentation • Improve error handling in main.py • Include more comprehensive examples ``` #### JSON Format ```json { "skill_path": "engineering/example-skill", "timestamp": "2026-02-16T16:41:00Z", "validation_results": { "structure_compliance": { "score": 0.95, "checks": { "skill_md_exists": true, "readme_exists": true, "scripts_directory": true, "references_directory": false } }, "overall_score": 85, "letter_grade": "B", "tier_recommendation": "STANDARD", "improvement_suggestions": [ "Add references/ directory", "Improve error handling", "Include comprehensive examples" ] } } ``` ## Quality Assurance Standards ### Code Quality Requirements - **Standard Library Only**: No external dependencies (pip packages) - **Error Handling**: Comprehensive exception handling with meaningful error messages - **Output Consistency**: Standardized JSON schema and human-readable formatting - **Performance**: Efficient validation algorithms with reasonable execution time - **Maintainability**: Clear code structure, comprehensive docstrings, type hints where appropriate ### Testing Standards - **Self-Testing**: The skill-tester validates itself (meta-validation) - **Sample Data Coverage**: Comprehensive test cases covering edge cases and error conditions - **Expected Output Verification**: All sample runs produce verifiable, reproducible outputs - **Timeout Protection**: Safe execution of potentially problematic scripts with timeout limits ### Documentation Standards - **Comprehensive Coverage**: All functions, classes, and modules documented - **Usage Examples**: Clear, practical examples for all use cases - **Integration Guides**: Step-by-step CI/CD and workflow integration instructions - **Reference Materials**: Complete specification documents for standards and requirements ## Integration Examples ### Pre-Commit Hook Setup ```bash #!/bin/bash # .git/hooks/pre-commit echo "Running skill validation..." python engineering/skill-tester/scripts/skill_validator.py engineering/new-skill --tier STANDARD if [ $? -ne 0 ]; then echo "Skill validation failed. Commit blocked." exit 1 fi echo "Validation passed. Proceeding with commit." ``` ### GitHub Actions Workflow ```yaml name: "skill-quality-gate" on: pull_request: paths: ['engineering/**'] jobs: validate-skills: runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - name: "setup-python" uses: actions/setup-python@v4 with: python-version: '3.11' - name: "validate-changed-skills" run: | changed_skills=$(git diff --name-only ${{ github.event.before }} | grep -E '^engineering/[^/]+/' | cut -d'/' -f1-2 | sort -u) for skill in $changed_skills; do echo "Validating $skill..." python engineering/skill-tester/scripts/skill_validator.py $skill --json python engineering/skill-tester/scripts/script_tester.py $skill python engineering/skill-tester/scripts/quality_scorer.py $skill --minimum-score 75 done ``` ### Continuous Quality Monitoring ```bash #!/bin/bash # Daily quality report generation echo "Generating daily skill quality report..." timestamp=$(date +"%Y-%m-%d") python engineering/skill-tester/scripts/quality_scorer.py engineering/ \ --batch --json > "reports/quality_report_${timestamp}.json" echo "Quality trends analysis..." python engineering/skill-tester/scripts/trend_analyzer.py reports/ \ --days 30 > "reports/quality_trends_${timestamp}.md" ``` ## Performance & Scalability ### Execution Performance - **Fast Validation**: Structure validation completes in <1 second per skill - **Efficient Testing**: Script testing with timeout protection (configurable, default 30s) - **Batch Processing**: Optimized for repository-wide analysis with parallel processing support - **Memory Efficiency**: Minimal memory footprint for large-scale repository analysis ### Scalability Considerations - **Repository Size**: Designed to handle repositories with 100+ skills - **Concurrent Execution**: Thread-safe implementation supports parallel validation - **Resource Management**: Automatic cleanup of temporary files and subprocess resources - **Configuration Flexibility**: Configurable timeouts, memory limits, and validation strictness ## Security & Safety ### Safe Execution Environment - **Sandboxed Testing**: Scripts execute in controlled environment with timeout protection - **Resource Limits**: Memory and CPU usage monitoring to prevent resource exhaustion - **Input Validation**: All inputs sanitized and validated before processing - **No Network Access**: Offline operation ensures no external dependencies or network calls ### Security Best Practices - **No Code Injection**: Static analysis only, no dynamic code generation - **Path Traversal Protection**: Secure file system access with path validation - **Minimal Privileges**: Operates with minimal required file system permissions - **Audit Logging**: Comprehensive logging for security monitoring and troubleshooting ## Troubleshooting & Support ### Common Issues & Solutions #### Validation Failures - **Missing Files**: Check directory structure against tier requirements - **Import Errors**: Ensure only standard library imports are used - **Documentation Issues**: Verify SKILL.md frontmatter and section completeness #### Script Testing Problems - **Timeout Errors**: Increase timeout limit or optimize script performance - **Execution Failures**: Check script syntax and import statement validity - **Output Format Issues**: Ensure proper JSON formatting and dual output support #### Quality Scoring Discrepancies - **Low Scores**: Review scoring rubric and improvement recommendations - **Tier Misclassification**: Verify skill complexity against tier requirements - **Inconsistent Results**: Check for recent changes in quality standards or scoring weights ### Debugging Support - **Verbose Mode**: Detailed logging and execution tracing available - **Dry Run Mode**: Validation without execution for debugging purposes - **Debug Output**: Comprehensive error reporting with file locations and suggestions ## Future Enhancements ### Planned Features - **Machine Learning Quality Prediction**: AI-powered quality assessment using historical data - **Performance Benchmarking**: Execution time and resource usage tracking across skills - **Dependency Analysis**: Automated detection and validation of skill interdependencies - **Quality Trend Analysis**: Historical quality tracking and regression detection ### Integration Roadmap - **IDE Plugins**: Real-time validation in popular development environments - **Web Dashboard**: Centralized quality monitoring and reporting interface - **API Endpoints**: RESTful API for external integration and automation - **Notification Systems**: Automated alerts for quality degradation or validation failures ## Conclusion The Skill Tester represents a critical infrastructure component for maintaining the high-quality standards of the claude-skills ecosystem. By providing comprehensive validation, testing, and scoring capabilities, it ensures that all skills meet or exceed the rigorous requirements for their respective tiers. This meta-skill not only serves as a quality gate but also as a development tool that guides skill authors toward best practices and helps maintain consistency across the entire repository. Through its integration capabilities and comprehensive reporting, it enables both manual and automated quality assurance workflows that scale with the growing claude-skills ecosystem. The combination of structural validation, runtime testing, and multi-dimensional quality scoring provides unparalleled visibility into skill quality while maintaining the flexibility needed for diverse skill types and complexity levels. As the claude-skills repository continues to grow, the Skill Tester will remain the cornerstone of quality assurance and ecosystem integrity. FILE:README.md # Skill Tester - Quality Assurance Meta-Skill A POWERFUL-tier skill that provides comprehensive validation, testing, and quality scoring for skills in the claude-skills ecosystem. ## Overview The Skill Tester is a meta-skill that ensures quality and consistency across all skills in the repository through: - **Structure Validation** - Verifies directory structure, file presence, and documentation standards - **Script Testing** - Tests Python scripts for syntax, functionality, and compliance - **Quality Scoring** - Provides comprehensive quality assessment across multiple dimensions ## Quick Start ### Validate a Skill ```bash # Basic validation python scripts/skill_validator.py engineering/my-skill # Validate against specific tier python scripts/skill_validator.py engineering/my-skill --tier POWERFUL --json ``` ### Test Scripts ```bash # Test all scripts in a skill python scripts/script_tester.py engineering/my-skill # Test with custom timeout python scripts/script_tester.py engineering/my-skill --timeout 60 --json ``` ### Score Quality ```bash # Get quality assessment python scripts/quality_scorer.py engineering/my-skill # Detailed scoring with improvement suggestions python scripts/quality_scorer.py engineering/my-skill --detailed --json ``` ## Components ### Scripts - **skill_validator.py** (700+ LOC) - Validates skill structure and compliance - **script_tester.py** (800+ LOC) - Tests script functionality and quality - **quality_scorer.py** (1100+ LOC) - Multi-dimensional quality assessment ### Reference Documentation - **skill-structure-specification.md** - Complete structural requirements - **tier-requirements-matrix.md** - Tier-specific quality standards - **quality-scoring-rubric.md** - Detailed scoring methodology ### Sample Assets - **sample-skill/** - Complete sample skill for testing the tester itself ## Features ### Validation Capabilities - SKILL.md format and content validation - Directory structure compliance checking - Python script syntax and import validation - Argparse implementation verification - Tier-specific requirement enforcement ### Testing Framework - Syntax validation using AST parsing - Import analysis for external dependencies - Runtime execution testing with timeout protection - Help functionality verification - Sample data processing validation - Output format compliance checking ### Quality Assessment - Documentation quality scoring (25%) - Code quality evaluation (25%) - Completeness assessment (25%) - Usability analysis (25%) - Letter grade assignment (A+ to F) - Tier recommendation generation - Improvement roadmap creation ## CI/CD Integration ### GitHub Actions Example ```yaml name: Skill Quality Gate on: pull_request: paths: ['engineering/**'] jobs: validate-skills: runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - name: Setup Python uses: actions/setup-python@v4 with: python-version: '3.11' - name: Validate Skills run: | for skill in $(git diff --name-only ${{ github.event.before }} | grep -E '^engineering/[^/]+/' | cut -d'/' -f1-2 | sort -u); do python engineering/skill-tester/scripts/skill_validator.py $skill --json python engineering/skill-tester/scripts/script_tester.py $skill python engineering/skill-tester/scripts/quality_scorer.py $skill --minimum-score 75 done ``` ### Pre-commit Hook ```bash #!/bin/bash # .git/hooks/pre-commit python engineering/skill-tester/scripts/skill_validator.py engineering/my-skill --tier STANDARD if [ $? -ne 0 ]; then echo "Skill validation failed. Commit blocked." exit 1 fi ``` ## Quality Standards ### All Scripts - **Zero External Dependencies** - Python standard library only - **Comprehensive Error Handling** - Meaningful error messages and recovery - **Dual Output Support** - Both JSON and human-readable formats - **Proper Documentation** - Comprehensive docstrings and comments - **CLI Best Practices** - Full argparse implementation with help text ### Validation Accuracy - **Structure Checks** - 100% accurate directory and file validation - **Content Analysis** - Deep parsing of SKILL.md and documentation - **Code Analysis** - AST-based Python code validation - **Compliance Scoring** - Objective, repeatable quality assessment ## Self-Testing The skill-tester can validate itself: ```bash # Validate the skill-tester structure python scripts/skill_validator.py . --tier POWERFUL # Test the skill-tester scripts python scripts/script_tester.py . # Score the skill-tester quality python scripts/quality_scorer.py . --detailed ``` ## Advanced Usage ### Batch Validation ```bash # Validate all skills in repository find engineering/ -maxdepth 1 -type d | while read skill; do echo "Validating $skill..." python engineering/skill-tester/scripts/skill_validator.py "$skill" done ``` ### Quality Monitoring ```bash # Generate quality report for all skills python engineering/skill-tester/scripts/quality_scorer.py engineering/ \ --batch --json > quality_report.json ``` ### Custom Scoring Thresholds ```bash # Enforce minimum quality scores python scripts/quality_scorer.py engineering/my-skill --minimum-score 80 # Exit code 0 = passed, 1 = failed, 2 = needs improvement ``` ## Error Handling All scripts provide comprehensive error handling: - **File System Errors** - Missing files, permission issues, invalid paths - **Content Errors** - Malformed YAML, invalid JSON, encoding issues - **Execution Errors** - Script timeouts, runtime failures, import errors - **Validation Errors** - Standards violations, compliance failures ## Output Formats ### Human-Readable ``` === SKILL VALIDATION REPORT === Skill: engineering/my-skill Overall Score: 85.2/100 (B+) Tier Recommendation: STANDARD STRUCTURE VALIDATION: ✓ PASS: SKILL.md found ✓ PASS: README.md found ✓ PASS: scripts/ directory found SUGGESTIONS: • Add references/ directory • Improve error handling in main.py ``` ### JSON Format ```json { "skill_path": "engineering/my-skill", "overall_score": 85.2, "letter_grade": "B+", "tier_recommendation": "STANDARD", "dimensions": { "Documentation": {"score": 88.5, "weight": 0.25}, "Code Quality": {"score": 82.0, "weight": 0.25}, "Completeness": {"score": 85.5, "weight": 0.25}, "Usability": {"score": 84.8, "weight": 0.25} } } ``` ## Requirements - **Python 3.7+** - No external dependencies required - **File System Access** - Read access to skill directories - **Execution Permissions** - Ability to run Python scripts for testing ## Contributing See [SKILL.md](SKILL.md) for comprehensive documentation and contribution guidelines. The skill-tester itself serves as a reference implementation of POWERFUL-tier quality standards. FILE:assets/sample-skill/README.md # Sample Text Processor A basic text processing skill that demonstrates BASIC tier requirements for the claude-skills ecosystem. ## Quick Start ```bash # Analyze a text file python scripts/text_processor.py analyze sample.txt # Get JSON output python scripts/text_processor.py analyze sample.txt --format json # Transform text to uppercase python scripts/text_processor.py transform sample.txt --mode upper # Process multiple files python scripts/text_processor.py batch text_files/ --verbose ``` ## Features - Word count and text statistics - Text transformations (upper, lower, title, reverse) - Batch file processing - JSON and human-readable output formats - Comprehensive error handling ## Requirements - Python 3.7 or later - No external dependencies (standard library only) ## Usage See [SKILL.md](SKILL.md) for comprehensive documentation and examples. ## Testing Sample data files are provided in the `assets/` directory for testing the functionality. FILE:assets/sample-skill/SKILL.md # Sample Text Processor --- **Name**: sample-text-processor **Tier**: BASIC **Category**: Text Processing **Dependencies**: None (Python Standard Library Only) **Author**: Claude Skills Engineering Team **Version**: 1.0.0 **Last Updated**: 2026-02-16 --- ## Description The Sample Text Processor is a simple skill designed to demonstrate the basic structure and functionality expected in the claude-skills ecosystem. This skill provides fundamental text processing capabilities including word counting, character analysis, and basic text transformations. This skill serves as a reference implementation for BASIC tier requirements and can be used as a template for creating new skills. It demonstrates proper file structure, documentation standards, and implementation patterns that align with ecosystem best practices. The skill processes text files and provides statistics and transformations in both human-readable and JSON formats, showcasing the dual output requirement for skills in the claude-skills repository. ## Features ### Core Functionality - **Word Count Analysis**: Count total words, unique words, and word frequency - **Character Statistics**: Analyze character count, line count, and special characters - **Text Transformations**: Convert text to uppercase, lowercase, or title case - **File Processing**: Process single text files or batch process directories - **Dual Output Formats**: Generate results in both JSON and human-readable formats ### Technical Features - Command-line interface with comprehensive argument parsing - Error handling for common file and processing issues - Progress reporting for batch operations - Configurable output formatting and verbosity levels - Cross-platform compatibility with standard library only dependencies ## Usage ### Basic Text Analysis ```bash python text_processor.py analyze document.txt python text_processor.py analyze document.txt --output results.json ``` ### Text Transformation ```bash python text_processor.py transform document.txt --mode uppercase python text_processor.py transform document.txt --mode title --output transformed.txt ``` ### Batch Processing ```bash python text_processor.py batch text_files/ --output results/ python text_processor.py batch text_files/ --format json --output batch_results.json ``` ## Examples ### Example 1: Basic Word Count ```bash $ python text_processor.py analyze sample.txt === TEXT ANALYSIS RESULTS === File: sample.txt Total words: 150 Unique words: 85 Total characters: 750 Lines: 12 Most frequent word: "the" (8 occurrences) ``` ### Example 2: JSON Output ```bash $ python text_processor.py analyze sample.txt --format json { "file": "sample.txt", "statistics": { "total_words": 150, "unique_words": 85, "total_characters": 750, "lines": 12, "most_frequent": { "word": "the", "count": 8 } } } ``` ### Example 3: Text Transformation ```bash $ python text_processor.py transform sample.txt --mode title Original: "hello world from the text processor" Transformed: "Hello World From The Text Processor" ``` ## Installation This skill requires only Python 3.7 or later with the standard library. No external dependencies are required. 1. Clone or download the skill directory 2. Navigate to the scripts directory 3. Run the text processor directly with Python ```bash cd scripts/ python text_processor.py --help ``` ## Configuration The text processor supports various configuration options through command-line arguments: - `--format`: Output format (json, text) - `--verbose`: Enable verbose output and progress reporting - `--output`: Specify output file or directory - `--encoding`: Specify text file encoding (default: utf-8) ## Architecture The skill follows a simple modular architecture: - **TextProcessor Class**: Core processing logic and statistics calculation - **OutputFormatter Class**: Handles dual output format generation - **FileManager Class**: Manages file I/O operations and batch processing - **CLI Interface**: Command-line argument parsing and user interaction ## Error Handling The skill includes comprehensive error handling for: - File not found or permission errors - Invalid encoding or corrupted text files - Memory limitations for very large files - Output directory creation and write permissions - Invalid command-line arguments and parameters ## Performance Considerations - Efficient memory usage for large text files through streaming - Optimized word counting using dictionary lookups - Batch processing with progress reporting for large datasets - Configurable encoding detection for international text ## Contributing This skill serves as a reference implementation and contributions are welcome to demonstrate best practices: 1. Follow PEP 8 coding standards 2. Include comprehensive docstrings 3. Add test cases with sample data 4. Update documentation for any new features 5. Ensure backward compatibility ## Limitations As a BASIC tier skill, some advanced features are intentionally omitted: - Complex text analysis (sentiment, language detection) - Advanced file format support (PDF, Word documents) - Database integration or external API calls - Parallel processing for very large datasets This skill demonstrates the essential structure and quality standards required for BASIC tier skills in the claude-skills ecosystem while remaining simple and focused on core functionality. FILE:assets/sample-skill/assets/sample_text.txt This is a sample text file for testing the text processor skill. It contains multiple lines of text with various words and punctuation. The quick brown fox jumps over the lazy dog. This sentence contains all 26 letters of the English alphabet. Some additional content: - Numbers: 123, 456, 789 - Special characters: !@#$%^&*() - Mixed case: CamelCase, snake_case, PascalCase Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco. This file serves as a basic test case for: 1. Word counting functionality 2. Character analysis 3. Line counting 4. Text transformations 5. Statistical analysis The text processor should handle this content correctly and produce meaningful statistics and transformations for testing purposes. FILE:assets/sample-skill/assets/test_data.csv name,age,city,country John Doe,25,New York,USA Jane Smith,30,London,UK Bob Johnson,22,Toronto,Canada Alice Brown,28,Sydney,Australia Charlie Wilson,35,Berlin,Germany This CSV file contains sample data with headers and multiple rows. It can be used to test the text processor's ability to handle structured data formats and count words across different content types. The file includes: - Header row with column names - Data rows with mixed text and numbers - Various city and country names - Different age values for statistical analysis FILE:assets/sample-skill/expected_outputs/sample_text_analysis.json { "file": "assets/sample_text.txt", "file_size": 855, "total_words": 116, "unique_words": 87, "total_characters": 855, "lines": 19, "average_word_length": 4.7, "most_frequent": { "word": "the", "count": 5 } } FILE:assets/sample-skill/references/api-reference.md # Text Processor API Reference ## Classes ### TextProcessor Main class for text processing operations. #### `__init__(self, encoding: str = 'utf-8')` Initialize the text processor with specified encoding. **Parameters:** - `encoding` (str): Character encoding for file operations. Default: 'utf-8' #### `analyze_text(self, text: str) -> Dict[str, Any]` Analyze text and return comprehensive statistics. **Parameters:** - `text` (str): Text content to analyze **Returns:** - `dict`: Statistics including word count, character count, lines, most frequent word **Example:** ```python processor = TextProcessor() stats = processor.analyze_text("Hello world") # Returns: {'total_words': 2, 'unique_words': 2, ...} ``` #### `transform_text(self, text: str, mode: str) -> str` Transform text according to specified mode. **Parameters:** - `text` (str): Text to transform - `mode` (str): Transformation mode ('upper', 'lower', 'title', 'reverse') **Returns:** - `str`: Transformed text **Raises:** - `ValueError`: If mode is not supported ### OutputFormatter Static methods for output formatting. #### `format_json(data: Dict[str, Any]) -> str` Format data as JSON string. #### `format_human_readable(data: Dict[str, Any]) -> str` Format data as human-readable text. ### FileManager Handles file operations and batch processing. #### `find_text_files(self, directory: str) -> List[str]` Find all text files in a directory recursively. **Supported Extensions:** - .txt - .md - .rst - .csv - .log ## Command Line Interface ### Commands #### `analyze` Analyze text file statistics. ```bash python text_processor.py analyze [options] ``` #### `transform` Transform text file content. ```bash python text_processor.py transform --mode [options] ``` #### `batch` Process multiple files in a directory. ```bash python text_processor.py batch [options] ``` ### Global Options - `--format {json,text}`: Output format (default: text) - `--output FILE`: Output file path (default: stdout) - `--encoding ENCODING`: Text file encoding (default: utf-8) - `--verbose`: Enable verbose output ## Error Handling The text processor handles several error conditions: - **FileNotFoundError**: When input file doesn't exist - **UnicodeDecodeError**: When file encoding doesn't match specified encoding - **PermissionError**: When file access is denied - **ValueError**: When invalid transformation mode is specified All errors are reported to stderr with descriptive messages. FILE:assets/sample-skill/scripts/text_processor.py #!/usr/bin/env python3 """ Sample Text Processor - Basic text analysis and transformation tool This script demonstrates the basic structure and functionality expected in BASIC tier skills. It provides text processing capabilities with proper argument parsing, error handling, and dual output formats. Usage: python text_processor.py analyze [options] python text_processor.py transform --mode [options] python text_processor.py batch [options] Author: Claude Skills Engineering Team Version: 1.0.0 Dependencies: Python Standard Library Only """ import argparse import json import os import sys from collections import Counter from pathlib import Path from typing import Dict, List, Any, Optional class TextProcessor: """Core text processing functionality""" def __init__(self, encoding: str = 'utf-8'): self.encoding = encoding def analyze_text(self, text: str) -> Dict[str, Any]: """Analyze text and return statistics""" lines = text.split('\n') words = text.lower().split() # Calculate basic statistics stats = { 'total_words': len(words), 'unique_words': len(set(words)), 'total_characters': len(text), 'lines': len(lines), 'average_word_length': sum(len(word) for word in words) / len(words) if words else 0 } # Find most frequent word if words: word_counts = Counter(words) most_common = word_counts.most_common(1)[0] stats['most_frequent'] = { 'word': most_common[0], 'count': most_common[1] } else: stats['most_frequent'] = {'word': '', 'count': 0} return stats def transform_text(self, text: str, mode: str) -> str: """Transform text according to specified mode""" if mode == 'upper': return text.upper() elif mode == 'lower': return text.lower() elif mode == 'title': return text.title() elif mode == 'reverse': return text[::-1] else: raise ValueError(f"Unknown transformation mode: {mode}") def process_file(self, file_path: str) -> Dict[str, Any]: """Process a single text file""" try: with open(file_path, 'r', encoding=self.encoding) as file: content = file.read() stats = self.analyze_text(content) stats['file'] = file_path stats['file_size'] = os.path.getsize(file_path) return stats except FileNotFoundError: raise FileNotFoundError(f"File not found: {file_path}") except UnicodeDecodeError: raise UnicodeDecodeError(f"Cannot decode file with {self.encoding} encoding: {file_path}") except PermissionError: raise PermissionError(f"Permission denied accessing file: {file_path}") class OutputFormatter: """Handles dual output format generation""" @staticmethod def format_json(data: Dict[str, Any]) -> str: """Format data as JSON""" return json.dumps(data, indent=2, ensure_ascii=False) @staticmethod def format_human_readable(data: Dict[str, Any]) -> str: """Format data as human-readable text""" lines = [] lines.append("=== TEXT ANALYSIS RESULTS ===") lines.append(f"File: {data.get('file', 'Unknown')}") lines.append(f"File size: {data.get('file_size', 0)} bytes") lines.append(f"Total words: {data.get('total_words', 0)}") lines.append(f"Unique words: {data.get('unique_words', 0)}") lines.append(f"Total characters: {data.get('total_characters', 0)}") lines.append(f"Lines: {data.get('lines', 0)}") lines.append(f"Average word length: {data.get('average_word_length', 0):.1f}") most_frequent = data.get('most_frequent', {}) lines.append(f"Most frequent word: \"{most_frequent.get('word', '')}\" ({most_frequent.get('count', 0)} occurrences)") return "\n".join(lines) class FileManager: """Manages file I/O operations and batch processing""" def __init__(self, verbose: bool = False): self.verbose = verbose def log_verbose(self, message: str): """Log verbose message if verbose mode enabled""" if self.verbose: print(f"[INFO] {message}", file=sys.stderr) def find_text_files(self, directory: str) -> List[str]: """Find all text files in directory""" text_extensions = {'.txt', '.md', '.rst', '.csv', '.log'} text_files = [] try: for file_path in Path(directory).rglob('*'): if file_path.is_file() and file_path.suffix.lower() in text_extensions: text_files.append(str(file_path)) except PermissionError: raise PermissionError(f"Permission denied accessing directory: {directory}") return text_files def write_output(self, content: str, output_path: Optional[str] = None): """Write content to file or stdout""" if output_path: try: # Create directory if needed output_dir = os.path.dirname(output_path) if output_dir and not os.path.exists(output_dir): os.makedirs(output_dir) with open(output_path, 'w', encoding='utf-8') as file: file.write(content) self.log_verbose(f"Output written to: {output_path}") except PermissionError: raise PermissionError(f"Permission denied writing to: {output_path}") else: print(content) def analyze_command(args: argparse.Namespace) -> int: """Handle analyze command""" try: processor = TextProcessor(args.encoding) file_manager = FileManager(args.verbose) file_manager.log_verbose(f"Analyzing file: {args.file}") # Process the file results = processor.process_file(args.file) # Format output if args.format == 'json': output = OutputFormatter.format_json(results) else: output = OutputFormatter.format_human_readable(results) # Write output file_manager.write_output(output, args.output) return 0 except FileNotFoundError as e: print(f"Error: {e}", file=sys.stderr) return 1 except UnicodeDecodeError as e: print(f"Error: {e}", file=sys.stderr) print(f"Try using --encoding option with different encoding", file=sys.stderr) return 1 except Exception as e: print(f"Error: {e}", file=sys.stderr) return 1 def transform_command(args: argparse.Namespace) -> int: """Handle transform command""" try: processor = TextProcessor(args.encoding) file_manager = FileManager(args.verbose) file_manager.log_verbose(f"Transforming file: {args.file}") # Read and transform the file with open(args.file, 'r', encoding=args.encoding) as file: content = file.read() transformed = processor.transform_text(content, args.mode) # Write transformed content file_manager.write_output(transformed, args.output) return 0 except FileNotFoundError as e: print(f"Error: {e}", file=sys.stderr) return 1 except ValueError as e: print(f"Error: {e}", file=sys.stderr) return 1 except Exception as e: print(f"Error: {e}", file=sys.stderr) return 1 def batch_command(args: argparse.Namespace) -> int: """Handle batch command""" try: processor = TextProcessor(args.encoding) file_manager = FileManager(args.verbose) file_manager.log_verbose(f"Finding text files in: {args.directory}") # Find all text files text_files = file_manager.find_text_files(args.directory) if not text_files: print(f"No text files found in directory: {args.directory}", file=sys.stderr) return 1 file_manager.log_verbose(f"Found {len(text_files)} text files") # Process all files all_results = [] for i, file_path in enumerate(text_files, 1): try: file_manager.log_verbose(f"Processing {i}/{len(text_files)}: {file_path}") results = processor.process_file(file_path) all_results.append(results) except Exception as e: print(f"Warning: Failed to process {file_path}: {e}", file=sys.stderr) continue if not all_results: print("Error: No files could be processed successfully", file=sys.stderr) return 1 # Format batch results batch_summary = { 'total_files': len(all_results), 'total_words': sum(r.get('total_words', 0) for r in all_results), 'total_characters': sum(r.get('total_characters', 0) for r in all_results), 'files': all_results } if args.format == 'json': output = OutputFormatter.format_json(batch_summary) else: lines = [] lines.append("=== BATCH PROCESSING RESULTS ===") lines.append(f"Total files processed: {batch_summary['total_files']}") lines.append(f"Total words across all files: {batch_summary['total_words']}") lines.append(f"Total characters across all files: {batch_summary['total_characters']}") lines.append("") lines.append("Individual file results:") for result in all_results: lines.append(f" {result['file']}: {result['total_words']} words") output = "\n".join(lines) # Write output file_manager.write_output(output, args.output) return 0 except PermissionError as e: print(f"Error: {e}", file=sys.stderr) return 1 except Exception as e: print(f"Error: {e}", file=sys.stderr) return 1 def main(): """Main entry point with argument parsing""" parser = argparse.ArgumentParser( description="Sample Text Processor - Basic text analysis and transformation", formatter_class=argparse.RawDescriptionHelpFormatter, epilog=""" Examples: Analysis: python text_processor.py analyze document.txt python text_processor.py analyze document.txt --format json --output results.json Transformation: python text_processor.py transform document.txt --mode upper python text_processor.py transform document.txt --mode title --output transformed.txt Batch processing: python text_processor.py batch text_files/ --verbose python text_processor.py batch text_files/ --format json --output batch_results.json Transformation modes: upper - Convert to uppercase lower - Convert to lowercase title - Convert to title case reverse - Reverse the text """ ) parser.add_argument('--format', choices=['json', 'text'], default='text', help='Output format (default: text)') parser.add_argument('--output', help='Output file path (default: stdout)') parser.add_argument('--encoding', default='utf-8', help='Text file encoding (default: utf-8)') parser.add_argument('--verbose', action='store_true', help='Enable verbose output') subparsers = parser.add_subparsers(dest='command', help='Available commands') # Analyze subcommand analyze_parser = subparsers.add_parser('analyze', help='Analyze text file statistics') analyze_parser.add_argument('file', help='Text file to analyze') # Transform subcommand transform_parser = subparsers.add_parser('transform', help='Transform text file') transform_parser.add_argument('file', help='Text file to transform') transform_parser.add_argument('--mode', required=True, choices=['upper', 'lower', 'title', 'reverse'], help='Transformation mode') # Batch subcommand batch_parser = subparsers.add_parser('batch', help='Process multiple files') batch_parser.add_argument('directory', help='Directory containing text files') args = parser.parse_args() if not args.command: parser.print_help() return 1 try: if args.command == 'analyze': return analyze_command(args) elif args.command == 'transform': return transform_command(args) elif args.command == 'batch': return batch_command(args) else: print(f"Unknown command: {args.command}", file=sys.stderr) return 1 except KeyboardInterrupt: print("\nOperation interrupted by user", file=sys.stderr) return 130 except Exception as e: print(f"Unexpected error: {e}", file=sys.stderr) return 1 if __name__ == "__main__": sys.exit(main()) FILE:expected_outputs/sample_validation_report.json { "skill_path": "assets/sample-skill", "timestamp": "2026-02-16T16:41:00Z", "overall_score": 85.0, "compliance_level": "GOOD", "checks": { "skill_md_exists": { "passed": true, "message": "SKILL.md found", "score": 1.0 }, "readme_exists": { "passed": true, "message": "README.md found", "score": 1.0 }, "skill_md_length": { "passed": true, "message": "SKILL.md has 145 lines (≥100)", "score": 1.0 }, "frontmatter_complete": { "passed": true, "message": "All required frontmatter fields present", "score": 1.0 }, "required_sections": { "passed": true, "message": "All required sections present", "score": 1.0 }, "dir_scripts_exists": { "passed": true, "message": "scripts/ directory found", "score": 1.0 }, "min_scripts_count": { "passed": true, "message": "Found 1 Python scripts (≥1)", "score": 1.0 }, "script_syntax_text_processor.py": { "passed": true, "message": "text_processor.py has valid Python syntax", "score": 1.0 }, "script_argparse_text_processor.py": { "passed": true, "message": "Uses argparse in text_processor.py", "score": 1.0 }, "script_main_guard_text_processor.py": { "passed": true, "message": "Has main guard in text_processor.py", "score": 1.0 }, "tier_compliance": { "passed": true, "message": "Meets BASIC tier requirements", "score": 1.0 } }, "warnings": [], "errors": [], "suggestions": [ "Consider adding optional directories: references, expected_outputs" ] } FILE:references/quality-scoring-rubric.md # Quality Scoring Rubric **Version**: 1.0.0 **Last Updated**: 2026-02-16 **Authority**: Claude Skills Engineering Team ## Overview This document defines the comprehensive quality scoring methodology used to assess skills within the claude-skills ecosystem. The scoring system evaluates four key dimensions, each weighted equally at 25%, to provide an objective and consistent measure of skill quality. ## Scoring Framework ### Overall Scoring Scale - **A+ (95-100)**: Exceptional quality, exceeds all standards - **A (90-94)**: Excellent quality, meets highest standards consistently - **A- (85-89)**: Very good quality, minor areas for improvement - **B+ (80-84)**: Good quality, meets most standards well - **B (75-79)**: Satisfactory quality, meets standards adequately - **B- (70-74)**: Below average, several areas need improvement - **C+ (65-69)**: Poor quality, significant improvements needed - **C (60-64)**: Minimal acceptable quality, major improvements required - **C- (55-59)**: Unacceptable quality, extensive rework needed - **D (50-54)**: Very poor quality, fundamental issues present - **F (0-49)**: Failing quality, does not meet basic standards ### Dimension Weights Each dimension contributes equally to the overall score: - **Documentation Quality**: 25% - **Code Quality**: 25% - **Completeness**: 25% - **Usability**: 25% ## Documentation Quality (25% Weight) ### Scoring Components #### SKILL.md Quality (40% of Documentation Score) **Component Breakdown:** - **Length and Depth (25%)**: Line count and content substance - **Frontmatter Quality (25%)**: Completeness and accuracy of YAML metadata - **Section Coverage (25%)**: Required and recommended section presence - **Content Depth (25%)**: Technical detail and comprehensiveness **Scoring Criteria:** | Score Range | Length | Frontmatter | Sections | Depth | |-------------|--------|-------------|----------|-------| | 90-100 | 400+ lines | All fields complete + extras | All required + 4+ recommended | Rich technical detail, examples | | 80-89 | 300-399 lines | All required fields complete | All required + 2-3 recommended | Good technical coverage | | 70-79 | 200-299 lines | Most required fields | All required + 1 recommended | Adequate technical content | | 60-69 | 150-199 lines | Some required fields | Most required sections | Basic technical information | | 50-59 | 100-149 lines | Minimal frontmatter | Some required sections | Limited technical detail | | Below 50 | <100 lines | Missing/invalid frontmatter | Few/no required sections | Insufficient content | #### README.md Quality (25% of Documentation Score) **Scoring Criteria:** - **Excellent (90-100)**: 1000+ chars, comprehensive usage guide, examples, troubleshooting - **Good (75-89)**: 500-999 chars, clear usage instructions, basic examples - **Satisfactory (60-74)**: 200-499 chars, minimal usage information - **Poor (40-59)**: <200 chars or confusing content - **Failing (0-39)**: Missing or completely inadequate #### Reference Documentation (20% of Documentation Score) **Scoring Criteria:** - **Excellent (90-100)**: Multiple comprehensive reference docs (2000+ chars total) - **Good (75-89)**: 2-3 reference files with substantial content - **Satisfactory (60-74)**: 1-2 reference files with adequate content - **Poor (40-59)**: Minimal reference content or poor quality - **Failing (0-39)**: No reference documentation #### Examples and Usage Clarity (15% of Documentation Score) **Scoring Criteria:** - **Excellent (90-100)**: 5+ diverse examples, clear usage patterns - **Good (75-89)**: 3-4 examples covering different scenarios - **Satisfactory (60-74)**: 2-3 basic examples - **Poor (40-59)**: 1-2 minimal examples - **Failing (0-39)**: No examples or unclear usage ## Code Quality (25% Weight) ### Scoring Components #### Script Complexity and Architecture (25% of Code Score) **Evaluation Criteria:** - Lines of code per script relative to tier requirements - Function and class organization - Code modularity and reusability - Algorithm sophistication **Scoring Matrix:** | Tier | Excellent (90-100) | Good (75-89) | Satisfactory (60-74) | Poor (Below 60) | |------|-------------------|--------------|---------------------|-----------------| | BASIC | 200-300 LOC, well-structured | 150-199 LOC, organized | 100-149 LOC, basic | <100 LOC, minimal | | STANDARD | 400-500 LOC, modular | 350-399 LOC, structured | 300-349 LOC, adequate | <300 LOC, basic | | POWERFUL | 600-800 LOC, sophisticated | 550-599 LOC, advanced | 500-549 LOC, solid | <500 LOC, simple | #### Error Handling Quality (25% of Code Score) **Scoring Criteria:** - **Excellent (90-100)**: Comprehensive exception handling, specific error types, recovery mechanisms - **Good (75-89)**: Good exception handling, meaningful error messages, logging - **Satisfactory (60-74)**: Basic try/except blocks, simple error messages - **Poor (40-59)**: Minimal error handling, generic exceptions - **Failing (0-39)**: No error handling or inappropriate handling **Error Handling Checklist:** - [ ] Try/except blocks for risky operations - [ ] Specific exception types (not just Exception) - [ ] Meaningful error messages for users - [ ] Proper error logging or reporting - [ ] Graceful degradation where possible - [ ] Input validation and sanitization #### Code Structure and Organization (25% of Code Score) **Evaluation Elements:** - Function decomposition and single responsibility - Class design and inheritance patterns - Import organization and dependency management - Documentation and comments quality - Consistent naming conventions - PEP 8 compliance **Scoring Guidelines:** - **Excellent (90-100)**: Exemplary structure, comprehensive docstrings, perfect style - **Good (75-89)**: Well-organized, good documentation, minor style issues - **Satisfactory (60-74)**: Adequate structure, basic documentation, some style issues - **Poor (40-59)**: Poor organization, minimal documentation, style problems - **Failing (0-39)**: No clear structure, no documentation, major style violations #### Output Format Support (25% of Code Score) **Required Capabilities:** - JSON output format support - Human-readable output format - Proper data serialization - Consistent output structure - Error output handling **Scoring Criteria:** - **Excellent (90-100)**: Dual format + custom formats, perfect serialization - **Good (75-89)**: Dual format support, good serialization - **Satisfactory (60-74)**: Single format well-implemented - **Poor (40-59)**: Basic output, formatting issues - **Failing (0-39)**: Poor or no structured output ## Completeness (25% Weight) ### Scoring Components #### Directory Structure Compliance (25% of Completeness Score) **Required Directories by Tier:** - **BASIC**: scripts/ (required), assets/ + references/ (recommended) - **STANDARD**: scripts/ + assets/ + references/ (required), expected_outputs/ (recommended) - **POWERFUL**: scripts/ + assets/ + references/ + expected_outputs/ (all required) **Scoring Calculation:** ``` Structure Score = (Required Present / Required Total) * 0.6 + (Recommended Present / Recommended Total) * 0.4 ``` #### Asset Availability and Quality (25% of Completeness Score) **Scoring Criteria:** - **Excellent (90-100)**: 5+ diverse assets, multiple file types, realistic data - **Good (75-89)**: 3-4 assets, some diversity, good quality - **Satisfactory (60-74)**: 2-3 assets, basic variety - **Poor (40-59)**: 1-2 minimal assets - **Failing (0-39)**: No assets or unusable assets **Asset Quality Factors:** - File diversity (JSON, CSV, YAML, etc.) - Data realism and complexity - Coverage of use cases - File size appropriateness - Documentation of asset purpose #### Expected Output Coverage (25% of Completeness Score) **Evaluation Criteria:** - Correspondence with asset files - Coverage of success and error scenarios - Output format variety - Reproducibility and accuracy **Scoring Matrix:** - **Excellent (90-100)**: Complete output coverage, all scenarios, verified accuracy - **Good (75-89)**: Good coverage, most scenarios, mostly accurate - **Satisfactory (60-74)**: Basic coverage, main scenarios - **Poor (40-59)**: Minimal coverage, some inaccuracies - **Failing (0-39)**: No expected outputs or completely inaccurate #### Test Coverage and Validation (25% of Completeness Score) **Assessment Areas:** - Sample data processing capability - Output verification mechanisms - Edge case handling - Error condition testing - Integration test scenarios **Scoring Guidelines:** - **Excellent (90-100)**: Comprehensive test coverage, automated validation - **Good (75-89)**: Good test coverage, manual validation possible - **Satisfactory (60-74)**: Basic testing capability - **Poor (40-59)**: Minimal testing support - **Failing (0-39)**: No testing or validation capability ## Usability (25% Weight) ### Scoring Components #### Installation and Setup Simplicity (25% of Usability Score) **Evaluation Factors:** - Dependency requirements (Python stdlib preferred) - Setup complexity - Environment requirements - Installation documentation clarity **Scoring Criteria:** - **Excellent (90-100)**: Zero external dependencies, single-file execution - **Good (75-89)**: Minimal dependencies, simple setup - **Satisfactory (60-74)**: Some dependencies, documented setup - **Poor (40-59)**: Complex dependencies, unclear setup - **Failing (0-39)**: Unable to install or excessive complexity #### Usage Clarity and Help Quality (25% of Usability Score) **Assessment Elements:** - Command-line help comprehensiveness - Usage example clarity - Parameter documentation quality - Error message helpfulness **Help Quality Checklist:** - [ ] Comprehensive --help output - [ ] Clear parameter descriptions - [ ] Usage examples included - [ ] Error messages are actionable - [ ] Progress indicators where appropriate **Scoring Matrix:** - **Excellent (90-100)**: Exemplary help, multiple examples, perfect error messages - **Good (75-89)**: Good help quality, clear examples, helpful errors - **Satisfactory (60-74)**: Adequate help, basic examples - **Poor (40-59)**: Minimal help, confusing interface - **Failing (0-39)**: No help or completely unclear interface #### Documentation Accessibility (25% of Usability Score) **Evaluation Criteria:** - README quick start effectiveness - SKILL.md navigation and structure - Reference material organization - Learning curve considerations **Accessibility Factors:** - Information hierarchy clarity - Cross-reference quality - Beginner-friendly explanations - Advanced user shortcuts - Troubleshooting guidance #### Practical Example Quality (25% of Usability Score) **Assessment Areas:** - Example realism and relevance - Complexity progression (simple to advanced) - Output demonstration - Common use case coverage - Integration scenarios **Scoring Guidelines:** - **Excellent (90-100)**: 5+ examples, perfect progression, real-world scenarios - **Good (75-89)**: 3-4 examples, good variety, practical scenarios - **Satisfactory (60-74)**: 2-3 examples, adequate coverage - **Poor (40-59)**: 1-2 examples, limited practical value - **Failing (0-39)**: No examples or completely impractical ## Scoring Calculations ### Dimension Score Calculation Each dimension score is calculated as a weighted average of its components: ```python def calculate_dimension_score(components): total_weighted_score = 0 total_weight = 0 for component_name, component_data in components.items(): score = component_data['score'] weight = component_data['weight'] total_weighted_score += score * weight total_weight += weight return total_weighted_score / total_weight if total_weight > 0 else 0 ``` ### Overall Score Calculation The overall score combines all dimensions with equal weighting: ```python def calculate_overall_score(dimensions): return sum(dimension.score * 0.25 for dimension in dimensions.values()) ``` ### Letter Grade Assignment ```python def assign_letter_grade(overall_score): if overall_score >= 95: return "A+" elif overall_score >= 90: return "A" elif overall_score >= 85: return "A-" elif overall_score >= 80: return "B+" elif overall_score >= 75: return "B" elif overall_score >= 70: return "B-" elif overall_score >= 65: return "C+" elif overall_score >= 60: return "C" elif overall_score >= 55: return "C-" elif overall_score >= 50: return "D" else: return "F" ``` ## Quality Improvement Recommendations ### Score-Based Recommendations #### For Scores Below 60 (C- or Lower) **Priority Actions:** 1. Address fundamental structural issues 2. Implement basic error handling 3. Add essential documentation sections 4. Create minimal viable examples 5. Fix critical functionality issues #### For Scores 60-74 (C+ to B-) **Improvement Areas:** 1. Expand documentation comprehensiveness 2. Enhance error handling sophistication 3. Add more diverse examples and use cases 4. Improve code organization and structure 5. Increase test coverage and validation #### For Scores 75-84 (B to B+) **Enhancement Opportunities:** 1. Refine documentation for expert-level quality 2. Implement advanced error recovery mechanisms 3. Add comprehensive reference materials 4. Optimize code architecture and performance 5. Develop extensive example library #### For Scores 85+ (A- or Higher) **Excellence Maintenance:** 1. Regular quality audits and updates 2. Community feedback integration 3. Best practice evolution tracking 4. Mentoring lower-quality skills 5. Innovation and cutting-edge feature adoption ### Dimension-Specific Improvement Strategies #### Low Documentation Scores - Expand SKILL.md with technical details - Add comprehensive API reference - Include architecture diagrams and explanations - Develop troubleshooting guides - Create contributor documentation #### Low Code Quality Scores - Refactor for better modularity - Implement comprehensive error handling - Add extensive code documentation - Apply advanced design patterns - Optimize performance and efficiency #### Low Completeness Scores - Add missing directories and files - Develop comprehensive sample datasets - Create expected output libraries - Implement automated testing - Add integration examples #### Low Usability Scores - Simplify installation process - Improve command-line interface design - Enhance help text and documentation - Create beginner-friendly tutorials - Add interactive examples ## Quality Assurance Process ### Automated Scoring The quality scorer runs automated assessments based on this rubric: 1. File system analysis for structure compliance 2. Content analysis for documentation quality 3. Code analysis for quality metrics 4. Asset inventory and quality assessment ### Manual Review Process Human reviewers validate automated scores and provide qualitative insights: 1. Content quality assessment beyond automated metrics 2. Usability testing with real-world scenarios 3. Technical accuracy verification 4. Community value assessment ### Continuous Improvement The scoring rubric evolves based on: - Community feedback and usage patterns - Industry best practice changes - Tool capability enhancements - Quality trend analysis This quality scoring rubric ensures consistent, objective, and comprehensive assessment of all skills within the claude-skills ecosystem while providing clear guidance for quality improvement. FILE:references/skill-structure-specification.md # Skill Structure Specification **Version**: 1.0.0 **Last Updated**: 2026-02-16 **Authority**: Claude Skills Engineering Team ## Overview This document defines the mandatory and optional components that constitute a well-formed skill within the claude-skills ecosystem. All skills must adhere to these structural requirements to ensure consistency, maintainability, and quality across the repository. ## Directory Structure ### Mandatory Components ``` skill-name/ ├── SKILL.md # Primary skill documentation (REQUIRED) ├── README.md # Usage instructions and quick start (REQUIRED) └── scripts/ # Python implementation scripts (REQUIRED) └── *.py # At least one Python script ``` ### Recommended Components ``` skill-name/ ├── SKILL.md ├── README.md ├── scripts/ │ └── *.py ├── assets/ # Sample data and input files (RECOMMENDED) │ ├── samples/ │ ├── examples/ │ └── data/ ├── references/ # Reference documentation (RECOMMENDED) │ ├── api-reference.md │ ├── specifications.md │ └── external-links.md └── expected_outputs/ # Expected results for testing (RECOMMENDED) ├── sample_output.json ├── example_results.txt └── test_cases/ ``` ### Optional Components ``` skill-name/ ├── [mandatory and recommended components] ├── tests/ # Unit tests and validation scripts ├── examples/ # Extended examples and tutorials ├── docs/ # Additional documentation ├── config/ # Configuration files └── templates/ # Template files for code generation ``` ## File Requirements ### SKILL.md Requirements The `SKILL.md` file serves as the primary documentation for the skill and must contain: #### Mandatory YAML Frontmatter ```yaml --- Name: skill-name Tier: [BASIC|STANDARD|POWERFUL] Category: [Category Name] Dependencies: [None|List of dependencies] Author: [Author Name] Version: [Semantic Version] Last Updated: [YYYY-MM-DD] --- ``` #### Required Sections - **Description**: Comprehensive overview of the skill's purpose and capabilities - **Features**: Detailed list of key features and functionality - **Usage**: Instructions for using the skill and its components - **Examples**: Practical usage examples with expected outcomes #### Recommended Sections - **Architecture**: Technical architecture and design decisions - **Installation**: Setup and installation instructions - **Configuration**: Configuration options and parameters - **Troubleshooting**: Common issues and solutions - **Contributing**: Guidelines for contributors - **Changelog**: Version history and changes #### Content Requirements by Tier - **BASIC**: Minimum 100 lines of substantial content - **STANDARD**: Minimum 200 lines of substantial content - **POWERFUL**: Minimum 300 lines of substantial content ### README.md Requirements The `README.md` file provides quick start instructions and must include: #### Mandatory Content - Brief description of the skill - Quick start instructions - Basic usage examples - Link to full SKILL.md documentation #### Recommended Content - Installation instructions - Prerequisites and dependencies - Command-line usage examples - Troubleshooting section - Contributing guidelines #### Length Requirements - Minimum 200 characters of substantial content - Recommended 500+ characters for comprehensive coverage ### Scripts Directory Requirements The `scripts/` directory contains all Python implementation files: #### Mandatory Requirements - At least one Python (.py) file - All scripts must be executable Python 3.7+ - No external dependencies outside Python standard library - Proper file naming conventions (lowercase, hyphens for separation) #### Script Content Requirements - **Shebang line**: `#!/usr/bin/env python3` - **Module docstring**: Comprehensive description of script purpose - **Argparse implementation**: Command-line argument parsing - **Main guard**: `if __name__ == "__main__":` protection - **Error handling**: Appropriate exception handling and user feedback - **Dual output**: Support for both JSON and human-readable output formats #### Script Size Requirements by Tier - **BASIC**: 100-300 lines of code per script - **STANDARD**: 300-500 lines of code per script - **POWERFUL**: 500-800 lines of code per script ### Assets Directory Structure The `assets/` directory contains sample data and supporting files: ``` assets/ ├── samples/ # Sample input data │ ├── simple_example.json │ ├── complex_dataset.csv │ └── test_configuration.yaml ├── examples/ # Example files demonstrating usage │ ├── basic_workflow.py │ ├── advanced_usage.sh │ └── integration_example.md └── data/ # Static data files ├── reference_data.json ├── lookup_tables.csv └── configuration_templates/ ``` #### Content Requirements - At least 2 sample files demonstrating different use cases - Files should represent realistic usage scenarios - Include both simple and complex examples where applicable - Provide diverse file formats (JSON, CSV, YAML, etc.) ### References Directory Structure The `references/` directory contains detailed reference documentation: ``` references/ ├── api-reference.md # Complete API documentation ├── specifications.md # Technical specifications and requirements ├── external-links.md # Links to related resources ├── algorithms.md # Algorithm descriptions and implementations └── best-practices.md # Usage best practices and patterns ``` #### Content Requirements - Each file should contain substantial technical content (500+ words) - Include code examples and technical specifications - Provide external references and links where appropriate - Maintain consistent documentation format and style ### Expected Outputs Directory Structure The `expected_outputs/` directory contains reference outputs for testing: ``` expected_outputs/ ├── basic_example_output.json ├── complex_scenario_result.txt ├── error_cases/ │ ├── invalid_input_error.json │ └── timeout_error.txt └── test_cases/ ├── unit_test_outputs/ └── integration_test_results/ ``` #### Content Requirements - Outputs correspond to sample inputs in assets/ directory - Include both successful and error case examples - Provide outputs in multiple formats (JSON, text, CSV) - Ensure outputs are reproducible and verifiable ## Naming Conventions ### Directory Names - Use lowercase letters only - Use hyphens (-) to separate words - Keep names concise but descriptive - Avoid special characters and spaces Examples: `data-processor`, `api-client`, `ml-trainer` ### File Names - Use lowercase letters for Python scripts - Use hyphens (-) to separate words in script names - Use underscores (_) only when required by Python conventions - Use descriptive names that indicate purpose Examples: `data-processor.py`, `api-client.py`, `quality_scorer.py` ### Script Internal Naming - Use PascalCase for class names - Use snake_case for function and variable names - Use UPPER_CASE for constants - Use descriptive names that indicate purpose ## Quality Standards ### Documentation Standards - All documentation must be written in clear, professional English - Use proper Markdown formatting and structure - Include code examples with syntax highlighting - Provide comprehensive coverage of all features - Maintain consistent terminology throughout ### Code Standards - Follow PEP 8 Python style guidelines - Include comprehensive docstrings for all functions and classes - Implement proper error handling with meaningful error messages - Use type hints where appropriate - Maintain reasonable code complexity and readability ### Testing Standards - Provide sample data that exercises all major functionality - Include expected outputs for verification - Cover both successful and error scenarios - Ensure reproducible results across different environments ## Validation Criteria Skills are validated against the following criteria: ### Structural Validation - All mandatory files and directories present - Proper file naming conventions followed - Directory structure matches specification - File permissions and accessibility correct ### Content Validation - SKILL.md meets minimum length and section requirements - README.md provides adequate quick start information - Scripts contain required components (argparse, main guard, etc.) - Sample data and expected outputs are complete and realistic ### Quality Validation - Documentation is comprehensive and accurate - Code follows established style and quality guidelines - Examples are practical and demonstrate real usage - Error handling is appropriate and user-friendly ## Compliance Levels ### Full Compliance - All mandatory components present and complete - All recommended components present with substantial content - Exceeds minimum quality thresholds for tier - Demonstrates best practices throughout ### Partial Compliance - All mandatory components present - Most recommended components present - Meets minimum quality thresholds for tier - Generally follows established patterns ### Non-Compliance - Missing mandatory components - Inadequate content quality or length - Does not meet minimum tier requirements - Significant deviations from established standards ## Migration and Updates ### Existing Skills Skills created before this specification should be updated to comply within: - **POWERFUL tier**: 30 days - **STANDARD tier**: 60 days - **BASIC tier**: 90 days ### Specification Updates - Changes to this specification require team consensus - Breaking changes must provide 90-day migration period - All changes must be documented with rationale and examples - Automated validation tools must be updated accordingly ## Tools and Automation ### Validation Tools - `skill_validator.py` - Validates structure and content compliance - `script_tester.py` - Tests script functionality and quality - `quality_scorer.py` - Provides comprehensive quality assessment ### Integration Points - Pre-commit hooks for basic validation - CI/CD pipeline integration for pull request validation - Automated quality reporting and tracking - Integration with code review processes ## Examples and Templates ### Minimal BASIC Tier Example ``` basic-skill/ ├── SKILL.md # 100+ lines ├── README.md # Basic usage instructions └── scripts/ └── main.py # 100-300 lines with argparse ``` ### Complete POWERFUL Tier Example ``` powerful-skill/ ├── SKILL.md # 300+ lines with comprehensive sections ├── README.md # Detailed usage and setup ├── scripts/ # Multiple sophisticated scripts │ ├── main_processor.py # 500-800 lines │ ├── data_analyzer.py # 500-800 lines │ └── report_generator.py # 500-800 lines ├── assets/ # Diverse sample data │ ├── samples/ │ ├── examples/ │ └── data/ ├── references/ # Comprehensive documentation │ ├── api-reference.md │ ├── specifications.md │ └── best-practices.md └── expected_outputs/ # Complete test outputs ├── json_outputs/ ├── text_reports/ └── error_cases/ ``` This specification serves as the authoritative guide for skill structure within the claude-skills ecosystem. Adherence to these standards ensures consistency, quality, and maintainability across all skills in the repository. FILE:references/tier-requirements-matrix.md # Tier Requirements Matrix **Version**: 1.0.0 **Last Updated**: 2026-02-16 **Authority**: Claude Skills Engineering Team ## Overview This document provides a comprehensive matrix of requirements for each skill tier within the claude-skills ecosystem. Skills are classified into three tiers based on complexity, functionality, and comprehensiveness: BASIC, STANDARD, and POWERFUL. ## Tier Classification Philosophy ### BASIC Tier Entry-level skills that provide fundamental functionality with minimal complexity. Suitable for simple automation tasks, basic data processing, or straightforward utilities. ### STANDARD Tier Intermediate skills that offer enhanced functionality with moderate complexity. Suitable for business processes, advanced data manipulation, or multi-step workflows. ### POWERFUL Tier Advanced skills that provide comprehensive functionality with sophisticated implementation. Suitable for complex systems, enterprise-grade tools, or mission-critical applications. ## Requirements Matrix | Component | BASIC | STANDARD | POWERFUL | |-----------|-------|----------|----------| | **SKILL.md Lines** | ≥100 | ≥200 | ≥300 | | **Scripts Count** | ≥1 | ≥1 | ≥2 | | **Script Size (LOC)** | 100-300 | 300-500 | 500-800 | | **Required Directories** | scripts | scripts, assets, references | scripts, assets, references, expected_outputs | | **Argparse Implementation** | Basic | Advanced | Complex with subcommands | | **Output Formats** | Human-readable | JSON + Human-readable | JSON + Human-readable + Custom | | **Error Handling** | Basic | Comprehensive | Advanced with recovery | | **Documentation Depth** | Functional | Comprehensive | Expert-level | | **Examples Provided** | ≥1 | ≥3 | ≥5 | | **Test Coverage** | Basic validation | Sample data testing | Comprehensive test suite | ## Detailed Requirements by Tier ### BASIC Tier Requirements #### Documentation Requirements - **SKILL.md**: Minimum 100 lines of substantial content - **Required Sections**: Name, Description, Features, Usage, Examples - **README.md**: Basic usage instructions (200+ characters) - **Content Quality**: Clear and functional documentation - **Examples**: At least 1 practical usage example #### Code Requirements - **Scripts**: Minimum 1 Python script (100-300 LOC) - **Argparse**: Basic command-line argument parsing - **Main Guard**: `if __name__ == "__main__":` protection - **Dependencies**: Python standard library only - **Output**: Human-readable format with clear messaging - **Error Handling**: Basic exception handling with user-friendly messages #### Structure Requirements - **Mandatory Directories**: `scripts/` - **Recommended Directories**: `assets/`, `references/` - **File Organization**: Logical file naming and structure - **Assets**: Optional sample data files #### Quality Standards - **Code Style**: Follows basic Python conventions - **Documentation**: Adequate coverage of functionality - **Usability**: Clear usage instructions and examples - **Completeness**: All essential components present ### STANDARD Tier Requirements #### Documentation Requirements - **SKILL.md**: Minimum 200 lines with comprehensive coverage - **Required Sections**: All BASIC sections plus Architecture, Installation - **README.md**: Detailed usage instructions (500+ characters) - **References**: Technical documentation in `references/` directory - **Content Quality**: Professional-grade documentation with technical depth - **Examples**: At least 3 diverse usage examples #### Code Requirements - **Scripts**: 1-2 Python scripts (300-500 LOC each) - **Argparse**: Advanced argument parsing with subcommands and validation - **Output Formats**: Both JSON and human-readable output support - **Error Handling**: Comprehensive exception handling with specific error types - **Code Structure**: Well-organized classes and functions - **Documentation**: Comprehensive docstrings for all functions #### Structure Requirements - **Mandatory Directories**: `scripts/`, `assets/`, `references/` - **Recommended Directories**: `expected_outputs/` - **Assets**: Multiple sample files demonstrating different use cases - **References**: Technical specifications and API documentation - **Expected Outputs**: Sample results for validation #### Quality Standards - **Code Quality**: Advanced Python patterns and best practices - **Documentation**: Expert-level technical documentation - **Testing**: Sample data processing with validation - **Integration**: Consideration for CI/CD and automation use ### POWERFUL Tier Requirements #### Documentation Requirements - **SKILL.md**: Minimum 300 lines with expert-level comprehensiveness - **Required Sections**: All STANDARD sections plus Troubleshooting, Contributing, Advanced Usage - **README.md**: Comprehensive guide with installation and setup (1000+ characters) - **References**: Multiple technical documents with specifications - **Content Quality**: Publication-ready documentation with architectural details - **Examples**: At least 5 examples covering simple to complex scenarios #### Code Requirements - **Scripts**: 2-3 Python scripts (500-800 LOC each) - **Argparse**: Complex argument parsing with multiple modes and configurations - **Output Formats**: JSON, human-readable, and custom format support - **Error Handling**: Advanced error handling with recovery mechanisms - **Code Architecture**: Sophisticated design patterns and modular structure - **Performance**: Optimized for efficiency and scalability #### Structure Requirements - **Mandatory Directories**: `scripts/`, `assets/`, `references/`, `expected_outputs/` - **Optional Directories**: `tests/`, `examples/`, `docs/` - **Assets**: Comprehensive sample data covering edge cases - **References**: Complete technical specification suite - **Expected Outputs**: Full test result coverage including error cases - **Testing**: Comprehensive validation and test coverage #### Quality Standards - **Enterprise Grade**: Production-ready code with enterprise patterns - **Documentation**: Comprehensive technical documentation suitable for technical teams - **Integration**: Full CI/CD integration capabilities - **Maintainability**: Designed for long-term maintenance and extension ## Tier Assessment Criteria ### Automatic Tier Classification Skills are automatically classified based on quantitative metrics: ```python def classify_tier(skill_metrics): if (skill_metrics['skill_md_lines'] >= 300 and skill_metrics['script_count'] >= 2 and skill_metrics['min_script_size'] >= 500 and all_required_dirs_present(['scripts', 'assets', 'references', 'expected_outputs'])): return 'POWERFUL' elif (skill_metrics['skill_md_lines'] >= 200 and skill_metrics['script_count'] >= 1 and skill_metrics['min_script_size'] >= 300 and all_required_dirs_present(['scripts', 'assets', 'references'])): return 'STANDARD' else: return 'BASIC' ``` ### Manual Tier Override Manual tier assignment may be considered when: - Skill provides exceptional value despite not meeting all quantitative requirements - Skill addresses critical infrastructure or security needs - Skill demonstrates innovative approaches or cutting-edge techniques - Skill provides essential integration or compatibility functions ### Tier Promotion Criteria Skills may be promoted to higher tiers when: - All quantitative requirements for higher tier are met - Quality assessment scores exceed tier thresholds - Community usage and feedback indicate higher value - Continuous integration and maintenance demonstrate reliability ### Tier Demotion Criteria Skills may be demoted to lower tiers when: - Quality degradation below tier standards - Lack of maintenance or updates - Compatibility issues or security vulnerabilities - Community feedback indicates reduced value ## Implementation Guidelines by Tier ### BASIC Tier Implementation ```python # Example argparse implementation for BASIC tier parser = argparse.ArgumentParser(description="Basic skill functionality") parser.add_argument("input", help="Input file or parameter") parser.add_argument("--output", help="Output destination") parser.add_argument("--verbose", action="store_true", help="Verbose output") # Basic error handling try: result = process_input(args.input) print(f"Processing completed: {result}") except FileNotFoundError: print("Error: Input file not found") sys.exit(1) except Exception as e: print(f"Error: {str(e)}") sys.exit(1) ``` ### STANDARD Tier Implementation ```python # Example argparse implementation for STANDARD tier parser = argparse.ArgumentParser( description="Standard skill with advanced functionality", formatter_class=argparse.RawDescriptionHelpFormatter, epilog="Examples:\n python script.py input.json --format json\n python script.py data/ --batch --output results/" ) parser.add_argument("input", help="Input file or directory") parser.add_argument("--format", choices=["json", "text"], default="json", help="Output format") parser.add_argument("--batch", action="store_true", help="Process multiple files") parser.add_argument("--output", help="Output destination") # Advanced error handling with specific exception types try: if args.batch: results = batch_process(args.input) else: results = single_process(args.input) if args.format == "json": print(json.dumps(results, indent=2)) else: print_human_readable(results) except FileNotFoundError as e: logging.error(f"File not found: {e}") sys.exit(1) except ValueError as e: logging.error(f"Invalid input: {e}") sys.exit(2) except Exception as e: logging.error(f"Unexpected error: {e}") sys.exit(1) ``` ### POWERFUL Tier Implementation ```python # Example argparse implementation for POWERFUL tier parser = argparse.ArgumentParser( description="Powerful skill with comprehensive functionality", formatter_class=argparse.RawDescriptionHelpFormatter, epilog=""" Examples: Basic usage: python script.py process input.json --output results/ Advanced batch processing: python script.py batch data/ --format json --parallel 4 --filter "*.csv" Custom configuration: python script.py process input.json --config custom.yaml --dry-run """ ) subparsers = parser.add_subparsers(dest="command", help="Available commands") # Process subcommand process_parser = subparsers.add_parser("process", help="Process single file") process_parser.add_argument("input", help="Input file path") process_parser.add_argument("--config", help="Configuration file") process_parser.add_argument("--dry-run", action="store_true", help="Show what would be done") # Batch subcommand batch_parser = subparsers.add_parser("batch", help="Process multiple files") batch_parser.add_argument("directory", help="Input directory") batch_parser.add_argument("--parallel", type=int, default=1, help="Number of parallel processes") batch_parser.add_argument("--filter", help="File filter pattern") # Comprehensive error handling with recovery try: if args.command == "process": result = process_with_recovery(args.input, args.config, args.dry_run) elif args.command == "batch": result = batch_process_with_monitoring(args.directory, args.parallel, args.filter) else: parser.print_help() sys.exit(1) # Multiple output format support output_formatter = OutputFormatter(args.format) output_formatter.write(result, args.output) except KeyboardInterrupt: logging.info("Processing interrupted by user") sys.exit(130) except ProcessingError as e: logging.error(f"Processing failed: {e}") if e.recoverable: logging.info("Attempting recovery...") # Recovery logic here sys.exit(1) except ValidationError as e: logging.error(f"Validation failed: {e}") logging.info("Check input format and try again") sys.exit(2) except Exception as e: logging.critical(f"Critical error: {e}") logging.info("Please report this issue") sys.exit(1) ``` ## Quality Scoring by Tier ### Scoring Thresholds - **POWERFUL Tier**: Overall score ≥80, all dimensions ≥75 - **STANDARD Tier**: Overall score ≥70, 3+ dimensions ≥65 - **BASIC Tier**: Overall score ≥60, meets minimum requirements ### Dimension Weights (All Tiers) - **Documentation**: 25% - **Code Quality**: 25% - **Completeness**: 25% - **Usability**: 25% ### Tier-Specific Quality Expectations #### BASIC Tier Quality Profile - Documentation: Functional and clear (60+ points expected) - Code Quality: Clean and maintainable (60+ points expected) - Completeness: Essential components present (60+ points expected) - Usability: Easy to understand and use (60+ points expected) #### STANDARD Tier Quality Profile - Documentation: Professional and comprehensive (70+ points expected) - Code Quality: Advanced patterns and best practices (70+ points expected) - Completeness: All recommended components (70+ points expected) - Usability: Well-designed user experience (70+ points expected) #### POWERFUL Tier Quality Profile - Documentation: Expert-level and publication-ready (80+ points expected) - Code Quality: Enterprise-grade implementation (80+ points expected) - Completeness: Comprehensive test and validation coverage (80+ points expected) - Usability: Exceptional user experience with extensive help (80+ points expected) ## Tier Migration Process ### Promotion Process 1. **Assessment**: Quality scorer evaluates skill against higher tier requirements 2. **Review**: Engineering team reviews assessment and implementation 3. **Testing**: Comprehensive testing against higher tier standards 4. **Approval**: Team consensus on tier promotion 5. **Update**: Skill metadata and documentation updated to reflect new tier ### Demotion Process 1. **Issue Identification**: Quality degradation or standards violation identified 2. **Assessment**: Current quality evaluated against tier requirements 3. **Notice**: Skill maintainer notified of potential demotion 4. **Grace Period**: 30-day period for remediation 5. **Final Review**: Re-assessment after grace period 6. **Action**: Tier adjustment or removal if standards not met ### Tier Change Communication - All tier changes logged in skill CHANGELOG.md - Repository-level tier change notifications - Integration with CI/CD systems for automated handling - Community notifications for significant changes ## Compliance Monitoring ### Automated Monitoring - Daily quality assessment scans - Tier compliance validation in CI/CD - Automated reporting of tier violations - Integration with code review processes ### Manual Review Process - Quarterly tier review cycles - Community feedback integration - Expert panel reviews for complex cases - Appeals process for tier disputes ### Enforcement Actions - **Warning**: First violation or minor issues - **Probation**: Repeated violations or moderate issues - **Demotion**: Serious violations or quality degradation - **Removal**: Critical violations or abandonment This tier requirements matrix serves as the definitive guide for skill classification and quality standards within the claude-skills ecosystem. Regular updates ensure alignment with evolving best practices and community needs. FILE:scripts/quality_scorer.py #!/usr/bin/env python3 """ Quality Scorer - Scores skills across multiple quality dimensions This script provides comprehensive quality assessment for skills in the claude-skills ecosystem by evaluating documentation, code quality, completeness, and usability. Generates letter grades, tier recommendations, and improvement roadmaps. Usage: python quality_scorer.py [--detailed] [--minimum-score SCORE] [--json] Author: Claude Skills Engineering Team Version: 1.0.0 Dependencies: Python Standard Library Only """ import argparse import ast import json import os import re import sys from datetime import datetime from pathlib import Path from typing import Dict, List, Any, Optional, Tuple try: import yaml except ImportError: # Minimal YAML subset: parse simple key: value frontmatter without pyyaml class _YamlStub: class YAMLError(Exception): pass @staticmethod def safe_load(text): result = {} for line in text.strip().splitlines(): if ':' in line: key, _, value = line.partition(':') result[key.strip()] = value.strip() return result if result else None yaml = _YamlStub() class QualityDimension: """Represents a quality scoring dimension""" def __init__(self, name: str, weight: float, description: str): self.name = name self.weight = weight self.description = description self.score = 0.0 self.max_score = 100.0 self.details = {} self.suggestions = [] def add_score(self, component: str, score: float, max_score: float, details: str = ""): """Add a component score""" self.details[component] = { "score": score, "max_score": max_score, "percentage": (score / max_score * 100) if max_score > 0 else 0, "details": details } def calculate_final_score(self): """Calculate the final weighted score for this dimension""" if not self.details: self.score = 0.0 return total_score = sum(detail["score"] for detail in self.details.values()) total_max = sum(detail["max_score"] for detail in self.details.values()) self.score = (total_score / total_max * 100) if total_max > 0 else 0.0 def add_suggestion(self, suggestion: str): """Add an improvement suggestion""" self.suggestions.append(suggestion) class QualityReport: """Container for quality assessment results""" def __init__(self, skill_path: str): self.skill_path = skill_path self.timestamp = datetime.utcnow().isoformat() + "Z" self.dimensions = {} self.overall_score = 0.0 self.letter_grade = "F" self.tier_recommendation = "BASIC" self.improvement_roadmap = [] self.summary_stats = {} def add_dimension(self, dimension: QualityDimension): """Add a quality dimension""" self.dimensions[dimension.name] = dimension def calculate_overall_score(self): """Calculate overall weighted score""" if not self.dimensions: return total_weighted_score = 0.0 total_weight = 0.0 for dimension in self.dimensions.values(): total_weighted_score += dimension.score * dimension.weight total_weight += dimension.weight self.overall_score = total_weighted_score / total_weight if total_weight > 0 else 0.0 # Calculate letter grade if self.overall_score >= 95: self.letter_grade = "A+" elif self.overall_score >= 90: self.letter_grade = "A" elif self.overall_score >= 85: self.letter_grade = "A-" elif self.overall_score >= 80: self.letter_grade = "B+" elif self.overall_score >= 75: self.letter_grade = "B" elif self.overall_score >= 70: self.letter_grade = "B-" elif self.overall_score >= 65: self.letter_grade = "C+" elif self.overall_score >= 60: self.letter_grade = "C" elif self.overall_score >= 55: self.letter_grade = "C-" elif self.overall_score >= 50: self.letter_grade = "D" else: self.letter_grade = "F" # Recommend tier based on overall score and specific criteria self._calculate_tier_recommendation() # Generate improvement roadmap self._generate_improvement_roadmap() # Calculate summary statistics self._calculate_summary_stats() def _calculate_tier_recommendation(self): """Calculate recommended tier based on quality scores""" doc_score = self.dimensions.get("Documentation", QualityDimension("", 0, "")).score code_score = self.dimensions.get("Code Quality", QualityDimension("", 0, "")).score completeness_score = self.dimensions.get("Completeness", QualityDimension("", 0, "")).score usability_score = self.dimensions.get("Usability", QualityDimension("", 0, "")).score # POWERFUL tier requirements (all dimensions must be strong) if (self.overall_score >= 80 and all(score >= 75 for score in [doc_score, code_score, completeness_score, usability_score])): self.tier_recommendation = "POWERFUL" # STANDARD tier requirements (most dimensions good) elif (self.overall_score >= 70 and sum(1 for score in [doc_score, code_score, completeness_score, usability_score] if score >= 65) >= 3): self.tier_recommendation = "STANDARD" # BASIC tier (minimum viable quality) else: self.tier_recommendation = "BASIC" def _generate_improvement_roadmap(self): """Generate prioritized improvement suggestions""" all_suggestions = [] # Collect suggestions from all dimensions with scores for dim_name, dimension in self.dimensions.items(): for suggestion in dimension.suggestions: priority = "HIGH" if dimension.score < 60 else "MEDIUM" if dimension.score < 75 else "LOW" all_suggestions.append({ "priority": priority, "dimension": dim_name, "suggestion": suggestion, "current_score": dimension.score }) # Sort by priority and score priority_order = {"HIGH": 0, "MEDIUM": 1, "LOW": 2} all_suggestions.sort(key=lambda x: (priority_order[x["priority"]], x["current_score"])) self.improvement_roadmap = all_suggestions[:10] # Top 10 suggestions def _calculate_summary_stats(self): """Calculate summary statistics""" scores = [dim.score for dim in self.dimensions.values()] self.summary_stats = { "highest_dimension": max(self.dimensions.items(), key=lambda x: x[1].score)[0] if scores else "None", "lowest_dimension": min(self.dimensions.items(), key=lambda x: x[1].score)[0] if scores else "None", "score_variance": sum((score - self.overall_score) ** 2 for score in scores) / len(scores) if scores else 0, "dimensions_above_70": sum(1 for score in scores if score >= 70), "dimensions_below_50": sum(1 for score in scores if score < 50) } class QualityScorer: """Main quality scoring engine""" def __init__(self, skill_path: str, detailed: bool = False, verbose: bool = False): self.skill_path = Path(skill_path).resolve() self.detailed = detailed self.verbose = verbose self.report = QualityReport(str(self.skill_path)) def log_verbose(self, message: str): """Log verbose message if verbose mode enabled""" if self.verbose: print(f"[VERBOSE] {message}", file=sys.stderr) def assess_quality(self) -> QualityReport: """Main quality assessment entry point""" try: self.log_verbose(f"Starting quality assessment for {self.skill_path}") # Check if skill path exists if not self.skill_path.exists(): raise ValueError(f"Skill path does not exist: {self.skill_path}") # Score each dimension self._score_documentation() self._score_code_quality() self._score_completeness() self._score_usability() # Calculate overall metrics self.report.calculate_overall_score() self.log_verbose(f"Quality assessment completed. Overall score: {self.report.overall_score:.1f}") except Exception as e: print(f"Quality assessment failed: {str(e)}", file=sys.stderr) raise return self.report def _score_documentation(self): """Score documentation quality (25% weight)""" self.log_verbose("Scoring documentation quality...") dimension = QualityDimension("Documentation", 0.25, "Quality of documentation and written materials") # Score SKILL.md self._score_skill_md(dimension) # Score README.md self._score_readme(dimension) # Score reference documentation self._score_references(dimension) # Score examples and usage clarity self._score_examples(dimension) dimension.calculate_final_score() self.report.add_dimension(dimension) def _score_skill_md(self, dimension: QualityDimension): """Score SKILL.md quality""" skill_md_path = self.skill_path / "SKILL.md" if not skill_md_path.exists(): dimension.add_score("skill_md_existence", 0, 25, "SKILL.md does not exist") dimension.add_suggestion("Create comprehensive SKILL.md file") return try: content = skill_md_path.read_text(encoding='utf-8') lines = [line for line in content.split('\n') if line.strip()] # Score based on length and depth line_count = len(lines) if line_count >= 400: length_score = 25 elif line_count >= 300: length_score = 20 elif line_count >= 200: length_score = 15 elif line_count >= 100: length_score = 10 else: length_score = 5 dimension.add_score("skill_md_length", length_score, 25, f"SKILL.md has {line_count} lines") if line_count < 300: dimension.add_suggestion("Expand SKILL.md with more detailed sections") # Score frontmatter quality frontmatter_score = self._score_frontmatter(content) dimension.add_score("skill_md_frontmatter", frontmatter_score, 25, "Frontmatter completeness and accuracy") # Score section completeness section_score = self._score_sections(content) dimension.add_score("skill_md_sections", section_score, 25, "Required and recommended section coverage") # Score content depth depth_score = self._score_content_depth(content) dimension.add_score("skill_md_depth", depth_score, 25, "Content depth and technical detail") except Exception as e: dimension.add_score("skill_md_readable", 0, 25, f"Error reading SKILL.md: {str(e)}") dimension.add_suggestion("Fix SKILL.md file encoding or format issues") def _score_frontmatter(self, content: str) -> float: """Score SKILL.md frontmatter quality""" required_fields = ["Name", "Tier", "Category", "Dependencies", "Author", "Version"] recommended_fields = ["Last Updated", "Description"] try: if not content.startswith('---'): return 5 # Partial credit for having some structure end_marker = content.find('---', 3) if end_marker == -1: return 5 frontmatter_text = content[3:end_marker].strip() frontmatter = yaml.safe_load(frontmatter_text) if not isinstance(frontmatter, dict): return 5 score = 0 # Required fields (15 points) present_required = sum(1 for field in required_fields if field in frontmatter) score += (present_required / len(required_fields)) * 15 # Recommended fields (5 points) present_recommended = sum(1 for field in recommended_fields if field in frontmatter) score += (present_recommended / len(recommended_fields)) * 5 # Quality of field values (5 points) quality_bonus = 0 for field, value in frontmatter.items(): if isinstance(value, str) and len(value.strip()) > 3: quality_bonus += 0.5 score += min(quality_bonus, 5) return min(score, 25) except yaml.YAMLError: return 5 # Some credit for attempting frontmatter def _score_sections(self, content: str) -> float: """Score section completeness""" required_sections = ["Description", "Features", "Usage", "Examples"] recommended_sections = ["Architecture", "Installation", "Troubleshooting", "Contributing"] score = 0 # Required sections (15 points) present_required = 0 for section in required_sections: if re.search(rf'^#+\s*{re.escape(section)}\s*$', content, re.MULTILINE | re.IGNORECASE): present_required += 1 score += (present_required / len(required_sections)) * 15 # Recommended sections (10 points) present_recommended = 0 for section in recommended_sections: if re.search(rf'^#+\s*{re.escape(section)}\s*$', content, re.MULTILINE | re.IGNORECASE): present_recommended += 1 score += (present_recommended / len(recommended_sections)) * 10 return score def _score_content_depth(self, content: str) -> float: """Score content depth and technical detail""" score = 0 # Code examples (8 points) code_blocks = len(re.findall(r'```[\w]*\n.*?\n```', content, re.DOTALL)) score += min(code_blocks * 2, 8) # Technical depth indicators (8 points) depth_indicators = ['API', 'algorithm', 'architecture', 'implementation', 'performance', 'scalability', 'security', 'integration', 'configuration', 'parameters'] depth_score = sum(1 for indicator in depth_indicators if indicator.lower() in content.lower()) score += min(depth_score * 0.8, 8) # Usage examples (9 points) example_patterns = [r'Example:', r'Usage:', r'```bash', r'```python', r'```yaml'] example_count = sum(len(re.findall(pattern, content, re.IGNORECASE)) for pattern in example_patterns) score += min(example_count * 1.5, 9) return score def _score_readme(self, dimension: QualityDimension): """Score README.md quality""" readme_path = self.skill_path / "README.md" if not readme_path.exists(): dimension.add_score("readme_existence", 10, 25, "README.md exists (partial credit)") dimension.add_suggestion("Create README.md with usage instructions") return try: content = readme_path.read_text(encoding='utf-8') # Length and substance if len(content.strip()) >= 1000: length_score = 25 elif len(content.strip()) >= 500: length_score = 20 elif len(content.strip()) >= 200: length_score = 15 else: length_score = 10 dimension.add_score("readme_quality", length_score, 25, f"README.md content quality ({len(content)} characters)") if len(content.strip()) < 500: dimension.add_suggestion("Expand README.md with more detailed usage examples") except Exception: dimension.add_score("readme_readable", 5, 25, "README.md exists but has issues") def _score_references(self, dimension: QualityDimension): """Score reference documentation quality""" references_dir = self.skill_path / "references" if not references_dir.exists(): dimension.add_score("references_existence", 0, 25, "No references directory") dimension.add_suggestion("Add references directory with documentation") return ref_files = list(references_dir.glob("*.md")) + list(references_dir.glob("*.txt")) if not ref_files: dimension.add_score("references_content", 5, 25, "References directory empty") dimension.add_suggestion("Add reference documentation files") return # Score based on number and quality of reference files score = min(len(ref_files) * 5, 20) # Up to 20 points for multiple files # Bonus for substantial content total_content = 0 for ref_file in ref_files: try: content = ref_file.read_text(encoding='utf-8') total_content += len(content.strip()) except: continue if total_content >= 2000: score += 5 # Bonus for substantial reference content dimension.add_score("references_quality", score, 25, f"References: {len(ref_files)} files, {total_content} chars") def _score_examples(self, dimension: QualityDimension): """Score examples and usage clarity""" score = 0 # Look for example files in various locations example_locations = ["examples", "assets", "scripts"] example_files = [] for location in example_locations: location_path = self.skill_path / location if location_path.exists(): example_files.extend(location_path.glob("*example*")) example_files.extend(location_path.glob("*sample*")) example_files.extend(location_path.glob("*demo*")) # Score based on example availability if len(example_files) >= 3: score = 25 elif len(example_files) >= 2: score = 20 elif len(example_files) >= 1: score = 15 else: score = 10 dimension.add_suggestion("Add more usage examples and sample files") dimension.add_score("examples_availability", score, 25, f"Found {len(example_files)} example/sample files") def _score_code_quality(self): """Score code quality (25% weight)""" self.log_verbose("Scoring code quality...") dimension = QualityDimension("Code Quality", 0.25, "Quality of Python scripts and implementation") scripts_dir = self.skill_path / "scripts" if not scripts_dir.exists(): dimension.add_score("scripts_existence", 0, 100, "No scripts directory") dimension.add_suggestion("Create scripts directory with Python files") dimension.calculate_final_score() self.report.add_dimension(dimension) return python_files = list(scripts_dir.glob("*.py")) if not python_files: dimension.add_score("python_scripts", 0, 100, "No Python scripts found") dimension.add_suggestion("Add Python scripts to scripts directory") dimension.calculate_final_score() self.report.add_dimension(dimension) return # Score script complexity and quality self._score_script_complexity(python_files, dimension) # Score error handling self._score_error_handling(python_files, dimension) # Score code structure self._score_code_structure(python_files, dimension) # Score output format support self._score_output_support(python_files, dimension) dimension.calculate_final_score() self.report.add_dimension(dimension) def _score_script_complexity(self, python_files: List[Path], dimension: QualityDimension): """Score script complexity and sophistication""" total_complexity = 0 script_count = len(python_files) for script_path in python_files: try: content = script_path.read_text(encoding='utf-8') # Count lines of code (excluding empty lines and comments) lines = content.split('\n') loc = len([line for line in lines if line.strip() and not line.strip().startswith('#')]) # Score based on LOC if loc >= 800: complexity_score = 25 elif loc >= 500: complexity_score = 20 elif loc >= 300: complexity_score = 15 elif loc >= 100: complexity_score = 10 else: complexity_score = 5 total_complexity += complexity_score except Exception: continue avg_complexity = total_complexity / script_count if script_count > 0 else 0 dimension.add_score("script_complexity", avg_complexity, 25, f"Average script complexity across {script_count} scripts") if avg_complexity < 15: dimension.add_suggestion("Consider expanding scripts with more functionality") def _score_error_handling(self, python_files: List[Path], dimension: QualityDimension): """Score error handling quality""" total_error_score = 0 script_count = len(python_files) for script_path in python_files: try: content = script_path.read_text(encoding='utf-8') error_score = 0 # Check for try/except blocks try_count = content.count('try:') error_score += min(try_count * 5, 15) # Up to 15 points for try/except # Check for specific exception handling exception_types = ['Exception', 'ValueError', 'FileNotFoundError', 'KeyError', 'TypeError'] for exc_type in exception_types: if exc_type in content: error_score += 2 # 2 points per specific exception type # Check for logging or error reporting if any(indicator in content for indicator in ['print(', 'logging.', 'sys.stderr']): error_score += 5 # 5 points for error reporting total_error_score += min(error_score, 25) # Cap at 25 per script except Exception: continue avg_error_score = total_error_score / script_count if script_count > 0 else 0 dimension.add_score("error_handling", avg_error_score, 25, f"Error handling quality across {script_count} scripts") if avg_error_score < 15: dimension.add_suggestion("Improve error handling with try/except blocks and meaningful error messages") def _score_code_structure(self, python_files: List[Path], dimension: QualityDimension): """Score code structure and organization""" total_structure_score = 0 script_count = len(python_files) for script_path in python_files: try: content = script_path.read_text(encoding='utf-8') structure_score = 0 # Check for functions and classes function_count = content.count('def ') class_count = content.count('class ') structure_score += min(function_count * 2, 10) # Up to 10 points for functions structure_score += min(class_count * 3, 9) # Up to 9 points for classes # Check for docstrings docstring_patterns = ['"""', "'''", 'def.*:\n.*"""', 'class.*:\n.*"""'] for pattern in docstring_patterns: if re.search(pattern, content): structure_score += 1 # 1 point per docstring indicator # Check for if __name__ == "__main__" if 'if __name__ == "__main__"' in content: structure_score += 3 # Check for imports organization if content.lstrip().startswith(('import ', 'from ')): structure_score += 2 # Imports at top total_structure_score += min(structure_score, 25) except Exception: continue avg_structure_score = total_structure_score / script_count if script_count > 0 else 0 dimension.add_score("code_structure", avg_structure_score, 25, f"Code structure quality across {script_count} scripts") if avg_structure_score < 15: dimension.add_suggestion("Improve code structure with more functions, classes, and documentation") def _score_output_support(self, python_files: List[Path], dimension: QualityDimension): """Score output format support""" total_output_score = 0 script_count = len(python_files) for script_path in python_files: try: content = script_path.read_text(encoding='utf-8') output_score = 0 # Check for JSON support if any(indicator in content for indicator in ['json.dump', 'json.load', '--json']): output_score += 12 # JSON support # Check for formatted output if any(indicator in content for indicator in ['print(f"', 'print("', '.format(', 'f"']): output_score += 8 # Human-readable output # Check for argparse help if '--help' in content or 'add_help=' in content: output_score += 5 # Help functionality total_output_score += min(output_score, 25) except Exception: continue avg_output_score = total_output_score / script_count if script_count > 0 else 0 dimension.add_score("output_support", avg_output_score, 25, f"Output format support across {script_count} scripts") if avg_output_score < 15: dimension.add_suggestion("Add support for both JSON and human-readable output formats") def _score_completeness(self): """Score completeness (25% weight)""" self.log_verbose("Scoring completeness...") dimension = QualityDimension("Completeness", 0.25, "Completeness of required components and assets") # Score directory structure self._score_directory_structure(dimension) # Score asset availability self._score_assets(dimension) # Score expected outputs self._score_expected_outputs(dimension) # Score test coverage self._score_test_coverage(dimension) dimension.calculate_final_score() self.report.add_dimension(dimension) def _score_directory_structure(self, dimension: QualityDimension): """Score directory structure completeness""" required_dirs = ["scripts"] recommended_dirs = ["assets", "references", "expected_outputs"] score = 0 # Required directories (15 points) for dir_name in required_dirs: if (self.skill_path / dir_name).exists(): score += 15 / len(required_dirs) # Recommended directories (10 points) present_recommended = 0 for dir_name in recommended_dirs: if (self.skill_path / dir_name).exists(): present_recommended += 1 score += (present_recommended / len(recommended_dirs)) * 10 dimension.add_score("directory_structure", score, 25, f"Directory structure completeness") missing_recommended = [d for d in recommended_dirs if not (self.skill_path / d).exists()] if missing_recommended: dimension.add_suggestion(f"Add recommended directories: {', '.join(missing_recommended)}") def _score_assets(self, dimension: QualityDimension): """Score asset availability and quality""" assets_dir = self.skill_path / "assets" if not assets_dir.exists(): dimension.add_score("assets_existence", 5, 25, "Assets directory missing") dimension.add_suggestion("Create assets directory with sample data") return asset_files = [f for f in assets_dir.rglob("*") if f.is_file()] if not asset_files: dimension.add_score("assets_content", 10, 25, "Assets directory empty") dimension.add_suggestion("Add sample data files to assets directory") return # Score based on number and diversity of assets score = min(len(asset_files) * 3, 20) # Up to 20 points for multiple assets # Bonus for diverse file types extensions = set(f.suffix.lower() for f in asset_files if f.suffix) if len(extensions) >= 3: score += 5 # Bonus for file type diversity dimension.add_score("assets_quality", score, 25, f"Assets: {len(asset_files)} files, {len(extensions)} types") def _score_expected_outputs(self, dimension: QualityDimension): """Score expected outputs availability""" expected_dir = self.skill_path / "expected_outputs" if not expected_dir.exists(): dimension.add_score("expected_outputs", 10, 25, "Expected outputs directory missing") dimension.add_suggestion("Add expected_outputs directory with sample results") return output_files = [f for f in expected_dir.rglob("*") if f.is_file()] if len(output_files) >= 3: score = 25 elif len(output_files) >= 2: score = 20 elif len(output_files) >= 1: score = 15 else: score = 10 dimension.add_suggestion("Add expected output files for testing") dimension.add_score("expected_outputs", score, 25, f"Expected outputs: {len(output_files)} files") def _score_test_coverage(self, dimension: QualityDimension): """Score test coverage and validation""" # This is a simplified scoring - in a more sophisticated system, # this would integrate with actual test runners score = 15 # Base score for having a structure # Check for test-related files test_indicators = ["test", "spec", "check"] test_files = [] for indicator in test_indicators: test_files.extend(self.skill_path.rglob(f"*{indicator}*")) if test_files: score += 10 # Bonus for test files dimension.add_score("test_coverage", score, 25, f"Test coverage indicators: {len(test_files)} files") if not test_files: dimension.add_suggestion("Add test files or validation scripts") def _score_usability(self): """Score usability (25% weight)""" self.log_verbose("Scoring usability...") dimension = QualityDimension("Usability", 0.25, "Ease of use and user experience") # Score installation simplicity self._score_installation(dimension) # Score usage clarity self._score_usage_clarity(dimension) # Score help and documentation accessibility self._score_help_accessibility(dimension) # Score practical examples self._score_practical_examples(dimension) dimension.calculate_final_score() self.report.add_dimension(dimension) def _score_installation(self, dimension: QualityDimension): """Score installation simplicity""" # Check for installation complexity indicators score = 25 # Start with full points for standard library only approach # Check for requirements.txt or setup.py (would reduce score) if (self.skill_path / "requirements.txt").exists(): score -= 5 # Minor penalty for external dependencies dimension.add_suggestion("Consider removing external dependencies for easier installation") if (self.skill_path / "setup.py").exists(): score -= 3 # Minor penalty for complex setup dimension.add_score("installation_simplicity", max(score, 15), 25, "Installation complexity assessment") def _score_usage_clarity(self, dimension: QualityDimension): """Score usage clarity""" score = 0 # Check README for usage instructions readme_path = self.skill_path / "README.md" if readme_path.exists(): try: content = readme_path.read_text(encoding='utf-8').lower() if 'usage' in content or 'how to' in content: score += 10 if 'example' in content: score += 5 except: pass # Check scripts for help text quality scripts_dir = self.skill_path / "scripts" if scripts_dir.exists(): python_files = list(scripts_dir.glob("*.py")) help_quality = 0 for script_path in python_files: try: content = script_path.read_text(encoding='utf-8') if 'argparse' in content and 'help=' in content: help_quality += 2 except: continue score += min(help_quality, 10) # Up to 10 points for help text dimension.add_score("usage_clarity", score, 25, "Usage instructions and help quality") if score < 15: dimension.add_suggestion("Improve usage documentation and help text") def _score_help_accessibility(self, dimension: QualityDimension): """Score help and documentation accessibility""" score = 0 # Check for comprehensive help in scripts scripts_dir = self.skill_path / "scripts" if scripts_dir.exists(): python_files = list(scripts_dir.glob("*.py")) for script_path in python_files: try: content = script_path.read_text(encoding='utf-8') # Check for detailed help text if 'epilog=' in content or 'description=' in content: score += 5 # Detailed help # Check for examples in help if 'examples:' in content.lower() or 'example:' in content.lower(): score += 3 # Examples in help except: continue # Check for documentation files doc_files = list(self.skill_path.glob("*.md")) if len(doc_files) >= 2: score += 5 # Multiple documentation files dimension.add_score("help_accessibility", min(score, 25), 25, "Help and documentation accessibility") if score < 15: dimension.add_suggestion("Add more comprehensive help text and documentation") def _score_practical_examples(self, dimension: QualityDimension): """Score practical examples quality""" score = 0 # Look for example files example_patterns = ["*example*", "*sample*", "*demo*", "*tutorial*"] example_files = [] for pattern in example_patterns: example_files.extend(self.skill_path.rglob(pattern)) # Score based on example availability and quality if len(example_files) >= 5: score = 25 elif len(example_files) >= 3: score = 20 elif len(example_files) >= 2: score = 15 elif len(example_files) >= 1: score = 10 else: score = 5 dimension.add_suggestion("Add more practical examples and sample files") dimension.add_score("practical_examples", score, 25, f"Practical examples: {len(example_files)} files") class QualityReportFormatter: """Formats quality reports for output""" @staticmethod def format_json(report: QualityReport) -> str: """Format report as JSON""" return json.dumps({ "skill_path": report.skill_path, "timestamp": report.timestamp, "overall_score": round(report.overall_score, 1), "letter_grade": report.letter_grade, "tier_recommendation": report.tier_recommendation, "summary_stats": report.summary_stats, "dimensions": { name: { "name": dim.name, "weight": dim.weight, "score": round(dim.score, 1), "description": dim.description, "details": dim.details, "suggestions": dim.suggestions } for name, dim in report.dimensions.items() }, "improvement_roadmap": report.improvement_roadmap }, indent=2) @staticmethod def format_human_readable(report: QualityReport, detailed: bool = False) -> str: """Format report as human-readable text""" lines = [] lines.append("=" * 70) lines.append("SKILL QUALITY ASSESSMENT REPORT") lines.append("=" * 70) lines.append(f"Skill: {report.skill_path}") lines.append(f"Timestamp: {report.timestamp}") lines.append(f"Overall Score: {report.overall_score:.1f}/100 ({report.letter_grade})") lines.append(f"Recommended Tier: {report.tier_recommendation}") lines.append("") # Dimension scores lines.append("QUALITY DIMENSIONS:") for name, dimension in report.dimensions.items(): lines.append(f" {name}: {dimension.score:.1f}/100 ({dimension.weight * 100:.0f}% weight)") if detailed and dimension.details: for component, details in dimension.details.items(): lines.append(f" • {component}: {details['score']:.1f}/{details['max_score']} - {details['details']}") lines.append("") # Summary statistics if report.summary_stats: lines.append("SUMMARY STATISTICS:") lines.append(f" Highest Dimension: {report.summary_stats['highest_dimension']}") lines.append(f" Lowest Dimension: {report.summary_stats['lowest_dimension']}") lines.append(f" Dimensions Above 70%: {report.summary_stats['dimensions_above_70']}") lines.append(f" Dimensions Below 50%: {report.summary_stats['dimensions_below_50']}") lines.append("") # Improvement roadmap if report.improvement_roadmap: lines.append("IMPROVEMENT ROADMAP:") for i, item in enumerate(report.improvement_roadmap[:5], 1): priority_symbol = "🔴" if item["priority"] == "HIGH" else "🟡" if item["priority"] == "MEDIUM" else "🟢" lines.append(f" {i}. {priority_symbol} [{item['dimension']}] {item['suggestion']}") lines.append("") return "\n".join(lines) def main(): """Main entry point""" parser = argparse.ArgumentParser( description="Score skill quality across multiple dimensions", formatter_class=argparse.RawDescriptionHelpFormatter, epilog=""" Examples: python quality_scorer.py engineering/my-skill python quality_scorer.py engineering/my-skill --detailed --json python quality_scorer.py engineering/my-skill --minimum-score 75 Quality Dimensions (each 25%): Documentation - SKILL.md quality, README, references, examples Code Quality - Script complexity, error handling, structure, output Completeness - Directory structure, assets, expected outputs, tests Usability - Installation simplicity, usage clarity, help accessibility Letter Grades: A+ (95+), A (90+), A- (85+), B+ (80+), B (75+), B- (70+), C+ (65+), C (60+), C- (55+), D (50+), F (<50) """ ) parser.add_argument("skill_path", help="Path to the skill directory to assess") parser.add_argument("--detailed", action="store_true", help="Show detailed component scores") parser.add_argument("--minimum-score", type=float, default=0, help="Minimum acceptable score (exit with error if below)") parser.add_argument("--json", action="store_true", help="Output results in JSON format") parser.add_argument("--verbose", action="store_true", help="Enable verbose logging") args = parser.parse_args() try: # Create scorer and assess quality scorer = QualityScorer(args.skill_path, args.detailed, args.verbose) report = scorer.assess_quality() # Format and output report if args.json: print(QualityReportFormatter.format_json(report)) else: print(QualityReportFormatter.format_human_readable(report, args.detailed)) # Check minimum score requirement if report.overall_score < args.minimum_score: print(f"\nERROR: Quality score {report.overall_score:.1f} is below minimum {args.minimum_score}", file=sys.stderr) sys.exit(1) # Exit with different codes based on grade if report.letter_grade in ["A+", "A", "A-"]: sys.exit(0) # Excellent elif report.letter_grade in ["B+", "B", "B-"]: sys.exit(0) # Good elif report.letter_grade in ["C+", "C", "C-"]: sys.exit(0) # Acceptable elif report.letter_grade == "D": sys.exit(2) # Needs improvement else: # F sys.exit(1) # Poor quality except KeyboardInterrupt: print("\nQuality assessment interrupted by user", file=sys.stderr) sys.exit(130) except Exception as e: print(f"Quality assessment failed: {str(e)}", file=sys.stderr) if args.verbose: import traceback traceback.print_exc() sys.exit(1) if __name__ == "__main__": main() FILE:scripts/script_tester.py #!/usr/bin/env python3 """ Script Tester - Tests Python scripts in a skill directory This script validates and tests Python scripts within a skill directory by checking syntax, imports, runtime execution, argparse functionality, and output formats. It ensures scripts meet quality standards and function correctly. Usage: python script_tester.py [--timeout SECONDS] [--json] [--verbose] Author: Claude Skills Engineering Team Version: 1.0.0 Dependencies: Python Standard Library Only """ import argparse import ast import json import os import subprocess import sys import tempfile import time from datetime import datetime from pathlib import Path from typing import Dict, List, Any, Optional, Tuple, Union import threading class TestError(Exception): """Custom exception for testing errors""" pass class ScriptTestResult: """Container for individual script test results""" def __init__(self, script_path: str): self.script_path = script_path self.script_name = Path(script_path).name self.timestamp = datetime.utcnow().isoformat() + "Z" self.tests = {} self.overall_status = "PENDING" self.execution_time = 0.0 self.errors = [] self.warnings = [] def add_test(self, test_name: str, passed: bool, message: str = "", details: Dict = None): """Add a test result""" self.tests[test_name] = { "passed": passed, "message": message, "details": details or {} } def add_error(self, error: str): """Add an error message""" self.errors.append(error) def add_warning(self, warning: str): """Add a warning message""" self.warnings.append(warning) def calculate_status(self): """Calculate overall test status""" if not self.tests: self.overall_status = "NO_TESTS" return failed_tests = [name for name, result in self.tests.items() if not result["passed"]] if not failed_tests: self.overall_status = "PASS" elif len(failed_tests) <= len(self.tests) // 2: self.overall_status = "PARTIAL" else: self.overall_status = "FAIL" class TestSuite: """Container for all test results""" def __init__(self, skill_path: str): self.skill_path = skill_path self.timestamp = datetime.utcnow().isoformat() + "Z" self.script_results = {} self.summary = {} self.global_errors = [] def add_script_result(self, result: ScriptTestResult): """Add a script test result""" self.script_results[result.script_name] = result def add_global_error(self, error: str): """Add a global error message""" self.global_errors.append(error) def calculate_summary(self): """Calculate summary statistics""" if not self.script_results: self.summary = { "total_scripts": 0, "passed": 0, "partial": 0, "failed": 0, "overall_status": "NO_SCRIPTS" } return statuses = [result.overall_status for result in self.script_results.values()] self.summary = { "total_scripts": len(self.script_results), "passed": statuses.count("PASS"), "partial": statuses.count("PARTIAL"), "failed": statuses.count("FAIL"), "no_tests": statuses.count("NO_TESTS") } # Determine overall status if self.summary["failed"] == 0 and self.summary["no_tests"] == 0: self.summary["overall_status"] = "PASS" elif self.summary["passed"] > 0: self.summary["overall_status"] = "PARTIAL" else: self.summary["overall_status"] = "FAIL" class ScriptTester: """Main script testing engine""" def __init__(self, skill_path: str, timeout: int = 30, verbose: bool = False): self.skill_path = Path(skill_path).resolve() self.timeout = timeout self.verbose = verbose self.test_suite = TestSuite(str(self.skill_path)) def log_verbose(self, message: str): """Log verbose message if verbose mode enabled""" if self.verbose: print(f"[VERBOSE] {message}", file=sys.stderr) def test_all_scripts(self) -> TestSuite: """Main entry point - test all scripts in the skill""" try: self.log_verbose(f"Starting script testing for {self.skill_path}") # Check if skill path exists if not self.skill_path.exists(): self.test_suite.add_global_error(f"Skill path does not exist: {self.skill_path}") return self.test_suite scripts_dir = self.skill_path / "scripts" if not scripts_dir.exists(): self.test_suite.add_global_error("No scripts directory found") return self.test_suite # Find all Python scripts python_files = list(scripts_dir.glob("*.py")) if not python_files: self.test_suite.add_global_error("No Python scripts found in scripts directory") return self.test_suite self.log_verbose(f"Found {len(python_files)} Python scripts to test") # Test each script for script_path in python_files: try: result = self.test_single_script(script_path) self.test_suite.add_script_result(result) except Exception as e: # Create a failed result for the script result = ScriptTestResult(str(script_path)) result.add_error(f"Failed to test script: {str(e)}") result.overall_status = "FAIL" self.test_suite.add_script_result(result) # Calculate summary self.test_suite.calculate_summary() except Exception as e: self.test_suite.add_global_error(f"Testing failed with exception: {str(e)}") return self.test_suite def test_single_script(self, script_path: Path) -> ScriptTestResult: """Test a single Python script comprehensively""" result = ScriptTestResult(str(script_path)) start_time = time.time() try: self.log_verbose(f"Testing script: {script_path.name}") # Read script content try: content = script_path.read_text(encoding='utf-8') except Exception as e: result.add_test("file_readable", False, f"Cannot read file: {str(e)}") result.add_error(f"Cannot read script file: {str(e)}") result.overall_status = "FAIL" return result result.add_test("file_readable", True, "Script file is readable") # Test 1: Syntax validation self._test_syntax(content, result) # Test 2: Import validation self._test_imports(content, result) # Test 3: Argparse validation self._test_argparse_implementation(content, result) # Test 4: Main guard validation self._test_main_guard(content, result) # Test 5: Runtime execution tests if result.tests.get("syntax_valid", {}).get("passed", False): self._test_script_execution(script_path, result) # Test 6: Help functionality if result.tests.get("syntax_valid", {}).get("passed", False): self._test_help_functionality(script_path, result) # Test 7: Sample data processing (if available) self._test_sample_data_processing(script_path, result) # Test 8: Output format validation self._test_output_formats(script_path, result) except Exception as e: result.add_error(f"Unexpected error during testing: {str(e)}") finally: result.execution_time = time.time() - start_time result.calculate_status() return result def _test_syntax(self, content: str, result: ScriptTestResult): """Test Python syntax validity""" self.log_verbose("Testing syntax...") try: ast.parse(content) result.add_test("syntax_valid", True, "Python syntax is valid") except SyntaxError as e: result.add_test("syntax_valid", False, f"Syntax error: {str(e)}", {"error": str(e), "line": getattr(e, 'lineno', 'unknown')}) result.add_error(f"Syntax error: {str(e)}") def _test_imports(self, content: str, result: ScriptTestResult): """Test import statements for external dependencies""" self.log_verbose("Testing imports...") try: tree = ast.parse(content) external_imports = self._find_external_imports(tree) if not external_imports: result.add_test("imports_valid", True, "Uses only standard library imports") else: result.add_test("imports_valid", False, f"Uses external imports: {', '.join(external_imports)}", {"external_imports": external_imports}) result.add_error(f"External imports detected: {', '.join(external_imports)}") except Exception as e: result.add_test("imports_valid", False, f"Error analyzing imports: {str(e)}") def _find_external_imports(self, tree: ast.AST) -> List[str]: """Find external (non-stdlib) imports""" # Comprehensive standard library module list stdlib_modules = { # Built-in modules 'argparse', 'ast', 'json', 'os', 'sys', 'pathlib', 'datetime', 'typing', 'collections', 're', 'math', 'random', 'itertools', 'functools', 'operator', 'csv', 'sqlite3', 'urllib', 'http', 'html', 'xml', 'email', 'base64', 'hashlib', 'hmac', 'secrets', 'tempfile', 'shutil', 'glob', 'fnmatch', 'subprocess', 'threading', 'multiprocessing', 'queue', 'time', 'calendar', 'locale', 'gettext', 'logging', 'warnings', 'unittest', 'doctest', 'pickle', 'copy', 'pprint', 'reprlib', 'enum', 'dataclasses', 'contextlib', 'abc', 'atexit', 'traceback', 'gc', 'weakref', 'types', 'decimal', 'fractions', 'statistics', 'cmath', 'platform', 'errno', 'io', 'codecs', 'unicodedata', 'stringprep', 'textwrap', 'string', 'struct', 'difflib', 'heapq', 'bisect', 'array', 'uuid', 'mmap', 'ctypes', 'winreg', 'msvcrt', 'winsound', 'posix', 'pwd', 'grp', 'crypt', 'termios', 'tty', 'pty', 'fcntl', 'resource', 'nis', 'syslog', 'signal', 'socket', 'ssl', 'select', 'selectors', 'asyncio', 'asynchat', 'asyncore', 'netrc', 'xdrlib', 'plistlib', 'mailbox', 'mimetypes', 'encodings', 'pkgutil', 'modulefinder', 'runpy', 'importlib', 'imp', 'zipimport', 'zipfile', 'tarfile', 'gzip', 'bz2', 'lzma', 'zlib', 'binascii', 'quopri', 'uu', 'configparser', 'netrc', 'xdrlib', 'plistlib', 'token', 'tokenize', 'keyword', 'heapq', 'bisect', 'array', 'weakref', 'types', 'copyreg', 'shelve', 'marshal', 'dbm', 'sqlite3', 'zoneinfo' } external_imports = [] for node in ast.walk(tree): if isinstance(node, ast.Import): for alias in node.names: module_name = alias.name.split('.')[0] if module_name not in stdlib_modules and not module_name.startswith('_'): external_imports.append(alias.name) elif isinstance(node, ast.ImportFrom) and node.module: module_name = node.module.split('.')[0] if module_name not in stdlib_modules and not module_name.startswith('_'): external_imports.append(node.module) return list(set(external_imports)) def _test_argparse_implementation(self, content: str, result: ScriptTestResult): """Test argparse implementation""" self.log_verbose("Testing argparse implementation...") try: tree = ast.parse(content) # Check for argparse import has_argparse_import = False has_parser_creation = False has_parse_args = False for node in ast.walk(tree): if isinstance(node, (ast.Import, ast.ImportFrom)): if (isinstance(node, ast.Import) and any(alias.name == 'argparse' for alias in node.names)): has_argparse_import = True elif (isinstance(node, ast.ImportFrom) and node.module == 'argparse'): has_argparse_import = True elif isinstance(node, ast.Call): # Check for ArgumentParser creation if (isinstance(node.func, ast.Attribute) and isinstance(node.func.value, ast.Name) and node.func.value.id == 'argparse' and node.func.attr == 'ArgumentParser'): has_parser_creation = True # Check for parse_args call if (isinstance(node.func, ast.Attribute) and node.func.attr == 'parse_args'): has_parse_args = True argparse_score = sum([has_argparse_import, has_parser_creation, has_parse_args]) if argparse_score == 3: result.add_test("argparse_implementation", True, "Complete argparse implementation found") elif argparse_score > 0: result.add_test("argparse_implementation", False, "Partial argparse implementation", {"missing_components": [ comp for comp, present in [ ("import", has_argparse_import), ("parser_creation", has_parser_creation), ("parse_args", has_parse_args) ] if not present ]}) result.add_warning("Incomplete argparse implementation") else: result.add_test("argparse_implementation", False, "No argparse implementation found") result.add_error("Script should use argparse for command-line arguments") except Exception as e: result.add_test("argparse_implementation", False, f"Error analyzing argparse: {str(e)}") def _test_main_guard(self, content: str, result: ScriptTestResult): """Test for if __name__ == '__main__' guard""" self.log_verbose("Testing main guard...") has_main_guard = 'if __name__ == "__main__"' in content or "if __name__ == '__main__'" in content if has_main_guard: result.add_test("main_guard", True, "Has proper main guard") else: result.add_test("main_guard", False, "Missing main guard") result.add_error("Script should have 'if __name__ == \"__main__\"' guard") def _test_script_execution(self, script_path: Path, result: ScriptTestResult): """Test basic script execution""" self.log_verbose("Testing script execution...") try: # Try to run the script with no arguments (should not crash immediately) process = subprocess.run( [sys.executable, str(script_path)], capture_output=True, text=True, timeout=self.timeout, cwd=script_path.parent ) # Script might exit with error code if no args provided, but shouldn't crash if process.returncode in (0, 1, 2): # 0=success, 1=general error, 2=misuse result.add_test("basic_execution", True, f"Script runs without crashing (exit code: {process.returncode})") else: result.add_test("basic_execution", False, f"Script crashed with exit code {process.returncode}", {"stdout": process.stdout, "stderr": process.stderr}) except subprocess.TimeoutExpired: result.add_test("basic_execution", False, f"Script execution timed out after {self.timeout} seconds") result.add_error(f"Script execution timeout ({self.timeout}s)") except Exception as e: result.add_test("basic_execution", False, f"Execution error: {str(e)}") result.add_error(f"Script execution failed: {str(e)}") def _test_help_functionality(self, script_path: Path, result: ScriptTestResult): """Test --help functionality""" self.log_verbose("Testing help functionality...") try: # Test --help flag process = subprocess.run( [sys.executable, str(script_path), '--help'], capture_output=True, text=True, timeout=self.timeout, cwd=script_path.parent ) if process.returncode == 0: help_output = process.stdout # Check for reasonable help content help_indicators = ['usage:', 'positional arguments:', 'optional arguments:', 'options:', 'description:', 'help'] has_help_content = any(indicator in help_output.lower() for indicator in help_indicators) if has_help_content and len(help_output.strip()) > 50: result.add_test("help_functionality", True, "Provides comprehensive help text") else: result.add_test("help_functionality", False, "Help text is too brief or missing key sections", {"help_output": help_output}) result.add_warning("Help text could be more comprehensive") else: result.add_test("help_functionality", False, f"Help command failed with exit code {process.returncode}", {"stderr": process.stderr}) result.add_error("--help flag does not work properly") except subprocess.TimeoutExpired: result.add_test("help_functionality", False, "Help command timed out") except Exception as e: result.add_test("help_functionality", False, f"Help test error: {str(e)}") def _test_sample_data_processing(self, script_path: Path, result: ScriptTestResult): """Test script against sample data if available""" self.log_verbose("Testing sample data processing...") assets_dir = self.skill_path / "assets" if not assets_dir.exists(): result.add_test("sample_data_processing", True, "No sample data to test (assets dir missing)") return # Look for sample input files sample_files = list(assets_dir.rglob("*sample*")) + list(assets_dir.rglob("*test*")) sample_files = [f for f in sample_files if f.is_file() and not f.name.startswith('.')] if not sample_files: result.add_test("sample_data_processing", True, "No sample data files found to test") return tested_files = 0 successful_tests = 0 for sample_file in sample_files[:3]: # Test up to 3 sample files try: self.log_verbose(f"Testing with sample file: {sample_file.name}") # Try to run script with the sample file as input process = subprocess.run( [sys.executable, str(script_path), str(sample_file)], capture_output=True, text=True, timeout=self.timeout, cwd=script_path.parent ) tested_files += 1 if process.returncode == 0: successful_tests += 1 else: self.log_verbose(f"Sample test failed for {sample_file.name}: {process.stderr}") except subprocess.TimeoutExpired: tested_files += 1 result.add_warning(f"Sample data test timed out for {sample_file.name}") except Exception as e: tested_files += 1 self.log_verbose(f"Sample test error for {sample_file.name}: {str(e)}") if tested_files == 0: result.add_test("sample_data_processing", True, "No testable sample data found") elif successful_tests == tested_files: result.add_test("sample_data_processing", True, f"Successfully processed all {tested_files} sample files") elif successful_tests > 0: result.add_test("sample_data_processing", False, f"Processed {successful_tests}/{tested_files} sample files", {"success_rate": successful_tests / tested_files}) result.add_warning("Some sample data processing failed") else: result.add_test("sample_data_processing", False, "Failed to process any sample data files") result.add_error("Script cannot process sample data") def _test_output_formats(self, script_path: Path, result: ScriptTestResult): """Test output format compliance""" self.log_verbose("Testing output formats...") # Test if script supports JSON output json_support = False human_readable_support = False try: # Read script content to check for output format indicators content = script_path.read_text(encoding='utf-8') # Look for JSON-related code if any(indicator in content.lower() for indicator in ['json.dump', 'json.load', '"json"', '--json']): json_support = True # Look for human-readable output indicators if any(indicator in content for indicator in ['print(', 'format(', 'f"', "f'"]): human_readable_support = True # Try running with --json flag if it looks like it supports it if '--json' in content: try: process = subprocess.run( [sys.executable, str(script_path), '--json', '--help'], capture_output=True, text=True, timeout=10, cwd=script_path.parent ) if process.returncode == 0: json_support = True except: pass # Evaluate dual output support if json_support and human_readable_support: result.add_test("output_formats", True, "Supports both JSON and human-readable output") elif json_support or human_readable_support: format_type = "JSON" if json_support else "human-readable" result.add_test("output_formats", False, f"Supports only {format_type} output", {"json_support": json_support, "human_readable_support": human_readable_support}) result.add_warning("Consider adding dual output format support") else: result.add_test("output_formats", False, "No clear output format support detected") result.add_warning("Output format support is unclear") except Exception as e: result.add_test("output_formats", False, f"Error testing output formats: {str(e)}") class TestReportFormatter: """Formats test reports for output""" @staticmethod def format_json(test_suite: TestSuite) -> str: """Format test suite as JSON""" return json.dumps({ "skill_path": test_suite.skill_path, "timestamp": test_suite.timestamp, "summary": test_suite.summary, "global_errors": test_suite.global_errors, "script_results": { name: { "script_path": result.script_path, "timestamp": result.timestamp, "overall_status": result.overall_status, "execution_time": round(result.execution_time, 2), "tests": result.tests, "errors": result.errors, "warnings": result.warnings } for name, result in test_suite.script_results.items() } }, indent=2) @staticmethod def format_human_readable(test_suite: TestSuite) -> str: """Format test suite as human-readable text""" lines = [] lines.append("=" * 60) lines.append("SCRIPT TESTING REPORT") lines.append("=" * 60) lines.append(f"Skill: {test_suite.skill_path}") lines.append(f"Timestamp: {test_suite.timestamp}") lines.append("") # Summary if test_suite.summary: lines.append("SUMMARY:") lines.append(f" Total Scripts: {test_suite.summary['total_scripts']}") lines.append(f" Passed: {test_suite.summary['passed']}") lines.append(f" Partial: {test_suite.summary['partial']}") lines.append(f" Failed: {test_suite.summary['failed']}") lines.append(f" Overall Status: {test_suite.summary['overall_status']}") lines.append("") # Global errors if test_suite.global_errors: lines.append("GLOBAL ERRORS:") for error in test_suite.global_errors: lines.append(f" • {error}") lines.append("") # Individual script results for script_name, result in test_suite.script_results.items(): lines.append(f"SCRIPT: {script_name}") lines.append(f" Status: {result.overall_status}") lines.append(f" Execution Time: {result.execution_time:.2f}s") lines.append("") # Tests if result.tests: lines.append(" TESTS:") for test_name, test_result in result.tests.items(): status = "✓ PASS" if test_result["passed"] else "✗ FAIL" lines.append(f" {status}: {test_result['message']}") lines.append("") # Errors if result.errors: lines.append(" ERRORS:") for error in result.errors: lines.append(f" • {error}") lines.append("") # Warnings if result.warnings: lines.append(" WARNINGS:") for warning in result.warnings: lines.append(f" • {warning}") lines.append("") lines.append("-" * 40) lines.append("") return "\n".join(lines) def main(): """Main entry point""" parser = argparse.ArgumentParser( description="Test Python scripts in a skill directory", formatter_class=argparse.RawDescriptionHelpFormatter, epilog=""" Examples: python script_tester.py engineering/my-skill python script_tester.py engineering/my-skill --timeout 60 --json python script_tester.py engineering/my-skill --verbose Test Categories: - Syntax validation (AST parsing) - Import validation (stdlib only) - Argparse implementation - Main guard presence - Basic execution testing - Help functionality - Sample data processing - Output format compliance """ ) parser.add_argument("skill_path", help="Path to the skill directory containing scripts to test") parser.add_argument("--timeout", type=int, default=30, help="Timeout for script execution tests in seconds (default: 30)") parser.add_argument("--json", action="store_true", help="Output results in JSON format") parser.add_argument("--verbose", action="store_true", help="Enable verbose logging") args = parser.parse_args() try: # Create tester and run tests tester = ScriptTester(args.skill_path, args.timeout, args.verbose) test_suite = tester.test_all_scripts() # Format and output results if args.json: print(TestReportFormatter.format_json(test_suite)) else: print(TestReportFormatter.format_human_readable(test_suite)) # Exit with appropriate code if test_suite.global_errors: sys.exit(1) elif test_suite.summary.get("overall_status") == "FAIL": sys.exit(1) elif test_suite.summary.get("overall_status") == "PARTIAL": sys.exit(2) # Partial success else: sys.exit(0) # Success except KeyboardInterrupt: print("\nTesting interrupted by user", file=sys.stderr) sys.exit(130) except Exception as e: print(f"Testing failed: {str(e)}", file=sys.stderr) if args.verbose: import traceback traceback.print_exc() sys.exit(1) if __name__ == "__main__": main() FILE:scripts/skill_validator.py #!/usr/bin/env python3 """ Skill Validator - Validates skill directories against quality standards This script validates a skill directory structure, documentation, and Python scripts against the claude-skills ecosystem standards. It checks for required files, proper formatting, and compliance with tier-specific requirements. Usage: python skill_validator.py [--tier TIER] [--json] [--verbose] Author: Claude Skills Engineering Team Version: 1.0.0 Dependencies: Python Standard Library Only """ import argparse import ast import json import re import sys try: import yaml except ImportError: # Minimal YAML subset: parse simple key: value frontmatter without pyyaml class _YamlStub: class YAMLError(Exception): pass @staticmethod def safe_load(text): result = {} for line in text.strip().splitlines(): if ':' in line: key, _, value = line.partition(':') result[key.strip()] = value.strip() return result if result else None yaml = _YamlStub() import datetime as dt from pathlib import Path from typing import Dict, List, Any, Optional, Tuple class ValidationError(Exception): """Custom exception for validation errors""" pass class ValidationReport: """Container for validation results""" def __init__(self, skill_path: str): self.skill_path = skill_path self.timestamp = dt.datetime.now(dt.timezone.utc).isoformat().replace("+00:00", "Z") self.checks = {} self.warnings = [] self.errors = [] self.suggestions = [] self.overall_score = 0.0 self.compliance_level = "FAIL" def add_check(self, check_name: str, passed: bool, message: str = "", score: float = 0.0): """Add a validation check result""" self.checks[check_name] = { "passed": passed, "message": message, "score": score } def add_warning(self, message: str): """Add a warning message""" self.warnings.append(message) def add_error(self, message: str): """Add an error message""" self.errors.append(message) def add_suggestion(self, message: str): """Add an improvement suggestion""" self.suggestions.append(message) def calculate_overall_score(self): """Calculate overall compliance score""" if not self.checks: self.overall_score = 0.0 return total_score = sum(check["score"] for check in self.checks.values()) max_score = len(self.checks) * 1.0 self.overall_score = (total_score / max_score) * 100 if max_score > 0 else 0.0 # Determine compliance level if self.overall_score >= 90: self.compliance_level = "EXCELLENT" elif self.overall_score >= 75: self.compliance_level = "GOOD" elif self.overall_score >= 60: self.compliance_level = "ACCEPTABLE" elif self.overall_score >= 40: self.compliance_level = "NEEDS_IMPROVEMENT" else: self.compliance_level = "POOR" class SkillValidator: """Main skill validation engine""" # Tier requirements TIER_REQUIREMENTS = { "BASIC": { "min_skill_md_lines": 100, "min_scripts": 1, "script_size_range": (100, 300), "required_dirs": ["scripts"], "optional_dirs": ["assets", "references", "expected_outputs"], "features_required": ["argparse", "main_guard"] }, "STANDARD": { "min_skill_md_lines": 200, "min_scripts": 1, "script_size_range": (300, 500), "required_dirs": ["scripts", "assets", "references"], "optional_dirs": ["expected_outputs"], "features_required": ["argparse", "main_guard", "json_output", "help_text"] }, "POWERFUL": { "min_skill_md_lines": 300, "min_scripts": 2, "script_size_range": (500, 800), "required_dirs": ["scripts", "assets", "references", "expected_outputs"], "optional_dirs": [], "features_required": ["argparse", "main_guard", "json_output", "help_text", "error_handling"] } } REQUIRED_SKILL_MD_SECTIONS = [ "Name", "Description", "Features", "Usage", "Examples" ] FRONTMATTER_REQUIRED_FIELDS = [ "Name", "Tier", "Category", "Dependencies", "Author", "Version" ] def __init__(self, skill_path: str, target_tier: Optional[str] = None, verbose: bool = False): self.skill_path = Path(skill_path).resolve() self.target_tier = target_tier self.verbose = verbose self.report = ValidationReport(str(self.skill_path)) def log_verbose(self, message: str): """Log verbose message if verbose mode enabled""" if self.verbose: print(f"[VERBOSE] {message}", file=sys.stderr) def validate_skill_structure(self) -> ValidationReport: """Main validation entry point""" try: self.log_verbose(f"Starting validation of {self.skill_path}") # Check if path exists if not self.skill_path.exists(): self.report.add_error(f"Skill path does not exist: {self.skill_path}") return self.report if not self.skill_path.is_dir(): self.report.add_error(f"Skill path is not a directory: {self.skill_path}") return self.report # Run all validation checks self._validate_required_files() self._validate_skill_md() self._validate_readme() self._validate_directory_structure() self._validate_python_scripts() self._validate_tier_compliance() # Calculate overall score self.report.calculate_overall_score() self.log_verbose(f"Validation completed. Score: {self.report.overall_score:.1f}") except Exception as e: self.report.add_error(f"Validation failed with exception: {str(e)}") return self.report def _validate_required_files(self): """Validate presence of required files""" self.log_verbose("Checking required files...") # Check SKILL.md skill_md_path = self.skill_path / "SKILL.md" if skill_md_path.exists(): self.report.add_check("skill_md_exists", True, "SKILL.md found", 1.0) else: self.report.add_check("skill_md_exists", False, "SKILL.md missing", 0.0) self.report.add_error("SKILL.md is required but missing") # Check README.md readme_path = self.skill_path / "README.md" if readme_path.exists(): self.report.add_check("readme_exists", True, "README.md found", 1.0) else: self.report.add_check("readme_exists", False, "README.md missing", 0.0) self.report.add_warning("README.md is recommended but missing") self.report.add_suggestion("Add README.md with usage instructions and examples") def _validate_skill_md(self): """Validate SKILL.md content and format""" self.log_verbose("Validating SKILL.md...") skill_md_path = self.skill_path / "SKILL.md" if not skill_md_path.exists(): return try: content = skill_md_path.read_text(encoding='utf-8') lines = content.split('\n') line_count = len([line for line in lines if line.strip()]) # Check line count min_lines = self._get_tier_requirement("min_skill_md_lines", 100) if line_count >= min_lines: self.report.add_check("skill_md_length", True, f"SKILL.md has {line_count} lines (≥{min_lines})", 1.0) else: self.report.add_check("skill_md_length", False, f"SKILL.md has {line_count} lines (<{min_lines})", 0.0) self.report.add_error(f"SKILL.md too short: {line_count} lines, minimum {min_lines}") # Validate frontmatter self._validate_frontmatter(content) # Check required sections self._validate_required_sections(content) except Exception as e: self.report.add_check("skill_md_readable", False, f"Error reading SKILL.md: {str(e)}", 0.0) self.report.add_error(f"Cannot read SKILL.md: {str(e)}") def _validate_frontmatter(self, content: str): """Validate SKILL.md frontmatter""" self.log_verbose("Validating frontmatter...") # Extract frontmatter if content.startswith('---'): try: end_marker = content.find('---', 3) if end_marker == -1: self.report.add_check("frontmatter_format", False, "Frontmatter closing marker not found", 0.0) return frontmatter_text = content[3:end_marker].strip() frontmatter = yaml.safe_load(frontmatter_text) if not isinstance(frontmatter, dict): self.report.add_check("frontmatter_format", False, "Frontmatter is not a valid dictionary", 0.0) return # Check required fields missing_fields = [] for field in self.FRONTMATTER_REQUIRED_FIELDS: if field not in frontmatter: missing_fields.append(field) if not missing_fields: self.report.add_check("frontmatter_complete", True, "All required frontmatter fields present", 1.0) else: self.report.add_check("frontmatter_complete", False, f"Missing fields: {', '.join(missing_fields)}", 0.0) self.report.add_error(f"Missing frontmatter fields: {', '.join(missing_fields)}") except yaml.YAMLError as e: self.report.add_check("frontmatter_format", False, f"Invalid YAML frontmatter: {str(e)}", 0.0) self.report.add_error(f"Invalid YAML frontmatter: {str(e)}") else: self.report.add_check("frontmatter_exists", False, "No frontmatter found", 0.0) self.report.add_error("SKILL.md must start with YAML frontmatter") def _validate_required_sections(self, content: str): """Validate required sections in SKILL.md""" self.log_verbose("Checking required sections...") missing_sections = [] for section in self.REQUIRED_SKILL_MD_SECTIONS: pattern = rf'^#+\s*{re.escape(section)}\s*$' if not re.search(pattern, content, re.MULTILINE | re.IGNORECASE): missing_sections.append(section) if not missing_sections: self.report.add_check("required_sections", True, "All required sections present", 1.0) else: self.report.add_check("required_sections", False, f"Missing sections: {', '.join(missing_sections)}", 0.0) self.report.add_error(f"Missing required sections: {', '.join(missing_sections)}") def _validate_readme(self): """Validate README.md content""" self.log_verbose("Validating README.md...") readme_path = self.skill_path / "README.md" if not readme_path.exists(): return try: content = readme_path.read_text(encoding='utf-8') # Check minimum content length if len(content.strip()) >= 200: self.report.add_check("readme_substantial", True, "README.md has substantial content", 1.0) else: self.report.add_check("readme_substantial", False, "README.md content is too brief", 0.5) self.report.add_suggestion("Expand README.md with more detailed usage instructions") except Exception as e: self.report.add_check("readme_readable", False, f"Error reading README.md: {str(e)}", 0.0) def _validate_directory_structure(self): """Validate directory structure against tier requirements""" self.log_verbose("Validating directory structure...") required_dirs = self._get_tier_requirement("required_dirs", ["scripts"]) optional_dirs = self._get_tier_requirement("optional_dirs", []) # Check required directories missing_required = [] for dir_name in required_dirs: dir_path = self.skill_path / dir_name if dir_path.exists() and dir_path.is_dir(): self.report.add_check(f"dir_{dir_name}_exists", True, f"{dir_name}/ directory found", 1.0) else: missing_required.append(dir_name) self.report.add_check(f"dir_{dir_name}_exists", False, f"{dir_name}/ directory missing", 0.0) if missing_required: self.report.add_error(f"Missing required directories: {', '.join(missing_required)}") # Check optional directories and provide suggestions missing_optional = [] for dir_name in optional_dirs: dir_path = self.skill_path / dir_name if not (dir_path.exists() and dir_path.is_dir()): missing_optional.append(dir_name) if missing_optional: self.report.add_suggestion(f"Consider adding optional directories: {', '.join(missing_optional)}") def _validate_python_scripts(self): """Validate Python scripts in the scripts directory""" self.log_verbose("Validating Python scripts...") scripts_dir = self.skill_path / "scripts" if not scripts_dir.exists(): return python_files = list(scripts_dir.glob("*.py")) min_scripts = self._get_tier_requirement("min_scripts", 1) # Check minimum number of scripts if len(python_files) >= min_scripts: self.report.add_check("min_scripts_count", True, f"Found {len(python_files)} Python scripts (≥{min_scripts})", 1.0) else: self.report.add_check("min_scripts_count", False, f"Found {len(python_files)} Python scripts (<{min_scripts})", 0.0) self.report.add_error(f"Insufficient scripts: {len(python_files)}, minimum {min_scripts}") # Validate each script for script_path in python_files: self._validate_single_script(script_path) def _validate_single_script(self, script_path: Path): """Validate a single Python script""" script_name = script_path.name self.log_verbose(f"Validating script: {script_name}") try: content = script_path.read_text(encoding='utf-8') # Count lines of code (excluding empty lines and comments) lines = content.split('\n') loc = len([line for line in lines if line.strip() and not line.strip().startswith('#')]) # Check script size against tier requirements size_range = self._get_tier_requirement("script_size_range", (100, 1000)) min_size, max_size = size_range if min_size <= loc <= max_size: self.report.add_check(f"script_size_{script_name}", True, f"{script_name} has {loc} LOC (within {min_size}-{max_size})", 1.0) else: self.report.add_check(f"script_size_{script_name}", False, f"{script_name} has {loc} LOC (outside {min_size}-{max_size})", 0.5) if loc < min_size: self.report.add_suggestion(f"Consider expanding {script_name} (currently {loc} LOC)") else: self.report.add_suggestion(f"Consider refactoring {script_name} (currently {loc} LOC)") # Parse and validate Python syntax try: tree = ast.parse(content) self.report.add_check(f"script_syntax_{script_name}", True, f"{script_name} has valid Python syntax", 1.0) # Check for required features self._validate_script_features(tree, script_name, content) except SyntaxError as e: self.report.add_check(f"script_syntax_{script_name}", False, f"{script_name} has syntax error: {str(e)}", 0.0) self.report.add_error(f"Syntax error in {script_name}: {str(e)}") except Exception as e: self.report.add_check(f"script_readable_{script_name}", False, f"Cannot read {script_name}: {str(e)}", 0.0) self.report.add_error(f"Cannot read {script_name}: {str(e)}") def _validate_script_features(self, tree: ast.AST, script_name: str, content: str): """Validate required script features""" required_features = self._get_tier_requirement("features_required", ["argparse", "main_guard"]) # Check for argparse usage if "argparse" in required_features: has_argparse = self._check_argparse_usage(tree) self.report.add_check(f"script_argparse_{script_name}", has_argparse, f"{'Uses' if has_argparse else 'Missing'} argparse in {script_name}", 1.0 if has_argparse else 0.0) if not has_argparse: self.report.add_error(f"{script_name} must use argparse for command-line arguments") # Check for main guard if "main_guard" in required_features: has_main_guard = 'if __name__ == "__main__"' in content self.report.add_check(f"script_main_guard_{script_name}", has_main_guard, f"{'Has' if has_main_guard else 'Missing'} main guard in {script_name}", 1.0 if has_main_guard else 0.0) if not has_main_guard: self.report.add_error(f"{script_name} must have 'if __name__ == \"__main__\"' guard") # Check for external imports (should only use stdlib) external_imports = self._check_external_imports(tree) if not external_imports: self.report.add_check(f"script_imports_{script_name}", True, f"{script_name} uses only standard library", 1.0) else: self.report.add_check(f"script_imports_{script_name}", False, f"{script_name} uses external imports: {', '.join(external_imports)}", 0.0) self.report.add_error(f"{script_name} uses external imports: {', '.join(external_imports)}") def _check_argparse_usage(self, tree: ast.AST) -> bool: """Check if the script uses argparse""" for node in ast.walk(tree): if isinstance(node, ast.Import): for alias in node.names: if alias.name == 'argparse': return True elif isinstance(node, ast.ImportFrom): if node.module == 'argparse': return True return False def _check_external_imports(self, tree: ast.AST) -> List[str]: """Check for external (non-stdlib) imports""" # Simplified check - a more comprehensive solution would use a stdlib module list stdlib_modules = { 'argparse', 'ast', 'json', 'os', 'sys', 'pathlib', 'datetime', 'typing', 'collections', 're', 'math', 'random', 'itertools', 'functools', 'operator', 'csv', 'sqlite3', 'urllib', 'http', 'html', 'xml', 'email', 'base64', 'hashlib', 'hmac', 'secrets', 'tempfile', 'shutil', 'glob', 'fnmatch', 'subprocess', 'threading', 'multiprocessing', 'queue', 'time', 'calendar', 'zoneinfo', 'locale', 'gettext', 'logging', 'warnings', 'unittest', 'doctest', 'pickle', 'copy', 'pprint', 'reprlib', 'enum', 'dataclasses', 'contextlib', 'abc', 'atexit', 'traceback', 'gc', 'weakref', 'types', 'copy', 'pprint', 'reprlib', 'enum', 'decimal', 'fractions', 'statistics', 'cmath', 'platform', 'errno', 'io', 'codecs', 'unicodedata', 'stringprep', 'textwrap', 'string', 'struct', 'difflib', 'heapq', 'bisect', 'array', 'weakref', 'types', 'copyreg', 'uuid', 'mmap', 'ctypes' } external_imports = [] for node in ast.walk(tree): if isinstance(node, ast.Import): for alias in node.names: module_name = alias.name.split('.')[0] if module_name not in stdlib_modules: external_imports.append(alias.name) elif isinstance(node, ast.ImportFrom) and node.module: module_name = node.module.split('.')[0] if module_name not in stdlib_modules: external_imports.append(node.module) return list(set(external_imports)) def _validate_tier_compliance(self): """Validate overall tier compliance""" if not self.target_tier: return self.log_verbose(f"Validating {self.target_tier} tier compliance...") # This is a summary check - individual checks are done in other methods critical_checks = ["skill_md_exists", "min_scripts_count", "skill_md_length"] failed_critical = [check for check in critical_checks if check in self.report.checks and not self.report.checks[check]["passed"]] if not failed_critical: self.report.add_check("tier_compliance", True, f"Meets {self.target_tier} tier requirements", 1.0) else: self.report.add_check("tier_compliance", False, f"Does not meet {self.target_tier} tier requirements", 0.0) self.report.add_error(f"Failed critical checks for {self.target_tier} tier: {', '.join(failed_critical)}") def _get_tier_requirement(self, requirement: str, default: Any) -> Any: """Get tier-specific requirement value""" if self.target_tier and self.target_tier in self.TIER_REQUIREMENTS: return self.TIER_REQUIREMENTS[self.target_tier].get(requirement, default) return default class ReportFormatter: """Formats validation reports for output""" @staticmethod def format_json(report: ValidationReport) -> str: """Format report as JSON""" return json.dumps({ "skill_path": report.skill_path, "timestamp": report.timestamp, "overall_score": round(report.overall_score, 1), "compliance_level": report.compliance_level, "checks": report.checks, "warnings": report.warnings, "errors": report.errors, "suggestions": report.suggestions }, indent=2) @staticmethod def format_human_readable(report: ValidationReport) -> str: """Format report as human-readable text""" lines = [] lines.append("=" * 60) lines.append("SKILL VALIDATION REPORT") lines.append("=" * 60) lines.append(f"Skill: {report.skill_path}") lines.append(f"Timestamp: {report.timestamp}") lines.append(f"Overall Score: {report.overall_score:.1f}/100 ({report.compliance_level})") lines.append("") # Group checks by category structure_checks = {k: v for k, v in report.checks.items() if k.startswith(('skill_md', 'readme', 'dir_'))} script_checks = {k: v for k, v in report.checks.items() if k.startswith('script_')} other_checks = {k: v for k, v in report.checks.items() if k not in structure_checks and k not in script_checks} if structure_checks: lines.append("STRUCTURE VALIDATION:") for check_name, result in structure_checks.items(): status = "✓ PASS" if result["passed"] else "✗ FAIL" lines.append(f" {status}: {result['message']}") lines.append("") if script_checks: lines.append("SCRIPT VALIDATION:") for check_name, result in script_checks.items(): status = "✓ PASS" if result["passed"] else "✗ FAIL" lines.append(f" {status}: {result['message']}") lines.append("") if other_checks: lines.append("OTHER CHECKS:") for check_name, result in other_checks.items(): status = "✓ PASS" if result["passed"] else "✗ FAIL" lines.append(f" {status}: {result['message']}") lines.append("") if report.errors: lines.append("ERRORS:") for error in report.errors: lines.append(f" • {error}") lines.append("") if report.warnings: lines.append("WARNINGS:") for warning in report.warnings: lines.append(f" • {warning}") lines.append("") if report.suggestions: lines.append("SUGGESTIONS:") for suggestion in report.suggestions: lines.append(f" • {suggestion}") lines.append("") return "\n".join(lines) def main(): """Main entry point""" parser = argparse.ArgumentParser( description="Validate skill directories against quality standards", formatter_class=argparse.RawDescriptionHelpFormatter, epilog=""" Examples: python skill_validator.py engineering/my-skill python skill_validator.py engineering/my-skill --tier POWERFUL --json python skill_validator.py engineering/my-skill --verbose Tier Options: BASIC - Basic skill requirements (100+ lines SKILL.md, 1+ script) STANDARD - Standard skill requirements (200+ lines, advanced features) POWERFUL - Powerful skill requirements (300+ lines, comprehensive features) """ ) parser.add_argument("skill_path", help="Path to the skill directory to validate") parser.add_argument("--tier", choices=["BASIC", "STANDARD", "POWERFUL"], help="Target tier for validation (optional)") parser.add_argument("--json", action="store_true", help="Output results in JSON format") parser.add_argument("--verbose", action="store_true", help="Enable verbose logging") args = parser.parse_args() try: # Create validator and run validation validator = SkillValidator(args.skill_path, args.tier, args.verbose) report = validator.validate_skill_structure() # Format and output report if args.json: print(ReportFormatter.format_json(report)) else: print(ReportFormatter.format_human_readable(report)) # Exit with error code if validation failed if report.errors or report.overall_score < 60: sys.exit(1) else: sys.exit(0) except KeyboardInterrupt: print("\nValidation interrupted by user", file=sys.stderr) sys.exit(130) except Exception as e: print(f"Validation failed: {str(e)}", file=sys.stderr) if args.verbose: import traceback traceback.print_exc() sys.exit(1) if __name__ == "__main__": main()

Skill Tester

Comments (0)