CI-templates/TESTING.md

CI Template Testing Guide

This document explains how to test and validate the CI workflow templates in this repository.

Overview

The testing infrastructure includes:

• Automated validation workflow (.gitea/workflows/validate-templates.yml)
• Manual validation script (scripts/validate-templates.sh)
• Test projects (test-projects/)
• Matrix testing across multiple configurations

Quick Start

1. Run Local Validation

# Make the script executable
chmod +x scripts/validate-templates.sh

# Validate all workflows
./scripts/validate-templates.sh

# List available templates
./scripts/validate-templates.sh list

# Show help
./scripts/validate-templates.sh help

2. Test with Sample Projects

# Navigate to a test project
cd test-projects/python-basic

# Install dependencies (if using UV)
uv sync --all-extras

# Run local tests
uv run pytest
uv run mypy .
uv run ruff check .

Testing Infrastructure

1. Validation Workflow (.gitea/workflows/validate-templates.yml)

Automatically runs on every push and pull request:

• Matrix testing across Python versions (3.10, 3.11, 3.12)
• Template validation for all workflow files
• Integration testing with sample projects
• Documentation checks

Triggered by:

• Push to main branch
• Pull requests
• Manual dispatch
• Daily schedule (2 AM UTC)

Test matrix:

strategy:
  matrix:
    python-version: ["3.10", "3.11", "3.12"]
    template: ["code-quality", "pytest"]
    include:
      - template: "formatting"
        python-version: "3.12"  # Only test once for web template

2. Validation Script (scripts/validate-templates.sh)

Features:

• YAML syntax validation with yamllint
• Workflow structure validation with yq
• Reusable workflow detection
• Input parameter validation
• Documentation checks
• Common issue detection

Usage:

# Basic validation
./scripts/validate-templates.sh

# List all templates
./scripts/validate-templates.sh list

# Help
./scripts/validate-templates.sh help

Dependencies:

• yamllint (optional): pip install yamllint
• yq (optional): brew install yq (macOS) or apt install yq (Ubuntu)

Note: The script works without dependencies but provides more detailed validation when they're available.

3. Test Projects

Python Basic (test-projects/python-basic/)

Purpose: Test Python-specific templates

Structure:

python-basic/
├── .gitea/workflows/
│   └── test-templates.yml      # Tests both code-quality and pytest templates
├── src/test_package/
│   ├── __init__.py
│   ├── calculator.py           # Sample module with type hints
│   └── utils.py                # Utility functions
├── tests/
│   ├── test_calculator.py      # Pytest tests
│   └── test_utils.py           # More tests
├── pyproject.toml              # UV-compatible project config
└── README.md

Features:

• Type hints for mypy testing
• Pytest with coverage
• Ruff-compatible code style
• UV package management

Web Basic (test-projects/web-basic/)

Purpose: Test web formatting template

Structure:

web-basic/
├── .gitea/workflows/
│   └── test-formatting.yml     # Tests formatting template
├── package.json                # Node.js project
├── index.js                    # JavaScript file
├── config.yml                  # YAML file for validation
├── .prettierrc.json           # Prettier configuration
└── README.md

Features:

• Prettier configuration
• YAML validation
• JSON validation
• Node.js setup

Testing Workflow Templates

1. Local Testing with Act

Install and use Act to test workflows locally:

# Install Act
brew install act  # macOS
# or
curl https://raw.githubusercontent.com/nektos/act/master/install.sh | sudo bash

# Test a specific workflow
act -W .gitea/workflows/python/code-quality.yml

# Test with inputs
act workflow_call -W .gitea/workflows/python/code-quality.yml \
  --input python-version=3.11 \
  --input working-directory=./test-projects/python-basic

2. Branch Testing

Test changes on feature branches:

# Create feature branch
git checkout -b feature/improve-python-template

# Make changes to templates
# ...

# Update test project to use feature branch
# In test-projects/python-basic/.gitea/workflows/test-templates.yml:
# uses: gitea.gt-proj.com/brian/CI-templates/.gitea/workflows/python/code-quality.yml@feature/improve-python-template

# Push and test
git push origin feature/improve-python-template

3. Matrix Testing

Test multiple configurations:

# Example matrix test
strategy:
  matrix:
    python-version: ["3.10", "3.11", "3.12"]
    os: ["ubuntu-latest", "ubuntu-20.04"]
    working-directory: [".", "./src"]

Validation Checklist

Workflow File Validation

• Valid YAML syntax
• Contains required fields: name, on, jobs
• Uses workflow_call trigger for reusable workflows
• All inputs have descriptions and types
• No hardcoded values that should be inputs
• Proper defaults for optional inputs

Template Testing

• All input combinations work correctly
• Error handling for invalid inputs
• Backwards compatibility maintained
• Documentation updated
• Test projects updated

Integration Testing

• Works with real projects
• Performance is acceptable
• No conflicts between templates
• Secrets handling works correctly

Common Issues and Solutions

1. YAML Syntax Errors

Problem: Invalid YAML syntax

Error: YAML syntax validation failed

Solution:

• Use yamllint to check syntax
• Validate indentation (use spaces, not tabs)
• Check for missing quotes around strings with special characters

2. Missing Required Fields

Problem: Workflow missing required fields

Error: Missing required field 'on' in workflow.yml

Solution:

• Ensure all workflows have name, on, and jobs fields
• For reusable workflows, use workflow_call trigger

3. Input Validation Issues

Problem: Missing input descriptions or types

Warning: Input 'python-version' missing description

Solution:

inputs:
  python-version:
    description: 'Python version to use (e.g., "3.12")'
    required: false
    type: string
    default: "3.12"

4. Test Project Issues

Problem: Tests fail in sample projects

Error: Module not found

Solutions:

• Check pyproject.toml configuration
• Verify package structure
• Ensure dependencies are correctly specified
• Check import paths

CI/CD Best Practices

1. Version Control

• Use semantic versioning for releases
• Tag stable versions: v1.0.0, v1.1.0, etc.
• Reference specific versions in production: @v1.0.0
• Use @main for development/testing

2. Backwards Compatibility

• Test with previous versions
• Provide migration guides for breaking changes
• Deprecate features gracefully
• Maintain changelog

3. Security

• Validate all inputs
• Use secrets for sensitive data
• Limit permissions to minimum required
• Regular security updates

4. Documentation

• Keep README files updated
• Document all input parameters
• Provide usage examples
• Include troubleshooting guides

Troubleshooting

Validation Script Issues

# Check dependencies
./scripts/validate-templates.sh

# Manual YAML validation
yamllint .gitea/workflows/python/code-quality.yml

# Manual structure check
yq eval '.on | has("workflow_call")' .gitea/workflows/python/code-quality.yml

Test Project Issues

# Python projects
cd test-projects/python-basic
uv sync --all-extras
uv run pytest --verbose

# Web projects  
cd test-projects/web-basic
npm install
npm run format:check

Workflow Issues

# Check workflow logs in Gitea
# Enable debug logging:
env:
  ACTIONS_STEP_DEBUG: true
  ACTIONS_RUNNER_DEBUG: true

Contributing

When adding new templates or making changes:

1. Create/update test projects to validate the changes
2. Run validation script locally before committing
3. Update documentation as needed
4. Test with feature branches before merging
5. Tag releases for stable versions

Resources

• Gitea Actions Documentation
• GitHub Actions Syntax
• Act - Local Testing Tool
• yamllint Documentation
• yq Documentation