389 lines
8.6 KiB
Markdown
389 lines
8.6 KiB
Markdown
# CI Template Testing Guide
|
|
|
|
This document explains how to test and validate the CI workflow templates in this repository.
|
|
|
|
## Overview
|
|
|
|
The testing infrastructure includes:
|
|
|
|
- **Automated validation workflow** (`.gitea/workflows/validate-templates.yml`)
|
|
- **Manual validation script** (`scripts/validate-templates.sh`)
|
|
- **Test projects** (`test-projects/`)
|
|
- **Matrix testing** across multiple configurations
|
|
|
|
## Quick Start
|
|
|
|
### 1. Run Local Validation
|
|
|
|
```bash
|
|
# Make the script executable
|
|
chmod +x scripts/validate-templates.sh
|
|
|
|
# Validate all workflows
|
|
./scripts/validate-templates.sh
|
|
|
|
# List available templates
|
|
./scripts/validate-templates.sh list
|
|
|
|
# Show help
|
|
./scripts/validate-templates.sh help
|
|
```
|
|
|
|
### 2. Test with Sample Projects
|
|
|
|
```bash
|
|
# Navigate to a test project
|
|
cd test-projects/python-basic
|
|
|
|
# Install dependencies (if using UV)
|
|
uv sync --all-extras
|
|
|
|
# Run local tests
|
|
uv run pytest
|
|
uv run mypy .
|
|
uv run ruff check .
|
|
```
|
|
|
|
## Testing Infrastructure
|
|
|
|
### 1. Validation Workflow (`.gitea/workflows/validate-templates.yml`)
|
|
|
|
Automatically runs on every push and pull request:
|
|
|
|
- **Matrix testing** across Python versions (3.10, 3.11, 3.12)
|
|
- **Template validation** for all workflow files
|
|
- **Integration testing** with sample projects
|
|
- **Documentation checks**
|
|
|
|
**Triggered by:**
|
|
|
|
- Push to `main` branch
|
|
- Pull requests
|
|
- Manual dispatch
|
|
- Daily schedule (2 AM UTC)
|
|
|
|
**Test matrix:**
|
|
|
|
```yaml
|
|
strategy:
|
|
matrix:
|
|
python-version: ["3.10", "3.11", "3.12"]
|
|
template: ["code-quality", "pytest"]
|
|
include:
|
|
- template: "formatting"
|
|
python-version: "3.12" # Only test once for web template
|
|
```
|
|
|
|
### 2. Validation Script (`scripts/validate-templates.sh`)
|
|
|
|
**Features:**
|
|
|
|
- YAML syntax validation with `yamllint`
|
|
- Workflow structure validation with `yq`
|
|
- Reusable workflow detection
|
|
- Input parameter validation
|
|
- Documentation checks
|
|
- Common issue detection
|
|
|
|
**Usage:**
|
|
|
|
```bash
|
|
# Basic validation
|
|
./scripts/validate-templates.sh
|
|
|
|
# List all templates
|
|
./scripts/validate-templates.sh list
|
|
|
|
# Help
|
|
./scripts/validate-templates.sh help
|
|
```
|
|
|
|
**Dependencies:**
|
|
|
|
- `yamllint` (optional): `pip install yamllint`
|
|
- `yq` (optional): `brew install yq` (macOS) or `apt install yq` (Ubuntu)
|
|
|
|
**Note:** The script works without dependencies but provides more detailed validation when they're available.
|
|
|
|
### 3. Test Projects
|
|
|
|
#### Python Basic (`test-projects/python-basic/`)
|
|
|
|
**Purpose:** Test Python-specific templates
|
|
|
|
**Structure:**
|
|
|
|
```text
|
|
python-basic/
|
|
├── .gitea/workflows/
|
|
│ └── test-templates.yml # Tests both code-quality and pytest templates
|
|
├── src/test_package/
|
|
│ ├── __init__.py
|
|
│ ├── calculator.py # Sample module with type hints
|
|
│ └── utils.py # Utility functions
|
|
├── tests/
|
|
│ ├── test_calculator.py # Pytest tests
|
|
│ └── test_utils.py # More tests
|
|
├── pyproject.toml # UV-compatible project config
|
|
└── README.md
|
|
```
|
|
|
|
**Features:**
|
|
|
|
- Type hints for mypy testing
|
|
- Pytest with coverage
|
|
- Ruff-compatible code style
|
|
- UV package management
|
|
|
|
#### Web Basic (`test-projects/web-basic/`)
|
|
|
|
**Purpose:** Test web formatting template
|
|
|
|
**Structure:**
|
|
|
|
```text
|
|
web-basic/
|
|
├── .gitea/workflows/
|
|
│ └── test-formatting.yml # Tests formatting template
|
|
├── package.json # Node.js project
|
|
├── index.js # JavaScript file
|
|
├── config.yml # YAML file for validation
|
|
├── .prettierrc.json # Prettier configuration
|
|
└── README.md
|
|
```
|
|
|
|
**Features:**
|
|
|
|
- Prettier configuration
|
|
- YAML validation
|
|
- JSON validation
|
|
- Node.js setup
|
|
|
|
## Testing Workflow Templates
|
|
|
|
### 1. Local Testing with Act
|
|
|
|
Install and use Act to test workflows locally:
|
|
|
|
```bash
|
|
# Install Act
|
|
brew install act # macOS
|
|
# or
|
|
curl https://raw.githubusercontent.com/nektos/act/master/install.sh | sudo bash
|
|
|
|
# Test a specific workflow
|
|
act -W .gitea/workflows/python/code-quality.yml
|
|
|
|
# Test with inputs
|
|
act workflow_call -W .gitea/workflows/python/code-quality.yml \
|
|
--input python-version=3.11 \
|
|
--input working-directory=./test-projects/python-basic
|
|
```
|
|
|
|
### 2. Branch Testing
|
|
|
|
Test changes on feature branches:
|
|
|
|
```bash
|
|
# Create feature branch
|
|
git checkout -b feature/improve-python-template
|
|
|
|
# Make changes to templates
|
|
# ...
|
|
|
|
# Update test project to use feature branch
|
|
# In test-projects/python-basic/.gitea/workflows/test-templates.yml:
|
|
# uses: gitea.gt-proj.com/brian/CI-templates/.gitea/workflows/python/code-quality.yml@feature/improve-python-template
|
|
|
|
# Push and test
|
|
git push origin feature/improve-python-template
|
|
```
|
|
|
|
### 3. Matrix Testing
|
|
|
|
Test multiple configurations:
|
|
|
|
```yaml
|
|
# Example matrix test
|
|
strategy:
|
|
matrix:
|
|
python-version: ["3.10", "3.11", "3.12"]
|
|
os: ["ubuntu-latest", "ubuntu-20.04"]
|
|
working-directory: [".", "./src"]
|
|
```
|
|
|
|
## Validation Checklist
|
|
|
|
### Workflow File Validation
|
|
|
|
- [ ] Valid YAML syntax
|
|
- [ ] Contains required fields: `name`, `on`, `jobs`
|
|
- [ ] Uses `workflow_call` trigger for reusable workflows
|
|
- [ ] All inputs have descriptions and types
|
|
- [ ] No hardcoded values that should be inputs
|
|
- [ ] Proper defaults for optional inputs
|
|
|
|
### Template Testing
|
|
|
|
- [ ] All input combinations work correctly
|
|
- [ ] Error handling for invalid inputs
|
|
- [ ] Backwards compatibility maintained
|
|
- [ ] Documentation updated
|
|
- [ ] Test projects updated
|
|
|
|
### Integration Testing
|
|
|
|
- [ ] Works with real projects
|
|
- [ ] Performance is acceptable
|
|
- [ ] No conflicts between templates
|
|
- [ ] Secrets handling works correctly
|
|
|
|
## Common Issues and Solutions
|
|
|
|
### 1. YAML Syntax Errors
|
|
|
|
**Problem:** Invalid YAML syntax
|
|
|
|
```bash
|
|
Error: YAML syntax validation failed
|
|
```
|
|
|
|
**Solution:**
|
|
|
|
- Use `yamllint` to check syntax
|
|
- Validate indentation (use spaces, not tabs)
|
|
- Check for missing quotes around strings with special characters
|
|
|
|
### 2. Missing Required Fields
|
|
|
|
**Problem:** Workflow missing required fields
|
|
|
|
```bash
|
|
Error: Missing required field 'on' in workflow.yml
|
|
```
|
|
|
|
**Solution:**
|
|
|
|
- Ensure all workflows have `name`, `on`, and `jobs` fields
|
|
- For reusable workflows, use `workflow_call` trigger
|
|
|
|
### 3. Input Validation Issues
|
|
|
|
**Problem:** Missing input descriptions or types
|
|
|
|
```bash
|
|
Warning: Input 'python-version' missing description
|
|
```
|
|
|
|
**Solution:**
|
|
|
|
```yaml
|
|
inputs:
|
|
python-version:
|
|
description: 'Python version to use (e.g., "3.12")'
|
|
required: false
|
|
type: string
|
|
default: "3.12"
|
|
```
|
|
|
|
### 4. Test Project Issues
|
|
|
|
**Problem:** Tests fail in sample projects
|
|
|
|
```bash
|
|
Error: Module not found
|
|
```
|
|
|
|
**Solutions:**
|
|
|
|
- Check `pyproject.toml` configuration
|
|
- Verify package structure
|
|
- Ensure dependencies are correctly specified
|
|
- Check import paths
|
|
|
|
## CI/CD Best Practices
|
|
|
|
### 1. Version Control
|
|
|
|
- Use semantic versioning for releases
|
|
- Tag stable versions: `v1.0.0`, `v1.1.0`, etc.
|
|
- Reference specific versions in production: `@v1.0.0`
|
|
- Use `@main` for development/testing
|
|
|
|
### 2. Backwards Compatibility
|
|
|
|
- Test with previous versions
|
|
- Provide migration guides for breaking changes
|
|
- Deprecate features gracefully
|
|
- Maintain changelog
|
|
|
|
### 3. Security
|
|
|
|
- Validate all inputs
|
|
- Use secrets for sensitive data
|
|
- Limit permissions to minimum required
|
|
- Regular security updates
|
|
|
|
### 4. Documentation
|
|
|
|
- Keep README files updated
|
|
- Document all input parameters
|
|
- Provide usage examples
|
|
- Include troubleshooting guides
|
|
|
|
## Troubleshooting
|
|
|
|
### Validation Script Issues
|
|
|
|
```bash
|
|
# Check dependencies
|
|
./scripts/validate-templates.sh
|
|
|
|
# Manual YAML validation
|
|
yamllint .gitea/workflows/python/code-quality.yml
|
|
|
|
# Manual structure check
|
|
yq eval '.on | has("workflow_call")' .gitea/workflows/python/code-quality.yml
|
|
```
|
|
|
|
### Test Project Issues
|
|
|
|
```bash
|
|
# Python projects
|
|
cd test-projects/python-basic
|
|
uv sync --all-extras
|
|
uv run pytest --verbose
|
|
|
|
# Web projects
|
|
cd test-projects/web-basic
|
|
npm install
|
|
npm run format:check
|
|
```
|
|
|
|
### Workflow Issues
|
|
|
|
```bash
|
|
# Check workflow logs in Gitea
|
|
# Enable debug logging:
|
|
env:
|
|
ACTIONS_STEP_DEBUG: true
|
|
ACTIONS_RUNNER_DEBUG: true
|
|
```
|
|
|
|
## Contributing
|
|
|
|
When adding new templates or making changes:
|
|
|
|
1. **Create/update test projects** to validate the changes
|
|
2. **Run validation script** locally before committing
|
|
3. **Update documentation** as needed
|
|
4. **Test with feature branches** before merging
|
|
5. **Tag releases** for stable versions
|
|
|
|
## Resources
|
|
|
|
- [Gitea Actions Documentation](https://docs.gitea.io/en-us/actions/)
|
|
- [GitHub Actions Syntax](https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions)
|
|
- [Act - Local Testing Tool](https://github.com/nektos/act)
|
|
- [yamllint Documentation](https://yamllint.readthedocs.io/)
|
|
- [yq Documentation](https://mikefarah.gitbook.io/yq/) |