added tests
This commit is contained in:
Brian Bjarke Jensen
+389
@@ -0,0 +1,389 @@
|
||||
# CI Template Testing Guide
|
||||
|
||||
This document explains how to test and validate the CI workflow templates in this repository.
|
||||
|
||||
## Overview
|
||||
|
||||
The testing infrastructure includes:
|
||||
|
||||
- **Automated validation workflow** (`.gitea/workflows/validate-templates.yml`)
|
||||
- **Manual validation script** (`scripts/validate-templates.sh`)
|
||||
- **Test projects** (`test-projects/`)
|
||||
- **Matrix testing** across multiple configurations
|
||||
|
||||
## Quick Start
|
||||
|
||||
### 1. Run Local Validation
|
||||
|
||||
```bash
|
||||
# Make the script executable
|
||||
chmod +x scripts/validate-templates.sh
|
||||
|
||||
# Validate all workflows
|
||||
./scripts/validate-templates.sh
|
||||
|
||||
# List available templates
|
||||
./scripts/validate-templates.sh list
|
||||
|
||||
# Show help
|
||||
./scripts/validate-templates.sh help
|
||||
```
|
||||
|
||||
### 2. Test with Sample Projects
|
||||
|
||||
```bash
|
||||
# Navigate to a test project
|
||||
cd test-projects/python-basic
|
||||
|
||||
# Install dependencies (if using UV)
|
||||
uv sync --all-extras
|
||||
|
||||
# Run local tests
|
||||
uv run pytest
|
||||
uv run mypy .
|
||||
uv run ruff check .
|
||||
```
|
||||
|
||||
## Testing Infrastructure
|
||||
|
||||
### 1. Validation Workflow (`.gitea/workflows/validate-templates.yml`)
|
||||
|
||||
Automatically runs on every push and pull request:
|
||||
|
||||
- **Matrix testing** across Python versions (3.10, 3.11, 3.12)
|
||||
- **Template validation** for all workflow files
|
||||
- **Integration testing** with sample projects
|
||||
- **Documentation checks**
|
||||
|
||||
**Triggered by:**
|
||||
|
||||
- Push to `main` branch
|
||||
- Pull requests
|
||||
- Manual dispatch
|
||||
- Daily schedule (2 AM UTC)
|
||||
|
||||
**Test matrix:**
|
||||
|
||||
```yaml
|
||||
strategy:
|
||||
matrix:
|
||||
python-version: ["3.10", "3.11", "3.12"]
|
||||
template: ["code-quality", "pytest"]
|
||||
include:
|
||||
- template: "formatting"
|
||||
python-version: "3.12" # Only test once for web template
|
||||
```
|
||||
|
||||
### 2. Validation Script (`scripts/validate-templates.sh`)
|
||||
|
||||
**Features:**
|
||||
|
||||
- YAML syntax validation with `yamllint`
|
||||
- Workflow structure validation with `yq`
|
||||
- Reusable workflow detection
|
||||
- Input parameter validation
|
||||
- Documentation checks
|
||||
- Common issue detection
|
||||
|
||||
**Usage:**
|
||||
|
||||
```bash
|
||||
# Basic validation
|
||||
./scripts/validate-templates.sh
|
||||
|
||||
# List all templates
|
||||
./scripts/validate-templates.sh list
|
||||
|
||||
# Help
|
||||
./scripts/validate-templates.sh help
|
||||
```
|
||||
|
||||
**Dependencies:**
|
||||
|
||||
- `yamllint` (optional): `pip install yamllint`
|
||||
- `yq` (optional): `brew install yq` (macOS) or `apt install yq` (Ubuntu)
|
||||
|
||||
**Note:** The script works without dependencies but provides more detailed validation when they're available.
|
||||
|
||||
### 3. Test Projects
|
||||
|
||||
#### Python Basic (`test-projects/python-basic/`)
|
||||
|
||||
**Purpose:** Test Python-specific templates
|
||||
|
||||
**Structure:**
|
||||
|
||||
```text
|
||||
python-basic/
|
||||
├── .gitea/workflows/
|
||||
│ └── test-templates.yml # Tests both code-quality and pytest templates
|
||||
├── src/test_package/
|
||||
│ ├── __init__.py
|
||||
│ ├── calculator.py # Sample module with type hints
|
||||
│ └── utils.py # Utility functions
|
||||
├── tests/
|
||||
│ ├── test_calculator.py # Pytest tests
|
||||
│ └── test_utils.py # More tests
|
||||
├── pyproject.toml # UV-compatible project config
|
||||
└── README.md
|
||||
```
|
||||
|
||||
**Features:**
|
||||
|
||||
- Type hints for mypy testing
|
||||
- Pytest with coverage
|
||||
- Ruff-compatible code style
|
||||
- UV package management
|
||||
|
||||
#### Web Basic (`test-projects/web-basic/`)
|
||||
|
||||
**Purpose:** Test web formatting template
|
||||
|
||||
**Structure:**
|
||||
|
||||
```text
|
||||
web-basic/
|
||||
├── .gitea/workflows/
|
||||
│ └── test-formatting.yml # Tests formatting template
|
||||
├── package.json # Node.js project
|
||||
├── index.js # JavaScript file
|
||||
├── config.yml # YAML file for validation
|
||||
├── .prettierrc.json # Prettier configuration
|
||||
└── README.md
|
||||
```
|
||||
|
||||
**Features:**
|
||||
|
||||
- Prettier configuration
|
||||
- YAML validation
|
||||
- JSON validation
|
||||
- Node.js setup
|
||||
|
||||
## Testing Workflow Templates
|
||||
|
||||
### 1. Local Testing with Act
|
||||
|
||||
Install and use Act to test workflows locally:
|
||||
|
||||
```bash
|
||||
# Install Act
|
||||
brew install act # macOS
|
||||
# or
|
||||
curl https://raw.githubusercontent.com/nektos/act/master/install.sh | sudo bash
|
||||
|
||||
# Test a specific workflow
|
||||
act -W .gitea/workflows/python/code-quality.yml
|
||||
|
||||
# Test with inputs
|
||||
act workflow_call -W .gitea/workflows/python/code-quality.yml \
|
||||
--input python-version=3.11 \
|
||||
--input working-directory=./test-projects/python-basic
|
||||
```
|
||||
|
||||
### 2. Branch Testing
|
||||
|
||||
Test changes on feature branches:
|
||||
|
||||
```bash
|
||||
# Create feature branch
|
||||
git checkout -b feature/improve-python-template
|
||||
|
||||
# Make changes to templates
|
||||
# ...
|
||||
|
||||
# Update test project to use feature branch
|
||||
# In test-projects/python-basic/.gitea/workflows/test-templates.yml:
|
||||
# uses: gitea.gt-proj.com/brian/CI-templates/.gitea/workflows/python/code-quality.yml@feature/improve-python-template
|
||||
|
||||
# Push and test
|
||||
git push origin feature/improve-python-template
|
||||
```
|
||||
|
||||
### 3. Matrix Testing
|
||||
|
||||
Test multiple configurations:
|
||||
|
||||
```yaml
|
||||
# Example matrix test
|
||||
strategy:
|
||||
matrix:
|
||||
python-version: ["3.10", "3.11", "3.12"]
|
||||
os: ["ubuntu-latest", "ubuntu-20.04"]
|
||||
working-directory: [".", "./src"]
|
||||
```
|
||||
|
||||
## Validation Checklist
|
||||
|
||||
### Workflow File Validation
|
||||
|
||||
- [ ] Valid YAML syntax
|
||||
- [ ] Contains required fields: `name`, `on`, `jobs`
|
||||
- [ ] Uses `workflow_call` trigger for reusable workflows
|
||||
- [ ] All inputs have descriptions and types
|
||||
- [ ] No hardcoded values that should be inputs
|
||||
- [ ] Proper defaults for optional inputs
|
||||
|
||||
### Template Testing
|
||||
|
||||
- [ ] All input combinations work correctly
|
||||
- [ ] Error handling for invalid inputs
|
||||
- [ ] Backwards compatibility maintained
|
||||
- [ ] Documentation updated
|
||||
- [ ] Test projects updated
|
||||
|
||||
### Integration Testing
|
||||
|
||||
- [ ] Works with real projects
|
||||
- [ ] Performance is acceptable
|
||||
- [ ] No conflicts between templates
|
||||
- [ ] Secrets handling works correctly
|
||||
|
||||
## Common Issues and Solutions
|
||||
|
||||
### 1. YAML Syntax Errors
|
||||
|
||||
**Problem:** Invalid YAML syntax
|
||||
|
||||
```bash
|
||||
Error: YAML syntax validation failed
|
||||
```
|
||||
|
||||
**Solution:**
|
||||
|
||||
- Use `yamllint` to check syntax
|
||||
- Validate indentation (use spaces, not tabs)
|
||||
- Check for missing quotes around strings with special characters
|
||||
|
||||
### 2. Missing Required Fields
|
||||
|
||||
**Problem:** Workflow missing required fields
|
||||
|
||||
```bash
|
||||
Error: Missing required field 'on' in workflow.yml
|
||||
```
|
||||
|
||||
**Solution:**
|
||||
|
||||
- Ensure all workflows have `name`, `on`, and `jobs` fields
|
||||
- For reusable workflows, use `workflow_call` trigger
|
||||
|
||||
### 3. Input Validation Issues
|
||||
|
||||
**Problem:** Missing input descriptions or types
|
||||
|
||||
```bash
|
||||
Warning: Input 'python-version' missing description
|
||||
```
|
||||
|
||||
**Solution:**
|
||||
|
||||
```yaml
|
||||
inputs:
|
||||
python-version:
|
||||
description: 'Python version to use (e.g., "3.12")'
|
||||
required: false
|
||||
type: string
|
||||
default: "3.12"
|
||||
```
|
||||
|
||||
### 4. Test Project Issues
|
||||
|
||||
**Problem:** Tests fail in sample projects
|
||||
|
||||
```bash
|
||||
Error: Module not found
|
||||
```
|
||||
|
||||
**Solutions:**
|
||||
|
||||
- Check `pyproject.toml` configuration
|
||||
- Verify package structure
|
||||
- Ensure dependencies are correctly specified
|
||||
- Check import paths
|
||||
|
||||
## CI/CD Best Practices
|
||||
|
||||
### 1. Version Control
|
||||
|
||||
- Use semantic versioning for releases
|
||||
- Tag stable versions: `v1.0.0`, `v1.1.0`, etc.
|
||||
- Reference specific versions in production: `@v1.0.0`
|
||||
- Use `@main` for development/testing
|
||||
|
||||
### 2. Backwards Compatibility
|
||||
|
||||
- Test with previous versions
|
||||
- Provide migration guides for breaking changes
|
||||
- Deprecate features gracefully
|
||||
- Maintain changelog
|
||||
|
||||
### 3. Security
|
||||
|
||||
- Validate all inputs
|
||||
- Use secrets for sensitive data
|
||||
- Limit permissions to minimum required
|
||||
- Regular security updates
|
||||
|
||||
### 4. Documentation
|
||||
|
||||
- Keep README files updated
|
||||
- Document all input parameters
|
||||
- Provide usage examples
|
||||
- Include troubleshooting guides
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Validation Script Issues
|
||||
|
||||
```bash
|
||||
# Check dependencies
|
||||
./scripts/validate-templates.sh
|
||||
|
||||
# Manual YAML validation
|
||||
yamllint .gitea/workflows/python/code-quality.yml
|
||||
|
||||
# Manual structure check
|
||||
yq eval '.on | has("workflow_call")' .gitea/workflows/python/code-quality.yml
|
||||
```
|
||||
|
||||
### Test Project Issues
|
||||
|
||||
```bash
|
||||
# Python projects
|
||||
cd test-projects/python-basic
|
||||
uv sync --all-extras
|
||||
uv run pytest --verbose
|
||||
|
||||
# Web projects
|
||||
cd test-projects/web-basic
|
||||
npm install
|
||||
npm run format:check
|
||||
```
|
||||
|
||||
### Workflow Issues
|
||||
|
||||
```bash
|
||||
# Check workflow logs in Gitea
|
||||
# Enable debug logging:
|
||||
env:
|
||||
ACTIONS_STEP_DEBUG: true
|
||||
ACTIONS_RUNNER_DEBUG: true
|
||||
```
|
||||
|
||||
## Contributing
|
||||
|
||||
When adding new templates or making changes:
|
||||
|
||||
1. **Create/update test projects** to validate the changes
|
||||
2. **Run validation script** locally before committing
|
||||
3. **Update documentation** as needed
|
||||
4. **Test with feature branches** before merging
|
||||
5. **Tag releases** for stable versions
|
||||
|
||||
## Resources
|
||||
|
||||
- [Gitea Actions Documentation](https://docs.gitea.io/en-us/actions/)
|
||||
- [GitHub Actions Syntax](https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions)
|
||||
- [Act - Local Testing Tool](https://github.com/nektos/act)
|
||||
- [yamllint Documentation](https://yamllint.readthedocs.io/)
|
||||
- [yq Documentation](https://mikefarah.gitbook.io/yq/)
|
||||
Reference in New Issue
Block a user