added full cli package and CI execution
This commit is contained in:
Brian Bjarke Jensen
@@ -1,3 +1,380 @@
|
||||
# energy-consumption-ingester
|
||||
# Energy Consumption Ingester
|
||||
|
||||
Scheduled pipeline that ingests energy consumption data.
|
||||
A Python package for ingesting energy consumption data from the Eloverblik API and storing it in PostgreSQL.
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
energy-consumption-ingester/
|
||||
├── src/
|
||||
│ └── energy_consumption_ingester/
|
||||
│ ├── __init__.py # Package initialization
|
||||
│ ├── eloverblik.py # Eloverblik API client
|
||||
│ ├── database.py # PostgreSQL storage
|
||||
│ ├── utils.py # Data transformation utilities
|
||||
│ ├── cli.py # Command line interface
|
||||
│ └── config.py # Configuration and examples
|
||||
├── scripts/
|
||||
│ └── create_db_user.sh # Database setup script
|
||||
├── example.py # Usage example
|
||||
├── main.py # Original script
|
||||
├── POC.ipynb # Jupyter notebook for development
|
||||
├── pyproject.toml # Package configuration
|
||||
├── README.md # This file
|
||||
└── .env # Environment variables (create this)
|
||||
```
|
||||
|
||||
## Installation
|
||||
|
||||
### Prerequisites
|
||||
|
||||
1. **PostgreSQL** - Make sure you have PostgreSQL installed and running
|
||||
2. **Python 3.12+** - Required Python version
|
||||
3. **uv** - Package manager
|
||||
|
||||
### Setup
|
||||
|
||||
1. **Clone and enter the repository:**
|
||||
|
||||
```bash
|
||||
cd energy-consumption-ingester
|
||||
```
|
||||
|
||||
2. **Install dependencies using uv:**
|
||||
|
||||
```bash
|
||||
uv sync
|
||||
```
|
||||
|
||||
3. **Create environment file:**
|
||||
|
||||
```bash
|
||||
cp .env.example .env
|
||||
# Edit .env with your credentials
|
||||
```
|
||||
|
||||
4. **Set up PostgreSQL database:**
|
||||
|
||||
```bash
|
||||
# Create database
|
||||
createdb energy_consumption
|
||||
|
||||
# Create user (optional - run the script)
|
||||
chmod +x scripts/create_db_user.sh
|
||||
./scripts/create_db_user.sh
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
Create a `.env` file in the project root:
|
||||
|
||||
```env
|
||||
# Eloverblik API Configuration
|
||||
ELOVERBLIK_API_TOKEN=your_refresh_token_here
|
||||
|
||||
# Database Configuration
|
||||
DB_HOST=localhost
|
||||
DB_NAME=energy_consumption
|
||||
DB_USER=energy_user
|
||||
DB_PASSWORD=your_password_here
|
||||
DB_PORT=5432
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### Command Line Interface
|
||||
|
||||
The package provides a CLI for common operations:
|
||||
|
||||
```bash
|
||||
# Incremental sync (recommended for CI/CD hourly jobs)
|
||||
uv run energy-ingester sync-incremental
|
||||
|
||||
# Full sync for initial setup or backfill
|
||||
uv run energy-ingester sync --days 30
|
||||
|
||||
# Health check
|
||||
uv run energy-ingester health-check
|
||||
|
||||
# Database setup (initial setup)
|
||||
uv run energy-ingester setup-database
|
||||
|
||||
# Database reset (DESTRUCTIVE - deletes all data)
|
||||
uv run energy-ingester setup-database --reset
|
||||
|
||||
# Manual sync with custom parameters
|
||||
uv run energy-ingester sync-incremental --hours-back 48 --verbose
|
||||
```
|
||||
|
||||
### CI/CD Integration
|
||||
|
||||
For **Gitea CI** (recommended for hourly sync):
|
||||
|
||||
```bash
|
||||
# This is the command your CI should run every hour
|
||||
uv run energy-ingester sync-incremental --verbose
|
||||
```
|
||||
|
||||
The incremental sync is optimized for frequent runs:
|
||||
|
||||
- Only syncs recent data (last 25 hours by default)
|
||||
- Detects and fills gaps automatically
|
||||
- Handles overlapping data gracefully
|
||||
- Fast execution suitable for hourly scheduling
|
||||
|
||||
### Python API
|
||||
|
||||
```python
|
||||
from energy_consumption_ingester import (
|
||||
EloverblikClient,
|
||||
DatabaseStorage,
|
||||
transform_metering_point_data,
|
||||
transform_consumption_data
|
||||
)
|
||||
|
||||
# Initialize clients from environment variables
|
||||
client = EloverblikClient.from_env()
|
||||
db = DatabaseStorage.from_env()
|
||||
|
||||
# Connect and setup database
|
||||
db.connect()
|
||||
db.create_tables()
|
||||
|
||||
# Get metering points
|
||||
metering_points = client.get_metering_points()
|
||||
|
||||
# Get consumption data
|
||||
consumption_data = client.get_consumption_data_last_days(
|
||||
metering_points[0]['meteringPointId'],
|
||||
days=7
|
||||
)
|
||||
|
||||
# Transform and store data
|
||||
mp_data = transform_metering_point_data(metering_points[0])
|
||||
db.store_metering_point(mp_data)
|
||||
|
||||
metering_point_id, readings = transform_consumption_data(consumption_data)
|
||||
db.store_consumption_readings(metering_point_id, readings)
|
||||
|
||||
db.close()
|
||||
```
|
||||
|
||||
### Example Script
|
||||
|
||||
Run the included example:
|
||||
|
||||
```bash
|
||||
uv run python example.py
|
||||
```
|
||||
|
||||
## Development
|
||||
|
||||
### Install development dependencies:
|
||||
|
||||
```bash
|
||||
uv sync --group dev
|
||||
```
|
||||
|
||||
### Code formatting and linting:
|
||||
|
||||
```bash
|
||||
# Format code
|
||||
uv run black src/ example.py
|
||||
|
||||
# Lint code
|
||||
uv run flake8 src/ example.py
|
||||
|
||||
# Type checking
|
||||
uv run mypy src/
|
||||
```
|
||||
|
||||
### Testing:
|
||||
|
||||
```bash
|
||||
uv run pytest
|
||||
```
|
||||
|
||||
## CI/CD Setup for Gitea
|
||||
|
||||
### Automated Workflows
|
||||
|
||||
The project includes pre-configured Gitea workflows in `.gitea/workflows/`:
|
||||
|
||||
1. **`hourly-sync.yml`** - Runs every hour to sync energy data
|
||||
2. **`daily-health-check.yml`** - Daily system health monitoring
|
||||
3. **`manual-full-sync.yml`** - Manual trigger for full data backfill
|
||||
4. **`database-setup.yml`** - Manual database setup and management
|
||||
|
||||
### Required Secrets
|
||||
|
||||
Configure these secrets in your Gitea repository settings:
|
||||
|
||||
```
|
||||
ELOVERBLIK_API_TOKEN # Your Eloverblik refresh token
|
||||
DB_HOST # Database host (e.g., your-db-server.com)
|
||||
DB_NAME # Database name (e.g., energy_consumption)
|
||||
DB_USER # Database username
|
||||
DB_PASSWORD # Database password
|
||||
DB_PORT # Database port (usually 5432)
|
||||
```
|
||||
|
||||
### Workflow Schedule
|
||||
|
||||
- **Hourly Sync**: Runs at 5 minutes past every hour (`5 * * * *`)
|
||||
- **Health Check**: Runs daily at 6:00 AM UTC (`0 6 * * *`)
|
||||
- **Manual Sync**: Can be triggered manually with custom day range
|
||||
|
||||
### Commands for CI
|
||||
|
||||
```bash
|
||||
# Recommended for hourly CI jobs (fast, incremental)
|
||||
uv run energy-ingester sync-incremental --verbose
|
||||
|
||||
# For initial setup or recovery (slower, comprehensive)
|
||||
uv run energy-ingester sync --days 30 --verbose
|
||||
|
||||
# For monitoring and alerting
|
||||
uv run energy-ingester health-check --verbose
|
||||
|
||||
# Database setup (run once initially or after schema changes)
|
||||
uv run energy-ingester setup-database --verbose
|
||||
```
|
||||
|
||||
### Database Setup Workflow
|
||||
|
||||
The `database-setup.yml` workflow provides safe database management:
|
||||
|
||||
- **Setup**: Create tables and schema (safe, idempotent)
|
||||
- **Migrate**: Re-run table creation (for schema updates)
|
||||
- **Reset**: Drop all tables and recreate (DESTRUCTIVE - requires confirmation)
|
||||
|
||||
To run the database setup in Gitea CI:
|
||||
|
||||
1. Go to Actions → Database Setup
|
||||
2. Choose action: `setup`, `migrate`, or `reset`
|
||||
3. For reset operations, type `yes` in the confirm field
|
||||
4. Run workflow
|
||||
|
||||
### Docker Deployment
|
||||
|
||||
For containerized deployment:
|
||||
|
||||
```bash
|
||||
# Build and run with Docker Compose
|
||||
docker-compose up -d
|
||||
|
||||
# Or build manually
|
||||
docker build -t energy-ingester .
|
||||
docker run -e ELOVERBLIK_API_TOKEN=your_token energy-ingester
|
||||
```
|
||||
|
||||
### Monitoring and Alerts
|
||||
|
||||
The workflows are designed to:
|
||||
|
||||
- Exit with error codes on failure (for CI/CD detection)
|
||||
- Log detailed information for debugging
|
||||
- Support notification integration (extend the workflows)
|
||||
|
||||
To add Slack/email notifications, modify the workflow files in `.gitea/workflows/`.
|
||||
|
||||
## Modules
|
||||
|
||||
### 1. `eloverblik.py` - API Client
|
||||
|
||||
**Purpose:** Handles all interactions with the Eloverblik API
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- Automatic token refresh (refresh token → access token)
|
||||
- Metering points discovery
|
||||
- Consumption data retrieval with flexible date ranges
|
||||
- Error handling and retry logic
|
||||
- Environment variable configuration
|
||||
|
||||
**Main Methods:**
|
||||
|
||||
- `get_access_token()` - Exchange refresh token for access token
|
||||
- `get_metering_points()` - Retrieve all metering points
|
||||
- `get_consumption_data()` - Get consumption data for date range
|
||||
- `get_consumption_data_last_days()` - Convenience method for recent data
|
||||
|
||||
### 2. `database.py` - PostgreSQL Storage
|
||||
|
||||
**Purpose:** Handles all database operations for storing energy data
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- Normalized database schema design
|
||||
- UPSERT operations (handles duplicates gracefully)
|
||||
- Proper indexing for performance
|
||||
- Transaction management
|
||||
- Connection pooling ready
|
||||
|
||||
**Main Methods:**
|
||||
|
||||
- `create_tables()` - Set up database schema
|
||||
- `store_metering_point()` - Store/update metering point info
|
||||
- `store_consumption_readings()` - Batch insert consumption data
|
||||
- `get_latest_reading_timestamp()` - Find last stored reading
|
||||
|
||||
**Database Schema:**
|
||||
|
||||
- `metering_points` - Meter metadata (location, consumer info, etc.)
|
||||
- `consumption_readings` - Time series consumption data
|
||||
|
||||
### 3. `utils.py` - Data Transformation
|
||||
|
||||
**Purpose:** Transform API responses to database-ready format
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- API data normalization
|
||||
- Timestamp parsing and timezone handling
|
||||
- Data validation
|
||||
- Statistics calculation
|
||||
|
||||
## Architecture Benefits
|
||||
|
||||
### 1. **Separation of Concerns**
|
||||
|
||||
- **API layer** (`eloverblik.py`) - Pure API interactions
|
||||
- **Storage layer** (`database.py`) - Pure database operations
|
||||
- **Transform layer** (`utils.py`) - Data processing logic
|
||||
|
||||
### 2. **Modular Design**
|
||||
|
||||
- Each module can be used independently
|
||||
- Easy to test individual components
|
||||
- Simple to extend or replace components
|
||||
|
||||
### 3. **Environment-Based Configuration**
|
||||
|
||||
- No hardcoded credentials
|
||||
- Easy deployment across environments
|
||||
- Secure credential management
|
||||
|
||||
### 4. **uv Package Management**
|
||||
|
||||
- Fast dependency resolution
|
||||
- Proper Python packaging
|
||||
- Development vs production dependencies
|
||||
- Script entry points
|
||||
|
||||
### 5. **Database Design**
|
||||
|
||||
- Normalized schema prevents data duplication
|
||||
- Proper indexing for time-series queries
|
||||
- UPSERT operations handle data refresh scenarios
|
||||
- Extensible for multiple meters/customers
|
||||
|
||||
## Best Practices Implemented
|
||||
|
||||
1. **Type Hints** - Full type annotation for better IDE support and validation
|
||||
2. **Error Handling** - Comprehensive exception handling with logging
|
||||
3. **Configuration Management** - Environment variables with validation
|
||||
4. **Documentation** - Comprehensive docstrings and README
|
||||
5. **CLI Interface** - User-friendly command line tool
|
||||
6. **Packaging** - Proper Python package structure with `pyproject.toml`
|
||||
|
||||
This structure gives you a professional, maintainable, and scalable foundation for your energy consumption ingestion project!
|
||||
|
||||
Reference in New Issue
Block a user