449 lines
12 KiB
Markdown
449 lines
12 KiB
Markdown
# Energy Consumption Ingester
|
|
|
|
A Python package for ingesting energy consumption data from the Eloverblik API and storing it in PostgreSQL.
|
|
|
|
## Project Structure
|
|
|
|
```
|
|
energy-consumption-ingester/
|
|
├── src/
|
|
│ └── energy_consumption_ingester/
|
|
│ ├── __init__.py # Package initialization
|
|
│ ├── eloverblik.py # Eloverblik API client
|
|
│ ├── database.py # PostgreSQL storage
|
|
│ ├── utils.py # Data transformation utilities
|
|
│ ├── cli.py # Command line interface
|
|
│ └── config.py # Configuration and examples
|
|
├── scripts/
|
|
│ └── create_db_user.sh # Database setup script
|
|
├── example.py # Usage example
|
|
├── main.py # Original script
|
|
├── POC.ipynb # Jupyter notebook for development
|
|
├── pyproject.toml # Package configuration
|
|
├── README.md # This file
|
|
└── .env # Environment variables (create this)
|
|
```
|
|
|
|
## Installation
|
|
|
|
### Prerequisites
|
|
|
|
1. **PostgreSQL** - Make sure you have PostgreSQL installed and running
|
|
2. **Python 3.12+** - Required Python version
|
|
3. **uv** - Package manager
|
|
|
|
### Setup
|
|
|
|
1. **Clone and enter the repository:**
|
|
|
|
```bash
|
|
cd energy-consumption-ingester
|
|
```
|
|
|
|
2. **Install dependencies using uv:**
|
|
|
|
```bash
|
|
uv sync
|
|
```
|
|
|
|
3. **Create environment file:**
|
|
|
|
```bash
|
|
cp .env.example .env
|
|
# Edit .env with your credentials
|
|
```
|
|
|
|
4. **Set up PostgreSQL database:**
|
|
|
|
```bash
|
|
# Connect to Postgres14 host
|
|
psql -U <POSTGRES_ADMIN> -h <POSTGERS_HOST> -p <POSTGERS_PORT>
|
|
|
|
# Create database user
|
|
CREATE USER <DB_USER> WITH PASSWORD <DB_PASSWORD>;
|
|
|
|
# Create database
|
|
CREATE DATABASE <DB_NAME>;
|
|
|
|
# Grant full privileges on database to user
|
|
GRANT ALL PRIVILEGES ON DATABASE <DB_NAME> TO <DB_USER>;
|
|
|
|
# (Optional) verify new user and database creation
|
|
\q
|
|
psql -U <DB_USER> -h <POSTGRES_HOST> -p <POSTGRES_PORT>
|
|
```
|
|
|
|
5. **OPTIONAL - Add read-only user to database:**
|
|
|
|
```bash
|
|
# Connect to PostgreSQL as admin user
|
|
psql -U <POSTGRES_ADMIN> -h <POSTGRES_HOST> -p <POSTGRES_PORT> -d <DB_NAME>
|
|
|
|
# Create read-only user
|
|
CREATE USER <READONLY_USER> WITH PASSWORD '<READONLY_PASSWORD>';
|
|
|
|
# Grant connect privilege to the database
|
|
GRANT CONNECT ON DATABASE <DB_NAME> TO <READONLY_USER>;
|
|
|
|
# Grant usage on the schema (assuming default 'public' schema)
|
|
GRANT USAGE ON SCHEMA public TO <READONLY_USER>;
|
|
|
|
# Grant select privileges on all existing tables
|
|
GRANT SELECT ON ALL TABLES IN SCHEMA public TO <READONLY_USER>;
|
|
|
|
# Grant select privileges on all future tables (important for new tables)
|
|
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON TABLES TO <READONLY_USER>;
|
|
|
|
# Grant usage on all existing sequences (needed for serial columns)
|
|
GRANT USAGE ON ALL SEQUENCES IN SCHEMA public TO <READONLY_USER>;
|
|
|
|
# Grant usage on all future sequences
|
|
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT USAGE ON SEQUENCES TO <READONLY_USER>;
|
|
|
|
# Verify the read-only user can connect and query
|
|
\q
|
|
psql -U <READONLY_USER> -h <POSTGRES_HOST> -p <POSTGRES_PORT> -d <DB_NAME>
|
|
|
|
# Test read access (should work)
|
|
SELECT COUNT(*) FROM metering_points;
|
|
SELECT COUNT(*) FROM consumption_readings;
|
|
|
|
# Test write access (should fail)
|
|
INSERT INTO metering_points (metering_point_id) VALUES ('test'); -- Should get permission denied
|
|
```
|
|
|
|
**For Home Assistant integration**, use these read-only credentials in your `configuration.yaml`:
|
|
|
|
```yaml
|
|
# configuration.yaml
|
|
sensor:
|
|
- platform: sql
|
|
db_url: postgresql://<READONLY_USER>:<READONLY_PASSWORD>@<POSTGRES_HOST>:<POSTGRES_PORT>/<DB_NAME>
|
|
queries:
|
|
- name: "Latest Energy Reading"
|
|
query: >
|
|
SELECT
|
|
consumption_kwh,
|
|
timestamp
|
|
FROM consumption_readings
|
|
ORDER BY timestamp DESC
|
|
LIMIT 1
|
|
column: consumption_kwh
|
|
unit_of_measurement: "kWh"
|
|
```
|
|
|
|
## Configuration
|
|
|
|
Create a `.env` file in the project root:
|
|
|
|
```env
|
|
# Eloverblik API Configuration
|
|
ELOVERBLIK_API_TOKEN=your_refresh_token_here
|
|
|
|
# Database Configuration
|
|
DB_HOST=localhost
|
|
DB_NAME=energy_consumption
|
|
DB_USER=energy_user
|
|
DB_PASSWORD=your_password_here
|
|
DB_PORT=5432
|
|
```
|
|
|
|
## Usage
|
|
|
|
### Command Line Interface
|
|
|
|
The package provides a CLI for common operations:
|
|
|
|
```bash
|
|
# Incremental sync (recommended for CI/CD hourly jobs)
|
|
uv run energy-ingester sync-incremental
|
|
|
|
# Full sync for initial setup or backfill
|
|
uv run energy-ingester sync --days 30
|
|
|
|
# Health check
|
|
uv run energy-ingester health-check
|
|
|
|
# Database setup (initial setup)
|
|
uv run energy-ingester setup-database
|
|
|
|
# Database reset (DESTRUCTIVE - deletes all data)
|
|
uv run energy-ingester setup-database --reset
|
|
|
|
# Manual sync with custom parameters
|
|
uv run energy-ingester sync-incremental --hours-back 48 --verbose
|
|
```
|
|
|
|
### CI/CD Integration
|
|
|
|
For **Gitea CI** (recommended for hourly sync):
|
|
|
|
```bash
|
|
# This is the command your CI should run every hour
|
|
uv run energy-ingester sync-incremental --verbose
|
|
```
|
|
|
|
The incremental sync is optimized for frequent runs:
|
|
|
|
- Only syncs recent data (last 25 hours by default)
|
|
- Detects and fills gaps automatically
|
|
- Handles overlapping data gracefully
|
|
- Fast execution suitable for hourly scheduling
|
|
|
|
### Python API
|
|
|
|
```python
|
|
from energy_consumption_ingester import (
|
|
EloverblikClient,
|
|
DatabaseStorage,
|
|
transform_metering_point_data,
|
|
transform_consumption_data
|
|
)
|
|
|
|
# Initialize clients from environment variables
|
|
client = EloverblikClient.from_env()
|
|
db = DatabaseStorage.from_env()
|
|
|
|
# Connect and setup database
|
|
db.connect()
|
|
db.create_tables()
|
|
|
|
# Get metering points
|
|
metering_points = client.get_metering_points()
|
|
|
|
# Get consumption data
|
|
consumption_data = client.get_consumption_data_last_days(
|
|
metering_points[0]['meteringPointId'],
|
|
days=7
|
|
)
|
|
|
|
# Transform and store data
|
|
mp_data = transform_metering_point_data(metering_points[0])
|
|
db.store_metering_point(mp_data)
|
|
|
|
metering_point_id, readings = transform_consumption_data(consumption_data)
|
|
db.store_consumption_readings(metering_point_id, readings)
|
|
|
|
db.close()
|
|
```
|
|
|
|
### Example Script
|
|
|
|
Run the included example:
|
|
|
|
```bash
|
|
uv run python example.py
|
|
```
|
|
|
|
## Development
|
|
|
|
### Install development dependencies:
|
|
|
|
```bash
|
|
uv sync --group dev
|
|
```
|
|
|
|
### Code formatting and linting:
|
|
|
|
```bash
|
|
# Format code
|
|
uv run black src/ example.py
|
|
|
|
# Lint code
|
|
uv run flake8 src/ example.py
|
|
|
|
# Type checking
|
|
uv run mypy src/
|
|
```
|
|
|
|
### Testing:
|
|
|
|
```bash
|
|
uv run pytest
|
|
```
|
|
|
|
## CI/CD Setup for Gitea
|
|
|
|
### Automated Workflows
|
|
|
|
The project includes pre-configured Gitea workflows in `.gitea/workflows/`:
|
|
|
|
1. **`hourly-sync.yml`** - Runs every hour to sync energy data
|
|
2. **`daily-health-check.yml`** - Daily system health monitoring
|
|
3. **`manual-full-sync.yml`** - Manual trigger for full data backfill
|
|
4. **`database-setup.yml`** - Manual database setup and management
|
|
|
|
### Required Secrets
|
|
|
|
Configure these secrets in your Gitea repository settings:
|
|
|
|
```
|
|
ELOVERBLIK_API_TOKEN # Your Eloverblik refresh token
|
|
DB_HOST # Database host (e.g., your-db-server.com)
|
|
DB_NAME # Database name (e.g., energy_consumption)
|
|
DB_USER # Database username
|
|
DB_PASSWORD # Database password
|
|
DB_PORT # Database port (usually 5432)
|
|
```
|
|
|
|
### Workflow Schedule
|
|
|
|
- **Hourly Sync**: Runs at 5 minutes past every hour (`5 * * * *`)
|
|
- **Health Check**: Runs daily at 6:00 AM UTC (`0 6 * * *`)
|
|
- **Manual Sync**: Can be triggered manually with custom day range
|
|
|
|
### Commands for CI
|
|
|
|
```bash
|
|
# Recommended for hourly CI jobs (fast, incremental)
|
|
uv run energy-ingester sync-incremental --verbose
|
|
|
|
# For initial setup or recovery (slower, comprehensive)
|
|
uv run energy-ingester sync --days 30 --verbose
|
|
|
|
# For monitoring and alerting
|
|
uv run energy-ingester health-check --verbose
|
|
|
|
# Database setup (run once initially or after schema changes)
|
|
uv run energy-ingester setup-database --verbose
|
|
```
|
|
|
|
### Database Setup Workflow
|
|
|
|
The `database-setup.yml` workflow provides safe database management:
|
|
|
|
- **Setup**: Create tables and schema (safe, idempotent)
|
|
- **Migrate**: Re-run table creation (for schema updates)
|
|
- **Reset**: Drop all tables and recreate (DESTRUCTIVE - requires confirmation)
|
|
|
|
To run the database setup in Gitea CI:
|
|
|
|
1. Go to Actions → Database Setup
|
|
2. Choose action: `setup`, `migrate`, or `reset`
|
|
3. For reset operations, type `yes` in the confirm field
|
|
4. Run workflow
|
|
|
|
### Docker Deployment
|
|
|
|
For containerized deployment:
|
|
|
|
```bash
|
|
# Build and run with Docker Compose
|
|
docker-compose up -d
|
|
|
|
# Or build manually
|
|
docker build -t energy-ingester .
|
|
docker run -e ELOVERBLIK_API_TOKEN=your_token energy-ingester
|
|
```
|
|
|
|
### Monitoring and Alerts
|
|
|
|
The workflows are designed to:
|
|
|
|
- Exit with error codes on failure (for CI/CD detection)
|
|
- Log detailed information for debugging
|
|
- Support notification integration (extend the workflows)
|
|
|
|
To add Slack/email notifications, modify the workflow files in `.gitea/workflows/`.
|
|
|
|
## Modules
|
|
|
|
### 1. `eloverblik.py` - API Client
|
|
|
|
**Purpose:** Handles all interactions with the Eloverblik API
|
|
|
|
**Key Features:**
|
|
|
|
- Automatic token refresh (refresh token → access token)
|
|
- Metering points discovery
|
|
- Consumption data retrieval with flexible date ranges
|
|
- Error handling and retry logic
|
|
- Environment variable configuration
|
|
|
|
**Main Methods:**
|
|
|
|
- `get_access_token()` - Exchange refresh token for access token
|
|
- `get_metering_points()` - Retrieve all metering points
|
|
- `get_consumption_data()` - Get consumption data for date range
|
|
- `get_consumption_data_last_days()` - Convenience method for recent data
|
|
|
|
### 2. `database.py` - PostgreSQL Storage
|
|
|
|
**Purpose:** Handles all database operations for storing energy data
|
|
|
|
**Key Features:**
|
|
|
|
- Normalized database schema design
|
|
- UPSERT operations (handles duplicates gracefully)
|
|
- Proper indexing for performance
|
|
- Transaction management
|
|
- Connection pooling ready
|
|
|
|
**Main Methods:**
|
|
|
|
- `create_tables()` - Set up database schema
|
|
- `store_metering_point()` - Store/update metering point info
|
|
- `store_consumption_readings()` - Batch insert consumption data
|
|
- `get_latest_reading_timestamp()` - Find last stored reading
|
|
|
|
**Database Schema:**
|
|
|
|
- `metering_points` - Meter metadata (location, consumer info, etc.)
|
|
- `consumption_readings` - Time series consumption data
|
|
|
|
### 3. `utils.py` - Data Transformation
|
|
|
|
**Purpose:** Transform API responses to database-ready format
|
|
|
|
**Key Features:**
|
|
|
|
- API data normalization
|
|
- Timestamp parsing and timezone handling
|
|
- Data validation
|
|
- Statistics calculation
|
|
|
|
## Architecture Benefits
|
|
|
|
### 1. **Separation of Concerns**
|
|
|
|
- **API layer** (`eloverblik.py`) - Pure API interactions
|
|
- **Storage layer** (`database.py`) - Pure database operations
|
|
- **Transform layer** (`utils.py`) - Data processing logic
|
|
|
|
### 2. **Modular Design**
|
|
|
|
- Each module can be used independently
|
|
- Easy to test individual components
|
|
- Simple to extend or replace components
|
|
|
|
### 3. **Environment-Based Configuration**
|
|
|
|
- No hardcoded credentials
|
|
- Easy deployment across environments
|
|
- Secure credential management
|
|
|
|
### 4. **uv Package Management**
|
|
|
|
- Fast dependency resolution
|
|
- Proper Python packaging
|
|
- Development vs production dependencies
|
|
- Script entry points
|
|
|
|
### 5. **Database Design**
|
|
|
|
- Normalized schema prevents data duplication
|
|
- Proper indexing for time-series queries
|
|
- UPSERT operations handle data refresh scenarios
|
|
- Extensible for multiple meters/customers
|
|
|
|
## Best Practices Implemented
|
|
|
|
1. **Type Hints** - Full type annotation for better IDE support and validation
|
|
2. **Error Handling** - Comprehensive exception handling with logging
|
|
3. **Configuration Management** - Environment variables with validation
|
|
4. **Documentation** - Comprehensive docstrings and README
|
|
5. **CLI Interface** - User-friendly command line tool
|
|
6. **Packaging** - Proper Python package structure with `pyproject.toml`
|
|
|
|
This structure gives you a professional, maintainable, and scalable foundation for your energy consumption ingestion project!
|