Energy Consumption Ingester

A Python package for ingesting energy consumption data from the Eloverblik API and storing it in PostgreSQL.

Project Structure

energy-consumption-ingester/
├── src/
│   └── energy_consumption_ingester/
│       ├── __init__.py          # Package initialization
│       ├── eloverblik.py        # Eloverblik API client
│       ├── database.py          # PostgreSQL storage
│       ├── utils.py             # Data transformation utilities
│       ├── cli.py               # Command line interface
│       └── config.py            # Configuration and examples
├── scripts/
│   └── create_db_user.sh        # Database setup script
├── example.py                   # Usage example
├── main.py                      # Original script
├── POC.ipynb                    # Jupyter notebook for development
├── pyproject.toml               # Package configuration
├── README.md                    # This file
└── .env                         # Environment variables (create this)

Installation

Prerequisites

1. PostgreSQL - Make sure you have PostgreSQL installed and running
2. Python 3.12+ - Required Python version
3. uv - Package manager

Setup

1. Clone and enter the repository:

   cd energy-consumption-ingester
   

2. Install dependencies using uv:

   uv sync
   

3. Create environment file:

   cp .env.example .env
   # Edit .env with your credentials
   

4. Set up PostgreSQL database:

   # Create database
   createdb energy_consumption
   
   # Create user (optional - run the script)
   chmod +x scripts/create_db_user.sh
   ./scripts/create_db_user.sh
   

Configuration

Create a .env file in the project root:

# Eloverblik API Configuration
ELOVERBLIK_API_TOKEN=your_refresh_token_here

# Database Configuration
DB_HOST=localhost
DB_NAME=energy_consumption
DB_USER=energy_user
DB_PASSWORD=your_password_here
DB_PORT=5432

Usage

Command Line Interface

The package provides a CLI for common operations:

# Incremental sync (recommended for CI/CD hourly jobs)
uv run energy-ingester sync-incremental

# Full sync for initial setup or backfill
uv run energy-ingester sync --days 30

# Health check
uv run energy-ingester health-check

# Database setup (initial setup)
uv run energy-ingester setup-database

# Database reset (DESTRUCTIVE - deletes all data)
uv run energy-ingester setup-database --reset

# Manual sync with custom parameters
uv run energy-ingester sync-incremental --hours-back 48 --verbose

CI/CD Integration

For Gitea CI (recommended for hourly sync):

# This is the command your CI should run every hour
uv run energy-ingester sync-incremental --verbose

The incremental sync is optimized for frequent runs:

• Only syncs recent data (last 25 hours by default)
• Detects and fills gaps automatically
• Handles overlapping data gracefully
• Fast execution suitable for hourly scheduling

Python API

from energy_consumption_ingester import (
    EloverblikClient,
    DatabaseStorage,
    transform_metering_point_data,
    transform_consumption_data
)

# Initialize clients from environment variables
client = EloverblikClient.from_env()
db = DatabaseStorage.from_env()

# Connect and setup database
db.connect()
db.create_tables()

# Get metering points
metering_points = client.get_metering_points()

# Get consumption data
consumption_data = client.get_consumption_data_last_days(
    metering_points[0]['meteringPointId'],
    days=7
)

# Transform and store data
mp_data = transform_metering_point_data(metering_points[0])
db.store_metering_point(mp_data)

metering_point_id, readings = transform_consumption_data(consumption_data)
db.store_consumption_readings(metering_point_id, readings)

db.close()

Example Script

Run the included example:

uv run python example.py

Development

Install development dependencies:

uv sync --group dev

Code formatting and linting:

# Format code
uv run black src/ example.py

# Lint code
uv run flake8 src/ example.py

# Type checking
uv run mypy src/

Testing:

uv run pytest

CI/CD Setup for Gitea

Automated Workflows

The project includes pre-configured Gitea workflows in .gitea/workflows/:

1. hourly-sync.yml - Runs every hour to sync energy data
2. daily-health-check.yml - Daily system health monitoring
3. manual-full-sync.yml - Manual trigger for full data backfill
4. database-setup.yml - Manual database setup and management

Required Secrets

Configure these secrets in your Gitea repository settings:

ELOVERBLIK_API_TOKEN    # Your Eloverblik refresh token
DB_HOST                 # Database host (e.g., your-db-server.com)
DB_NAME                 # Database name (e.g., energy_consumption)
DB_USER                 # Database username
DB_PASSWORD             # Database password
DB_PORT                 # Database port (usually 5432)

Workflow Schedule

• Hourly Sync: Runs at 5 minutes past every hour (5 * * * *)
• Health Check: Runs daily at 6:00 AM UTC (0 6 * * *)
• Manual Sync: Can be triggered manually with custom day range

Commands for CI

# Recommended for hourly CI jobs (fast, incremental)
uv run energy-ingester sync-incremental --verbose

# For initial setup or recovery (slower, comprehensive)
uv run energy-ingester sync --days 30 --verbose

# For monitoring and alerting
uv run energy-ingester health-check --verbose

# Database setup (run once initially or after schema changes)
uv run energy-ingester setup-database --verbose

Database Setup Workflow

The database-setup.yml workflow provides safe database management:

• Setup: Create tables and schema (safe, idempotent)
• Migrate: Re-run table creation (for schema updates)
• Reset: Drop all tables and recreate (DESTRUCTIVE - requires confirmation)

To run the database setup in Gitea CI:

1. Go to Actions → Database Setup
2. Choose action: setup, migrate, or reset
3. For reset operations, type yes in the confirm field
4. Run workflow

Docker Deployment

For containerized deployment:

# Build and run with Docker Compose
docker-compose up -d

# Or build manually
docker build -t energy-ingester .
docker run -e ELOVERBLIK_API_TOKEN=your_token energy-ingester

Monitoring and Alerts

The workflows are designed to:

• Exit with error codes on failure (for CI/CD detection)
• Log detailed information for debugging
• Support notification integration (extend the workflows)

To add Slack/email notifications, modify the workflow files in .gitea/workflows/.

Modules

1. eloverblik.py - API Client

Purpose: Handles all interactions with the Eloverblik API

Key Features:

• Automatic token refresh (refresh token → access token)
• Metering points discovery
• Consumption data retrieval with flexible date ranges
• Error handling and retry logic
• Environment variable configuration

Main Methods:

• get_access_token() - Exchange refresh token for access token
• get_metering_points() - Retrieve all metering points
• get_consumption_data() - Get consumption data for date range
• get_consumption_data_last_days() - Convenience method for recent data

2. database.py - PostgreSQL Storage

Purpose: Handles all database operations for storing energy data

Key Features:

• Normalized database schema design
• UPSERT operations (handles duplicates gracefully)
• Proper indexing for performance
• Transaction management
• Connection pooling ready

Main Methods:

• create_tables() - Set up database schema
• store_metering_point() - Store/update metering point info
• store_consumption_readings() - Batch insert consumption data
• get_latest_reading_timestamp() - Find last stored reading

Database Schema:

• metering_points - Meter metadata (location, consumer info, etc.)
• consumption_readings - Time series consumption data

3. utils.py - Data Transformation

Purpose: Transform API responses to database-ready format

Key Features:

• API data normalization
• Timestamp parsing and timezone handling
• Data validation
• Statistics calculation

Architecture Benefits

1. Separation of Concerns

• API layer (eloverblik.py) - Pure API interactions
• Storage layer (database.py) - Pure database operations
• Transform layer (utils.py) - Data processing logic

2. Modular Design

• Each module can be used independently
• Easy to test individual components
• Simple to extend or replace components

3. Environment-Based Configuration

• No hardcoded credentials
• Easy deployment across environments
• Secure credential management

4. uv Package Management

• Fast dependency resolution
• Proper Python packaging
• Development vs production dependencies
• Script entry points

5. Database Design

• Normalized schema prevents data duplication
• Proper indexing for time-series queries
• UPSERT operations handle data refresh scenarios
• Extensible for multiple meters/customers

Best Practices Implemented

1. Type Hints - Full type annotation for better IDE support and validation
2. Error Handling - Comprehensive exception handling with logging
3. Configuration Management - Environment variables with validation
4. Documentation - Comprehensive docstrings and README
5. CLI Interface - User-friendly command line tool
6. Packaging - Proper Python package structure with pyproject.toml

This structure gives you a professional, maintainable, and scalable foundation for your energy consumption ingestion project!