# Energy Consumption Ingester

A Python package for ingesting energy consumption data from the Eloverblik API and storing it in PostgreSQL.

## Project Structure

```
energy-consumption-ingester/
├── src/
│   └── energy_consumption_ingester/
│       ├── __init__.py          # Package initialization
│       ├── eloverblik.py        # Eloverblik API client
│       ├── database.py          # PostgreSQL storage
│       ├── utils.py             # Data transformation utilities
│       ├── cli.py               # Command line interface
│       └── config.py            # Configuration and examples
├── scripts/
│   └── create_db_user.sh        # Database setup script
├── example.py                   # Usage example
├── main.py                      # Original script
├── POC.ipynb                    # Jupyter notebook for development
├── pyproject.toml               # Package configuration
├── README.md                    # This file
└── .env                         # Environment variables (create this)
```

## Installation

### Prerequisites

1. **PostgreSQL** - Make sure you have PostgreSQL installed and running
2. **Python 3.12+** - Required Python version
3. **uv** - Package manager

### Setup

1. **Clone and enter the repository:**

   ```bash
   cd energy-consumption-ingester
   ```

2. **Install dependencies using uv:**

   ```bash
   uv sync
   ```

3. **Create environment file:**

   ```bash
   cp .env.example .env
   # Edit .env with your credentials
   ```

4. **Set up PostgreSQL database:**

   ```bash
   # Connect to Postgres14 host
   psql -U <POSTGRES_ADMIN> -h <POSTGERS_HOST> -p <POSTGERS_PORT>

   # Create database user
   CREATE USER <DB_USER> WITH PASSWORD <DB_PASSWORD>;

   # Create database
   CREATE DATABASE <DB_NAME>;

   # Grant full privileges on database to user
   GRANT ALL PRIVILEGES ON DATABASE <DB_NAME> TO <DB_USER>;

   # (Optional) verify new user and database creation
   \q
   psql -U <DB_USER> -h <POSTGRES_HOST> -p <POSTGRES_PORT>
   ```

5. **OPTIONAL - Add read-only user to database:**

   ```bash
   # Connect to PostgreSQL as admin user
   psql -U <POSTGRES_ADMIN> -h <POSTGRES_HOST> -p <POSTGRES_PORT> -d <DB_NAME>

   # Create read-only user
   CREATE USER <READONLY_USER> WITH PASSWORD '<READONLY_PASSWORD>';

   # Grant connect privilege to the database
   GRANT CONNECT ON DATABASE <DB_NAME> TO <READONLY_USER>;

   # Grant usage on the schema (assuming default 'public' schema)
   GRANT USAGE ON SCHEMA public TO <READONLY_USER>;

   # Grant select privileges on all existing tables
   GRANT SELECT ON ALL TABLES IN SCHEMA public TO <READONLY_USER>;

   # Grant select privileges on all future tables (important for new tables)
   ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON TABLES TO <READONLY_USER>;

   # Grant usage on all existing sequences (needed for serial columns)
   GRANT USAGE ON ALL SEQUENCES IN SCHEMA public TO <READONLY_USER>;

   # Grant usage on all future sequences
   ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT USAGE ON SEQUENCES TO <READONLY_USER>;

   # Verify the read-only user can connect and query
   \q
   psql -U <READONLY_USER> -h <POSTGRES_HOST> -p <POSTGRES_PORT> -d <DB_NAME>
   
   # Test read access (should work)
   SELECT COUNT(*) FROM metering_points;
   SELECT COUNT(*) FROM consumption_readings;
   
   # Test write access (should fail)
   INSERT INTO metering_points (metering_point_id) VALUES ('test'); -- Should get permission denied
   ```

   **For Home Assistant integration**, use these read-only credentials in your `configuration.yaml`:

   ```yaml
   # configuration.yaml
   sensor:
     - platform: sql
       db_url: postgresql://<READONLY_USER>:<READONLY_PASSWORD>@<POSTGRES_HOST>:<POSTGRES_PORT>/<DB_NAME>
       queries:
         - name: "Latest Energy Reading"
           query: >
             SELECT 
               consumption_kwh,
               timestamp
             FROM consumption_readings 
             ORDER BY timestamp DESC 
             LIMIT 1
           column: consumption_kwh
           unit_of_measurement: "kWh"
   ```

## Configuration

Create a `.env` file in the project root:

```env
# Eloverblik API Configuration
ELOVERBLIK_API_TOKEN=your_refresh_token_here

# Database Configuration
DB_HOST=localhost
DB_NAME=energy_consumption
DB_USER=energy_user
DB_PASSWORD=your_password_here
DB_PORT=5432
```

## Usage

### Command Line Interface

The package provides a CLI for common operations:

```bash
# Incremental sync (recommended for CI/CD hourly jobs)
uv run energy-ingester sync-incremental

# Full sync for initial setup or backfill
uv run energy-ingester sync --days 30

# Health check
uv run energy-ingester health-check

# Database setup (initial setup)
uv run energy-ingester setup-database

# Database reset (DESTRUCTIVE - deletes all data)
uv run energy-ingester setup-database --reset

# Manual sync with custom parameters
uv run energy-ingester sync-incremental --hours-back 48 --verbose
```

### CI/CD Integration

For **Gitea CI** (recommended for hourly sync):

```bash
# This is the command your CI should run every hour
uv run energy-ingester sync-incremental --verbose
```

The incremental sync is optimized for frequent runs:

- Only syncs recent data (last 25 hours by default)
- Detects and fills gaps automatically
- Handles overlapping data gracefully
- Fast execution suitable for hourly scheduling

### Python API

```python
from energy_consumption_ingester import (
    EloverblikClient,
    DatabaseStorage,
    transform_metering_point_data,
    transform_consumption_data
)

# Initialize clients from environment variables
client = EloverblikClient.from_env()
db = DatabaseStorage.from_env()

# Connect and setup database
db.connect()
db.create_tables()

# Get metering points
metering_points = client.get_metering_points()

# Get consumption data
consumption_data = client.get_consumption_data_last_days(
    metering_points[0]['meteringPointId'],
    days=7
)

# Transform and store data
mp_data = transform_metering_point_data(metering_points[0])
db.store_metering_point(mp_data)

metering_point_id, readings = transform_consumption_data(consumption_data)
db.store_consumption_readings(metering_point_id, readings)

db.close()
```

### Example Script

Run the included example:

```bash
uv run python example.py
```

## Development

### Install development dependencies:

```bash
uv sync --group dev
```

### Code formatting and linting:

```bash
# Format code
uv run black src/ example.py

# Lint code
uv run flake8 src/ example.py

# Type checking
uv run mypy src/
```

### Testing:

```bash
uv run pytest
```

## CI/CD Setup for Gitea

### Automated Workflows

The project includes pre-configured Gitea workflows in `.gitea/workflows/`:

1. **`hourly-sync.yml`** - Runs every hour to sync energy data
2. **`daily-health-check.yml`** - Daily system health monitoring
3. **`manual-full-sync.yml`** - Manual trigger for full data backfill
4. **`database-setup.yml`** - Manual database setup and management

### Required Secrets

Configure these secrets in your Gitea repository settings:

```
ELOVERBLIK_API_TOKEN    # Your Eloverblik refresh token
DB_HOST                 # Database host (e.g., your-db-server.com)
DB_NAME                 # Database name (e.g., energy_consumption)
DB_USER                 # Database username
DB_PASSWORD             # Database password
DB_PORT                 # Database port (usually 5432)
```

### Workflow Schedule

- **Hourly Sync**: Runs at 5 minutes past every hour (`5 * * * *`)
- **Health Check**: Runs daily at 6:00 AM UTC (`0 6 * * *`)
- **Manual Sync**: Can be triggered manually with custom day range

### Commands for CI

```bash
# Recommended for hourly CI jobs (fast, incremental)
uv run energy-ingester sync-incremental --verbose

# For initial setup or recovery (slower, comprehensive)
uv run energy-ingester sync --days 30 --verbose

# For monitoring and alerting
uv run energy-ingester health-check --verbose

# Database setup (run once initially or after schema changes)
uv run energy-ingester setup-database --verbose
```

### Database Setup Workflow

The `database-setup.yml` workflow provides safe database management:

- **Setup**: Create tables and schema (safe, idempotent)
- **Migrate**: Re-run table creation (for schema updates)
- **Reset**: Drop all tables and recreate (DESTRUCTIVE - requires confirmation)

To run the database setup in Gitea CI:

1. Go to Actions → Database Setup
2. Choose action: `setup`, `migrate`, or `reset`
3. For reset operations, type `yes` in the confirm field
4. Run workflow

### Docker Deployment

For containerized deployment:

```bash
# Build and run with Docker Compose
docker-compose up -d

# Or build manually
docker build -t energy-ingester .
docker run -e ELOVERBLIK_API_TOKEN=your_token energy-ingester
```

### Monitoring and Alerts

The workflows are designed to:

- Exit with error codes on failure (for CI/CD detection)
- Log detailed information for debugging
- Support notification integration (extend the workflows)

To add Slack/email notifications, modify the workflow files in `.gitea/workflows/`.

## Modules

### 1. `eloverblik.py` - API Client

**Purpose:** Handles all interactions with the Eloverblik API

**Key Features:**

- Automatic token refresh (refresh token → access token)
- Metering points discovery
- Consumption data retrieval with flexible date ranges
- Error handling and retry logic
- Environment variable configuration

**Main Methods:**

- `get_access_token()` - Exchange refresh token for access token
- `get_metering_points()` - Retrieve all metering points
- `get_consumption_data()` - Get consumption data for date range
- `get_consumption_data_last_days()` - Convenience method for recent data

### 2. `database.py` - PostgreSQL Storage

**Purpose:** Handles all database operations for storing energy data

**Key Features:**

- Normalized database schema design
- UPSERT operations (handles duplicates gracefully)
- Proper indexing for performance
- Transaction management
- Connection pooling ready

**Main Methods:**

- `create_tables()` - Set up database schema
- `store_metering_point()` - Store/update metering point info
- `store_consumption_readings()` - Batch insert consumption data
- `get_latest_reading_timestamp()` - Find last stored reading

**Database Schema:**

- `metering_points` - Meter metadata (location, consumer info, etc.)
- `consumption_readings` - Time series consumption data

### 3. `utils.py` - Data Transformation

**Purpose:** Transform API responses to database-ready format

**Key Features:**

- API data normalization
- Timestamp parsing and timezone handling
- Data validation
- Statistics calculation

## Architecture Benefits

### 1. **Separation of Concerns**

- **API layer** (`eloverblik.py`) - Pure API interactions
- **Storage layer** (`database.py`) - Pure database operations
- **Transform layer** (`utils.py`) - Data processing logic

### 2. **Modular Design**

- Each module can be used independently
- Easy to test individual components
- Simple to extend or replace components

### 3. **Environment-Based Configuration**

- No hardcoded credentials
- Easy deployment across environments
- Secure credential management

### 4. **uv Package Management**

- Fast dependency resolution
- Proper Python packaging
- Development vs production dependencies
- Script entry points

### 5. **Database Design**

- Normalized schema prevents data duplication
- Proper indexing for time-series queries
- UPSERT operations handle data refresh scenarios
- Extensible for multiple meters/customers

## Best Practices Implemented

1. **Type Hints** - Full type annotation for better IDE support and validation
2. **Error Handling** - Comprehensive exception handling with logging
3. **Configuration Management** - Environment variables with validation
4. **Documentation** - Comprehensive docstrings and README
5. **CLI Interface** - User-friendly command line tool
6. **Packaging** - Proper Python package structure with `pyproject.toml`

This structure gives you a professional, maintainable, and scalable foundation for your energy consumption ingestion project!