Add QueueRepositoryInterface with memory and file-backed adapters.
PR Title Check / check-title (pull_request) Successful in 9s
Code Quality Pipeline / code-quality (pull_request) Failing after 53s
Test Python Package / unit-tests (pull_request) Successful in 1m1s
Test Python Package / integration-tests (pull_request) Successful in 1m44s
Test Python Package / coverage-report (pull_request) Successful in 13s

Provide a generic disk-backed FIFO queue with configurable path, retention, and dedup keys so consumers can buffer items across restarts without optional extras.

Co-authored-by: Cursor <[email protected]>
This commit is contained in:
Brian Bjarke Jensen
2026-07-16 20:20:52 +02:00
co-authored by Cursor
parent 88ea7f06ab
commit 4f58c32dd6
15 changed files with 1092 additions and 13 deletions
+55 -8
View File
@@ -6,11 +6,11 @@ Subclass an adapter in your own repository to add domain-specific methods while
## Architecture
| Layer | Responsibility |
| ---------------- | ------------------------------------------------------------------------------------ |
| **Interfaces** | Abstract contracts for connection, context, and CRUD |
| **Adapters** | Technology-specific base classes (`RedisAdapter`, `MinioAdapter`, `PostgresAdapter`) |
| **Your project** | Subclass an adapter and add domain methods |
| Layer | Responsibility |
| ---------------- | ------------------------------------------------------------------------------------------------------- |
| **Interfaces** | Abstract contracts for connection, context, CRUD, and queues |
| **Adapters** | Technology-specific base classes (`RedisAdapter`, `MinioAdapter`, `PostgresAdapter`, queue adapters) |
| **Your project** | Subclass an adapter and add domain methods |
Each public interface is a `@runtime_checkable` `Protocol` with `@abstractmethod` members. **Subclass an adapter** when you need connection management and shared behavior — explicit subclasses get runtime instantiation guards and inherited default methods (e.g. `scan_keys`). **Type-annotate against an interface** when you want loose coupling — any object with the right methods satisfies the contract for mypy and `isinstance()` checks, without inheriting from this package.
@@ -22,7 +22,7 @@ The current API is synchronous. Async repository interfaces and adapters may be
## Optional dependencies
Repository **interfaces** import with the base package. **Adapters** require the matching extra; importing an adapter without its extra raises `ImportError` with install instructions.
Repository **interfaces** import with the base package. Networked **adapters** (`RedisAdapter`, `MinioAdapter`, `PostgresAdapter`) require the matching extra; importing one without its extra raises `ImportError` with install instructions. Queue adapters (`MemoryQueueAdapter`, `FileBackedQueueAdapter`) need no extra.
Install with the extras you need:
@@ -76,14 +76,50 @@ Requires PostgreSQL 10 or later; tested against PostgreSQL 16 in CI. Tables are
| `POSTGRES_TABLE` | Table name for CRUD operations |
| `POSTGRES_PRIMARY_KEY` | Primary key column name (default: `id`) |
Copy [`.env.example`](.env.example) to `.env` for local development. `RedisConfig.from_env()`, `MinioConfig.from_env()`, and `PostgresConfig.from_env()` load `.env` automatically when resolving configuration from the environment.
### File-backed queue (`QueueRepositoryInterface`)
In-process and disk-backed FIFO queues for buffering dict items. No optional extra required.
| Adapter | Role |
| ------------------------ | -------------------------------------------------------------------- |
| `MemoryQueueAdapter` | Thread-safe in-memory buffer with optional dedup and age eviction |
| `FileBackedQueueAdapter` | Mirrors memory to a JSONL file; replays on `connect()`, compacts on dequeue/evict |
| Environment variable | Description |
| ---------------------------- | -------------------------------------------------------- |
| `FILE_QUEUE_PATH` | Path to the JSONL queue file |
| `FILE_QUEUE_MAX_AGE_HOURS` | Retention window in hours (default: `24`) |
Dedup keys and the age field name are domain-specific: pass them when constructing `FileQueueConfig` (or as kwargs to `FileQueueConfig.from_env(...)`). Callers choose the file path and item schema.
```python
from pathlib import Path
from python_repositories import FileBackedQueueAdapter, FileQueueConfig
config = FileQueueConfig(
path=Path("/var/lib/my-service/queue/items.jsonl"),
max_age_hours=24,
dedup_keys=("id",),
age_key="created_at",
)
with FileBackedQueueAdapter(config=config) as queue:
queue.enqueue([{"id": 1, "created_at": "2024-01-01T00:00:00+00:00"}])
batch = queue.dequeue_batch(max_items=100)
```
Copy [`.env.example`](.env.example) to `.env` for local development. `RedisConfig.from_env()`, `MinioConfig.from_env()`, `PostgresConfig.from_env()`, and `FileQueueConfig.from_env()` load `.env` automatically when resolving configuration from the environment.
## Configuration injection
Adapters accept optional `config` and `client` keyword arguments for explicit setup and testing:
Adapters accept optional `config` and `client` keyword arguments for explicit setup and testing. Queue adapters accept `config` (and an optional injected `memory` buffer for `FileBackedQueueAdapter`):
```python
from pathlib import Path
from python_repositories import (
FileBackedQueueAdapter,
FileQueueConfig,
MinioAdapter,
MinioConfig,
PostgresAdapter,
@@ -110,6 +146,13 @@ postgres = PostgresAdapter(
primary_key="user_id",
)
)
queue = FileBackedQueueAdapter(
config=FileQueueConfig(
path=Path("/var/lib/my-service/queue/items.jsonl"),
dedup_keys=("id",),
age_key="created_at",
)
)
```
When both `config` and `client` are provided, `connect()` skips client creation (the caller owns the client lifecycle). `config` is required whenever `client` is injected.
@@ -186,12 +229,16 @@ class UserRepository(RedisAdapter):
from python_repositories import (
ConnectionAwareInterface,
ContextAwareInterface,
FileBackedQueueAdapter,
FileQueueConfig,
JsonRepositoryInterface,
MemoryQueueAdapter,
MinioAdapter,
MinioConfig,
ObjectRepositoryInterface,
PostgresAdapter,
PostgresConfig,
QueueRepositoryInterface,
RedisAdapter,
RedisConfig,
TableRepositoryInterface,