Add QueueRepositoryInterface with memory and file-backed adapters.
PR Title Check / check-title (pull_request) Successful in 9s
Code Quality Pipeline / code-quality (pull_request) Failing after 53s
Test Python Package / unit-tests (pull_request) Successful in 1m1s
Test Python Package / integration-tests (pull_request) Successful in 1m44s
Test Python Package / coverage-report (pull_request) Successful in 13s
PR Title Check / check-title (pull_request) Successful in 9s
Code Quality Pipeline / code-quality (pull_request) Failing after 53s
Test Python Package / unit-tests (pull_request) Successful in 1m1s
Test Python Package / integration-tests (pull_request) Successful in 1m44s
Test Python Package / coverage-report (pull_request) Successful in 13s
Provide a generic disk-backed FIFO queue with configurable path, retention, and dedup keys so consumers can buffer items across restarts without optional extras. Co-authored-by: Cursor <[email protected]>
This commit is contained in:
co-authored by
Cursor
parent
88ea7f06ab
commit
4f58c32dd6
@@ -6,11 +6,11 @@ Subclass an adapter in your own repository to add domain-specific methods while
|
||||
|
||||
## Architecture
|
||||
|
||||
| Layer | Responsibility |
|
||||
| ---------------- | ------------------------------------------------------------------------------------ |
|
||||
| **Interfaces** | Abstract contracts for connection, context, and CRUD |
|
||||
| **Adapters** | Technology-specific base classes (`RedisAdapter`, `MinioAdapter`, `PostgresAdapter`) |
|
||||
| **Your project** | Subclass an adapter and add domain methods |
|
||||
| Layer | Responsibility |
|
||||
| ---------------- | ------------------------------------------------------------------------------------------------------- |
|
||||
| **Interfaces** | Abstract contracts for connection, context, CRUD, and queues |
|
||||
| **Adapters** | Technology-specific base classes (`RedisAdapter`, `MinioAdapter`, `PostgresAdapter`, queue adapters) |
|
||||
| **Your project** | Subclass an adapter and add domain methods |
|
||||
|
||||
Each public interface is a `@runtime_checkable` `Protocol` with `@abstractmethod` members. **Subclass an adapter** when you need connection management and shared behavior — explicit subclasses get runtime instantiation guards and inherited default methods (e.g. `scan_keys`). **Type-annotate against an interface** when you want loose coupling — any object with the right methods satisfies the contract for mypy and `isinstance()` checks, without inheriting from this package.
|
||||
|
||||
@@ -22,7 +22,7 @@ The current API is synchronous. Async repository interfaces and adapters may be
|
||||
|
||||
## Optional dependencies
|
||||
|
||||
Repository **interfaces** import with the base package. **Adapters** require the matching extra; importing an adapter without its extra raises `ImportError` with install instructions.
|
||||
Repository **interfaces** import with the base package. Networked **adapters** (`RedisAdapter`, `MinioAdapter`, `PostgresAdapter`) require the matching extra; importing one without its extra raises `ImportError` with install instructions. Queue adapters (`MemoryQueueAdapter`, `FileBackedQueueAdapter`) need no extra.
|
||||
|
||||
Install with the extras you need:
|
||||
|
||||
@@ -76,14 +76,50 @@ Requires PostgreSQL 10 or later; tested against PostgreSQL 16 in CI. Tables are
|
||||
| `POSTGRES_TABLE` | Table name for CRUD operations |
|
||||
| `POSTGRES_PRIMARY_KEY` | Primary key column name (default: `id`) |
|
||||
|
||||
Copy [`.env.example`](.env.example) to `.env` for local development. `RedisConfig.from_env()`, `MinioConfig.from_env()`, and `PostgresConfig.from_env()` load `.env` automatically when resolving configuration from the environment.
|
||||
### File-backed queue (`QueueRepositoryInterface`)
|
||||
|
||||
In-process and disk-backed FIFO queues for buffering dict items. No optional extra required.
|
||||
|
||||
| Adapter | Role |
|
||||
| ------------------------ | -------------------------------------------------------------------- |
|
||||
| `MemoryQueueAdapter` | Thread-safe in-memory buffer with optional dedup and age eviction |
|
||||
| `FileBackedQueueAdapter` | Mirrors memory to a JSONL file; replays on `connect()`, compacts on dequeue/evict |
|
||||
|
||||
| Environment variable | Description |
|
||||
| ---------------------------- | -------------------------------------------------------- |
|
||||
| `FILE_QUEUE_PATH` | Path to the JSONL queue file |
|
||||
| `FILE_QUEUE_MAX_AGE_HOURS` | Retention window in hours (default: `24`) |
|
||||
|
||||
Dedup keys and the age field name are domain-specific: pass them when constructing `FileQueueConfig` (or as kwargs to `FileQueueConfig.from_env(...)`). Callers choose the file path and item schema.
|
||||
|
||||
```python
|
||||
from pathlib import Path
|
||||
|
||||
from python_repositories import FileBackedQueueAdapter, FileQueueConfig
|
||||
|
||||
config = FileQueueConfig(
|
||||
path=Path("/var/lib/my-service/queue/items.jsonl"),
|
||||
max_age_hours=24,
|
||||
dedup_keys=("id",),
|
||||
age_key="created_at",
|
||||
)
|
||||
with FileBackedQueueAdapter(config=config) as queue:
|
||||
queue.enqueue([{"id": 1, "created_at": "2024-01-01T00:00:00+00:00"}])
|
||||
batch = queue.dequeue_batch(max_items=100)
|
||||
```
|
||||
|
||||
Copy [`.env.example`](.env.example) to `.env` for local development. `RedisConfig.from_env()`, `MinioConfig.from_env()`, `PostgresConfig.from_env()`, and `FileQueueConfig.from_env()` load `.env` automatically when resolving configuration from the environment.
|
||||
|
||||
## Configuration injection
|
||||
|
||||
Adapters accept optional `config` and `client` keyword arguments for explicit setup and testing:
|
||||
Adapters accept optional `config` and `client` keyword arguments for explicit setup and testing. Queue adapters accept `config` (and an optional injected `memory` buffer for `FileBackedQueueAdapter`):
|
||||
|
||||
```python
|
||||
from pathlib import Path
|
||||
|
||||
from python_repositories import (
|
||||
FileBackedQueueAdapter,
|
||||
FileQueueConfig,
|
||||
MinioAdapter,
|
||||
MinioConfig,
|
||||
PostgresAdapter,
|
||||
@@ -110,6 +146,13 @@ postgres = PostgresAdapter(
|
||||
primary_key="user_id",
|
||||
)
|
||||
)
|
||||
queue = FileBackedQueueAdapter(
|
||||
config=FileQueueConfig(
|
||||
path=Path("/var/lib/my-service/queue/items.jsonl"),
|
||||
dedup_keys=("id",),
|
||||
age_key="created_at",
|
||||
)
|
||||
)
|
||||
```
|
||||
|
||||
When both `config` and `client` are provided, `connect()` skips client creation (the caller owns the client lifecycle). `config` is required whenever `client` is injected.
|
||||
@@ -186,12 +229,16 @@ class UserRepository(RedisAdapter):
|
||||
from python_repositories import (
|
||||
ConnectionAwareInterface,
|
||||
ContextAwareInterface,
|
||||
FileBackedQueueAdapter,
|
||||
FileQueueConfig,
|
||||
JsonRepositoryInterface,
|
||||
MemoryQueueAdapter,
|
||||
MinioAdapter,
|
||||
MinioConfig,
|
||||
ObjectRepositoryInterface,
|
||||
PostgresAdapter,
|
||||
PostgresConfig,
|
||||
QueueRepositoryInterface,
|
||||
RedisAdapter,
|
||||
RedisConfig,
|
||||
TableRepositoryInterface,
|
||||
|
||||
Reference in New Issue
Block a user