Compare commits

..
13 Commits
Author SHA1 Message Date
CI Bot bd4794d1c0 chore: release v2.0.1 [skip ci] 2026-07-09 14:38:25 +00:00
brian 3dc412cd15 Merge pull request '[patch] Align Redis and MinIO connect() idempotency' (#37) from cursor/connect-idempotency-alignment into main
Test Python Package / coverage-report (push) Successful in 17s
Release on merge to main / release (push) Successful in 21s
Test Python Package / integration-tests (push) Successful in 1m17s
Test Python Package / unit-tests (push) Successful in 1m26s
Code Quality Pipeline / code-quality (push) Successful in 1m36s
Reviewed-on: https://gitea.lille-vemmelund.dk/brian/python-repositories/pulls/37
2026-07-09 16:38:05 +02:00
Brian Bjarke JensenandCursor b8d88e6703 Cover injected-client connect when client was cleared by disconnect.
PR Title Check / check-title (pull_request) Successful in 5s
Code Quality Pipeline / code-quality (pull_request) Successful in 26s
Test Python Package / integration-tests (pull_request) Successful in 1m5s
Test Python Package / unit-tests (pull_request) Successful in 29s
Test Python Package / coverage-report (pull_request) Successful in 41s
Restore 100% coverage for the _validate_injected_client early-return paths in Redis and MinIO adapters.

Co-authored-by: Cursor <[email protected]>
2026-07-09 15:57:07 +02:00
Brian Bjarke JensenandCursor 67bb88fbb4 Align Redis and MinIO connect() idempotency in ConnectionAwareAdapter.
PR Title Check / check-title (pull_request) Successful in 5s
Test Python Package / unit-tests (pull_request) Successful in 13s
Test Python Package / integration-tests (pull_request) Successful in 30s
Test Python Package / coverage-report (pull_request) Failing after 15s
Code Quality Pipeline / code-quality (pull_request) Successful in 53s
Move shared connect orchestration into the base adapter so both backends short-circuit when already healthy and only reconnect after a failed probe.

Co-authored-by: Cursor <[email protected]>
2026-07-09 15:46:39 +02:00
CI Bot c2e6a96c5d chore: release v2.0.0 [skip ci] 2026-07-09 13:17:19 +00:00
brian 547a1b3a90 Merge pull request '[breaking] Make MinIO bucket creation opt-in with production-safe defaults' (#36) from cursor/minio-bucket-creation-opt-in into main
Code Quality Pipeline / code-quality (push) Successful in 41s
Release on merge to main / release (push) Successful in 25s
Test Python Package / unit-tests (push) Successful in 17s
Test Python Package / integration-tests (push) Successful in 1m44s
Test Python Package / coverage-report (push) Successful in 23s
Reviewed-on: https://gitea.lille-vemmelund.dk/brian/python-repositories/pulls/36
2026-07-09 15:16:54 +02:00
Brian Bjarke JensenandCursor 39baf9badc Fix examples integration test by provisioning MinIO bucket in fixture.
Test Python Package / unit-tests (pull_request) Successful in 18s
PR Title Check / check-title (pull_request) Successful in 34s
Test Python Package / integration-tests (pull_request) Successful in 33s
Code Quality Pipeline / code-quality (pull_request) Successful in 49s
Test Python Package / coverage-report (pull_request) Successful in 51s
Ensure set_example_env depends on raw_minio_client so ArtifactObjectRepository connects against an existing bucket under the new production-safe default.

Co-authored-by: Cursor <[email protected]>
2026-07-09 15:06:30 +02:00
Brian Bjarke JensenandCursor 4a5b3b631f Make MinIO bucket creation opt-in with production-safe defaults.
PR Title Check / check-title (pull_request) Successful in 6s
Test Python Package / unit-tests (pull_request) Successful in 18s
Code Quality Pipeline / code-quality (pull_request) Successful in 1m28s
Test Python Package / integration-tests (pull_request) Failing after 1m54s
Test Python Package / coverage-report (pull_request) Has been skipped
Require buckets to exist by default on connect, extract env_bool parsing, and document local dev overrides for secure and auto-creation settings.

Co-authored-by: Cursor <[email protected]>
2026-07-09 14:52:14 +02:00
brian 1431f455c1 Merge pull request 'Move Redis test container helpers into integration conftest' (#35) from cursor/move-redis-container-to-conftest into main
Release on merge to main / release (push) Successful in 6s
Test Python Package / unit-tests (push) Successful in 21s
Code Quality Pipeline / code-quality (push) Successful in 28s
Test Python Package / integration-tests (push) Successful in 50s
Test Python Package / coverage-report (push) Successful in 1m19s
Reviewed-on: https://gitea.lille-vemmelund.dk/brian/python-repositories/pulls/35
2026-07-08 21:23:48 +02:00
Brian Bjarke JensenandCursor eafc717045 Move Redis test container helpers into integration conftest.
PR Title Check / check-title (pull_request) Successful in 5s
Test Python Package / unit-tests (pull_request) Successful in 14s
Code Quality Pipeline / code-quality (pull_request) Successful in 31s
Test Python Package / integration-tests (pull_request) Successful in 1m9s
Test Python Package / coverage-report (pull_request) Successful in 29s
Colocate RedisTestContainer with the fixtures that use it and remove the misleading redis_container_test module plus collect_ignore workaround.

Co-authored-by: Cursor <[email protected]>
2026-07-08 21:17:33 +02:00
CI Bot 9a9b7b2985 chore: release v1.1.0 [skip ci] 2026-07-08 19:15:45 +00:00
brian 5b9573d499 Merge pull request '[minor] Add Redis scan_keys iterator API' (#34) from cursor/redis-scan-keys into main
Test Python Package / integration-tests (push) Successful in 52s
Test Python Package / unit-tests (push) Failing after 13s
Code Quality Pipeline / code-quality (push) Successful in 24s
Test Python Package / coverage-report (push) Has been skipped
Release on merge to main / release (push) Successful in 1m18s
Reviewed-on: https://gitea.lille-vemmelund.dk/brian/python-repositories/pulls/34
2026-07-08 21:14:24 +02:00
Brian Bjarke JensenandCursor 50444af982 Add Redis scan_keys iterator API.
Test Python Package / unit-tests (pull_request) Successful in 14s
Code Quality Pipeline / code-quality (pull_request) Successful in 32s
Test Python Package / coverage-report (pull_request) Successful in 16s
PR Title Check / check-title (pull_request) Successful in 6s
Test Python Package / integration-tests (pull_request) Successful in 1m3s
Provide a SCAN-based key iterator for Redis adapters so callers can enumerate large keyspaces without relying on blocking KEYS lookups.

Co-authored-by: Cursor <[email protected]>
2026-07-08 21:06:13 +02:00
21 changed files with 487 additions and 133 deletions
+1
View File
@@ -7,3 +7,4 @@ MINIO_ACCESS_KEY=minioadmin
MINIO_SECRET_KEY=minioadmin MINIO_SECRET_KEY=minioadmin
MINIO_BUCKET=my-bucket MINIO_BUCKET=my-bucket
MINIO_SECURE=false MINIO_SECURE=false
MINIO_CREATE_BUCKET_IF_MISSING=true
+24 -8
View File
@@ -12,7 +12,7 @@ Subclass an adapter in your own repository to add domain-specific methods while
| **Adapters** | Technology-specific base classes (`RedisAdapter`, `MinioAdapter`) | | **Adapters** | Technology-specific base classes (`RedisAdapter`, `MinioAdapter`) |
| **Your project** | Subclass an adapter and add domain methods | | **Your project** | Subclass an adapter and add domain methods |
Connection adapters expose `connect()`, `disconnect()`, and `is_connected()`. The latter verifies backend reachability with a cached health probe (default TTL: 1 second). Subclasses may override `health_check_ttl_seconds`. Connection adapters expose `connect()`, `disconnect()`, and `is_connected()`. The latter verifies backend reachability with a cached health probe (default TTL: 1 second). Subclasses may override `health_check_ttl_seconds`. `connect()` is idempotent: calling it while already connected and healthy is a no-op.
## Optional dependencies ## Optional dependencies
@@ -34,18 +34,33 @@ Requires Redis with the RedisJSON module (e.g. redis-stack).
| -------------------- | ---------------------------------------------------- | | -------------------- | ---------------------------------------------------- |
| `REDIS_URI` | Redis connection URL (e.g. `redis://localhost:6379`) | | `REDIS_URI` | Redis connection URL (e.g. `redis://localhost:6379`) |
For key discovery:
- `list_keys(pattern)` is simple and returns a `list[str]`, but it uses Redis `KEYS` and may block on large datasets.
- `scan_keys(pattern, *, count=None)` is preferred for production use and yields keys incrementally via Redis `SCAN`.
Example:
```python
for key in repo.scan_keys("user:*"):
print(key)
```
### MinIO (`ObjectRepositoryInterface`) ### MinIO (`ObjectRepositoryInterface`)
| Environment variable | Description | | Environment variable | Description |
| -------------------- | ------------------------------------------- | | -------------------------------- | ------------------------------------------------------------------------ |
| `MINIO_ENDPOINT` | MinIO server endpoint | | `MINIO_ENDPOINT` | MinIO server endpoint |
| `MINIO_ACCESS_KEY` | Access key | | `MINIO_ACCESS_KEY` | Access key |
| `MINIO_SECRET_KEY` | Secret key | | `MINIO_SECRET_KEY` | Secret key |
| `MINIO_BUCKET` | Bucket name (created on connect if missing) | | `MINIO_BUCKET` | Bucket name |
| `MINIO_SECURE` | Use HTTPS (`true`/`false`; default: `true`) | | `MINIO_SECURE` | Use HTTPS (`true`/`false`; default: `true`) |
| `MINIO_CREATE_BUCKET_IF_MISSING` | Auto-create `MINIO_BUCKET` on connect (`true`/`false`; default: `false`) |
Copy [`.env.example`](.env.example) to `.env` for local development. `RedisConfig.from_env()` and `MinioConfig.from_env()` load `.env` automatically when resolving configuration from the environment. Copy [`.env.example`](.env.example) to `.env` for local development. `RedisConfig.from_env()` and `MinioConfig.from_env()` load `.env` automatically when resolving configuration from the environment.
For production, it is recommended to leave `MINIO_CREATE_BUCKET_IF_MISSING` unset so that `connect()` fails fast if the expected bucket is missing. For local development, you will often want `MINIO_SECURE=false` and `MINIO_CREATE_BUCKET_IF_MISSING=true`.
## Configuration injection ## Configuration injection
Adapters accept optional `config` and `client` keyword arguments for explicit setup and testing: Adapters accept optional `config` and `client` keyword arguments for explicit setup and testing:
@@ -61,6 +76,7 @@ minio = MinioAdapter(
secret_key="minioadmin", secret_key="minioadmin",
bucket="my-bucket", bucket="my-bucket",
secure=False, secure=False,
# create_bucket_if_missing=True, # convenient for local dev
) )
) )
``` ```
+1 -1
View File
@@ -1,6 +1,6 @@
[project] [project]
name = "python-repositories" name = "python-repositories"
version = "1.0.0" version = "2.0.1"
description = "Various python repository interfaces exposed as a python package." description = "Various python repository interfaces exposed as a python package."
authors = [ authors = [
{ name = "Brian Bjarke Jensen", email = "[email protected]" } { name = "Brian Bjarke Jensen", email = "[email protected]" }
@@ -22,6 +22,7 @@ class ConnectionAwareAdapter(ConnectionAwareInterface, ContextAwareInterface):
def __init__(self) -> None: def __init__(self) -> None:
self.logger = structlog.get_logger(self.__class__.__name__) self.logger = structlog.get_logger(self.__class__.__name__)
self._client_injected = False
self._health_check_at: float | None = None self._health_check_at: float | None = None
self._health_check_ok: bool = False self._health_check_ok: bool = False
@@ -57,6 +58,27 @@ class ConnectionAwareAdapter(ConnectionAwareInterface, ContextAwareInterface):
def _probe_connection(self) -> bool: def _probe_connection(self) -> bool:
"""Backend-specific liveness check; called only when client is ready.""" """Backend-specific liveness check; called only when client is ready."""
@abstractmethod
def _validate_injected_client(self) -> None:
"""Verify an injected client is reachable; raise ConnectionError on failure."""
@abstractmethod
def _establish_connection(self) -> None:
"""Create a backend client and set internal connection state."""
def connect(self) -> None:
"""Connect to the backend; idempotent when already connected and healthy."""
if self._client_injected:
self._validate_injected_client()
self._invalidate_health_cache()
return
if self._is_client_ready() and self.is_connected():
self.logger.info(f"Already connected to {self.connection_name}")
return
self.disconnect()
self._establish_connection()
self._invalidate_health_cache()
def is_connected(self) -> bool: def is_connected(self) -> bool:
"""Check if connected to the backend.""" """Check if connected to the backend."""
if not self._is_client_ready(): if not self._is_client_ready():
+18 -19
View File
@@ -26,6 +26,7 @@ class MinioAdapter(ObjectRepositoryInterface, ConnectionAwareAdapter):
secret_key_env_var_name: str = "MINIO_SECRET_KEY" secret_key_env_var_name: str = "MINIO_SECRET_KEY"
bucket_env_var_name: str = "MINIO_BUCKET" bucket_env_var_name: str = "MINIO_BUCKET"
secure_env_var_name: str = "MINIO_SECURE" secure_env_var_name: str = "MINIO_SECURE"
create_bucket_if_missing_env_var_name: str = "MINIO_CREATE_BUCKET_IF_MISSING"
chunk_size: int = 5 * 2**20 # 5 MiB chunk_size: int = 5 * 2**20 # 5 MiB
connection_name: str = "Minio" connection_name: str = "Minio"
@@ -36,15 +37,16 @@ class MinioAdapter(ObjectRepositoryInterface, ConnectionAwareAdapter):
client: minio.Minio | None = None, client: minio.Minio | None = None,
) -> None: ) -> None:
super().__init__() super().__init__()
if client is not None and config is None:
raise ValueError("config is required when client is provided")
if config is None: if config is None:
if client is not None:
raise ValueError("config is required when client is provided")
config = MinioConfig.from_env( config = MinioConfig.from_env(
self.endpoint_env_var_name, self.endpoint_env_var_name,
self.access_key_env_var_name, self.access_key_env_var_name,
self.secret_key_env_var_name, self.secret_key_env_var_name,
self.bucket_env_var_name, self.bucket_env_var_name,
self.secure_env_var_name, self.secure_env_var_name,
self.create_bucket_if_missing_env_var_name,
) )
self._config = config self._config = config
self._client_injected = client is not None self._client_injected = client is not None
@@ -56,23 +58,17 @@ class MinioAdapter(ObjectRepositoryInterface, ConnectionAwareAdapter):
def _is_client_ready(self) -> bool: def _is_client_ready(self) -> bool:
return self._client is not None and self._bucket_name is not None return self._client is not None and self._bucket_name is not None
def connect(self) -> None: def _validate_injected_client(self) -> None:
"""Connect to the Minio server.""" if self._client is None:
if self._client_injected:
if self._client is not None:
try:
_ = self._client.list_buckets()
except Exception as exc: # pylint: disable=broad-except
raise ConnectionError(
f"Could not connect to Minio at {self._config.endpoint}"
) from exc
self._invalidate_health_cache()
return return
if self._client is not None and self.is_connected(): try:
self.logger.info("Already connected to Minio") _ = self._client.list_buckets()
return except Exception as exc: # pylint: disable=broad-except
if self._client is not None: raise ConnectionError(
self.disconnect() f"Could not connect to Minio at {self._config.endpoint}"
) from exc
def _establish_connection(self) -> None:
endpoint = self._config.endpoint endpoint = self._config.endpoint
access_key = self._config.access_key access_key = self._config.access_key
secret_key = self._config.secret_key secret_key = self._config.secret_key
@@ -88,11 +84,14 @@ class MinioAdapter(ObjectRepositoryInterface, ConnectionAwareAdapter):
except Exception as exc: # pylint: disable=broad-except except Exception as exc: # pylint: disable=broad-except
raise ConnectionError(f"Could not connect to Minio at {endpoint}") from exc raise ConnectionError(f"Could not connect to Minio at {endpoint}") from exc
if not client.bucket_exists(bucket): if not client.bucket_exists(bucket):
if not self._config.create_bucket_if_missing:
raise ConnectionError(
f"Bucket '{bucket}' does not exist on Minio at {endpoint}"
)
self.logger.info(f"Creating bucket '{bucket}'") self.logger.info(f"Creating bucket '{bucket}'")
client.make_bucket(bucket) client.make_bucket(bucket)
self._client = client self._client = client
self._bucket_name = bucket self._bucket_name = bucket
self._invalidate_health_cache()
def disconnect(self) -> None: def disconnect(self) -> None:
"""Disconnect from the Minio server.""" """Disconnect from the Minio server."""
+47 -24
View File
@@ -1,6 +1,7 @@
"""Definition of RedisAdapter class.""" """Definition of RedisAdapter class."""
from __future__ import annotations from __future__ import annotations
from collections.abc import Iterator
from typing import cast from typing import cast
from python_repositories.adapters.connection_aware_adapter import ( from python_repositories.adapters.connection_aware_adapter import (
@@ -34,9 +35,9 @@ class RedisAdapter(JsonRepositoryInterface, ConnectionAwareAdapter):
client: redis.Redis | None = None, client: redis.Redis | None = None,
) -> None: ) -> None:
super().__init__() super().__init__()
if client is not None and config is None:
raise ValueError("config is required when client is provided")
if config is None: if config is None:
if client is not None:
raise ValueError("config is required when client is provided")
config = RedisConfig.from_env(self.uri_env_var_name) config = RedisConfig.from_env(self.uri_env_var_name)
self._config = config self._config = config
self._client_injected = client is not None self._client_injected = client is not None
@@ -48,25 +49,20 @@ class RedisAdapter(JsonRepositoryInterface, ConnectionAwareAdapter):
def _is_client_ready(self) -> bool: def _is_client_ready(self) -> bool:
return self._client is not None return self._client is not None
def connect(self) -> None: def _validate_injected_client(self) -> None:
"""Connect to the Redis server.""" if self._client is None:
if self._client_injected:
if self._client is not None:
try:
if not self._client.ping():
raise ConnectionError(
f"Could not connect to Redis at {self._config.uri}"
)
except (redis.ConnectionError, redis.TimeoutError) as exc:
raise ConnectionError(
f"Could not connect to Redis at {self._config.uri}"
) from exc
self._invalidate_health_cache()
return return
if self._client is not None: try:
self._client.close() if not self._client.ping():
self._client = None raise ConnectionError(
self._invalidate_health_cache() f"Could not connect to Redis at {self._config.uri}"
)
except (redis.ConnectionError, redis.TimeoutError) as exc:
raise ConnectionError(
f"Could not connect to Redis at {self._config.uri}"
) from exc
def _establish_connection(self) -> None:
uri = self._config.uri uri = self._config.uri
try: try:
client = redis.Redis.from_url( client = redis.Redis.from_url(
@@ -78,7 +74,6 @@ class RedisAdapter(JsonRepositoryInterface, ConnectionAwareAdapter):
except (redis.ConnectionError, redis.TimeoutError) as exc: except (redis.ConnectionError, redis.TimeoutError) as exc:
raise ConnectionError(f"Could not connect to Redis at {uri}") from exc raise ConnectionError(f"Could not connect to Redis at {uri}") from exc
self._client = client self._client = client
self._invalidate_health_cache()
def disconnect(self) -> None: def disconnect(self) -> None:
"""Disconnect from the Redis server.""" """Disconnect from the Redis server."""
@@ -136,11 +131,13 @@ class RedisAdapter(JsonRepositoryInterface, ConnectionAwareAdapter):
self._client.json().delete(key) self._client.json().delete(key)
self.logger.debug(f"Deleted {key}") self.logger.debug(f"Deleted {key}")
def list_keys(self, pattern: str) -> list[str]: def _validate_pattern(self, pattern: str) -> None:
"""List keys in Redis matching a pattern."""
# Check input
if not isinstance(pattern, str) or len(pattern) == 0: if not isinstance(pattern, str) or len(pattern) == 0:
raise ValueError("Pattern must be a non-empty string") raise ValueError("Pattern must be a non-empty string")
def list_keys(self, pattern: str) -> list[str]:
"""List keys in Redis using KEYS; may block on large datasets."""
self._validate_pattern(pattern)
# Check connection # Check connection
self._require_connected() self._require_connected()
assert self._client is not None assert self._client is not None
@@ -152,3 +149,29 @@ class RedisAdapter(JsonRepositoryInterface, ConnectionAwareAdapter):
keys: list[str] = [key.decode(self.encoding) for key in keys_raw] keys: list[str] = [key.decode(self.encoding) for key in keys_raw]
self.logger.debug(f"Got {keys} matching {pattern}") self.logger.debug(f"Got {keys} matching {pattern}")
return keys return keys
def scan_keys(
self,
pattern: str,
*,
count: int | None = None,
) -> Iterator[str]:
"""Yield keys in Redis using SCAN to avoid blocking large datasets."""
self._validate_pattern(pattern)
self._require_connected()
assert self._client is not None
client = self._client
def _decode(key: bytes | str) -> str:
return key if isinstance(key, str) else key.decode(self.encoding)
def _iter() -> Iterator[str]:
scan_iter = (
client.scan_iter(match=pattern, count=count)
if count is not None
else client.scan_iter(match=pattern)
)
for key_raw in scan_iter:
yield _decode(key_raw)
return _iter()
+21
View File
@@ -0,0 +1,21 @@
"""Parse boolean values from environment variables."""
from __future__ import annotations
import os
_TRUTHY = frozenset({"1", "true", "yes", "on"})
_FALSY = frozenset({"0", "false", "no", "off"})
def env_bool(name: str, default: bool) -> bool:
"""Parse an environment variable as a boolean value."""
raw = os.getenv(name)
if raw is None:
return default
normalized = raw.strip().lower()
if normalized in _TRUTHY:
return True
if normalized in _FALSY:
return False
raise ValueError(f"Invalid boolean value for {name}: {raw!r}")
+8 -16
View File
@@ -8,21 +8,7 @@ from dataclasses import dataclass
from python_utils import check_env from python_utils import check_env
from python_repositories.config.dotenv_loader import load_dotenv from python_repositories.config.dotenv_loader import load_dotenv
from python_repositories.config.env_bool import env_bool
_TRUTHY = frozenset({"1", "true", "yes", "on"})
_FALSY = frozenset({"0", "false", "no", "off"})
def _env_bool(name: str, default: bool) -> bool:
raw = os.getenv(name)
if raw is None:
return default
normalized = raw.strip().lower()
if normalized in _TRUTHY:
return True
if normalized in _FALSY:
return False
raise ValueError(f"Invalid boolean value for {name}: {raw!r}")
@dataclass(frozen=True) @dataclass(frozen=True)
@@ -34,6 +20,7 @@ class MinioConfig:
secret_key: str secret_key: str
bucket: str bucket: str
secure: bool = True secure: bool = True
create_bucket_if_missing: bool = False
@classmethod @classmethod
def from_env( def from_env(
@@ -43,6 +30,7 @@ class MinioConfig:
secret_key_env_var_name: str = "MINIO_SECRET_KEY", secret_key_env_var_name: str = "MINIO_SECRET_KEY",
bucket_env_var_name: str = "MINIO_BUCKET", bucket_env_var_name: str = "MINIO_BUCKET",
secure_env_var_name: str = "MINIO_SECURE", secure_env_var_name: str = "MINIO_SECURE",
create_bucket_if_missing_env_var_name: str = ("MINIO_CREATE_BUCKET_IF_MISSING"),
*, *,
use_dotenv: bool = True, use_dotenv: bool = True,
) -> MinioConfig: ) -> MinioConfig:
@@ -61,5 +49,9 @@ class MinioConfig:
access_key=str(os.getenv(access_key_env_var_name)), access_key=str(os.getenv(access_key_env_var_name)),
secret_key=str(os.getenv(secret_key_env_var_name)), secret_key=str(os.getenv(secret_key_env_var_name)),
bucket=str(os.getenv(bucket_env_var_name)), bucket=str(os.getenv(bucket_env_var_name)),
secure=_env_bool(secure_env_var_name, default=True), secure=env_bool(secure_env_var_name, default=True),
create_bucket_if_missing=env_bool(
create_bucket_if_missing_env_var_name,
default=False,
),
) )
@@ -8,7 +8,11 @@ class ConnectionAwareInterface(ABC):
@abstractmethod @abstractmethod
def connect(self) -> None: def connect(self) -> None:
"""Connect to resource.""" """Connect to resource.
Implementations should be idempotent: calling connect while already
connected and healthy is a no-op.
"""
... ...
@abstractmethod @abstractmethod
@@ -1,5 +1,6 @@
"""Definition of JsonRepositoryInterface abstract base class.""" """Definition of JsonRepositoryInterface abstract base class."""
from collections.abc import Iterator
from abc import ABC, abstractmethod from abc import ABC, abstractmethod
@@ -25,3 +26,13 @@ class JsonRepositoryInterface(ABC):
def list_keys(self, pattern: str) -> list[str]: def list_keys(self, pattern: str) -> list[str]:
"""List keys matching a glob pattern.""" """List keys matching a glob pattern."""
... ...
def scan_keys(
self,
pattern: str,
*,
count: int | None = None,
) -> Iterator[str]:
"""Yield keys matching a glob pattern incrementally."""
del count
yield from self.list_keys(pattern)
+31 -2
View File
@@ -2,23 +2,52 @@
import logging import logging
from collections.abc import Generator from collections.abc import Generator
from typing import Any, cast
import pytest import pytest
import redis import redis
import structlog import structlog
from minio import Minio from minio import Minio
from testcontainers.core.container import DockerContainer
from testcontainers.core.waiting_utils import WaitStrategy, WaitStrategyTarget
from testcontainers.minio import MinioContainer from testcontainers.minio import MinioContainer
from python_repositories.config import MinioConfig, RedisConfig from python_repositories.config import MinioConfig, RedisConfig
from tests.integration.redis_container_test import REDIS_PORT, RedisTestContainer
collect_ignore = ["redis_container_test.py"] REDIS_PORT = 6379
MINIO_ACCESS_KEY = "minioadmin" MINIO_ACCESS_KEY = "minioadmin"
MINIO_SECRET_KEY = "minioadmin" MINIO_SECRET_KEY = "minioadmin"
MINIO_BUCKET = "test-bucket" MINIO_BUCKET = "test-bucket"
class _RedisPingWaitStrategy(WaitStrategy):
def __init__(self) -> None:
super().__init__()
self.with_transient_exceptions(redis.exceptions.ConnectionError)
def wait_until_ready(self, container: WaitStrategyTarget) -> None:
redis_container = cast("RedisTestContainer", container)
if not self._poll(lambda: redis_container.get_client().ping()):
raise redis.exceptions.ConnectionError("Could not connect to Redis")
class RedisTestContainer(DockerContainer):
"""Redis container using wait strategies instead of the deprecated decorator."""
def __init__(self, image: str, port: int = REDIS_PORT) -> None:
super().__init__(image, _wait_strategy=_RedisPingWaitStrategy())
self.port = port
self.with_exposed_ports(self.port)
def get_client(self, **kwargs: Any) -> redis.Redis:
return redis.Redis(
host=self.get_container_host_ip(),
port=self.get_exposed_port(self.port),
**kwargs,
)
@pytest.fixture(scope="session", autouse=True) @pytest.fixture(scope="session", autouse=True)
def configure_logging() -> None: def configure_logging() -> None:
"""Configure logging for the test session.""" """Configure logging for the test session."""
+3
View File
@@ -6,6 +6,7 @@ import os
import random import random
import pytest import pytest
from minio import Minio
from python_repositories.examples.artifact_object_repository import ( from python_repositories.examples.artifact_object_repository import (
ArtifactObjectRepository, ArtifactObjectRepository,
@@ -19,8 +20,10 @@ pytestmark = pytest.mark.integration
def set_example_env( def set_example_env(
redis_container: str, redis_container: str,
minio_container: dict[str, str], minio_container: dict[str, str],
raw_minio_client: Minio,
) -> Generator[None, None, None]: ) -> Generator[None, None, None]:
"""Set env vars so example repositories can use from_env() defaults.""" """Set env vars so example repositories can use from_env() defaults."""
_ = raw_minio_client
env_vars = {"REDIS_URI": redis_container, **minio_container} env_vars = {"REDIS_URI": redis_container, **minio_container}
for key, value in env_vars.items(): for key, value in env_vars.items():
os.environ[key] = value os.environ[key] = value
+24 -3
View File
@@ -1,6 +1,7 @@
"""Integration tests for the MinioAdapter.""" """Integration tests for the MinioAdapter."""
from collections.abc import Generator from collections.abc import Generator
from dataclasses import replace
import logging import logging
import random import random
from io import BytesIO from io import BytesIO
@@ -118,17 +119,37 @@ def test_should_raise_connection_error_when_unable_to_connect() -> None:
assert not adapter.is_connected() assert not adapter.is_connected()
def test_should_log_info_when_creating_expected_bucket( def test_connect_raises_when_bucket_missing(
raw_minio_client: Minio,
minio_config: MinioConfig,
) -> None:
"""Test that connect fails when the configured bucket is missing."""
bucket_name = minio_config.bucket
raw_minio_client.remove_bucket(bucket_name)
adapter = MinioAdapter(config=minio_config)
try:
with pytest.raises(ConnectionError, match="does not exist"):
adapter.connect()
assert not adapter.is_connected()
finally:
raw_minio_client.make_bucket(bucket_name)
def test_connect_creates_bucket_when_create_bucket_if_missing_enabled(
raw_minio_client: Minio, raw_minio_client: Minio,
minio_config: MinioConfig, minio_config: MinioConfig,
caplog: pytest.LogCaptureFixture, caplog: pytest.LogCaptureFixture,
) -> None: ) -> None:
"""Test that the MinioAdapter logs info when creating the expected bucket.""" """Test that connect can create the configured bucket when enabled."""
bucket_name = minio_config.bucket bucket_name = minio_config.bucket
raw_minio_client.remove_bucket(bucket_name) raw_minio_client.remove_bucket(bucket_name)
adapter = MinioAdapter(config=minio_config) config = replace(minio_config, create_bucket_if_missing=True)
adapter = MinioAdapter(config=config)
with caplog.at_level(logging.INFO): with caplog.at_level(logging.INFO):
adapter.connect() adapter.connect()
assert f"Creating bucket '{bucket_name}'" in caplog.text assert f"Creating bucket '{bucket_name}'" in caplog.text
+40
View File
@@ -1,6 +1,7 @@
"""Integration tests for the RedisAdapter.""" """Integration tests for the RedisAdapter."""
from collections.abc import Generator from collections.abc import Generator
import logging
import pytest import pytest
import redis import redis
@@ -48,6 +49,16 @@ def clear_redis(raw_redis_client: redis.Redis) -> None:
raw_redis_client.flushall() raw_redis_client.flushall()
def test_should_log_info_when_already_connected(
redis_adapter: RedisAdapter,
caplog: pytest.LogCaptureFixture,
) -> None:
"""Test that the RedisAdapter logs info when connect is called while already connected."""
with caplog.at_level(logging.INFO):
redis_adapter.connect()
assert "Already connected to Redis" in caplog.text
def test_should_raise_connection_error_when_unable_to_connect() -> None: def test_should_raise_connection_error_when_unable_to_connect() -> None:
"""Test that the RedisAdapter raises ConnectionError when unable to connect.""" """Test that the RedisAdapter raises ConnectionError when unable to connect."""
adapter = RedisAdapter(config=RedisConfig(uri="redis://invalid:6379")) adapter = RedisAdapter(config=RedisConfig(uri="redis://invalid:6379"))
@@ -254,5 +265,34 @@ def test_should_raise_connection_error_on_list_keys_when_not_connected(
adapter.list_keys("some_pattern") adapter.list_keys("some_pattern")
def test_should_scan_keys(
redis_adapter: RedisAdapter,
) -> None:
"""Test scanning keys matching a pattern returns correct keys."""
redis_adapter.set("key1", {"a": 1})
redis_adapter.set("key2", {"b": 2})
keys = set(redis_adapter.scan_keys("key*"))
assert keys == {"key1", "key2"}
def test_should_raise_value_error_on_invalid_scan_keys_pattern(
redis_adapter: RedisAdapter,
) -> None:
"""Test that the RedisAdapter raises ValueError when scanning keys with an invalid pattern."""
invalid_patterns = ["", 123, None]
for pattern in invalid_patterns:
with pytest.raises(ValueError):
list(redis_adapter.scan_keys(pattern)) # type: ignore[arg-type]
def test_should_raise_connection_error_on_scan_keys_when_not_connected(
redis_config: RedisConfig,
) -> None:
"""Test that the RedisAdapter raises ConnectionError when scanning keys while not connected."""
adapter = RedisAdapter(config=redis_config)
with pytest.raises(ConnectionError):
list(adapter.scan_keys("some_pattern"))
if __name__ == "__main__": if __name__ == "__main__":
pytest.main(["-s", "-v", __file__]) pytest.main(["-s", "-v", __file__])
-36
View File
@@ -1,36 +0,0 @@
"""Redis test container without testcontainers' deprecated wait decorator."""
from typing import Any, cast
import redis
from testcontainers.core.container import DockerContainer
from testcontainers.core.waiting_utils import WaitStrategy, WaitStrategyTarget
REDIS_PORT = 6379
class _RedisPingWaitStrategy(WaitStrategy):
def __init__(self) -> None:
super().__init__()
self.with_transient_exceptions(redis.exceptions.ConnectionError)
def wait_until_ready(self, container: WaitStrategyTarget) -> None:
redis_container = cast("RedisTestContainer", container)
if not self._poll(lambda: redis_container.get_client().ping()):
raise redis.exceptions.ConnectionError("Could not connect to Redis")
class RedisTestContainer(DockerContainer):
"""Redis container using wait strategies instead of the deprecated decorator."""
def __init__(self, image: str, port: int = REDIS_PORT) -> None:
super().__init__(image, _wait_strategy=_RedisPingWaitStrategy())
self.port = port
self.with_exposed_ports(self.port)
def get_client(self, **kwargs: Any) -> redis.Redis:
return redis.Redis(
host=self.get_container_host_ip(),
port=self.get_exposed_port(self.port),
**kwargs,
)
+43
View File
@@ -0,0 +1,43 @@
"""Unit tests for env_bool."""
from __future__ import annotations
import pytest
from python_repositories.config.env_bool import env_bool
@pytest.mark.parametrize(
("value", "expected"),
[
("true", True),
("1", True),
("yes", True),
("on", True),
("false", False),
("0", False),
("no", False),
("off", False),
],
)
def test_env_bool_parses_truthy_and_falsy(
monkeypatch: pytest.MonkeyPatch,
value: str,
expected: bool,
) -> None:
monkeypatch.setenv("TEST_BOOL", value)
assert env_bool("TEST_BOOL", default=not expected) is expected
def test_env_bool_returns_default_when_unset(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.delenv("TEST_BOOL", raising=False)
assert env_bool("TEST_BOOL", default=True) is True
assert env_bool("TEST_BOOL", default=False) is False
def test_env_bool_raises_for_invalid_value(
monkeypatch: pytest.MonkeyPatch,
) -> None:
monkeypatch.setenv("TEST_BOOL", "not-a-bool")
with pytest.raises(ValueError, match="TEST_BOOL"):
env_bool("TEST_BOOL", default=False)
@@ -80,3 +80,24 @@ def test_instantiation_fails_when_list_keys_not_implemented() -> None:
with pytest.raises(TypeError): with pytest.raises(TypeError):
_ = Incomplete() # type: ignore _ = Incomplete() # type: ignore
def test_scan_keys_defaults_to_list_keys() -> None:
"""Test that the default scan_keys implementation delegates to list_keys."""
class Complete(JsonRepositoryInterface):
def get(self, key: str) -> dict | None:
return None
def set(self, key: str, data: dict) -> None:
pass
def delete(self, key: str) -> None:
pass
def list_keys(self, pattern: str) -> list[str]:
return [f"{pattern}-1", f"{pattern}-2"]
repository = Complete()
assert list(repository.scan_keys("user")) == ["user-1", "user-2"]
+70
View File
@@ -2,6 +2,7 @@
from __future__ import annotations from __future__ import annotations
from dataclasses import replace
from unittest.mock import MagicMock from unittest.mock import MagicMock
import pytest import pytest
@@ -74,6 +75,16 @@ def test_connect_with_injected_client_raises_on_failure() -> None:
adapter.connect() adapter.connect()
def test_connect_with_injected_client_skips_validation_when_client_cleared() -> None:
mock_client = MagicMock(spec=Minio)
adapter = MinioAdapter(config=TEST_MINIO_CONFIG, client=mock_client)
adapter.disconnect()
adapter.connect()
mock_client.list_buckets.assert_not_called()
def test_connect_disconnects_before_reconnect( def test_connect_disconnects_before_reconnect(
monkeypatch: pytest.MonkeyPatch, monkeypatch: pytest.MonkeyPatch,
) -> None: ) -> None:
@@ -94,6 +105,65 @@ def test_connect_disconnects_before_reconnect(
new_client.list_buckets.assert_called_once() new_client.list_buckets.assert_called_once()
def test_connect_skips_reconnect_when_already_connected(
monkeypatch: pytest.MonkeyPatch,
) -> None:
stale_client = MagicMock(spec=Minio)
stale_client.bucket_exists.return_value = True
adapter = MinioAdapter(config=TEST_MINIO_CONFIG)
adapter._client = stale_client
adapter._bucket_name = TEST_MINIO_CONFIG.bucket
minio_ctor = MagicMock()
monkeypatch.setattr(
"python_repositories.adapters.minio_adapter.minio.Minio",
minio_ctor,
)
adapter.connect()
minio_ctor.assert_not_called()
assert adapter._client is stale_client
def test_connect_raises_when_bucket_missing(
monkeypatch: pytest.MonkeyPatch,
) -> None:
mock_client = MagicMock(spec=Minio)
mock_client.bucket_exists.return_value = False
monkeypatch.setattr(
"python_repositories.adapters.minio_adapter.minio.Minio",
lambda *args, **kwargs: mock_client,
)
adapter = MinioAdapter(config=TEST_MINIO_CONFIG)
with pytest.raises(ConnectionError, match="does not exist"):
adapter.connect()
mock_client.make_bucket.assert_not_called()
def test_connect_creates_bucket_when_create_bucket_if_missing_enabled(
monkeypatch: pytest.MonkeyPatch,
) -> None:
mock_client = MagicMock(spec=Minio)
mock_client.bucket_exists.return_value = False
config = replace(TEST_MINIO_CONFIG, create_bucket_if_missing=True)
monkeypatch.setattr(
"python_repositories.adapters.minio_adapter.minio.Minio",
lambda *args, **kwargs: mock_client,
)
adapter = MinioAdapter(config=config)
adapter.connect()
mock_client.bucket_exists.assert_called_once_with(config.bucket)
mock_client.make_bucket.assert_called_once_with(config.bucket)
def test_get_closes_response_on_success() -> None: def test_get_closes_response_on_success() -> None:
"""get() must close and release the get_object HTTP response.""" """get() must close and release the get_object HTTP response."""
mock_client = MagicMock(spec=Minio) mock_client = MagicMock(spec=Minio)
+16 -21
View File
@@ -22,6 +22,7 @@ def test_from_env_loads_all_fields(monkeypatch: pytest.MonkeyPatch) -> None:
assert config.secret_key == "secret" assert config.secret_key == "secret"
assert config.bucket == "my-bucket" assert config.bucket == "my-bucket"
assert config.secure is True assert config.secure is True
assert config.create_bucket_if_missing is False
def test_from_env_defaults_secure_to_true_when_unset( def test_from_env_defaults_secure_to_true_when_unset(
@@ -33,33 +34,27 @@ def test_from_env_defaults_secure_to_true_when_unset(
assert config.secure is True assert config.secure is True
@pytest.mark.parametrize( def test_from_env_reads_secure_from_env(monkeypatch: pytest.MonkeyPatch) -> None:
("value", "expected"), _set_required_minio_env(monkeypatch)
[ monkeypatch.setenv("MINIO_SECURE", "false")
("true", True), assert MinioConfig.from_env(use_dotenv=False).secure is False
("1", True),
("yes", True),
("false", False), def test_from_env_reads_create_bucket_if_missing_from_env(
("0", False),
("no", False),
],
)
def test_from_env_parses_secure(
monkeypatch: pytest.MonkeyPatch, monkeypatch: pytest.MonkeyPatch,
value: str,
expected: bool,
) -> None: ) -> None:
_set_required_minio_env(monkeypatch) _set_required_minio_env(monkeypatch)
monkeypatch.setenv("MINIO_SECURE", value) monkeypatch.setenv("MINIO_CREATE_BUCKET_IF_MISSING", "true")
config = MinioConfig.from_env(use_dotenv=False) assert MinioConfig.from_env(use_dotenv=False).create_bucket_if_missing is True
assert config.secure is expected
def test_from_env_raises_for_invalid_secure(monkeypatch: pytest.MonkeyPatch) -> None: def test_from_env_defaults_create_bucket_if_missing_to_false_when_unset(
monkeypatch: pytest.MonkeyPatch,
) -> None:
_set_required_minio_env(monkeypatch) _set_required_minio_env(monkeypatch)
monkeypatch.setenv("MINIO_SECURE", "not-a-bool") monkeypatch.delenv("MINIO_CREATE_BUCKET_IF_MISSING", raising=False)
with pytest.raises(ValueError, match="MINIO_SECURE"): config = MinioConfig.from_env(use_dotenv=False)
MinioConfig.from_env(use_dotenv=False) assert config.create_bucket_if_missing is False
def test_from_env_raises_when_endpoint_missing( def test_from_env_raises_when_endpoint_missing(
+80 -1
View File
@@ -99,10 +99,21 @@ def test_connect_with_injected_client_raises_on_redis_error() -> None:
adapter.connect() adapter.connect()
def test_connect_closes_existing_non_injected_client( def test_connect_with_injected_client_skips_validation_when_client_cleared() -> None:
mock_client = MagicMock(spec=redis.Redis)
adapter = RedisAdapter(config=TEST_REDIS_CONFIG, client=mock_client)
adapter.disconnect()
adapter.connect()
mock_client.ping.assert_not_called()
def test_connect_reconnects_when_existing_client_unhealthy(
monkeypatch: pytest.MonkeyPatch, monkeypatch: pytest.MonkeyPatch,
) -> None: ) -> None:
stale_client = MagicMock(spec=redis.Redis) stale_client = MagicMock(spec=redis.Redis)
stale_client.ping.side_effect = redis.ConnectionError("connection lost")
new_client = MagicMock(spec=redis.Redis) new_client = MagicMock(spec=redis.Redis)
new_client.ping.return_value = True new_client.ping.return_value = True
adapter = RedisAdapter(config=TEST_REDIS_CONFIG) adapter = RedisAdapter(config=TEST_REDIS_CONFIG)
@@ -114,3 +125,71 @@ def test_connect_closes_existing_non_injected_client(
stale_client.close.assert_called_once() stale_client.close.assert_called_once()
assert adapter._client is new_client assert adapter._client is new_client
def test_connect_skips_reconnect_when_already_connected(
monkeypatch: pytest.MonkeyPatch,
) -> None:
stale_client = MagicMock(spec=redis.Redis)
stale_client.ping.return_value = True
adapter = RedisAdapter(config=TEST_REDIS_CONFIG)
adapter._client = stale_client
from_url = MagicMock()
monkeypatch.setattr("redis.Redis.from_url", from_url)
adapter.connect()
stale_client.close.assert_not_called()
from_url.assert_not_called()
assert adapter._client is stale_client
def test_scan_keys_yields_decoded_keys() -> None:
mock_client = MagicMock(spec=redis.Redis)
mock_client.scan_iter.return_value = iter([b"key1", b"key2"])
adapter = RedisAdapter(config=TEST_REDIS_CONFIG, client=mock_client)
keys = list(adapter.scan_keys("key*"))
assert keys == ["key1", "key2"]
mock_client.scan_iter.assert_called_once_with(match="key*")
mock_client.keys.assert_not_called()
def test_scan_keys_forwards_count() -> None:
mock_client = MagicMock(spec=redis.Redis)
mock_client.scan_iter.return_value = iter([b"key1"])
adapter = RedisAdapter(config=TEST_REDIS_CONFIG, client=mock_client)
keys = list(adapter.scan_keys("key*", count=50))
assert keys == ["key1"]
mock_client.scan_iter.assert_called_once_with(match="key*", count=50)
def test_list_keys_raises_value_error_on_invalid_pattern() -> None:
mock_client = MagicMock(spec=redis.Redis)
adapter = RedisAdapter(config=TEST_REDIS_CONFIG, client=mock_client)
invalid_patterns = ["", 123, None]
for pattern in invalid_patterns:
with pytest.raises(ValueError, match="Pattern must be a non-empty string"):
adapter.list_keys(pattern) # type: ignore[arg-type]
def test_scan_keys_raises_value_error_on_invalid_pattern() -> None:
mock_client = MagicMock(spec=redis.Redis)
adapter = RedisAdapter(config=TEST_REDIS_CONFIG, client=mock_client)
invalid_patterns = ["", 123, None]
for pattern in invalid_patterns:
with pytest.raises(ValueError, match="Pattern must be a non-empty string"):
list(adapter.scan_keys(pattern)) # type: ignore[arg-type]
def test_scan_keys_raises_connection_error_when_not_connected() -> None:
adapter = RedisAdapter(config=TEST_REDIS_CONFIG)
with pytest.raises(ConnectionError):
list(adapter.scan_keys("key*"))
Generated
+1 -1
View File
@@ -1056,7 +1056,7 @@ wheels = [
[[package]] [[package]]
name = "python-repositories" name = "python-repositories"
version = "1.0.0" version = "2.0.1"
source = { editable = "." } source = { editable = "." }
dependencies = [ dependencies = [
{ name = "python-dotenv" }, { name = "python-dotenv" },