Pytest diagnostic policy#
Pytest Receptor owns the pytest result and the canonical JSONL artifact. SMonitor
adds bounded diagnostic extension records. smonitor pytest-check is an
optional, separate QA gate over a completed artifact; it does not modify
pytest’s result, install a reporter, or upload evidence.
Install smonitor[pytest], create the artifact directory, then run pytest:
mkdir -p .pytest-receptor
python -m pytest --receptor=ci --receptor-events=.pytest-receptor/events.jsonl
smonitor pytest-check .pytest-receptor/events.jsonl \
--provider-root ./my_library \
--forbid-code MYLIB-DEPRECATED-001
--provider-root is a trusted package directory containing _smonitor.py.
SMonitor validates its CODES and SIGNALS declarations with the same
validator used for provider registration, and checks that emitted codes are
declared there. The configuration file is Python code, so inspect untrusted
repositories before loading it. The check does not claim to validate every
runtime template substitution; test each catalog-backed diagnostic with its
representative structured fields.
The command prints a versioned JSON result. Exit status is 0 when policy
passes, 1 for policy violations, and 2 for unreadable artifacts, invalid
rules, or missing optional dependencies. pytest_outcome is reported separately:
CI must still check pytest’s own exit status. The policy never changes it.
Accepted identity baseline#
After reviewing a representative passing run, create a baseline once:
smonitor pytest-baseline .pytest-receptor/events.jsonl \
--out tests/accepted_diagnostics.json
Then compare later runs, including xdist runs:
smonitor pytest-check .pytest-receptor/events.jsonl \
--baseline tests/accepted_diagnostics.json \
--provider-root ./my_library
The baseline is a sorted set of (code, source, fingerprint) for SMonitor
warning, error, and critical records. New identities fail the gate; disappeared
ones are reported for review. Repeated occurrences, worker IDs, test nodes and
phase assignments do not affect it. This is important when worker-local caches
change warning counts under xdist. Baselines cannot be created from a failed,
partial, truncated, or extension-incomplete run. Empty baselines require
--allow-empty; they give no proof that the SMonitor plugin was active.
SMonitor’s fingerprint also includes selected structured fields such as
caller, resource, and retry_attempt. A baseline is reproducible only when
those fields are stable for the covered workflow. Review a representative run
before accepting it, and use code-level forbidden rules for a diagnostic whose
resource or retry context legitimately varies.
Use --require-diagnostics on pytest-check when a smoke test must emit at
least one SMonitor record. Without that explicit witness, an artifact containing
zero diagnostics cannot distinguish a clean run from a missing producer.
An artifact with invalid integrity, incomplete execution, truncated evidence,
or dropped/incomplete extensions fails the gate because missing records cannot
prove a forbidden identity was absent. The receptor’s read_artifact validates
its digest before policy evaluation. The policy uses only uibcdf.smonitor@1
records with the smonitor.pytest-event@1 schema; other producers are ignored.
Test assertions and support bundles#
Use pytest.warns, pytest.raises, and SMonitor’s existing structured tests
for assertions inside individual tests. The receptor already associates
extensions with the node, phase, worker and attempt. A new SMonitor fixture or
marker would duplicate those contracts, so this policy adds neither.
The standalone SMonitor support bundle is optional and has different retention and redaction settings from the receptor artifact. Do not insert it into the canonical JSONL stream or infer that the two files cover the same history. If a workflow publishes both, its link manifest must name each relative artifact, record a SHA-256 for each complete file, identify the receptor schema and the SMonitor bundle profile/schema, state receptor integrity and execution completeness, and state the bundle’s redaction and coverage limits. The rich bundle currently has no versioned schema or guaranteed complete history; a generic automatic link would overstate its reliability. SMonitor therefore does not publish one until a consumer demonstrates a concrete paired-bundle workflow.