Chunked storage, tagged observations and verified append#

These provisional additions after 0.28.1 extend QuantityRecord without changing existing qrec/0.3 or bundle bytes. They do not migrate a consumer’s files or establish a stable 1.0 contract. First public availability belongs to the corresponding release notes.

Operation

Supported contract

Zarr

Homogeneous snapshots, complete or contiguous first-axis reads

Tagged layout

Equal-shaped source quantities with local unit/kind descriptors

Append

A new homogeneous in-memory snapshot, extending axis zero

Arrow/Parquet

Complete homogeneous or tagged records in one typed table/file

Optional installation and import#

The qualified optional development profile selects Zarr 3.1.6 and PyArrow 25.0.0. Both are published on conda-forge; Zarr 3.1.6 retains Python 3.11 compatibility, whereas newer Zarr releases may have a narrower Python range.

conda install -c conda-forge zarr=3.1.6 pyarrow=25.0.0

Neither provider is a required PyUnitWizard runtime dependency. Importing pyunitwizard.storage.zarr or .arrow imports neither provider. Operations check the selected provider through DepDigest and give an installation diagnostic for absence; transitive import failures remain distinct. Baseline forms, parsers and automatic backend selection are unchanged.

Verified Zarr snapshots and partial reads#

import numpy as np
import zarr
import pyunitwizard as puw
from pyunitwizard.record import QuantityRecord
from pyunitwizard.storage import zarr as storage

record = QuantityRecord.from_quantity(
    puw.quantity(np.zeros((100, 22, 3), dtype=np.float32), 'nanometer', form='pint'),
    field='coordinates',
)
root = zarr.open_group('trajectory.zarr', mode='a')
storage.write(root, 'coordinates', record, chunk_rows=16)
saved = storage.read(root['coordinates'], selection=slice(20, 30),
                     field='coordinates', unit='nanometer')
coordinates = saved.to_quantity(unit='angstrom', form='pint')

qrec-zarr/0.1 stores one numeric array with the original sealed qrec metadata, an ordered table of independently sealed chunks and a binding digest covering their association. The in-group units attribute must match the authoritative manifest; array-level secondary metadata is refused. Values, dtype, shape, physical chunk shape and available chunk inventory are checked. Zero chunks are written explicitly so a missing object cannot silently become default fill.

A reader must declare an expected unit or dimensionality; field and kind provide additional checks. The returned inert record retains its stored unit. Conversion is a separate explicit operation. Scalars, empty arrays, NaN/Inf, signed zero and the codec’s numeric dtypes are supported.

Partial reads accept unit-stride slice selections on axis zero. They validate the binding and inventory, then read and verify each complete intersecting chunk. They return a newly sealed subset, not a certificate of unread chunk contents. Metadata/inventory work still scales with chunk count. A full read verifies all chunks and the original qrec seal. An empty selection still checks the unit descriptor and reader expectations.

Writes require a new direct child and never overwrite existing data. The final authoritative attribute is written only after the array is complete; a caught failure removes that newly owned child. A reader refuses an incomplete child. The caller owns the store, permissions, lifecycle and concurrency. This route targets Zarr format 3 without shards and uses provider-default compression; remote object-store behavior, concurrent writers, crash recovery, tagged arrays, and in-place/streaming append are outside this qualification.

One class, a tagged layout#

observations = QuantityRecord.from_tagged(
    [puw.quantity(15000.0, 'nM', form='pint'),
     puw.quantity(50.0, 'percent', form='pint')],
    field='activities', kinds=['Ki', 'Inhibition'],
)
payload = observations.to_dict(encoding='base64')
restored = QuantityRecord.from_dict(payload)
groups = restored.split()
potency = groups[0].to_quantity(field='activities', kind='Ki', unit='uM', form='pint')

qrec-tagged/0.1 identifies this layout explicitly. Its manifest contains layout: tagged, field, dtype, shape and a local table of canonical unit/SI descriptions with optional application-owned kinds. Read-only uint16 codes identify one descriptor per value. Each input scalar or equal-shaped array contributes an outer-axis entry; codes broadcast across that input’s values. The writer deduplicates descriptors in first-occurrence order. At most 65,536 descriptors are supported. Original magnitudes stay unchanged. Mixed source dtypes promote to one common supported dtype only if their values survive round-trip casting exactly; unsafe integer-to-float promotion is refused.

The seal binds manifest, descriptor table, numeric bytes and code bytes. JSON codes follow the value shape; base64 encodes numeric values as in homogeneous records. Descriptors are detached copies, and values/codes have immutable backing. Wire shapes contain explicit nonnegative integer sizes; inferred negative sizes are refused even when an external writer recomputes the seal. Existing homogeneous writers still emit exactly qrec/0.3. Older readers may refuse the new identifier; they must not reinterpret it as the older format.

For computation, call to_quantity(unit=...) with an explicit common unit. Every descriptor must have compatible dimensions and the same declared kind. Otherwise use split(), which returns flattened homogeneous records keyed by descriptor code; codes == key retains their original-position association. Kinds are source declarations, not inferred ontology mappings. Source relation, uncertainty, identifiers and domain validity remain consumer responsibilities.

For ordinary independent fields, a bundle remains useful. Tagged records serve a heterogeneous collection; they are not a new live arithmetic backend. Direct core unit introspection/implicit conversion of a heterogeneous record is refused; use its explicit to_quantity/split route. Existing bundle, HDF5 and Zarr bindings remain homogeneous rather than silently acquiring tagged semantics.

Append without mutating a published snapshot#

first = QuantityRecord.from_quantity(
    puw.quantity(np.zeros((2, 22, 3), dtype=np.float32), 'nanometer', form='pint'),
    field='coordinates',
)
later = puw.quantity(np.ones((3, 22, 3), dtype=np.float32), 'angstrom', form='pint')
combined = first.append(later)
assert first.values.shape == (2, 22, 3)
assert combined.values.shape == (5, 22, 3)

The homogeneous qrec/0.3 manifest already defines ordered blocks. append converts incoming rows to the receiver’s unit, checks exact source dtype and trailing shape, and seals a new block and root. A record source must match field and kind; a computing quantity is explicitly bound to the receiver’s context. Integer conversions that lose values/range and finite floating-point overflow are refused. Affine conversion uses the actual unit provider. An empty append is a validated no-op.

The old snapshot and block digests remain unchanged on success or failure. Only new blocks need hashing, but immutable array concatenation still copies O(total values) bytes; this is not constant-cost streaming. Scalars and tagged records are refused. This codec operation does not add an append method to HDF5/Zarr files or promise disk transactions.

Arrow tables and Parquet files#

from pyunitwizard.storage import arrow

table = arrow.to_table(record)
saved = arrow.from_table(table, field='coordinates', unit='nanometer')
arrow.write('coordinates.parquet', record)
saved = arrow.read('coordinates.parquet', field='coordinates', unit='nanometer')

table = arrow.to_table(observations)
saved = arrow.from_table(table, field='activities',
                         units=['nM', 'percent'], kinds=['Ki', 'Inhibition'])

qrec-arrow/0.1 stores flattened typed values and, for tagged records, uint16 codes columns. Original shape, descriptors and codec seals travel in one authoritative schema-metadata envelope with its own binding digest. The code column is checked against the tagged seal; its data is not duplicated inside the Arrow envelope. Extra/reordered columns and secondary field metadata are refused. Removing metadata, projecting a column or editing values/codes does not yield a verified record.

Homogeneous readers require unit or dimensionality. Tagged readers require units in descriptor-code order and may declare kinds in that same order. The returned record retains its original values/units; these arguments are a handshake, not a hidden numerical normalization.

Nulls are refused because current qrec has no missing-value bitmap. Numeric NaN/Inf remain numeric values and never become nulls. NumPy dtype, shape and signed zero are preserved, including scalar and empty records. This route requires complete tables/files; it does not claim verified projection, row-group selection, partitioned datasets, dataframe schema inference or zero-copy verification.

Parquet uses the actual PyArrow writer with stored Arrow schema metadata. Local publication writes an owned same-directory temporary file, then creates the final path exclusively without overwriting existing data; staging is removed after success/failure. This relies on local hard-link semantics and does not qualify remote/cloud filesystems or crash-durability guarantees.

Evidence and scientific boundaries#

Consumer-derived tests use retained H5MSM coordinates and 343 ChEMBL numerical observations from Sabueso fixtures, with original file hashes and source commits. They exercise non-default session policy. Source relation and assay context stay in the fixture provenance; a record seal does not validate an assay or turn a bound into an exact measurement. Provider tests do not migrate those clients.

devtools/benchmark_record_extensions.py records raw operation times, input provenance, source hashes and bytes. Its tiled coordinate stress case repeats one real frame and is explicitly synthetic. Compare the same operation/scope; results are not a universal speed or compression claim. Optional installed matrix evidence and public package qualification remain separate from local development measurements. Digests detect inconsistency, not authenticity or a source that is already scientifically wrong.