Module 11: Iterating Systems#

Welcome back, Apprentice Master. In Module 10: Modifying Molecular Attributes, you learned how to modify attributes in place and work with modifiable forms. Now we turn to memory-efficient streaming: Iterating Systems using msm.Iterator().

Extracting all attributes at once with msm.get() is fast for small systems, but when analyzing multi-gigabyte trajectories or processing thousands of groups, loading everything into RAM can exhaust system memory. msm.Iterator() allows you to process systems incrementally—element by element, chunk by chunk, or structure by structure—with total memory efficiency.

Learning Outcomes

By the end of this module, you will be able to:

Iterate over element levels (group, molecule, chain) yielding requested attributes.
Process large datasets in batches using the chunk parameter.
Stream multi-structure trajectories structure by structure without loading full files into memory.

1. Element Loops#

Let’s begin by importing MolSysMT and loading our T4 Lysozyme demonstration system.

import molsysmt as msm
from molsysmt import systems

# Load T4 Lysozyme file
lysozyme = systems['T4 lysozyme L99A']['181l.bcif.gz']

You can instantiate msm.Iterator() to loop over any element level (group, molecule, chain). The iterator yields requested attributes for each unit sequentially:

# Iterate over the first 5 groups yielding group_name and atom_index
for name, atoms in msm.Iterator(lysozyme, element='group', selection=[0, 1, 2, 3, 4], group_name=True, atom_index=True):
    print(f"Processing group {name[0]:<3} | Atom count: {len(atoms[0])}")

Processing group MET | Atom count: 8
Processing group ASN | Atom count: 8
Processing group ILE | Atom count: 8
Processing group PHE | Atom count: 11
Processing group GLU | Atom count: 9

Hint

msm.Iterator(): Memory-efficient generator for looping over molecular elements or trajectory structures in memory or on disk. See API doc: molsysmt.basic.Iterator().

2. Chunked Iteration#

For high-throughput calculations, yielding single items can introduce Python loop overhead. You can specify a chunk size to process items in batches:

# Process groups in batches of 50
for names in msm.Iterator(lysozyme, element='group', chunk=50, group_name=True):
    print(f"Batch received containing {len(names)} groups.")

Batch received containing 50 groups.
Batch received containing 50 groups.
Batch received containing 50 groups.
Batch received containing 50 groups.
Batch received containing 50 groups.
Batch received containing 50 groups.
Batch received containing 2 groups.

3. Structure Iteration#

The primary application of msm.Iterator() is streaming trajectory structures sequentially from disk without consuming excessive memory:

# Load Villin Headpiece trajectory file
villin_traj = systems['chicken villin HP35']['traj_chicken_villin_HP35_solvated.dcd']

# Iterate over structures 1 by 1
for struct_idx, coords in enumerate(msm.Iterator(villin_traj, coordinates=True)):
    if struct_idx % 5 == 0:
        print(f"Processing structure {struct_idx:>2} | Coordinates shape: {coords.shape}")

Processing structure  0 | Coordinates shape: (1, 4369, 3)
Processing structure  5 | Coordinates shape: (1, 4369, 3)
Processing structure 10 | Coordinates shape: (1, 4369, 3)
Processing structure 15 | Coordinates shape: (1, 4369, 3)

🏆 Challenge 11: The Efficient Coder#

Load the T4 Lysozyme system (systems['T4 lysozyme L99A']['181l.bcif.gz']).
Use msm.Iterator() to loop over its Chains yielding chain_id and n_groups.
Print the chain ID and group count for each chain.

Incremental iteration gives you the power to process datasets of any scale. In Module 12: System Auditing and Curing, we will explore how to audit structural anomalies and validate molecular systems.