Path B - Module 48: Scalability & Heavy Trajectories#

High-throughput enzyme engineering generates massive amounts of data. If you have a 100GB trajectory of a PETase variant, you cannot load it with msm.convert() because it will fill your RAM and crash your computer.

In this module, you will learn to use Iterators to process any file size by reading data in small chunks.

import molsysmt as msm
from molsysmt import systems

# Even with small demo files, the Scalability API follows a specific pattern
villin_traj = systems['chicken villin HP35']['traj_chicken_villin_HP35_solvated.dcd']
villin_topo = systems['chicken villin HP35']['traj_chicken_villin_HP35_solvated.h5msm']

1. The Iterator Interface#

An Iterator acts as a stream. You define what you want to extract, and then you loop over the chunks.

# Create an iterator for the trajectory, reading 10 structures at a time
it = msm.Iterator([villin_topo, villin_traj], selection='molecule_type=="peptide"', 
                  chunk=10, coordinates=True)

for chunk_idx, coordinates in enumerate(it):
    print(f"Processing chunk {chunk_idx}. Data size in memory: {coordinates.nbytes / 1024:.1f} KB")
    # Perform your analysis on each chunk
    if chunk_idx == 4: break # Stop early for the demo

2. High-Performance Mapping#

Instead of writing the for loop yourself, you can use the map() method of the iterator to apply a function to every chunk automatically.

def analyze_stability(molsys):
    # This function will be applied to every chunk
    return msm.structure.get_radius_of_gyration(molsys, heavy_mode='off')

# Re-initialize the iterator and apply the function to each chunk
rg_results = []
structure_start = 0
for coordinates in msm.Iterator(
    [villin_topo, villin_traj], chunk=25, coordinates=True
):
    structure_indices = range(structure_start, structure_start + len(coordinates))
    chunk = msm.extract(
        [villin_topo, villin_traj], structure_indices=structure_indices
    )
    rg_results.extend(analyze_stability(chunk))
    structure_start += len(coordinates)

print(f"Analyzed stability for {len(rg_results)} total structures.")

3. Constant Memory Footprint#

By using this pattern, you can process 1,000,000 structures using the same amount of RAM as you would for 10 structures. This is the hallmark of a professional industrial pipeline.

🏆 Path B Challenge: The Heavy Auditor#

  1. Create an Iterator for the PETase trajectory with a chunk of 5.

  2. Inside a loop, calculate the Minimum Distance between the protein and the substrate for every chunk.

  3. Keep track of the global minimum distance found across all chunks.

  4. Compare the time to initialize an Iterator vs the time to convert() the whole system.