Skip to content

m3flow_analysis

Provider for analysis tasks: trajectories, MSD/diffusion, RDF, densities, and other property extraction from simulation artifacts.

m3flow_analysis.main

m3flow-analysis: analysis/validation tasks (MDAnalysis + freud + numpy).

Implements the analysis/validation/utility tasks of tasks/analysis/. Result artifacts carry their payload both as data (for conditions and downstream references) and as a result.json file (portability).

Statistics conventions
  • equilibration_fraction is always the fraction of samples DISCARDED from the start of a series; tail_fraction is the fraction KEPT at the end. Both are validated, and 0 is a legal value (never replaced by a default).
  • sem is the standard error of the mean of a time series, estimated by Flyvbjerg-Petersen blocking with the Lee-Morales-Umrigar optimal-block criterion (time-correlated samples are not independent). std is the sample standard deviation of the samples. A statistic that cannot be estimated is null with an explicit status — never NaN.
  • Time axes come from artifact metadata (actual sampling intervals); a trajectory without them is rejected rather than given a default.
  • Reduced (lj) inputs produce reduced results unless an explicit physical mapping (sigma, tau, ...) is supplied.

blocking_sem(x)

Standard error of the mean of a correlated series.

Flyvbjerg-Petersen blocking: repeatedly average neighbouring pairs and take the SEM at each level; the Lee-Morales-Umrigar criterion picks the smallest block size B with B^3 > 2 n (sem_B / sem_0)^4. When no level qualifies (series short compared with its correlation time) the largest estimate over levels with >= 4 blocks is returned and converged is False — a conservative, flagged answer instead of an optimistic one.

Source code in providers/src/m3flow_analysis/main.py
def blocking_sem(x):
    """Standard error of the mean of a correlated series.

    Flyvbjerg-Petersen blocking: repeatedly average neighbouring pairs and
    take the SEM at each level; the Lee-Morales-Umrigar criterion picks the
    smallest block size B with B^3 > 2 n (sem_B / sem_0)^4. When no level
    qualifies (series short compared with its correlation time) the largest
    estimate over levels with >= 4 blocks is returned and `converged` is
    False — a conservative, flagged answer instead of an optimistic one.
    """
    x = np.asarray(x, dtype=float)
    n = len(x)
    if n < 2:
        return {"sem": None, "status": "insufficient_data", "n": int(n),
                "method": SEM_METHOD}
    levels = []
    y = x
    while len(y) >= 2:
        levels.append((len(y), float(np.std(y, ddof=1) / np.sqrt(len(y)))))
        half = len(y) // 2
        if half < 2:
            break
        y = 0.5 * (y[0:2 * half:2] + y[1:2 * half:2])
    sem0 = levels[0][1]
    std = float(np.std(x, ddof=1))
    if sem0 == 0.0:
        return {"sem": 0.0, "status": "ok", "converged": True, "block_size": 1,
                "n": int(n), "n_eff": float(n), "method": SEM_METHOD}
    best = None
    for i, (_, sem) in enumerate(levels):
        if (2 ** i) ** 3 > 2 * n * (sem / sem0) ** 4:
            best = i
            break
    if best is not None:
        sem = levels[best][1]
        converged = True
    else:
        usable = [s for nb, s in levels if nb >= 4] or [levels[0][1]]
        sem = max(usable)
        best = int(np.argmax([s for _, s in levels]))
        converged = False
    n_eff = (std / sem) ** 2 if sem > 0 else float(n)
    return {"sem": float(sem), "status": "ok", "converged": converged,
            "block_size": int(2 ** best), "n": int(n),
            "n_eff": float(min(n_eff, n)), "method": SEM_METHOD}

msd_fft(x)

Time-origin-averaged MSD for lags 0..T-1 of x[T, N, 3] (Kneller / Calandrini FFT algorithm; identical to the direct multi-origin average up to round-off, at O(T log T) instead of O(T^2)).

Source code in providers/src/m3flow_analysis/main.py
def msd_fft(x):
    """Time-origin-averaged MSD for lags 0..T-1 of x[T, N, 3] (Kneller /
    Calandrini FFT algorithm; identical to the direct multi-origin average
    up to round-off, at O(T log T) instead of O(T^2))."""
    x = np.asarray(x, dtype=float)
    T = x.shape[0]
    d = np.square(x).sum(axis=-1)                       # T x N
    d = np.concatenate([d, np.zeros((1,) + d.shape[1:])])
    q = 2.0 * d[:T].sum(axis=0)
    s1 = np.empty((T,) + d.shape[1:])
    for m in range(T):
        q = q - d[m - 1] - d[T - m]
        s1[m] = q / (T - m)
    f = np.fft.rfft(x, n=2 * T, axis=0)
    ac = np.fft.irfft(f * f.conj(), axis=0)[:T].sum(axis=-1)   # T x N
    s2 = ac / (T - np.arange(T))[:, None]
    return (s1 - 2.0 * s2).mean(axis=1)

check_polymer_equilibration(req)

Thermodynamic stationarity gate for one sampling window.

Each check is passed / failed / insufficient_data / error; the report is equilibrated only when every required check passed. Missing evidence never counts as a pass. The report is bound to the evidence it examined (artifact ids + content hashes + producing task run) and, when given, to the state it certifies.

Source code in providers/src/m3flow_analysis/main.py
def check_polymer_equilibration(req):
    """Thermodynamic stationarity gate for one sampling window.

    Each check is passed / failed / insufficient_data / error; the report is
    `equilibrated` only when every required check passed. Missing evidence
    never counts as a pass. The report is bound to the evidence it examined
    (artifact ids + content hashes + producing task run) and, when given, to
    the state it certifies.
    """
    p = req["parameters"]
    thermo_in = req["inputs"]["thermo"]
    rows = _read_csv(thermo_in["files"]["csv"])
    drift_threshold = _num(p, "density_drift_threshold", 0.01, lo=0.0, lo_open=True)
    sigma_mult = _num(p, "stationarity_sigma", 3.0, lo=0.0, lo_open=True)
    tail_frac = _keep_fraction(p)

    time, time_unit = _time_axis(rows)
    tt, start = _keep_tail(time, tail_frac)
    checks, details = {}, {}
    metrics = {"stationarity_sigma": sigma_mult,
               "density_drift_threshold": drift_threshold,
               "n_tail_samples": int(len(tt))}

    rho, rho_unit = _series(rows, "density")
    tail = rho[start:]
    mean_rho = float(np.mean(tail))
    metrics["density_mean"] = mean_rho
    metrics["density_unit"] = rho_unit
    if rho_unit == "g_cm3":
        metrics["density_mean_g_cm3"] = mean_rho
    if len(tail) < MIN_TAIL_SAMPLES:
        checks["density_drift"] = INSUFFICIENT
    else:
        slope, _ = np.polyfit(tt, tail, 1)
        total_drift = abs(float(slope)) * float(tt[-1] - tt[0]) / abs(mean_rho) \
            if mean_rho else float("inf")
        metrics["density_relative_drift"] = None if math.isinf(total_drift) else total_drift
        checks["density_drift"] = PASSED if total_drift < drift_threshold else FAILED
    checks["density_stationarity"], details["density_stationarity"] = \
        _shift_check(tail, sigma_mult)

    observables = p.get("observables") or ["energy", "volume"]
    columns = {"energy": "etotal", "potential_energy": "pe", "volume": "vol"}
    for obs in observables:
        if obs not in columns:
            raise _bad(f"unknown observable '{obs}' (use {', '.join(columns)})")
        x, unit = _series(rows, columns[obs], required=False)
        if x is None:
            checks[f"{obs}_trend"] = INSUFFICIENT
            details[f"{obs}_trend"] = {"error": f"no '{columns[obs]}_*' column"}
            continue
        xt = x[start:]
        checks[f"{obs}_trend"], details[f"{obs}_trend"] = _trend_check(tt, xt, sigma_mult)
        checks[f"{obs}_stationarity"], details[f"{obs}_stationarity"] = \
            _shift_check(xt, sigma_mult)
        metrics[f"{obs}_mean"] = float(np.mean(xt))
        metrics[f"{obs}_unit"] = unit

    traj = req["inputs"].get("trajectory")
    if traj:
        checks["rg_stable"], details["rg_stable"] = \
            _structural_rg_check(traj, tail_frac, drift_threshold, p)
        if "rg_relative_drift" in details["rg_stable"]:
            metrics["rg_relative_drift"] = details["rg_stable"]["rg_relative_drift"]

    equilibrated = all(v == PASSED for v in checks.values())
    state_in = req["inputs"].get("state")
    payload = {
        "equilibrated": equilibrated,
        "criteria_version": CRITERIA_VERSION,
        "checks": checks,
        "check_details": details,
        "metrics": metrics,
        "scope": ("stationarity of the listed thermodynamic observables (and Rg "
                  "when a trajectory is given) over this sampling window; not "
                  "evidence of conformational, entanglement or long-time "
                  "dynamical equilibration"),
        "sampling_window": {"time_start": float(tt[0]), "time_end": float(tt[-1]),
                            "time_unit": time_unit,
                            "n_samples": int(len(tt)), "tail_fraction": tail_frac},
        "evidence": {"thermo": _artifact_ref(thermo_in),
                     "trajectory": _artifact_ref(traj)},
        "subject": None if not state_in else {
            "state_id": state_in.get("id"),
            "state_content_hash": state_in.get("content_hash"),
            "state_producer": state_in.get("producer")},
    }
    Path("report.json").write_text(dumps(payload, indent=1))
    out = artifact("EquilibrationReport", files={"json": "report.json"}, data=payload)
    return {"outputs": {"report": out}}

compute_adhesion(req)

Interfacial interaction-energy density -/(n A).

This is the group/group interaction energy per interface area. It is NOT the reversible work of adhesion (no entropy, surface reconstruction or separation path); the payload says which energy terms it contains.

Source code in providers/src/m3flow_analysis/main.py
def compute_adhesion(req):
    """Interfacial interaction-energy density -<E_int>/(n A).

    This is the group/group interaction energy per interface area. It is NOT
    the reversible work of adhesion (no entropy, surface reconstruction or
    separation path); the payload says which energy terms it contains.
    """
    p = req["parameters"]
    thermo_in = req["inputs"]["thermo"]
    meta = thermo_in.get("metadata") or {}
    rows = _read_csv(thermo_in["files"]["csv"])
    e, e_unit = _series(rows, "e_interaction", required=False)
    if e is None:
        cols = list(rows[0].keys()) if rows else []
        raise ProviderFailure(
            "input_invalid", "input_error",
            f"series lacks an e_interaction_<unit> column (has {cols}); "
            "run the MD step with the 'interaction' parameter set")
    frac = _discard_fraction(p)
    tail, skipped = _discard_head(e, frac)
    s = _summary(tail)
    n_if = _int(p, "n_interfaces", 1, lo=1)
    reduced = e_unit == "lj"

    area_q = p.get("interface_area")
    normal = p.get("interface_normal") or "z"
    if normal not in ("x", "y", "z"):
        raise _bad("interface_normal must be x, y or z")
    lat = [a for a in "xyz" if a != normal]
    box_area = None
    la, _ = _series(rows, f"l{lat[0]}", required=False)
    lb, _ = _series(rows, f"l{lat[1]}", required=False)
    if la is not None and lb is not None:
        a_series = (la * lb)[skipped:]
        box_area = {"mean": float(a_series.mean()),
                    "relative_std": float(a_series.std() / a_series.mean())
                    if len(a_series) > 1 else 0.0}
    if area_q is not None:
        area = float(area_q["value"]) if isinstance(area_q, dict) else float(area_q)
        area_source = "parameter"
    elif box_area is not None:
        area = box_area["mean"]
        area_source = f"box l{lat[0]} x l{lat[1]} (tail mean)"
    else:
        raise _bad("interface_area not given and the series has no box lengths")
    if area <= 0:
        raise _bad("interface area must be positive")

    scale = 1.0 if reduced else KCAL_MOL_A2_TO_MJ_M2
    w = -s["mean"] / (n_if * area) * scale
    w_sem = None if s["sem"] is None else s["sem"] / (n_if * area) * scale
    interaction = meta.get("interaction") or {}
    payload = {"value": float(w), "sem": w_sem, "sem_status": s["sem_status"],
               "unit": "epsilon/sigma2" if reduced else "mJ/m2",
               "observable": "interaction_energy_per_area",
               "definition": "-<E_int(group_a, group_b)> / (n_interfaces * area)",
               "note": "interaction-energy density, not the reversible work of adhesion",
               "kspace_included": interaction.get("kspace_included"),
               "energy_terms": interaction.get("observable", "unknown (series lacks metadata)"),
               "interaction_energy": s["mean"], "interaction_energy_sem": s["sem"],
               "interaction_energy_unit": "lj" if reduced else "kcal/mol",
               "n_interfaces": n_if, "interface_area": area,
               "interface_area_source": area_source, "interface_normal": normal,
               "box_area": box_area, "group_charges": interaction.get("group_charges"),
               "n_samples": int(len(tail)), "n_skipped": int(skipped)}
    for comp in ("pair", "kspace"):
        x, _ = _series(rows, f"e_interaction_{comp}", required=False)
        if x is not None:
            payload[f"interaction_energy_{comp}"] = float(np.mean(x[skipped:]))
    if not reduced:
        payload["interaction_energy_kcal_mol"] = s["mean"]
        payload["interface_area_A2"] = area
    warnings = []
    if interaction.get("kspace_included") is None:
        warnings.append("thermo series records no interaction metadata; whether "
                        "long-range (kspace) terms are included is unknown")
    out = {"outputs": {"result": _result(req, "AdhesionResult", payload)}}
    if warnings:
        out["warnings"] = warnings
    return out

compute_species_msd(req)

Species-resolved MSD from an extxyz trajectory (mlip provider format).

v1 supports trajectories written as unwrapped extxyz frame series plus a topology structure file (cif). LAMMPS dcd inputs are rejected with a structured error.

Source code in providers/src/m3flow_analysis/main.py
def compute_species_msd(req):
    """Species-resolved MSD from an extxyz trajectory (mlip provider format).

    v1 supports trajectories written as unwrapped extxyz frame series plus a
    topology structure file (cif). LAMMPS dcd inputs are rejected with a
    structured error.
    """
    from ase.io import read as ase_read

    traj_inp = req["inputs"]["trajectory"]
    meta = traj_inp.get("metadata") or {}
    files = traj_inp["files"]
    fmt = meta.get("format")
    if fmt != "extxyz" or "extxyz" not in files:
        raise ProviderFailure(
            "input_invalid", "input_error",
            "compute_species_msd v1 requires an extxyz Trajectory "
            "(metadata.format == 'extxyz'); got format=%r" % fmt)
    specie = req["parameters"].get("specie")
    if not specie:
        raise ProviderFailure("input_invalid", "input_error",
                              "parameter 'specie' is required")
    max_lag_frac = _num(req["parameters"], "max_lag_fraction", 0.5,
                        lo=0.0, hi=1.0, lo_open=True)
    frame_dt_fs, _ = _trajectory_time(traj_inp)

    frames = ase_read(files["extxyz"], index=":")
    if not frames:
        raise ProviderFailure("engine_crash", "engine_error",
                              "extxyz trajectory has no frames")
    symbols = frames[0].get_chemical_symbols()
    idx = [i for i, s in enumerate(symbols) if s == specie]
    if not idx:
        raise ProviderFailure(
            "input_invalid", "input_error",
            f"specie '{specie}' not present; available: {sorted(set(symbols))}")
    arr = np.array([f.get_positions()[idx] for f in frames], dtype=float)
    lag_frames, msd = _msd_lags(arr, max_lag_frac)
    lags = [float(k * frame_dt_fs) for k in lag_frames]
    msd = [float(m) for m in msd]
    payload = {"lag": lags, "lag_fs": lags, "time_unit": "fs", "msd": msd,
               "unit": "angstrom2", "algorithm": "fft",
               "specie": specie, "n_species_atoms": len(idx),
               "n_frames": int(arr.shape[0])}
    art = _result(req, "MSDResult", payload)
    art["metadata"] = {"specie": specie}
    if meta.get("temperature_K") is not None:
        art["metadata"]["temperature_K"] = float(meta["temperature_K"])
    with open("msd.csv", "w") as f:
        f.write("lag_fs,msd\n")
        for l, m in zip(lags, msd):
            f.write(f"{l:.6g},{m:.8g}\n")
    art["files"]["csv"] = "msd.csv"
    return {"outputs": {"result": art},
            "validation": [verdict("trajectory_readable", True),
                           verdict("no_nan", bool(np.isfinite(msd).all()))]}

fit_arrhenius(req)

Per-temperature Einstein fits over fanned-in MSDResults, then Arrhenius.

Source code in providers/src/m3flow_analysis/main.py
def fit_arrhenius(req):
    """Per-temperature Einstein fits over fanned-in MSDResults, then Arrhenius."""
    series = req["inputs"]["msd_series"]
    if not isinstance(series, list):
        series = [series]
    p = req["parameters"]
    f0 = _num(p, "fit_start_fraction", 0.2, lo=0.0, hi=1.0)
    f1 = _num(p, "fit_end_fraction", 0.8, lo=0.0, hi=1.0)
    min_points = _int(p, "min_points", 4, lo=2)
    KB_EV = 8.617333262e-5  # eV/K

    points = []
    for art in series:
        data = art.get("data") or {}
        meta = art.get("metadata") or {}
        temp = meta.get("temperature_K")
        if temp is None:
            raise ProviderFailure(
                "input_invalid", "input_error",
                "each MSDResult needs data.lag/msd and metadata.temperature_K")
        lag, y, time_unit = _msd_axes(data)
        if data.get("unit") != "angstrom2" or time_unit != "fs":
            raise _bad("fit_arrhenius needs physical MSDs (angstrom2 vs fs)")
        i0, i1 = _fit_window(len(lag), f0, f1)
        slope, _ = np.polyfit(lag[i0:i1], y[i0:i1], 1)
        d_cm2_s = slope / 6.0 * 0.1
        points.append({"temperature_K": float(temp), "D_cm2_s": float(d_cm2_s)})
    points.sort(key=lambda x: x["temperature_K"])
    if len(points) < min_points:
        raise ProviderFailure(
            "convergence_failed", "validation_failed",
            f"need >= {min_points} temperatures, got {len(points)}",
            recoverable=False)
    if any(pt["D_cm2_s"] <= 0 for pt in points):
        raise ProviderFailure(
            "convergence_failed", "validation_failed",
            "non-positive diffusivity in series", recoverable=False)
    inv_t = np.array([1.0 / pt["temperature_K"] for pt in points])
    ln_d = np.log([pt["D_cm2_s"] for pt in points])
    slope, intercept = np.polyfit(inv_t, ln_d, 1)
    pred = slope * inv_t + intercept
    ss_res = float(((ln_d - pred) ** 2).sum())
    ss_tot = float(((ln_d - ln_d.mean()) ** 2).sum())
    r2 = 1.0 - ss_res / ss_tot if ss_tot > 0 else 1.0
    ea = -slope * KB_EV
    d0 = float(np.exp(intercept))
    payload = {"activation_energy": float(ea), "activation_energy_unit": "eV",
               "d0": d0, "d0_unit": "cm2/s", "r_squared": r2,
               "diffusivities": points}
    art = _result(req, "ArrheniusResult", payload)
    with open("arrhenius.csv", "w") as f:
        f.write("temperature_K,D_cm2_s\n")
        for pt in points:
            f.write(f"{pt['temperature_K']:.1f},{pt['D_cm2_s']:.6e}\n")
    art["files"]["csv"] = "arrhenius.csv"
    return {"outputs": {"result": art},
            "validation": [
                verdict("enough_temperatures", len(points) >= min_points),
                verdict("positive_diffusivities",
                        all(pt["D_cm2_s"] > 0 for pt in points))]}