Skip to content

M3Flow — The Complete Manual

A task-centric, artifact-driven, provenance-first workflow runtime for molecular simulation. M3Flow sits between coding agents (Claude Code, etc.) or human researchers and simulation software (AutoPoly, LAMMPS), providing a stable, discoverable, composable, and fully tracked execution environment.

SystemSpec ──▶ tasks ──▶ artifacts ──▶ workflows ──▶ results
                  │            │
                  └── provenance + content-addressed cache ──┘

This manual consolidates everything: concepts, the CLI, the JSON contract, the provider protocol, the task and workflow libraries, project configuration, the TUI, and the reference runs. Shorter focused documents live alongside it in docs/; this file is the single-page reference.


Table of contents

  1. Concepts
  2. Installation
  3. Quickstart
  4. The artifact type system
  5. Projects and configuration
  6. The runtime
  7. Provenance
  8. The CLI
  9. The agent-facing JSON contract
  10. The provider protocol (m3flow-provider/1)
  11. The task library
  12. The workflow library
  13. Authoring tasks and providers
  14. Authoring workflows
  15. The TUI
  16. Reference runs
  17. Design invariants
  18. Repository layout

1. Concepts

Everything in M3Flow is one of four primitives: task, artifact, workflow, run.

1.1 Tasks

A task is the atomic unit of work: versioned (name@semver), typed, validated. Tasks are declared in YAML against the task/v1 schema and loaded from the embedded library (tasks/) or a project's own tasks/ directory — a project task shadows the builtin at the same name@version.

A task spec declares:

  • inputs / outputs — named ports with an artifact type; inputs may be required and/or many (a fan-in list).
  • parameters — typed: string | integer | number | boolean | enum | string_list | number_list | map, plus quantity types temperature | pressure | time | length | density | energy | area. Quantity values must carry units: {value: 300, unit: K} or the string form "300 K". Bare numbers are rejected. Canonical units: K, bar, fs, Å, g/cm³, kcal/mol, Ų.
  • requirements — scientific preconditions (ensemble, observables, needs_time_axis, dynamics_sensitive, resolution).
  • validation — validator names the provider MUST report in every response; a failing or missing validator fails the task with scientific_validation.
  • implementations — which providers can execute the task; default: true picks the one used unless overridden.

Run tasks (run_nvt, run_npt, run_nve, run_deform, …) take exactly one of system: SimulationSystem or state: SimulationState and produce a new state — chains of MD stages are state → state edges.

1.2 Artifacts

Artifacts are immutable, typed, content-addressed records:

{id: art_…, type, schema_version, files, metadata, data, content_hash,
 producer, created_at}
  • Files live in the content-addressed store (CAS) at .m3flow/artifacts/sha256/<2>/<hash>; identical bytes are stored once.
  • content_hash = hash(type + schema_version + per-file sha256): the file identity (deduplication, lineage). Metadata and data are excluded.
  • The execution fingerprint of an artifact — what cache keys use for inputs — adds the data payload and the semantic metadata (time axes, unit systems, strides, …); only presentation keys (description, display_name, label, labels, notes, tags) are excluded. Two trajectories with identical bytes but different frame_interval_fs are different task inputs.
  • data is a small inline JSON payload (e.g. a DensityResult's value/unit) that workflow conditions and ${step.out.data.field} references read.
  • The type hierarchy is nominal subtyping (§4): a SimulationState input accepts an EquilibratedState.

1.3 Workflows

A workflow (workflow/v1 YAML) composes tasks — and other workflows — into a DAG with a small expression language. The compiler expands everything statically; there are no runtime loops.

  • Declaration order is semantic — a step may only reference earlier steps.
  • stages — MD shorthand: {ensemble: npt, temperature: 300 K, duration: 50 ps} expands to a chained run task. minimize's duration maps to relax_duration. Temperature and pressure ramp linearly with the engine's native fix keywords: temperature_start/temperature_end (nvt/npt) and pressure_start/pressure_end (npt). NPT barostat control: pressure_style: iso|aniso|tri|xyz, couple: xyz|xy|xz|yz|none (xyz style only), and per-axis pressure_x/y/z (+_end) — unset axes are not barostated, matching LAMMPS fix nh semantics.
  • foreach — foreach: ${params.temperatures} as: T expands statically at compile time into step__0, step__1, … (max 256). Referencing the base name (${sweep.thermo}) in a many: true input collects all expansions in order.
  • conditions — "${check.report.equilibrated}", "not … and …" — a false condition marks the node SKIPPED, and dependents are SKIPPED transitively.
  • subworkflows — workflow: name inlines the child at compile time with step. id prefixing; child inputs bind to parent expressions, child outputs become ${step.output} in the parent. Nesting depth ≤ 8.
  • Protocol workflows carry scientific metadata (domain, purpose, references, assumptions) and are immutable once published — fork a new name@version to change a schedule.

1.4 The equilibration gate

An EquilibratedState can only come into existence through a passing EquilibrationReport — via the promote_equilibrated_state task. Finishing a protocol (the last MD stage completing) is never treated as proof of equilibration. This is enforced by the type system plus the runtime, not by convention:

  • every check of the report is passed / failed / insufficient_data / error, and only an all-passed report certifies — missing evidence (e.g. an unreadable trajectory) never counts as a pass;
  • the report records the evidence it examined (artifact ids, content hashes, producing task run), its criteria version and sampling window, and — when the check receives state — the state it certifies;
  • promotion requires the report to be bound to the promoted state (its subject fingerprint, or thermo evidence from the task run that produced the state), and the runtime re-verifies that binding and that the state's files are carried unchanged;
  • certified types cannot be registered by hand or emitted by any other task; forced promotion (require_pass: false) was removed — to continue without certification, use the SimulationState directly.

A pass certifies stationarity of the checked observables over one sampling window, for the stated scope. It is not evidence of conformational, entanglement or long-time dynamical equilibration.


2. Installation

From a release (recommended):

# 1. Download the m3flow binary for your platform from
#    https://github.com/Chenghao-Wu/m3flow/releases/latest
#    (tarball for Linux/macOS, zip for Windows; the linux-musl build
#    is fully static and works on older clusters).
tar xzf m3flow-<version>-<target>.tar.gz
sudo install m3flow-<version>-<target>/m3flow /usr/local/bin/

# 2. Providers (Python; JSON-protocol processes on PATH)
pip install m3flow-providers   # or: download the .whl from the release
                               # page and `pip install` that file

# 3. Engines
#   AutoPoly: importable in the provider's Python environment
#   LAMMPS:   `lmp` on PATH, or configure in m3flow.yaml:
#             providers: {lammps: {engine: {executable: /path/to/lmp}}}

From source:

# Rust core + CLI + TUI
cargo build --release          # binary: target/release/m3flow

# Providers (Python; JSON-protocol processes on PATH)
pip install -e providers/      # m3flow-autopoly, m3flow-lammps, m3flow-analysis

Verify:

m3flow provider list       # all providers "ok", engine versions shown
m3flow task list           # the 19-task atomic library
m3flow workflow list       # the 18-workflow library

3. Quickstart

m3flow init --name demo
cd demo

cat > systems/ethanol.yaml <<'EOF'
schema: system/v1
name: ethanol
components:
  - id: ethanol
    type: molecule
    representation: {type: smiles, value: "CCO"}
    count: 50
environment:
  type: bulk
  target_density: {value: 0.789, unit: g/cm3}
resolution: {type: atomistic, force_field: oplsaa}
EOF

m3flow workflow run construct_system --input specification=@systems/ethanol.yaml
m3flow workflow run simple_equilibration --input system=art_<from previous>
m3flow run list && m3flow run inspect <wr_…>
m3flow tui                    # live cockpit

Discovery is part of the contract — everything an agent needs is one CLI call away:

m3flow task search polymer                        # full-text task search
m3flow task inspect run_npt                       # full typed contract
m3flow workflow plan polymer_21step_equilibration # static expansion
m3flow schema show system                         # SystemSpec JSON schema
m3flow artifact lineage art_…                     # full upstream tree

Every command accepts --json; see §9.


4. The artifact type system

Nominal subtyping over five families (Spec, System, State, Dataset, Result) plus the open root Artifact. Compatibility: have <: want — a subtype is accepted wherever a supertype is declared (m3flow artifact compatible art_… <task> answers this directly).

Artifact
├── Spec
│   └── SystemSpec
├── System
│   ├── MolecularSystem
│   ├── ParameterizedSystem
│   └── SimulationSystem
├── State
│   └── SimulationState
│       └── EquilibratedState
├── Dataset
│   ├── Trajectory
│   │   └── ProductionTrajectory
│   ├── SimulationLog
│   ├── ThermodynamicSeries
│   │   └── TemperatureSeries
│   └── StressStrainSeries
└── Result
    ├── DensityResult      ├── RDFResult        ├── RgResult
    ├── ReeResult          ├── MSDResult        ├── DiffusionResult
    ├── CTEResult          ├── TgResult         ├── AdhesionResult
    ├── ModulusResult      └── EquilibrationReport

m3flow schema list prints this from the binary itself; the hierarchy is defined in crates/core/src/atypes.rs.

Type packs: extending the hierarchy without touching m3flow

Projects declare additional types in types/v1 documents — YAML files placed in a types/ directory under the project root (or in any directory listed under registries: in m3flow.yaml):

schema: types/v1
pack: atomicsim
version: 0.1.0
types:
  AtomicStructure: System          # NewType: ParentType
  RelaxedStructure: AtomicStructure
  LabeledStructureSet: StructureSet
  StructureSet: Dataset

Rules:

  • The parent must be a builtin type or one the pack itself defines (any declaration order; intra-pack chains resolve automatically). Cycles are therefore impossible — the tree only grows downward.
  • Redefining a known type is an error, never an override — unlike task specs, type names live inside artifact records and provenance chains.
  • Family is inherited from the parent chain (RelaxedStructure is a System, LabeledStructureSet a Dataset).
  • Type names match [A-Z][A-Za-z0-9]*.
  • Type packs register before specs at load time, so a project's tasks may reference types its own types/ directory defines.
  • The builtin table is frozen per release: once a type ships builtin it stays builtin.

m3flow schema list annotates pack types with their origin file. The JSON schema for pack documents is schemas/types.schema.json (m3flow schema show types).


5. Projects and configuration

m3flow init creates a project directory rooted at m3flow.yaml:

schema: m3flow-project/v1
name: my_project
registries:                    # extra directories to load specs from
  tasks: [./tasks]
  workflows: [./workflows]
providers:
  lammps:
    executable: /path/to/m3flow-lammps   # default: m3flow-<name> on PATH
    python: /path/to/python              # for Python providers
    engine: {executable: /path/to/lmp, mpi: false, np: 8}
    extra: {...}                          # free-form, forwarded in `config`
    executor: local                       # per-provider override: local | slurm
executor:
  type: local                             # local (default) | slurm
  slurm:                                  # see docs/slurm.md
    partition: gpua800
    qos: 4gpus
    poll_interval_secs: 15
    setup_commands: [module load anaconda3, source activate autopoly]
defaults: {...}

For the LAMMPS provider, the effective core count is the task/step resources.cpu if declared, else engine.np, else 1 (serial). With more than one core the engine runs via mpirun -np N (override the launcher with engine.launcher); engine.mpi: false selects OpenMP threading (OMP_NUM_THREADS + -sf omp) instead of MPI ranks.

The executor section selects where provider jobs run (see Slurm execution). Executor selection is a scheduling concern: like resources, it never joins cache keys, spec hashes, or artifact identity. Precedence: --executor CLI flag > providers.<name>.executor > executor.type > local.

Project layout after a run:

m3flow.yaml
systems/           workflows/        tasks/         # user-authored specs
.m3flow/
  artifacts/sha256/<2>/<hash>   # content-addressed store
  runs/<wr_…>/<tr_…>/           # per-task workdirs, request/response, logs
  m3flow.db                     # SQLite provenance store

The CLI discovers the project by walking upward from the cwd; outside a project, registry commands still work with builtins.

Production mode: freezing the extension surface

A project in development lets you add artifact types (types/), task and workflow specs (tasks/, workflows/, registries: paths), and dispatch any installed provider. Once a study is producing publishable results you usually want that vocabulary frozen — type names and spec versions live inside artifact records and provenance chains, so quiet additions or edits are a reproducibility hazard. mode: production locks all three:

mode: production        # default: development

In production mode:

  • Any types/v1 pack found in project sources is a load error naming the file (not a silent skip), as is any project task/v1/workflow/v1 document. Builtin specs and types are unaffected.
  • Only providers declared under providers: in m3flow.yaml may be dispatched; a workflow step needing an undeclared provider fails with provider_not_pinned before anything runs.

Per-kind overrides win over the mode default, e.g. freeze specs but keep adding types:

mode: production
extensions:
  types: allow            # allow | deny
  tasks: deny             # covers both tasks/ and workflows/
  providers: any          # any | pinned

Independently of mode, a provider entry may pin a version; the provider's self-reported version (also what joins the cache key) must match exactly, else the step fails with provider_version_mismatch:

providers:
  analysis:
    version: 0.4.0

This is a guardrail against accidental drift, not a security boundary — m3flow.yaml is a writable project file, and flipping it back to development is the intended escape hatch.


6. The runtime

m3flow workflow run performs: compile → bind → execute.

  1. Compile — static expansion of stages/foreach/subworkflows into a DAG; every reference resolved; full type check against the artifact hierarchy. m3flow workflow plan <name> shows exactly this compiled form before anything executes.
  2. Bind — inputs come from --input name=art_… (existing artifact) or name=@file. SystemSpec files are schema-validated and registered as Spec artifacts, so provenance chains start at the human-authored spec.
  3. Execute — a concurrent scheduler runs ready nodes (--max-concurrency to cap), dispatching each node to its provider process over m3flow-provider/1 and ingesting returned files into the CAS. With executor.type: slurm the same provider call is submitted to the cluster via sbatch instead of run locally (see Slurm execution); --max-concurrency then caps in-flight Slurm jobs.

6.1 Caching

Before executing, a node looks up:

hash("m3flow-cache/2", task@version, TaskSpec execution fingerprint,
     provider@version, engine descriptor fingerprint,
     effective engine config, input execution fingerprints, canonical params)
  • The engine descriptor is the provider's whole describe.engine object (version, resolved executable, compiled packages, library versions), and describe is called with the configured engine config so it probes the engine execute would actually launch.
  • The engine config forwarded to the provider joins the key, minus keys the provider declares scheduling_config_keys (e.g. MPI rank count).
  • A provider that cannot identify its engine (unknown / unavailable) bypasses the cache entirely — such results are never shared.
  • Keys of the older m3flow-cache/1 scheme never match; old entries are simply unused (m3flow cache clear removes them).

A hit marks the node CACHED and links the original artifacts — provenance stays complete, and a repeated workflow is nearly instant (the reference polymer_multi run is 35/35 CACHED in ~4 s). --no-cache bypasses; m3flow cache clear drops entries (artifacts are kept).

Versioning discipline still matters: if a task's logic changes meaningfully, bump its version (and the provider's version when provider code changes); the engine descriptor and config join the key automatically.

6.2 Failure taxonomy, retry, resume

Every failure carries a structured error_type + category (input_error, provider_error, execution_error, scientific_validation, environment_error) — agents branch on these programmatically. Per-step retry: {max_attempts, on: [categories]} retries recoverable categories automatically.

  • m3flow run resume <wr> — keep completed steps, re-run the rest. Resume reuses the run's original execution options (executor override, concurrency, cache policy) and refuses to continue if the run's execution closure (expanded graph, parameters, referenced task definitions) changed since submission — same-name@version edits are allowed in development projects (with a warning), but not mid-run.
  • m3flow run retry <wr> <step> — re-execute one step and everything downstream of it.
  • m3flow run cancel <wr> — request cancellation of the run's current execution: local provider process groups (including MPI children) are terminated, Slurm jobs are scanceled. A later resume runs as a new execution generation and is not affected by the old request.
  • One process executes a run at a time: it holds an exclusive OS lock on runs/<wr>/OWNER (released automatically when the process exits), so a second resume/retry fails while the owner is alive and a crashed owner never blocks recovery. A scheduler error only finalizes its own run.
  • Every dispatch runs in a fresh runs/<wr>/<step>/attempt-NNN/ directory; earlier attempts stay on disk for diagnosis.
  • resources.walltime is enforced for local jobs too (walltime_exceeded, resource_error, recoverable).

Provider task failures include raw_log (the engine log tail) and m3flow run logs <wr> [--step …] surfaces the full request/response and workdir files.


7. Provenance

.m3flow/m3flow.db records every workflow_run, task_run (status, attempts, timestamps, params, validation verdicts, engine), artifact (+files), and every input/output edge.

m3flow artifact lineage art_c8399702   # recursive tree back to the SystemSpec
m3flow artifact inspect art_…          # files, metadata, data payload, producer
m3flow run graph <wr_…>                # nodes, edges, statuses

Because artifacts are content-addressed and cache hits link rather than copy, the lineage of any result is always complete and exact — you can answer "which equilibration protocol, at what temperature, with which engine build produced this density?" for any artifact in the store.


8. The CLI

Global: --json on every command (stdout = one JSON document; progress to stderr). Exit codes for runs: 0 COMPLETED, 2 FAILED, 3 CANCELLED; 1 for command errors.

command what it does
init [dir] [--name N] initialize a project
task list [--category C] list registered tasks
task search <query> full-text search over names/descriptions/tags
task inspect <name> full task spec as JSON
workflow list list registered workflows
workflow inspect <name> full workflow spec as JSON
workflow validate <name_or_file> schema-validate a workflow
workflow plan <name> [--param k=v] compiled DAG, no execution
workflow run <name> [--input n=art/@f] [--param k=v] [--no-cache] [--max-concurrency N] [--dry-run] execute
run list [--limit N] recent runs
run status <wr> one-call composite: run + steps + failure + suggested next command
run inspect <wr> run record + per-step status
run logs <wr> [--step prefix] per-step response documents, workdir files
run graph <wr> dependency graph with statuses
run resume <wr> resume, keeping completed steps
run retry <wr> <step> re-execute one step + downstream
run cancel <wr> request cancellation
artifact list [--type T] [--limit N] newest artifacts
artifact inspect <art> files/metadata/data/producer
artifact lineage <art> recursive provenance tree
artifact compatible <art> <task> can this artifact feed that task?
artifact register --type T --file name=path [--meta JSON] [--data JSON] register existing files
schema list artifact type hierarchy + document schemas
schema show <task\|workflow\|system\|artifact> print a JSON schema
provider list providers, availability, engine versions
provider diagnose <name> locate + describe a provider
cache stats / cache clear cache inspection and management
results sync [--run <wr>] rebuild friendly results/ tree(s) from the provenance DB
run label <wr> [name] set/clear a run's study label (moves its results/ folder)
tui [wr] interactive execution cockpit

The results/ tree

Every run materializes a human-browsable view of its outputs as it executes (disable per-run with workflow run --no-materialize, or project-wide via defaults: {materialize: {enabled: false}}):

results/<group>/<YYYY-MM-DD>_<HH-MM>_<workflow>_<wr_id>/
  run.json                     run summary, resolved params (workflow + per-step), step → artifact map
  _inputs/<name>/              workflow inputs
  NN_<step>/<ArtifactType>/    symlinks to output files + _artifact.yaml

<group> is the run's study label (workflow run --label peo-5k-screen) when it has one, or the workflow name when it doesn't — a study folder collects runs across workflow types and reruns, while unlabeled runs behave exactly as before. Labels are validated ([A-Za-z0-9._-], max 64 chars, starting with a letter or digit) and are purely presentational: they never enter spec hashes, cache keys, or artifact identity. Rename a study with m3flow run label <wr> <new-name> (omit the name to clear) — the run's folder moves immediately. Do not mv folders by hand: the label lives in the provenance DB and the next results sync would resurrect the old name.

The timestamp is the run's created_at in UTC (same clock as run.json and the database records).

Files are relative symlinks into the content-addressed store, and _artifact.yaml in each directory carries the artifact id, type, content hash, and producer — the tree is operable: feed any artifact id you find here straight into the next workflow. Nothing under results/ is authoritative (the DB + store are truth); store blobs are read-only, so the view can never mutate them. Rebuild any tree any time with m3flow results sync — without --run it wipes results/ first, so orphans from deleted runs or renamed studies disappear.

| tui [wr] | interactive execution cockpit |

IDs: art_… artifact, tr_… task run, wr_… workflow run. Commands take full ids (8 hex chars); run logs --step accepts a unique step prefix.


9. The agent-facing JSON contract

M3Flow is designed to be driven programmatically. With --json, stdout carries exactly one JSON document; diagnostics go to stderr.

Success documents (stable keys):

  • workflow run --json → the WorkflowRun record {id, name, version, status, inputs, params, outputs, error, created_at, started_at, ended_at, spec_hash, workdir, git}. outputs maps names to artifact ids; outputs of SKIPPED branches are absent.
  • run inspect --json → {run: {...}, tasks: [TaskRunRecord…]} with {id, node_id, task_name, task_version, provider, status, attempts, params, error, validation, engine, started_at, ended_at, cache_key}.
  • run status --json → the agent-oriented composite, answering "did it fail, where, and what now?" in one call (replaces the inspect + logs + graph loop): {run: {id, workflow, label, status, created_at, started_at, ended_at}, results_dir, materialized, steps: [{node, task, status, attempts, outputs: {role: art_id}}], failure: {step, category, recoverable, message} | null, next: "m3flow run retry …" | null}. next is the single most useful follow-up command: retry for recoverable failures, logs for permanent ones, resume for cancelled runs, null when there is nothing to do.
  • run graph --json → {nodes: [{id, task, label, status}], edges: [{from, to}]}.
  • artifact inspect --json → {id, type, schema_version, content_hash, files, metadata, data, producer}.
  • artifact lineage --json → recursive {id, type, producer: {task_run…, inputs: [...]}}.
  • artifact compatible --json → {compatible, matching_inputs: [{input, declared_type}]}.

Errors — a single document:

{
  "error_type": "schema | type | workflow | task | provider | execution |
                 scientific_validation | artifact_compatibility | not_found |
                 io | internal",
  "category": "input_error | protocol_error | task_error | provider_error |
               execution_error | scientific_validation | compatibility_error |
               not_found | environment_error | internal",
  "recoverable": false,
  "message": "human-readable",
  "...": "variant-specific fields"
}

Branch on category (retry policy) and error_type (semantics). Provider failures add raw_log with the engine log tail.


10. The provider protocol

Providers adapt external tools (AutoPoly, LAMMPS, Python analysis) to M3Flow tasks. They are independent OS processes named m3flow-<provider>, communicating exclusively through JSON on stdout/stdin — languages fully decoupled. Protocol: m3flow-provider/1.

m3flow-lammps describe  [--json]
m3flow-lammps validate  REQUEST.json
m3flow-lammps execute   REQUEST.json
m3flow-lammps diagnose  REQUEST.json

Rules: exactly one JSON document on stdout (human chatter to stderr); exit 0 for any well-formed response including scientific failures; non-zero only for protocol violations.

  • describe [CONFIG.json] → {protocol, provider: {name, version}, engine: {name, version, ...}, scheduling_config_keys, tasks: [...]}. The optional argument is the engine config the runtime will forward to execute; probe that engine. The whole engine object joins the cache key — report it accurately, and report unknown when you cannot tell (such results are never cached).
  • validate → {valid, errors[]} — check an execute request without running it.
  • execute — request carries {task, workflow_run_id, task_run_id, attempt, workdir, inputs, parameters, resources, config}; each input artifact carries {id, type, content_hash, producer, files, metadata, data}. Input files are absolute CAS paths, never written to; outputs must be staged inside workdir and returned as workdir-relative paths (no .., no symlinks out); quantities arrive canonicalized ({value, unit}). Responses are strict JSON: NaN/Infinity are protocol errors — report missing statistics as null with a status. Success response: {status: "success", outputs, validation: [{name, passed, detail}], engine, warnings}. Every validator named by the TaskSpec must appear with a verdict. Scientific failure: {status: "error", error: {error_type, category, recoverable, message, details, raw_log}, partial_outputs}.
  • diagnose — best-effort post-mortem: {diagnostics, suggestions}. M3Flow reports facts; repair strategy belongs to the caller.

Standard error_types: lost_atoms, nan_detected, energy_blowup, trajectory_corrupt, simulation_incomplete, engine_crash, engine_missing, input_invalid, builder_failed, type_check_failed, validation_failed. Standard categories: simulation_instability, input_error, environment_error, resource_error, protocol_error, scientific_validation.

The protocol version is sent in every request and returned by describe; the runtime refuses providers whose major protocol differs.


11. The task library

19 atomic tasks in three groups. Full typed contract of each: m3flow task inspect <name>.

construction (AutoPoly)

task summary
build_system force-field-agnostic geometry from a SystemSpec
parameterize_system assign force-field atom types and charges
prepare_simulation_system pack into a simulation box → runnable SimulationSystem

simulation (LAMMPS)

task summary
energy_minimize potential-energy minimization of a system or state
run_nvt canonical (NVT) MD
run_npt isothermal-isobaric (NPT) MD
run_nve microcanonical (NVE) MD — typical for dynamics production
run_soft_pushoff soft-potential push-off for CG/overlapping systems
run_deform uniaxial tensile deformation (fix deform, NVT)

analysis, sampling, validation

task summary
compute_density mean density + blocked standard error after discarding equilibration_fraction of the series
compute_msd mean-squared displacement vs lag time (unwrapped coords)
compute_rdf g(r), optionally between atom-type selections
compute_rg radius of gyration of polymer chains (by molecule id)
compute_ree mean end-to-end distance of linear chains (ends from the bond graph or an explicit selection)
compute_adhesion interfacial interaction-energy density −⟨E_int⟩/(n·A) (pair + kspace terms; not the reversible work of adhesion)
collect_thermo_series fan-in a temperature sweep into one TemperatureSeries
fit_diffusion Einstein-relation linear fit of MSD → D (cm²/s; reduced units for lj unless sigma/tau are given)
fit_cte volumetric thermal expansion coefficient α
fit_tg bilinear V(T) fit → glass transition temperature
fit_modulus Young's modulus from the linear stress-strain region
check_polymer_equilibration density drift/shift, energy + volume trend/shift, Rg stability → evidence-bound EquilibrationReport
promote_equilibrated_state SimulationState → EquilibratedState, gated on a passing report

12. The workflow library

18 workflows in three layers. Inspect any of them with m3flow workflow inspect / workflow plan.

preparation

workflow summary
construct_system SystemSpec → runnable SimulationSystem (all three construction stages)
simple_equilibration minimize → NVT → NPT at the target state point
polymer_21step_equilibration Larsen 21-step compression/decompression for polymer melts
equilibrate_polymer adaptive: run the base protocol, check, extend only if the check fails
cg_push_off CG melt preparation: minimize, soft push-off, NVT
interface_relaxation polymer/substrate interface: gentle minimize + low-NVT relaxation

production

workflow summary
npt_thermodynamic_production NPT production from an equilibrated state → ThermodynamicSeries
dynamics_production NVE production → unwrapped ProductionTrajectory
temperature_sweep NPT at each of a list of temperatures → TemperatureSeries
mechanical_deformation tensile deformation → StressStrainSeries

properties

workflow summary
density mean density from an NPT series (equilibrated tail)
diffusion MSD from an unwrapped trajectory + Einstein fit
rdf radial distribution function
cte volumetric thermal expansion coefficient
tg glass transition temperature from bilinear V(T)
adhesion interaction-energy density from an interface series
mechanical_properties Young's modulus from a stress-strain series
polymer_basic_properties one-shot panel for an equilibrated polymer

Protocol workflows are versioned and immutable — fork name@new-version, never edit a published schedule in place.


13. Authoring tasks and providers

A task = a YAML contract (task/v1) + at least one provider implementation.

schema: task/v1
name: compute_viscosity
version: 1.0.0
description: Green-Kubo viscosity from an NVE trajectory.
category: analysis            # construction|simulation|sampling|analysis|validation|utility
inputs:
  trajectory: {type: Trajectory, required: true}
parameters:
  temperature: {type: temperature, required: true}
  block_length: {type: time, default: 10 ps}
outputs:
  result: {type: ViscosityResult}   # new types go in crates/core/src/atypes.rs
requirements:
  needs_time_axis: true
  dynamics_sensitive: true
  resolution: [atomistic]
validation: []
implementations:
  - {provider: analysis, default: true}
resources: {cpu: 2, walltime: 30 min}

Rules of thumb:

  • Outputs come back as files + a data payload for anything other steps/conditions reference (${step.result.data.value}).
  • Validators are the scientific gate — declare only what the provider can honestly check.
  • Bump version when logic changes meaningfully; cache keys include name@version and the provider version.

In Python, the shared runtime (providers/src/m3flow_provider/) does the plumbing:

from m3flow_provider import Provider, ProviderFailure, artifact, verdict

def compute_viscosity(req):
    # req: protocol, task, workdir, inputs (ABSOLUTE store paths),
    #      parameters (canonical), resources, config
    traj = req["inputs"]["trajectory"]["files"]["dcd"]
    temp = req["parameters"]["temperature"]["value"]   # always Kelvin
    ...
    return {
        "outputs": {"result": artifact(
            "ViscosityResult",
            files={"json": "result.json"},       # workdir-relative
            data={"value": eta, "unit": "mPa*s"})},
        "validation": [verdict("trajectory_long_enough", ok, detail)],
    }

def cli():
    raise SystemExit(Provider(
        name="analysis", version="0.3.2",
        engine=lambda: {"name": "mdanalysis", "version": ...},
        tasks={"compute_viscosity": compute_viscosity},
    ).cli())

Contract points:

  • stdout carries exactly one JSON document; logs go to stderr/files.
  • Staged output files are ingested into the CAS by the runtime — never move/rename them after responding.
  • Raise ProviderFailure(error_type, category, message, recoverable=…, details=…, raw_log=…); unexpected exceptions become engine_crash/provider_error. Categories drive scheduler retry.
  • Input artifact files are extensionless CAS paths — stage copies with proper filenames if your parser sniffs extensions.

Register by dropping the YAML into a project's tasks/ (shadows builtins) or the library tasks/, then:

m3flow task inspect compute_viscosity
m3flow-analysis describe
m3flow workflow validate workflows/x.yaml

14. Authoring workflows

schema: workflow/v1
name: my_protocol
version: 1.0.0
description: What it does, when to use it.
kind: scientific_protocol
domain: [polymer, atomistic]
purpose: [equilibration]
references: ["citation strings"]
assumptions: ["explicit scientific assumptions"]

inputs:
  system: {type: SimulationSystem, required: true}
parameters:
  temperature: {type: temperature, default: 300 K}
  seed: {type: integer, default: 12345}

stages:                      # shorthand for linear MD protocols
  - {ensemble: minimize, name: min, duration: 5 ps, timestep: 0.5 fs}
  - {ensemble: nvt, name: nvt, temperature: "${params.temperature}", duration: 50 ps}

steps:
  check:
    task: check_polymer_equilibration
    inputs: {thermo: "${nvt.thermo}"}

outputs:
  state: {value: "${nvt.state}"}
  report: {value: "${check.report}"}

Native ramps and barostat control (LAMMPS fix nh keywords, linear start→stop over the stage):

stages:
  # heat 10 K -> 300 K during NVT
  - {ensemble: nvt, name: heat,
     temperature: 10 K, temperature_end: 300 K, duration: 50 ps}
  # decompress 10 bar -> 1 bar during NPT (isotropic)
  - {ensemble: npt, name: decompress,
     temperature: 300 K, pressure: 10 bar, pressure_end: 1 bar, duration: 500 ps}
  # per-axis barostat, e.g. slab/interface: z-only pressure control
  - {ensemble: npt, name: slab,
     temperature: 300 K, pressure_style: xyz, pressure_z: 1 bar, duration: 100 ps}
  # coupled in-plane + independent normal, x/y ramping 10 -> 1 bar
  - {ensemble: npt, name: membrane,
     temperature: 300 K, pressure_style: xyz, couple: xy,
     pressure_x: 10 bar, pressure_x_end: 1 bar,
     pressure_y: 10 bar, pressure_y_end: 1 bar,
     pressure_z: 1 bar, duration: 200 ps}

References: ${inputs.x} (workflow input), ${step.output} (artifact of an earlier step), ${step.output.data.field} / .metadata.field (values inside the payload, usable in parameters and conditions), ${params.x}. Whole-string refs substitute the value; embedded refs interpolate into strings ("${T} K" — building quantities from loop variables).

foreach:

steps:
  sweep:
    task: run_npt
    foreach: "${params.temperatures}"   # a JSON list
    as: T
    inputs: {state: "${inputs.state}"}
    parameters: {temperature: "${T} K"}
  collect:
    task: collect_thermo_series          # input declared many: true
    inputs: {series: "${sweep.thermo}"}  # gathers sweep__0..n in order

conditions: condition: "not ${check.report.equilibrated}" — boolean mini-language (and or not == != < <= > >=, parens, literals, ${...} refs). False → SKIPPED, transitively. This is how equilibrate_polymer extends NPT only when the first check fails.

Discipline:

  • m3flow workflow plan <name> before every run — the compiled DAG is the contract; type errors and bad references surface there, not mid-run.
  • Immutable once used: fork name@new-version to change a protocol.
  • Keep quantities typed end-to-end ("${params.temperature}", never pre-baked numbers).

15. The TUI

m3flow tui [wr] — the execution cockpit: run header, status-colored node list, detail pane; auto-refreshes while the run is RUNNING.

key action
j / k navigate nodes
l logs pane
a artifacts pane
p provenance pane
g graph pane
r retry selected node
R resume run
q quit

16. Reference runs

examples/ref/ holds five reduced-scale end-to-end validations, all reproducible via ./run_all.sh:

run demonstrates result
ethanol_diffusion SMILES → equilibrate → NVE → MSD → D full pipeline
peo_density Larsen 21-step equilibration → density 1.069 g/cm³ (lit. ~1.1)
polymer_multi property fan-out + cache 3rd run 35/35 CACHED in ~4 s
peo_silica_adhesion quartz slab + film → interaction-energy density W = 101 mJ/m² (measured before the kspace fix: pair terms only)
cg_melt bead-spring CG construct → push-off → NVT CG path

Reduced scale demonstrates the platform, not converged physics: ps-scale tails carry real statistical drift, and the equilibration gate says so — that honesty is the feature.


17. Design invariants

Do not break these; correctness and trust depend on them.

  • Quantities carry units. Bare numbers are rejected for quantity parameters — {value, unit} or "300 K" strings, canonicalized (K, bar, fs, Å, g/cm³, kcal/mol, Ų).
  • Artifact identity = type + schema version + per-file content hashes. Task inputs are identified by their execution fingerprint (identity + data + semantic metadata).
  • Cache key = task@version + TaskSpec fingerprint + provider@version + engine descriptor + effective engine config + input execution fingerprints + canonical params. Anything that can change a result must join the key — including passthrough config dicts.
  • EquilibratedState only via promote_equilibrated_state from a passing EquilibrationReport bound to that state (runtime-verified).
  • A resume executes the definition the run started with.
  • Providers are separate processes speaking m3flow-provider/1 (one JSON document on stdout); the runtime owns ingestion into the CAS.
  • Protocols are immutable once published — fork versions, never edit schedules in place.
  • Discovery is part of the contract — every spec, schema, and status is reachable through the CLI with --json; agents never need to read source.

18. Repository layout

schemas/     JSON schemas (task/v1, workflow/v1, system/v1, artifact/v1)
tasks/       the 19-task atomic library (autopoly / lammps / analysis)
workflows/   the 18-workflow library (preparation / production / properties)
crates/      Rust workspace: core, registry, runtime, cli, tui
providers/   Python provider processes (m3flow-provider/1)
docs/        this manual + focused documents
examples/    reference projects (reduced-scale end-to-end runs)

Focused documents: concepts.md, json-contract.md, provider-protocol.md, writing-tasks.md, writing-workflows.md, agent-benchmark.md (a transcript of an agent completing "build a PEO melt, equilibrate it, calculate density and diffusion at 300 K" using CLI discovery only).