M3Flow — The Complete Manual¶
A task-centric, artifact-driven, provenance-first workflow runtime for molecular simulation. M3Flow sits between coding agents (Claude Code, etc.) or human researchers and simulation software (AutoPoly, LAMMPS), providing a stable, discoverable, composable, and fully tracked execution environment.
SystemSpec ──▶ tasks ──▶ artifacts ──▶ workflows ──▶ results
│ │
└── provenance + content-addressed cache ──┘
This manual consolidates everything: concepts, the CLI, the JSON contract,
the provider protocol, the task and workflow libraries, project
configuration, the TUI, and the reference runs. Shorter focused documents
live alongside it in docs/; this file is the single-page reference.
Table of contents¶
- Concepts
- Installation
- Quickstart
- The artifact type system
- Projects and configuration
- The runtime
- Provenance
- The CLI
- The agent-facing JSON contract
- The provider protocol (m3flow-provider/1)
- The task library
- The workflow library
- Authoring tasks and providers
- Authoring workflows
- The TUI
- Reference runs
- Design invariants
- Repository layout
1. Concepts¶
Everything in M3Flow is one of four primitives: task, artifact, workflow, run.
1.1 Tasks¶
A task is the atomic unit of work: versioned (name@semver), typed,
validated. Tasks are declared in YAML against the task/v1 schema and
loaded from the embedded library (tasks/) or a project's own tasks/
directory — a project task shadows the builtin at the same name@version.
A task spec declares:
- inputs / outputs — named ports with an artifact
type; inputs may berequiredand/ormany(a fan-in list). - parameters — typed:
string | integer | number | boolean | enum | string_list | number_list | map, plus quantity typestemperature | pressure | time | length | density | energy | area. Quantity values must carry units:{value: 300, unit: K}or the string form"300 K". Bare numbers are rejected. Canonical units: K, bar, fs, Å, g/cm³, kcal/mol, Ų. - requirements — scientific preconditions (
ensemble,observables,needs_time_axis,dynamics_sensitive,resolution). - validation — validator names the provider MUST report in every
response; a failing or missing validator fails the task with
scientific_validation. - implementations — which providers can execute the task;
default: truepicks the one used unless overridden.
Run tasks (run_nvt, run_npt, run_nve, run_deform, …) take exactly one
of system: SimulationSystem or state: SimulationState and produce a new
state — chains of MD stages are state → state edges.
1.2 Artifacts¶
Artifacts are immutable, typed, content-addressed records:
- Files live in the content-addressed store (CAS) at
.m3flow/artifacts/sha256/<2>/<hash>; identical bytes are stored once. content_hash= hash(type + schema_version + per-file sha256): the file identity (deduplication, lineage). Metadata and data are excluded.- The execution fingerprint of an artifact — what cache keys use for
inputs — adds the
datapayload and the semantic metadata (time axes, unit systems, strides, …); only presentation keys (description,display_name,label,labels,notes,tags) are excluded. Two trajectories with identical bytes but differentframe_interval_fsare different task inputs. datais a small inline JSON payload (e.g. a DensityResult'svalue/unit) that workflow conditions and${step.out.data.field}references read.- The type hierarchy is nominal subtyping (§4): a
SimulationStateinput accepts anEquilibratedState.
1.3 Workflows¶
A workflow (workflow/v1 YAML) composes tasks — and other workflows — into
a DAG with a small expression language. The compiler expands everything
statically; there are no runtime loops.
- Declaration order is semantic — a step may only reference earlier steps.
- stages — MD shorthand:
{ensemble: npt, temperature: 300 K, duration: 50 ps}expands to a chained run task.minimize'sdurationmaps torelax_duration. Temperature and pressure ramp linearly with the engine's native fix keywords:temperature_start/temperature_end(nvt/npt) andpressure_start/pressure_end(npt). NPT barostat control:pressure_style: iso|aniso|tri|xyz,couple: xyz|xy|xz|yz|none(xyz style only), and per-axispressure_x/y/z(+_end) — unset axes are not barostated, matching LAMMPS fix nh semantics. - foreach —
foreach: ${params.temperatures} as: Texpands statically at compile time intostep__0,step__1, … (max 256). Referencing the base name (${sweep.thermo}) in amany: trueinput collects all expansions in order. - conditions —
"${check.report.equilibrated}","not … and …"— a false condition marks the node SKIPPED, and dependents are SKIPPED transitively. - subworkflows —
workflow: nameinlines the child at compile time withstep.id prefixing; child inputs bind to parent expressions, child outputs become${step.output}in the parent. Nesting depth ≤ 8. - Protocol workflows carry scientific metadata (
domain,purpose,references,assumptions) and are immutable once published — fork a newname@versionto change a schedule.
1.4 The equilibration gate¶
An EquilibratedState can only come into existence through a passing
EquilibrationReport — via the promote_equilibrated_state task.
Finishing a protocol (the last MD stage completing) is never treated as
proof of equilibration. This is enforced by the type system plus the
runtime, not by convention:
- every check of the report is
passed/failed/insufficient_data/error, and only an all-passedreport certifies — missing evidence (e.g. an unreadable trajectory) never counts as a pass; - the report records the evidence it examined (artifact ids, content
hashes, producing task run), its criteria version and sampling window,
and — when the check receives
state— the state it certifies; - promotion requires the report to be bound to the promoted state (its subject fingerprint, or thermo evidence from the task run that produced the state), and the runtime re-verifies that binding and that the state's files are carried unchanged;
- certified types cannot be registered by hand or emitted by any other
task; forced promotion (
require_pass: false) was removed — to continue without certification, use theSimulationStatedirectly.
A pass certifies stationarity of the checked observables over one sampling window, for the stated scope. It is not evidence of conformational, entanglement or long-time dynamical equilibration.
2. Installation¶
From a release (recommended):
# 1. Download the m3flow binary for your platform from
# https://github.com/Chenghao-Wu/m3flow/releases/latest
# (tarball for Linux/macOS, zip for Windows; the linux-musl build
# is fully static and works on older clusters).
tar xzf m3flow-<version>-<target>.tar.gz
sudo install m3flow-<version>-<target>/m3flow /usr/local/bin/
# 2. Providers (Python; JSON-protocol processes on PATH)
pip install m3flow-providers # or: download the .whl from the release
# page and `pip install` that file
# 3. Engines
# AutoPoly: importable in the provider's Python environment
# LAMMPS: `lmp` on PATH, or configure in m3flow.yaml:
# providers: {lammps: {engine: {executable: /path/to/lmp}}}
From source:
# Rust core + CLI + TUI
cargo build --release # binary: target/release/m3flow
# Providers (Python; JSON-protocol processes on PATH)
pip install -e providers/ # m3flow-autopoly, m3flow-lammps, m3flow-analysis
Verify:
m3flow provider list # all providers "ok", engine versions shown
m3flow task list # the 19-task atomic library
m3flow workflow list # the 18-workflow library
3. Quickstart¶
m3flow init --name demo
cd demo
cat > systems/ethanol.yaml <<'EOF'
schema: system/v1
name: ethanol
components:
- id: ethanol
type: molecule
representation: {type: smiles, value: "CCO"}
count: 50
environment:
type: bulk
target_density: {value: 0.789, unit: g/cm3}
resolution: {type: atomistic, force_field: oplsaa}
EOF
m3flow workflow run construct_system --input specification=@systems/ethanol.yaml
m3flow workflow run simple_equilibration --input system=art_<from previous>
m3flow run list && m3flow run inspect <wr_…>
m3flow tui # live cockpit
Discovery is part of the contract — everything an agent needs is one CLI call away:
m3flow task search polymer # full-text task search
m3flow task inspect run_npt # full typed contract
m3flow workflow plan polymer_21step_equilibration # static expansion
m3flow schema show system # SystemSpec JSON schema
m3flow artifact lineage art_… # full upstream tree
Every command accepts --json; see §9.
4. The artifact type system¶
Nominal subtyping over five families (Spec, System, State, Dataset,
Result) plus the open root Artifact. Compatibility: have <: want — a
subtype is accepted wherever a supertype is declared
(m3flow artifact compatible art_… <task> answers this directly).
Artifact
├── Spec
│ └── SystemSpec
├── System
│ ├── MolecularSystem
│ ├── ParameterizedSystem
│ └── SimulationSystem
├── State
│ └── SimulationState
│ └── EquilibratedState
├── Dataset
│ ├── Trajectory
│ │ └── ProductionTrajectory
│ ├── SimulationLog
│ ├── ThermodynamicSeries
│ │ └── TemperatureSeries
│ └── StressStrainSeries
└── Result
├── DensityResult ├── RDFResult ├── RgResult
├── ReeResult ├── MSDResult ├── DiffusionResult
├── CTEResult ├── TgResult ├── AdhesionResult
├── ModulusResult └── EquilibrationReport
m3flow schema list prints this from the binary itself; the hierarchy is
defined in crates/core/src/atypes.rs.
Type packs: extending the hierarchy without touching m3flow¶
Projects declare additional types in types/v1 documents — YAML files placed
in a types/ directory under the project root (or in any directory listed
under registries: in m3flow.yaml):
schema: types/v1
pack: atomicsim
version: 0.1.0
types:
AtomicStructure: System # NewType: ParentType
RelaxedStructure: AtomicStructure
LabeledStructureSet: StructureSet
StructureSet: Dataset
Rules:
- The parent must be a builtin type or one the pack itself defines (any declaration order; intra-pack chains resolve automatically). Cycles are therefore impossible — the tree only grows downward.
- Redefining a known type is an error, never an override — unlike task specs, type names live inside artifact records and provenance chains.
- Family is inherited from the parent chain (
RelaxedStructureis aSystem,LabeledStructureSetaDataset). - Type names match
[A-Z][A-Za-z0-9]*. - Type packs register before specs at load time, so a project's tasks may
reference types its own
types/directory defines. - The builtin table is frozen per release: once a type ships builtin it stays builtin.
m3flow schema list annotates pack types with their origin file. The JSON
schema for pack documents is schemas/types.schema.json (m3flow schema
show types).
5. Projects and configuration¶
m3flow init creates a project directory rooted at m3flow.yaml:
schema: m3flow-project/v1
name: my_project
registries: # extra directories to load specs from
tasks: [./tasks]
workflows: [./workflows]
providers:
lammps:
executable: /path/to/m3flow-lammps # default: m3flow-<name> on PATH
python: /path/to/python # for Python providers
engine: {executable: /path/to/lmp, mpi: false, np: 8}
extra: {...} # free-form, forwarded in `config`
executor: local # per-provider override: local | slurm
executor:
type: local # local (default) | slurm
slurm: # see docs/slurm.md
partition: gpua800
qos: 4gpus
poll_interval_secs: 15
setup_commands: [module load anaconda3, source activate autopoly]
defaults: {...}
For the LAMMPS provider, the effective core count is the task/step
resources.cpu if declared, else engine.np, else 1 (serial). With more
than one core the engine runs via mpirun -np N (override the launcher with
engine.launcher); engine.mpi: false selects OpenMP threading
(OMP_NUM_THREADS + -sf omp) instead of MPI ranks.
The executor section selects where provider jobs run (see
Slurm execution). Executor selection is a scheduling concern:
like resources, it never joins cache keys, spec hashes, or artifact
identity. Precedence: --executor CLI flag > providers.<name>.executor >
executor.type > local.
Project layout after a run:
m3flow.yaml
systems/ workflows/ tasks/ # user-authored specs
.m3flow/
artifacts/sha256/<2>/<hash> # content-addressed store
runs/<wr_…>/<tr_…>/ # per-task workdirs, request/response, logs
m3flow.db # SQLite provenance store
The CLI discovers the project by walking upward from the cwd; outside a project, registry commands still work with builtins.
Production mode: freezing the extension surface¶
A project in development lets you add artifact types (types/), task and
workflow specs (tasks/, workflows/, registries: paths), and dispatch
any installed provider. Once a study is producing publishable results you
usually want that vocabulary frozen — type names and spec versions live
inside artifact records and provenance chains, so quiet additions or edits
are a reproducibility hazard. mode: production locks all three:
In production mode:
- Any
types/v1pack found in project sources is a load error naming the file (not a silent skip), as is any projecttask/v1/workflow/v1document. Builtin specs and types are unaffected. - Only providers declared under
providers:inm3flow.yamlmay be dispatched; a workflow step needing an undeclared provider fails withprovider_not_pinnedbefore anything runs.
Per-kind overrides win over the mode default, e.g. freeze specs but keep adding types:
mode: production
extensions:
types: allow # allow | deny
tasks: deny # covers both tasks/ and workflows/
providers: any # any | pinned
Independently of mode, a provider entry may pin a version; the provider's
self-reported version (also what joins the cache key) must match exactly,
else the step fails with provider_version_mismatch:
This is a guardrail against accidental drift, not a security boundary —
m3flow.yaml is a writable project file, and flipping it back to
development is the intended escape hatch.
6. The runtime¶
m3flow workflow run performs: compile → bind → execute.
- Compile — static expansion of stages/foreach/subworkflows into a
DAG; every reference resolved; full type check against the artifact
hierarchy.
m3flow workflow plan <name>shows exactly this compiled form before anything executes. - Bind — inputs come from
--input name=art_…(existing artifact) orname=@file. SystemSpec files are schema-validated and registered asSpecartifacts, so provenance chains start at the human-authored spec. - Execute — a concurrent scheduler runs ready nodes
(
--max-concurrencyto cap), dispatching each node to its provider process overm3flow-provider/1and ingesting returned files into the CAS. Withexecutor.type: slurmthe same provider call is submitted to the cluster viasbatchinstead of run locally (see Slurm execution);--max-concurrencythen caps in-flight Slurm jobs.
6.1 Caching¶
Before executing, a node looks up:
hash("m3flow-cache/2", task@version, TaskSpec execution fingerprint,
provider@version, engine descriptor fingerprint,
effective engine config, input execution fingerprints, canonical params)
- The engine descriptor is the provider's whole
describe.engineobject (version, resolved executable, compiled packages, library versions), anddescribeis called with the configured engine config so it probes the engineexecutewould actually launch. - The engine config forwarded to the provider joins the key, minus keys the
provider declares
scheduling_config_keys(e.g. MPI rank count). - A provider that cannot identify its engine (
unknown/unavailable) bypasses the cache entirely — such results are never shared. - Keys of the older
m3flow-cache/1scheme never match; old entries are simply unused (m3flow cache clearremoves them).
A hit marks the node CACHED and links the original artifacts —
provenance stays complete, and a repeated workflow is nearly instant (the
reference polymer_multi run is 35/35 CACHED in ~4 s). --no-cache
bypasses; m3flow cache clear drops entries (artifacts are kept).
Versioning discipline still matters: if a task's logic changes
meaningfully, bump its version (and the provider's version when provider
code changes); the engine descriptor and config join the key
automatically.
6.2 Failure taxonomy, retry, resume¶
Every failure carries a structured error_type + category
(input_error, provider_error, execution_error,
scientific_validation, environment_error) — agents branch on these
programmatically. Per-step retry: {max_attempts, on: [categories]}
retries recoverable categories automatically.
m3flow run resume <wr>— keep completed steps, re-run the rest. Resume reuses the run's original execution options (executor override, concurrency, cache policy) and refuses to continue if the run's execution closure (expanded graph, parameters, referenced task definitions) changed since submission — same-name@versionedits are allowed in development projects (with a warning), but not mid-run.m3flow run retry <wr> <step>— re-execute one step and everything downstream of it.m3flow run cancel <wr>— request cancellation of the run's current execution: local provider process groups (including MPI children) are terminated, Slurm jobs arescanceled. A later resume runs as a new execution generation and is not affected by the old request.- One process executes a run at a time: it holds an exclusive OS lock on
runs/<wr>/OWNER(released automatically when the process exits), so a second resume/retry fails while the owner is alive and a crashed owner never blocks recovery. A scheduler error only finalizes its own run. - Every dispatch runs in a fresh
runs/<wr>/<step>/attempt-NNN/directory; earlier attempts stay on disk for diagnosis. resources.walltimeis enforced for local jobs too (walltime_exceeded,resource_error, recoverable).
Provider task failures include raw_log (the engine log tail) and
m3flow run logs <wr> [--step …] surfaces the full request/response and
workdir files.
7. Provenance¶
.m3flow/m3flow.db records every workflow_run, task_run (status, attempts,
timestamps, params, validation verdicts, engine), artifact (+files), and
every input/output edge.
m3flow artifact lineage art_c8399702 # recursive tree back to the SystemSpec
m3flow artifact inspect art_… # files, metadata, data payload, producer
m3flow run graph <wr_…> # nodes, edges, statuses
Because artifacts are content-addressed and cache hits link rather than copy, the lineage of any result is always complete and exact — you can answer "which equilibration protocol, at what temperature, with which engine build produced this density?" for any artifact in the store.
8. The CLI¶
Global: --json on every command (stdout = one JSON document; progress to
stderr). Exit codes for runs: 0 COMPLETED, 2 FAILED, 3 CANCELLED;
1 for command errors.
| command | what it does |
|---|---|
init [dir] [--name N] |
initialize a project |
task list [--category C] |
list registered tasks |
task search <query> |
full-text search over names/descriptions/tags |
task inspect <name> |
full task spec as JSON |
workflow list |
list registered workflows |
workflow inspect <name> |
full workflow spec as JSON |
workflow validate <name_or_file> |
schema-validate a workflow |
workflow plan <name> [--param k=v] |
compiled DAG, no execution |
workflow run <name> [--input n=art/@f] [--param k=v] [--no-cache] [--max-concurrency N] [--dry-run] |
execute |
run list [--limit N] |
recent runs |
run status <wr> |
one-call composite: run + steps + failure + suggested next command |
run inspect <wr> |
run record + per-step status |
run logs <wr> [--step prefix] |
per-step response documents, workdir files |
run graph <wr> |
dependency graph with statuses |
run resume <wr> |
resume, keeping completed steps |
run retry <wr> <step> |
re-execute one step + downstream |
run cancel <wr> |
request cancellation |
artifact list [--type T] [--limit N] |
newest artifacts |
artifact inspect <art> |
files/metadata/data/producer |
artifact lineage <art> |
recursive provenance tree |
artifact compatible <art> <task> |
can this artifact feed that task? |
artifact register --type T --file name=path [--meta JSON] [--data JSON] |
register existing files |
schema list |
artifact type hierarchy + document schemas |
schema show <task\|workflow\|system\|artifact> |
print a JSON schema |
provider list |
providers, availability, engine versions |
provider diagnose <name> |
locate + describe a provider |
cache stats / cache clear |
cache inspection and management |
results sync [--run <wr>] |
rebuild friendly results/ tree(s) from the provenance DB |
run label <wr> [name] |
set/clear a run's study label (moves its results/ folder) |
tui [wr] |
interactive execution cockpit |
The results/ tree¶
Every run materializes a human-browsable view of its outputs as it
executes (disable per-run with workflow run --no-materialize, or
project-wide via defaults: {materialize: {enabled: false}}):
results/<group>/<YYYY-MM-DD>_<HH-MM>_<workflow>_<wr_id>/
run.json run summary, resolved params (workflow + per-step), step → artifact map
_inputs/<name>/ workflow inputs
NN_<step>/<ArtifactType>/ symlinks to output files + _artifact.yaml
<group> is the run's study label (workflow run --label
peo-5k-screen) when it has one, or the workflow name when it doesn't — a
study folder collects runs across workflow types and reruns, while
unlabeled runs behave exactly as before. Labels are validated
([A-Za-z0-9._-], max 64 chars, starting with a letter or digit) and are
purely presentational: they never enter spec hashes, cache keys, or
artifact identity. Rename a study with m3flow run label <wr> <new-name>
(omit the name to clear) — the run's folder moves immediately. Do not
mv folders by hand: the label lives in the provenance DB and the next
results sync would resurrect the old name.
The timestamp is the run's created_at in UTC (same clock as run.json
and the database records).
Files are relative symlinks into the content-addressed store, and
_artifact.yaml in each directory carries the artifact id, type, content
hash, and producer — the tree is operable: feed any artifact id you
find here straight into the next workflow. Nothing under results/ is
authoritative (the DB + store are truth); store blobs are read-only, so
the view can never mutate them. Rebuild any tree any time with
m3flow results sync — without --run it wipes results/ first, so
orphans from deleted runs or renamed studies disappear.
| tui [wr] | interactive execution cockpit |
IDs: art_… artifact, tr_… task run, wr_… workflow run. Commands take
full ids (8 hex chars); run logs --step accepts a unique step prefix.
9. The agent-facing JSON contract¶
M3Flow is designed to be driven programmatically. With --json, stdout
carries exactly one JSON document; diagnostics go to stderr.
Success documents (stable keys):
workflow run --json→ the WorkflowRun record{id, name, version, status, inputs, params, outputs, error, created_at, started_at, ended_at, spec_hash, workdir, git}.outputsmaps names to artifact ids; outputs of SKIPPED branches are absent.run inspect --json→{run: {...}, tasks: [TaskRunRecord…]}with{id, node_id, task_name, task_version, provider, status, attempts, params, error, validation, engine, started_at, ended_at, cache_key}.run status --json→ the agent-oriented composite, answering "did it fail, where, and what now?" in one call (replaces the inspect + logs + graph loop):{run: {id, workflow, label, status, created_at, started_at, ended_at}, results_dir, materialized, steps: [{node, task, status, attempts, outputs: {role: art_id}}], failure: {step, category, recoverable, message} | null, next: "m3flow run retry …" | null}.nextis the single most useful follow-up command: retry for recoverable failures, logs for permanent ones, resume for cancelled runs, null when there is nothing to do.run graph --json→{nodes: [{id, task, label, status}], edges: [{from, to}]}.artifact inspect --json→{id, type, schema_version, content_hash, files, metadata, data, producer}.artifact lineage --json→ recursive{id, type, producer: {task_run…, inputs: [...]}}.artifact compatible --json→{compatible, matching_inputs: [{input, declared_type}]}.
Errors — a single document:
{
"error_type": "schema | type | workflow | task | provider | execution |
scientific_validation | artifact_compatibility | not_found |
io | internal",
"category": "input_error | protocol_error | task_error | provider_error |
execution_error | scientific_validation | compatibility_error |
not_found | environment_error | internal",
"recoverable": false,
"message": "human-readable",
"...": "variant-specific fields"
}
Branch on category (retry policy) and error_type (semantics). Provider
failures add raw_log with the engine log tail.
10. The provider protocol¶
Providers adapt external tools (AutoPoly, LAMMPS, Python analysis) to
M3Flow tasks. They are independent OS processes named m3flow-<provider>,
communicating exclusively through JSON on stdout/stdin — languages fully
decoupled. Protocol: m3flow-provider/1.
m3flow-lammps describe [--json]
m3flow-lammps validate REQUEST.json
m3flow-lammps execute REQUEST.json
m3flow-lammps diagnose REQUEST.json
Rules: exactly one JSON document on stdout (human chatter to stderr);
exit 0 for any well-formed response including scientific failures;
non-zero only for protocol violations.
- describe
[CONFIG.json]→{protocol, provider: {name, version}, engine: {name, version, ...}, scheduling_config_keys, tasks: [...]}. The optional argument is the engine config the runtime will forward toexecute; probe that engine. The wholeengineobject joins the cache key — report it accurately, and reportunknownwhen you cannot tell (such results are never cached). - validate →
{valid, errors[]}— check an execute request without running it. - execute — request carries
{task, workflow_run_id, task_run_id, attempt, workdir, inputs, parameters, resources, config}; each input artifact carries{id, type, content_hash, producer, files, metadata, data}. Input files are absolute CAS paths, never written to; outputs must be staged insideworkdirand returned as workdir-relative paths (no.., no symlinks out); quantities arrive canonicalized ({value, unit}). Responses are strict JSON: NaN/Infinity are protocol errors — report missing statistics asnullwith a status. Success response:{status: "success", outputs, validation: [{name, passed, detail}], engine, warnings}. Every validator named by the TaskSpec must appear with a verdict. Scientific failure:{status: "error", error: {error_type, category, recoverable, message, details, raw_log}, partial_outputs}. - diagnose — best-effort post-mortem:
{diagnostics, suggestions}. M3Flow reports facts; repair strategy belongs to the caller.
Standard error_types: lost_atoms, nan_detected, energy_blowup,
trajectory_corrupt, simulation_incomplete, engine_crash,
engine_missing, input_invalid, builder_failed, type_check_failed,
validation_failed. Standard categories: simulation_instability,
input_error, environment_error, resource_error, protocol_error,
scientific_validation.
The protocol version is sent in every request and returned by describe;
the runtime refuses providers whose major protocol differs.
11. The task library¶
19 atomic tasks in three groups. Full typed contract of each:
m3flow task inspect <name>.
construction (AutoPoly)¶
| task | summary |
|---|---|
build_system |
force-field-agnostic geometry from a SystemSpec |
parameterize_system |
assign force-field atom types and charges |
prepare_simulation_system |
pack into a simulation box → runnable SimulationSystem |
simulation (LAMMPS)¶
| task | summary |
|---|---|
energy_minimize |
potential-energy minimization of a system or state |
run_nvt |
canonical (NVT) MD |
run_npt |
isothermal-isobaric (NPT) MD |
run_nve |
microcanonical (NVE) MD — typical for dynamics production |
run_soft_pushoff |
soft-potential push-off for CG/overlapping systems |
run_deform |
uniaxial tensile deformation (fix deform, NVT) |
analysis, sampling, validation¶
| task | summary |
|---|---|
compute_density |
mean density + blocked standard error after discarding equilibration_fraction of the series |
compute_msd |
mean-squared displacement vs lag time (unwrapped coords) |
compute_rdf |
g(r), optionally between atom-type selections |
compute_rg |
radius of gyration of polymer chains (by molecule id) |
compute_ree |
mean end-to-end distance of linear chains (ends from the bond graph or an explicit selection) |
compute_adhesion |
interfacial interaction-energy density −⟨E_int⟩/(n·A) (pair + kspace terms; not the reversible work of adhesion) |
collect_thermo_series |
fan-in a temperature sweep into one TemperatureSeries |
fit_diffusion |
Einstein-relation linear fit of MSD → D (cm²/s; reduced units for lj unless sigma/tau are given) |
fit_cte |
volumetric thermal expansion coefficient α |
fit_tg |
bilinear V(T) fit → glass transition temperature |
fit_modulus |
Young's modulus from the linear stress-strain region |
check_polymer_equilibration |
density drift/shift, energy + volume trend/shift, Rg stability → evidence-bound EquilibrationReport |
promote_equilibrated_state |
SimulationState → EquilibratedState, gated on a passing report |
12. The workflow library¶
18 workflows in three layers. Inspect any of them with
m3flow workflow inspect / workflow plan.
preparation¶
| workflow | summary |
|---|---|
construct_system |
SystemSpec → runnable SimulationSystem (all three construction stages) |
simple_equilibration |
minimize → NVT → NPT at the target state point |
polymer_21step_equilibration |
Larsen 21-step compression/decompression for polymer melts |
equilibrate_polymer |
adaptive: run the base protocol, check, extend only if the check fails |
cg_push_off |
CG melt preparation: minimize, soft push-off, NVT |
interface_relaxation |
polymer/substrate interface: gentle minimize + low-NVT relaxation |
production¶
| workflow | summary |
|---|---|
npt_thermodynamic_production |
NPT production from an equilibrated state → ThermodynamicSeries |
dynamics_production |
NVE production → unwrapped ProductionTrajectory |
temperature_sweep |
NPT at each of a list of temperatures → TemperatureSeries |
mechanical_deformation |
tensile deformation → StressStrainSeries |
properties¶
| workflow | summary |
|---|---|
density |
mean density from an NPT series (equilibrated tail) |
diffusion |
MSD from an unwrapped trajectory + Einstein fit |
rdf |
radial distribution function |
cte |
volumetric thermal expansion coefficient |
tg |
glass transition temperature from bilinear V(T) |
adhesion |
interaction-energy density from an interface series |
mechanical_properties |
Young's modulus from a stress-strain series |
polymer_basic_properties |
one-shot panel for an equilibrated polymer |
Protocol workflows are versioned and immutable — fork name@new-version,
never edit a published schedule in place.
13. Authoring tasks and providers¶
A task = a YAML contract (task/v1) + at least one provider implementation.
schema: task/v1
name: compute_viscosity
version: 1.0.0
description: Green-Kubo viscosity from an NVE trajectory.
category: analysis # construction|simulation|sampling|analysis|validation|utility
inputs:
trajectory: {type: Trajectory, required: true}
parameters:
temperature: {type: temperature, required: true}
block_length: {type: time, default: 10 ps}
outputs:
result: {type: ViscosityResult} # new types go in crates/core/src/atypes.rs
requirements:
needs_time_axis: true
dynamics_sensitive: true
resolution: [atomistic]
validation: []
implementations:
- {provider: analysis, default: true}
resources: {cpu: 2, walltime: 30 min}
Rules of thumb:
- Outputs come back as files + a
datapayload for anything other steps/conditions reference (${step.result.data.value}). - Validators are the scientific gate — declare only what the provider can honestly check.
- Bump
versionwhen logic changes meaningfully; cache keys includename@versionand the provider version.
In Python, the shared runtime (providers/src/m3flow_provider/) does the
plumbing:
from m3flow_provider import Provider, ProviderFailure, artifact, verdict
def compute_viscosity(req):
# req: protocol, task, workdir, inputs (ABSOLUTE store paths),
# parameters (canonical), resources, config
traj = req["inputs"]["trajectory"]["files"]["dcd"]
temp = req["parameters"]["temperature"]["value"] # always Kelvin
...
return {
"outputs": {"result": artifact(
"ViscosityResult",
files={"json": "result.json"}, # workdir-relative
data={"value": eta, "unit": "mPa*s"})},
"validation": [verdict("trajectory_long_enough", ok, detail)],
}
def cli():
raise SystemExit(Provider(
name="analysis", version="0.3.2",
engine=lambda: {"name": "mdanalysis", "version": ...},
tasks={"compute_viscosity": compute_viscosity},
).cli())
Contract points:
- stdout carries exactly one JSON document; logs go to stderr/files.
- Staged output files are ingested into the CAS by the runtime — never move/rename them after responding.
- Raise
ProviderFailure(error_type, category, message, recoverable=…, details=…, raw_log=…); unexpected exceptions becomeengine_crash/provider_error. Categories drive scheduler retry. - Input artifact files are extensionless CAS paths — stage copies with proper filenames if your parser sniffs extensions.
Register by dropping the YAML into a project's tasks/ (shadows builtins)
or the library tasks/, then:
m3flow task inspect compute_viscosity
m3flow-analysis describe
m3flow workflow validate workflows/x.yaml
14. Authoring workflows¶
schema: workflow/v1
name: my_protocol
version: 1.0.0
description: What it does, when to use it.
kind: scientific_protocol
domain: [polymer, atomistic]
purpose: [equilibration]
references: ["citation strings"]
assumptions: ["explicit scientific assumptions"]
inputs:
system: {type: SimulationSystem, required: true}
parameters:
temperature: {type: temperature, default: 300 K}
seed: {type: integer, default: 12345}
stages: # shorthand for linear MD protocols
- {ensemble: minimize, name: min, duration: 5 ps, timestep: 0.5 fs}
- {ensemble: nvt, name: nvt, temperature: "${params.temperature}", duration: 50 ps}
steps:
check:
task: check_polymer_equilibration
inputs: {thermo: "${nvt.thermo}"}
outputs:
state: {value: "${nvt.state}"}
report: {value: "${check.report}"}
Native ramps and barostat control (LAMMPS fix nh keywords, linear start→stop over the stage):
stages:
# heat 10 K -> 300 K during NVT
- {ensemble: nvt, name: heat,
temperature: 10 K, temperature_end: 300 K, duration: 50 ps}
# decompress 10 bar -> 1 bar during NPT (isotropic)
- {ensemble: npt, name: decompress,
temperature: 300 K, pressure: 10 bar, pressure_end: 1 bar, duration: 500 ps}
# per-axis barostat, e.g. slab/interface: z-only pressure control
- {ensemble: npt, name: slab,
temperature: 300 K, pressure_style: xyz, pressure_z: 1 bar, duration: 100 ps}
# coupled in-plane + independent normal, x/y ramping 10 -> 1 bar
- {ensemble: npt, name: membrane,
temperature: 300 K, pressure_style: xyz, couple: xy,
pressure_x: 10 bar, pressure_x_end: 1 bar,
pressure_y: 10 bar, pressure_y_end: 1 bar,
pressure_z: 1 bar, duration: 200 ps}
References: ${inputs.x} (workflow input), ${step.output} (artifact
of an earlier step), ${step.output.data.field} / .metadata.field
(values inside the payload, usable in parameters and conditions),
${params.x}. Whole-string refs substitute the value; embedded refs
interpolate into strings ("${T} K" — building quantities from loop
variables).
foreach:
steps:
sweep:
task: run_npt
foreach: "${params.temperatures}" # a JSON list
as: T
inputs: {state: "${inputs.state}"}
parameters: {temperature: "${T} K"}
collect:
task: collect_thermo_series # input declared many: true
inputs: {series: "${sweep.thermo}"} # gathers sweep__0..n in order
conditions: condition: "not ${check.report.equilibrated}" — boolean
mini-language (and or not == != < <= > >=, parens, literals, ${...}
refs). False → SKIPPED, transitively. This is how equilibrate_polymer
extends NPT only when the first check fails.
Discipline:
m3flow workflow plan <name>before every run — the compiled DAG is the contract; type errors and bad references surface there, not mid-run.- Immutable once used: fork
name@new-versionto change a protocol. - Keep quantities typed end-to-end (
"${params.temperature}", never pre-baked numbers).
15. The TUI¶
m3flow tui [wr] — the execution cockpit: run header, status-colored node
list, detail pane; auto-refreshes while the run is RUNNING.
| key | action |
|---|---|
j / k |
navigate nodes |
l |
logs pane |
a |
artifacts pane |
p |
provenance pane |
g |
graph pane |
r |
retry selected node |
R |
resume run |
q |
quit |
16. Reference runs¶
examples/ref/ holds five reduced-scale end-to-end validations, all
reproducible via ./run_all.sh:
| run | demonstrates | result |
|---|---|---|
ethanol_diffusion |
SMILES → equilibrate → NVE → MSD → D | full pipeline |
peo_density |
Larsen 21-step equilibration → density | 1.069 g/cm³ (lit. ~1.1) |
polymer_multi |
property fan-out + cache | 3rd run 35/35 CACHED in ~4 s |
peo_silica_adhesion |
quartz slab + film → interaction-energy density | W = 101 mJ/m² (measured before the kspace fix: pair terms only) |
cg_melt |
bead-spring CG construct → push-off → NVT | CG path |
Reduced scale demonstrates the platform, not converged physics: ps-scale tails carry real statistical drift, and the equilibration gate says so — that honesty is the feature.
17. Design invariants¶
Do not break these; correctness and trust depend on them.
- Quantities carry units. Bare numbers are rejected for quantity
parameters —
{value, unit}or"300 K"strings, canonicalized (K, bar, fs, Å, g/cm³, kcal/mol, Ų). - Artifact identity = type + schema version + per-file content hashes. Task inputs are identified by their execution fingerprint (identity + data + semantic metadata).
- Cache key = task@version + TaskSpec fingerprint + provider@version + engine descriptor + effective engine config + input execution fingerprints + canonical params. Anything that can change a result must join the key — including passthrough config dicts.
- EquilibratedState only via
promote_equilibrated_statefrom a passing EquilibrationReport bound to that state (runtime-verified). - A resume executes the definition the run started with.
- Providers are separate processes speaking
m3flow-provider/1(one JSON document on stdout); the runtime owns ingestion into the CAS. - Protocols are immutable once published — fork versions, never edit schedules in place.
- Discovery is part of the contract — every spec, schema, and status is
reachable through the CLI with
--json; agents never need to read source.
18. Repository layout¶
schemas/ JSON schemas (task/v1, workflow/v1, system/v1, artifact/v1)
tasks/ the 19-task atomic library (autopoly / lammps / analysis)
workflows/ the 18-workflow library (preparation / production / properties)
crates/ Rust workspace: core, registry, runtime, cli, tui
providers/ Python provider processes (m3flow-provider/1)
docs/ this manual + focused documents
examples/ reference projects (reduced-scale end-to-end runs)
Focused documents: concepts.md, json-contract.md,
provider-protocol.md, writing-tasks.md, writing-workflows.md,
agent-benchmark.md (a transcript of an agent completing "build a PEO
melt, equilibrate it, calculate density and diffusion at 300 K" using CLI
discovery only).