An Open Episode Interchange Format for Surgical Physical AI

Surgical data science has a data-abundance problem hiding inside a data-scarcity story. Over the last decade the field has produced genuinely excellent per-task corpora: Cholec80 and CholecT50 for laparoscopic cholecystectomy, CATARACTS for cataract surgery, and the EndoVis challenge datasets across a dozen procedures. Each is meticulously annotated by people who know the OR. And each ships in its own label format — its own phase taxonomy, its own tool vocabulary, its own file layout, its own idea of what a "step" even is.

What the field does not have is a shared episode interchange format: a neutral on-disk shape for one annotated workflow episode that any of these corpora could serialize into and any trainer could read. General robot learning has this. Open X-Embodiment standardized on RLDS; LeRobot gave the Hugging Face ecosystem a common dataset object; DROID showed a large multi-institution corpus could agree on one schema. Those formats became load-bearing — you can pull data from several labs and train without writing several loaders. Surgical AI has no such layer, so every project starts with a week of glue code, and every glue script is one more place a phase boundary quietly shifts by a frame.

We want to propose a candidate for that missing layer. It is v0.1, pre-capture, and explicitly a proposal inviting critique — not an adopted standard. But it is real enough to install and run today, and it already round-trips through real surgical labels. Here is the shape of it and how to tear it apart.

The proposal: an episode in four layers

The Kindly Episode Schema describes one episode as four independent label tracks over a single timeline. You can consume any subset.

• L1 — Workflow phase / step. A two-level hierarchy over the timeline: phases decomposed into ordered steps. This is the surface most surgical-video datasets stop at. We treat it as the easiest, lowest layer — necessary, but not where the differentiated value lives.
• L2 — Action triplets. <actor-role, verb, object> intervals with start/end timestamps. The first slot is who acts (tech, nurse, assist, robot, surgeon), because in workflow automation who acts is a control variable, not a footnote. Actions are annotated as intervals, not per-frame classes, so extent is preserved for behavior cloning and goal-conditioned imitation.
• L3 — Instrument / object state. Per-object tracks that persist across the episode — location zone, sterility state, count events — emitted as discrete transitions (added, removed, recount, relocated, missing-confirmed) with piecewise-constant state in between. This is what turns a video into a manipulable world state a policy can reason over.
• L4 — Workflow-context events. Typed events for the things that make clinical work real: interruptions, corrections and rework, protocol deviations, and handoffs.

L4 is the differentiator, and it is the layer almost every existing corpus omits. A policy trained only on clean runs learns the happy path and fails the moment reality diverges. L4 explicitly localizes the recoveries, deviations, and handoffs — exactly the transitions a robust policy must imitate, and exactly what a phase-classification corpus structurally cannot provide. It is also the hardest supervision to collect, because it requires clinical judgment to label, not just a pixel classifier.

Every episode serializes natively to the RLDS episode/step model and round-trips to LeRobot. Crucially, the per-episode JSON sidecar (the L1–L4 intervals and events) is the source of truth; the per-step RLDS flattening is a derived, regenerable view. There's no lossy "everything must be per-step" trap: your existing OXE or LeRobot loader ingests episodes without a custom parser.

Does it survive contact with real data?

A schema that only validates its author's synthetic examples has proven nothing. So the honest test: can it represent labels somebody else already published?

We built a worked episode that maps the published annotation taxonomy of Cholec80 and CholecT50 into the four layers. L1 uses Cholec80's seven-phase taxonomy and documented phase order. L2 uses CholecT50's <instrument, verb, target> triplet classes (6 instruments, 10 verbs, 15 targets; 100 triplet classes), each carrying its ivtmetrics class id. L3 uses Cholec80's seven-tool presence set, mapped to a coarse in-field / absent zone. And L4 is deliberately empty — because the public corpora do not annotate workflow-context at all. That empty track is the argument: it is precisely the layer this schema adds.

→ Inspect the real-data episode (JSON) — or explore it interactively alongside two synthetic episodes.

Two things must be crystal clear. First, this is a mapping demonstration, not captured or redistributed data. Cholec80, CholecT50, and the ivtmetrics map are the work of CAMMA, University of Strasbourg / IHU Strasbourg, released under CC BY-NC-SA 4.0 (ivtmetrics under BSD-2-Clause). Kindly did not capture, own, or license that footage and has no affiliation with the authors. No video and no restricted per-frame ground-truth files are reproduced — only the published label taxonomy and documented phase order; per-frame timings are representative placements, not copied ground truth. Full attribution and citations (Twinanda et al. 2017; Nwoye et al. 2022) live in the episode's source block. These are valuable per-task corpora; the interchange format complements them, it doesn't replace them.

Second, the other two sample episodes — an SPD tray reassembly and an OR instrument count — are synthetic, labeled as such, every value fabricated for illustration pending our own capture. Kindly has captured no real clinical data. That is the accurate state of things, and we would rather say it plainly.

Adopt it, or tear it apart

The fastest way to engage is to validate an episode against the schema. The open CLI bundles the v0.1 schema, so it's one offline command with no Kindly account:

pip install foodforthought-cli
ate episode validate your-episode.json

# Start from a valid skeleton, then fill it in:
ate episode scaffold --out episode.json

The schema is permissive by design — object nodes allow additional properties, so a superset of your fields still validates. It interoperates rather than locking you in. The machine-readable JSON Schema (draft 2020-12) has a canonical $id of https://kindlyrobotics.com/clinical/kindly-episode.schema-v0.1.json, and the full prose standard — envelope, conformance, and SemVer policy — is at /foodforthought/clinical/schema.

This sits deliberately alongside existing surgical conventions like SAGES workflow descriptions and the OntoSPM ontology — it is an interchange serialization, not a competing ontology, and it is meant to be argued with. If the layer boundaries are wrong, if L2's actor-role slot is over-engineered, if L4's event types miss something your procedure needs — that is the feedback that makes a v0.2 worth publishing. Map one of your own episodes and tell us where it broke. A proposed standard earns the name only by surviving other people's data.