framepin
v0.1 alpha — snapshot / diff / track / regress work, tested

A lockfile for your video/sequence-ML datasets.
When a metric moves: the code, or the data?

framepin pins the exact data a training run saw, links it to the run's params/metrics/commit, and answers that question directly. Zero dependencies, no server, no data copies — just JSON that commits cleanly to git.

Star on GitHub pip install framepin 
$ framepin regress <old_run> <new_run> -m val_loss
regress 02189c0fbd14 -> ab09cc84a35a
  val_loss: 0.21 -> 0.28  (Δ +0.07)

  ⚠ DATA CHANGED between these runs — the dataset version differs.
    A metric move here cannot be attributed to code alone.
That last line is the whole point. framepin separates "the code regressed" from "the data changed under you."

The problem

If you train models on video or sequence data, this has happened to you.

  • A run from three weeks ago had a better val_loss. Which dataset version was it? You re-sampled frames and re-labeled twice since. Gone.
  • val_loss got worse after a change. Was it your code … or did someone swap out 30% of the clips? Nobody can say quickly.
  • wandb/mlflow track the run but treat the dataset as an opaque string. dvc versions the data but is heavy and lives apart from your metrics. Neither answers "is this run reproducible from these exact bytes?"

Video datasets are big, churn constantly, and get reorganized — so a directory reshuffle looks like "half my data changed" in a naive diff.

Before / after

Before — a string you hope is true

run.log({"dataset": "data/clips (v3?)",
         "val_loss": 0.21})
# 3 weeks later: what was v3? which
# frames? was it re-labeled? 🤷

After — a content hash you can diff

with framepin.track(name="baseline") as run:
    run.use_dataset("data/clips")
    # -> version c29aa729c669
    run.log_metric("val_loss", 0.21)

How it works

Four commands. Everything lives under .framepin/ as plain, sorted JSON — commit it like any other file.

1

snapshot

framepin snapshot data/clips

Walks a directory, streams a SHA-256 over each file, and records a deterministic manifest keyed by a Merkle root of (path, content-hash) pairs. Same bytes → same version id, regardless of walk order or timestamps. Nothing is moved or copied.

2

diff

framepin diff <v1> <v2>

Classifies changes into added / removed / modified / moved, so a reorg of identical clips reads as moves, not churn.

3

track

framepin.track(...)

Python context manager that records a run's params, metrics, git commit, and the dataset version(s) it consumed into .framepin/runs/<id>.json.

4

regress

framepin regress <a> <b> -m metric

Compares two runs' metrics and their pinned dataset versions — the code-vs-data answer, in one command.

Also: framepin init (creates .framepin/), framepin runs (every run + its data version), framepin show <run> (full lineage: run → dataset version → files).

Why not W&B / MLflow / DVC?

framepin is deliberately small. It is not a full experiment platform — it does the one thing those tools under-serve: making the dataset version a first-class, reproducible, diffable part of every run. Use it alongside W&B if you like; framepin just owns the data-lineage question.

framepinW&B / MLflowDVC
Pins run ↔ exact dataset version ✓ first-class ⚠ manual string/artifact ⚠ separate from metrics
"Code vs data" regression answer ✓ regress — —
Detects moved/renamed files ✓ — partial
Copies your data never (hashes only) ⚠ artifacts ✓ cache/remote
Server / account required no ⚠ (or self-host) no
Runtime dependencies 0 many several
Git-friendly plain-JSON store ✓ — ✓ (pointers)

Install

Pure standard library. Nothing else gets installed.

$ pip install framepin

Requires Python ≥ 3.9. No third-party packages.

$ git clone https://github.com/boogy-ro/framepin
$ cd framepin
$ pip install -e .          # or just add to PYTHONPATH — no deps
$ python -m unittest discover -s tests

Try it, break it, tell me where it falls short

v0.3 — snapshot/diff/track/regress plus a CI gate with allow-lists, committed pinfiles, per-split manifests, and MLflow/W&B export. APIs may still shift before 1.0. Video/sequence ML data lineage is a narrow niche; feedback decides whether this is worth building further.

★ Star on GitHub Open an issue
pip install framepin