Audio#Model inference

TubeletGraph keeps tracking objects after they are cut, peeled or split

Official NeurIPS 2025 code composes SAM 2.1, FC-CLIP and GPT-4.1 zero-shot to track objects through shape changes and build a state graph from one annotation.

Project facts

GitHub Ecosystem
Repositorygithub.com/YihongSun/TubeletGraph
License
MIT
Language
Python
Stars
269
Data checked
2026-11-20

Snapshot figures reflect the check date and may change over time.

Tracking an object that changes form is much harder than tracking a pedestrian crossing a street: when an apple is cut, a cloth is torn or a shrimp is peeled, the number of shapes on screen changes, and conventional trackers — SAM 2 included — either lose the object or treat the pieces as new ones. TubeletGraph is the official implementation of the NeurIPS 2025 paper “Tracking and Understanding Object Transformations,” built for exactly this case. GitHub’s API reported 269 stars on November 20, 2026.

Core features

  • Zero-shot composition: no retraining; it combines pretrained models such as SAM 2.1 (with multi-mask prediction), FC-CLIP and GPT-4.1. Annotate the target in the first frame and the pipeline tracks it through the rest of the video.
  • Records transformations, not just positions: beyond per-frame masks, the framework builds a state graph recording when each sub-object separated, through which action, and from which parent — “cut,” “peel,” “pour.”
  • Benchmarks for shape change: the paper evaluates on VOST, VSCOS and M³-VOS, covering cut, torn and partially removed objects; the repository ships the evaluation scripts and visualizations.
  • Extensible data path: custom-dataset instructions and examples make it possible to apply the same tracking flow to quality inspection or robotic manipulation.

The relationship to SAM 2 is easy to misread: TubeletGraph does not replace it but composes SAM 2’s multi-mask prediction, FC-CLIP’s semantics and GPT-4.1’s visual-language judgment into an observe — decide — record loop. Which frames trigger a separation decision, and how the state graph merges, are detailed in the paper and project page. The evaluation scope matters too: VOST and similar benchmarks measure tracking quality given a first-frame mask, so detection and segmentation errors upstream are out of scope, and GPT-4.1 adds inference cost and a network dependency.

Typical use cases

  • Industrial inspection and food processing: cutting, peeling and portioning change object state; a state graph records where each piece came from, which is more useful for traceability than detection boxes alone.
  • Robotics and embodied research: grasping and cutting change object topology, and the state graph gives downstream policies an object history rather than a single frame. For an AI-native video workflow instead, see OpenFilm.
  • Annotation and editing assistance: automatically produced object trajectories with provenance help editors pick and compose footage.

Quick start

The README’s environment uses conda and PyTorch 2.7 (CUDA 12.6):

conda create -n tubeletgraph python=3.10 -y
conda activate tubeletgraph
pip install torch==2.7.0 torchvision==0.22.0 --index-url https://download.pytorch.org/whl/cu126
pip install -r requirements.txt

Then install SAM 2 with multi-mask prediction, CropFormer (detectron2) and FC-CLIP as described, and set an OpenAI API key (GPT-4.1 queries require it). The repo includes a Quick Start and notebooks; run the example video before pointing it at your own data.

Summary

TubeletGraph is research code rather than a plug-and-play product: the dependency chain is long (SAM 2, detectron2, FC-CLIP, an OpenAI key) and it expects a GPU and some setup effort. But it addresses a real gap — existing trackers assume object shape stays constant, while cutting, peeling and pouring break that assumption. GitHub’s API reported Python, MIT, 269 stars, an unarchived repository and a push on November 20, 2026.

This entry is based on the README and the paper abstract; we did not deploy it or reproduce the experiments. Benchmark numbers follow the paper and project page — verify on your own footage before production use.