Behind the Scenes: AURORA

October 2024 — Appendix to the AURORA paper

A word-by-word excerpt from the Behind the Scenes appendix of Learning Action and Reasoning-Centric Image Editing from Videos and Simulations (NeurIPS 2024, Datasets & Benchmarks, Spotlight). This was the first paper where I included such a section; it has since become a tradition for all my first-author papers. See also the Behind the Scenes of my whole PhD.

In this section we want to document the whole research process and not just the final product. Specifically, we show how the paper went from first idea to what you read here. By doing this, we also show the things that did not work and might be insightful for other researchers.

We started the project in January 2024 coming out of the Christmas holidays: The first author had realized that another project wasn’t going anywhere,¹ and was going through some other ideas with the more senior authors. We settled on the broad direction of “How can we leverage nearby video frames for image editing?” because it seemed like learning from videos for these sorts of tasks is the next step that not many have explored, and because it fitted into the story of previous PhD papers of the first author.

Most of January, February and March were spent scouting various datasets:

Discarded/unsuccessful datasets

Simply repurposing change caption or contrastive datasets (Park et al., 2019; Tu et al., 2023; Krojer et al., 2022) that contain two images and some captions of how they are different is not enough: To give one example, change captions are often underspecified (“the car changed its location”, “the man is not touching the shelf”) with nondescript prompts or images containing more changes than described in prompts.

We also experimented with the initial pre-training stage on noisier but large scale data such InstructPix2Pix, i.e. adding noisier video-data or HQ-Edit to the mix. But we found these datasets not very helpful via manual inspection of outputs: they often diverge strongly from the source image, or artifacts of automatic data curation (GenHowTo)

We came across Michel et al., 2023 too late. It might have either complemented or even replaced Kubric for the synthetic generation part.

We even collected almost 10K change/edit descriptions on top of nearby frames from MSR-VTT videos (Xu et al., 2016), see the appendix of the paper for 16 samples from this collected data. At that point, in late March we finally began to realize how much harder this task was than we initially expected: Most videos are not really suitable, so even with high-quality human annotation it won’t lead to a strong editing model. Videos are hard to learn from!

Initially we had set out to build a very general editing model, thus looking at very general video datasets from all of YouTube or lots of movies. We ended up using narrower videos depicting well-defined actions since videos in the wild do not depict meaningful edits. This narrowed scope was initially frustrating but ultimately led to a better paper.

Reflections on choosing and managing a research project (from a first person perspective of the first author)

This project taught me how important it is to really really define your research question and contribution as early as possible, and for that it helps knowing the literature and what it means to contribute to science. I should know early whether my main contribution is methodological, a new training dataset or better evaluation, or something else! From there, I should have one (or maybe two) clear research question in mind (and not five!). This doesn’t mean there can be other questions but ideally they should be sub-questions of the main one and emerge as ablation studies later on. Most of these lessions were painfully learned during the final writing which becomes very hard when there is too many things you wanted to contribute and the project was many different things at different points.

Tips for anyone working on something similar

1. Learning from videos is hard so don’t naively assume large-scale YouTube will solve anything. Lots of curation is needed.
2. I am curious if video generation is a more viable approach to the edits we are looking at. However it is more costly if all you care about is the edit so maybe there is some way to compress/distill the video model into an editing model?
3. Good metrics are almost non-existant. Please don’t use old ones simply because someone else did. Rely on human ratings, and do them well. Let’s build better metrics!
4. Image editing seems like a more intriguing problem for people interested in vision-and-language reasoning, compared to the more mainstream text-to-image generation.