Sep 2014 - NEW! We have the extended version of the ICMR paper in IJMIR. Check it out.
Story-based Video Retrieval in TV series using Plot SynopsesMakarand Tapaswi, Martin Baeuml and Rainer Stiefelhagen
ACM International Conference on Multimedia Retrieval (ICMR Oral, Full paper) Glasgow, Scotland, Apr 2014
[paper]
[presentation]
[demo poster]
[HQ video]
Demo
A demo of the proposed retrieval system was presented at CVPR 2014 and can also be found
here. Please contact me to obtain a login. Below is a video of the retrieval in action
Contributions
Story-based video retrieval in TV series
Alignment of Plot Synopsis (such as from Wikipedia) to shots in the video
Person identification and subtitles used as guidelines for alignment
Annotation and evaluation on the entire Season 5 of Buffy the Vampire Slayer (TV series)
Dataset
Buffy the Vampire Slayer (Season 5, all Episodes 1 to 22)
The dataset package contains the plot synopses used in this work (as they might have changed on Wiki)
and the annotations from one user indicating which shot is assigned to which sentence.
PRACTICAL NOTE: The frame numbers and time-stamps are for Region 2 DVDs (PAL) and have a 25 fps frame rate.
Video Eventsvidevents.tar.gz
Contains a list of automatically detected video events: shots, title song, credits.
Format: start_frame, start_time, TYPE, [end_frame,] [end_time]
Plot Synopsissynopses.tar.gz
Contains plot synopsis extracted from Wikipedia and stored in simple text format
Shot to Sentence annotationannots.tar.gz
Contains annotations for plot synopsis matching them to shots in the video
The files are encoded in JSON format and contain the sentence, start/end shot number, and the list of characters that appear in that sentence as inferred by reading the plot synopsis only
Links
At a glimpse
A snapshot of all multimedia related projects can be found here. This is the link you will come to from any of our papers too!
Multimedia Resources
I hope to collect links at one place to make it easy to browse the wonderful works.
CV, MM Papers on the Web
Meta-resources for Computer Vision conferences CVPapers, ACM Multimedia MM papers, and Multimedia Information Retrieval papers MIR.