Back
In this work, we analyze and summarize full-length movies from multimodal input (i.e., video, text, audio). We first hypothesize that identifying the narrative structure of movies is a precondition for summarizing them. According to screenwriting theory, turning points (e.g., change of plans, major setback, climax) are crucial narrative moments within a movie that define the narrative structure and determine its progression and thematic units. Therefore, we introduce the task of Turning Point (TP) identification and leverage it for movie summarization and trailer generation. Next, we propose graph-based methods for addressing the downstream tasks, where we construct sparse graphs that represent relations between scenes and/or shots in the movie. Finally, we demonstrate that knowledge about the narrative structure, as well as learning sparse graph structures for representing the events in a movie, offer performance improvements in both tasks according to human judges.
Pinelopi Papalampidi (University of Edinburgh)
PhD student
Pinelopi (Nelly) Papalampidi is a final-year PhD student at the University of Edinburgh under the supervision of Mirella Lapata and Frank Keller. Her PhD thesis focuses on structure-aware movie understanding and summarization via multimodal and graph-based methods. Before that, she completed her diploma (combined BEng and MEng) at the National Technical University of Athens, Department of Electrical and Computer Engineering. Recently, she also interned at DeepMind, focusing on narrative generation, and at MetaAI, where she addressed the task of multimodal abstractive video summarization.