MTVG: Multi-text Video Generation with Text-to-Video Models

[arxiv] 

Abstract

Recently, video generation has attracted massive attention and yielded noticeable outcomes. Concerning the characteristics of video, multi-text conditioning incorporating sequential events is necessary for next-step video generation. In this work, we propose a novel multi-text video generation~(MTVG) by directly utilizing a pre-trained diffusion-based text-to-video~(T2V) generation model without additional fine-tuning. To generate consecutive video segments, visual consistency generated by distinct prompts is necessary with diverse variations, such as motion and content-related transitions. Our proposed MTVG includes Dynamic Noise and Last Frame Aware Inversion which reinitialize the noise latent to preserve visual coherence between videos of different prompts and prevent repetitive motion or contents. Furthermore, we present Structure Guiding Sampling to maintain the global appearance across the frames in a single video clip, where we leverage iterative latent updates across the preceding frame. Additionally, our Prompt Generator allows for arbitrary format of text conditions consisting of diverse events. As a result, our extensive experiments, including diverse transitions of descriptions, demonstrate that our proposed methods show superior generated outputs in terms of semantically coherent and temporally seamless video.Video examples are available in our project page: https://kuai-lab.github.io/mtvg-page.

@misc{oh2023mtvg,
      title={MTVG : Multi-text Video Generation with Text-to-Video Models}, 
      author={Gyeongrok Oh and Jaehwan Jeong and Sieun Kim and Wonmin Byeon and Jinkyu Kim and Sungwoong Kim and Hyeokmin Kwon and Sangpil Kim},
      year={2023},
      eprint={2312.04086},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}