This paper proposes two network architectures to perform video generation from captions using Variational Autoencoders. We adopt a new perspective towards video generation where we use attention as well as allow the captions to be combined with the long-term and short-term dependencies between video frames and thus generate a video in an incremental manner. Our experiments demonstrate our network architectures’ ability to distinguish between objects, actions and interactions in a video and combine them to generate videos for unseen captions. Our second network also exhibits the capability to perform spatio-temporal style transfer when asked to generate videos for a sequence of captions. We also show that the network’s ability to learn a latent representation allows it generate videos in an unsupervised manner and perform other tasks such as action recognition. © 2018, Springer Nature Singapore Pte Ltd.