Meta has shared another contestant in the AI video race that’s seemingly taken over much of the industry in recent months. The tech giant released a new model called Movie Gen, which, as the name indicates, generates movies. It’s notably more comprehensive in its feature list than many others in its initial rollout, comparable to OpenAI’s Sora model, which garnered so much attention upon its initial unveiling. That said, Movie Gen also shares with Sora a limitation on access to specific filmmakers partnering with Meta rather than a public rollout.
Movie Gen is impressive based on the demonstrations of its ability to produce movies from text prompts, as seen above. The model can make 16-second videos and upscale them to 1080p resolution. The caveat is that the video comes out at 16 frames per second, a speed slower than any filming standard. For a more normal 24 fps, the film clip can’t be more than 10 seconds long.
Movie Gen Action
Still, 10 seconds can be plenty with the right prompt. Meta gave Movie Gen a fun personalization feature reminiscent of its Imagine tool for making images with you in them. Movie Gen can do the same with a video, using a reference image to put a real person into a clip. If the model can regularly match the demonstration, a lot of filmmakers might be eager to try it.
And the videos aren’t just limited to a prompt that then has to be rewritten to make another video that you hope will be better. Movie Gen has a text-based editing feature where a prompt can narrowly adjust one bit of the film or change an aspect of it as a whole. You might ask for characters to wear different outfits or set the background to a different location. That flexibility is impressive. The adjustments extend to the camera moves too, with panning and tracking requests understood by the AI and incorporated into the video or its later edits. The awareness of objects and their movements is likely borne out of the SAM 2 model Meta recently released, which is capable of tagging and tracking objects in videos.
Audio AI Future
Good visuals are all too common now among AI video makers, but Meta is going for the audio side of filmmaking too. Movie Gen will use the text prompts for the video to produce a soundtrack that blends with the sight, putting rain sounds in a rainy scene or car engines revving to go with a film set in a traffic jam. It will even create new music to play in the background and try to match the mood of the prompted video. Human speech is not currently part of Movie Gen’s repertoire.
Meta has kept impressive AI engines from the public before, most notably with an AI song generator it said was too good to release due to concerns over misuse. The company didn’t claim that as the reason for keeping Movie Gen away from most people, but it wouldn’t be surprising if it was a contributing factor.
Still, going the OpenAI Sora route means Meta has to ignore the possibility of a more open rival winning some of its market share. And there are an awful lot of AI video generators out or coming soon. That includes the new or recently upgraded models from Runway, Pika, Stability AI, Hotshot, and Luma Labs’ Dream Machine, among many others.
Meta’s new AI model tags and tracks every object in your videosHotshot is heating up AI video-making – here’s how you can try it for freeByteDance hits play on AI video creator while Sora seems paused