AI Outtakes, Vol. 58 📽️

Video!

While the current state of AI-generated video is modest, mostly due to the short duration of output (AI videos from popular models are only a few seconds long), there have been some impressive outputs.

This week we're taking some of our classic images from past editions of AI Outtakes and seeing what the current crop of AI Video platforms can do with them.

For this edition we're using basedlabs.ai . I don't have any affiliation with them - they were just the most user-friendly platform I could find that offered a web interface to popular video models. They have three on their platform: Luma Labs Dream Machine, Stable Diffusion Video (SDV), and i2vgen.

Guess which one of those dev teams didn't hire a marketer.

Each of them work the same way. You start with an image, and add a text prompt explaining what the image is (except for SDV, which handles this automatically), and press go. A little while later you get your video.

For all of the examples we'll show the original image, the text prompt if we used one, and the model we selected. Then we'll show the video exactly as it was generated.

This should be fun!

First up is a classic prompt that dates all the way to the Goth IHOP edition, Volume 11: Goths eating pancakes. The original image:

Given the image was created with Stable Diffusion, let's see what SDV can do first:

A goth eating a stack of three pancakes in an IHOP restaurant.

Basically just doing a camera pan? Still, interesting that it could generate a reasonable extension of the background outside the image and apply it seamlessly.

Luma Labs, what you got?

This one actually managed to visualize the goth eating! Visually, it's quite artificial, but the arm motion and mouth movement are relatively realistic.

Okay, let's give our third contender, i2vgen, a try:

No arm movement, but it had a decent implementation of chewing.

---

That was a pretty good head-to-head comparison.Let's try some other images. The Goth IHOP edition had a couple of other real-life implementations of memes, so let's try the "This is Fine" dog:

Luma had the best output from the first image, so let's give that model a try with this prompt:

A dog wearing a hat, sitting on a chair. The room is smoky, as though there is a fire. A plate with two oranges is on the table next to the chair.

This is actually pretty good! We didn't prompt the dog to do anything specifically, but it added motion and even made the smoke in the background move.

Let's try another meme image from that edition, "Surprised Pikachu" IRL:

Okay Luma, let's see what you do with this one.

a pikachu looking surprised

No idea at all what's going on with the tail. It was briefly a hobby horse, but that's just chaos.

I think the big takeaway for prompting with Luma is to use the prompt to describe motion you'd like to see in the video, but also given how short the duration is, the description and motion can't be too long or complex.

This time, let's use an image from one of our most popular editions, Volume 5, where we introduce 'Waldo':

This time let's go back to SDV and see what it decides to do with this image:

three businesspeople in an office meeting

They look like NPCs from a mid-2010s video game.

Okay, last shot. Let's take an original image that didn't turn out so well and see what AI video does to it. This time we'll use a more recent edition, Volume 51 (the breakdancing one). Here's "Ed Sheeran breakdancing":

Let's see what AI can do here. First up is SDV:

ed sheeran breakdancing in an alley

Okay, AI does NOT know breakdancing.

---

That's all for this week! The staff here at AI Outtakes really learned a lot from this foray into video, and we hope you did too. Let's close out with the "waving goodbye" image from AI Outtakes Vol. 40:

Let's close out with the video model that performed "best" in this edition, Luma.

three aliens in Times Square, waving goodbye