How to Use Sora from OpenAI

Sora is making waves across the internet, and it's so popular that even my grandma, after catching a CNN news segment, asked me if my job would be at risk in the coming years. The concerns and uncertainty surrounding Sora are valid, but they also spark creativity and open up exciting new possibilities.

What is Sora?

You've probably heard it before, but let me reiterate: Sora is an AI video generator that takes your text prompt and creates a video that matches the description. Sound similar to established names such as Runway ML and Pika labs? Sora exceeds in these aspects compared to previous models:

• Sora is capable of creating up to a minute-long video.
• Sora can generate coherent and realistic videos with dynamic camera movement and different angles.
• Sora can simulate simple actions that affect the state of the world, such as a painter leaving brushstrokes on a canvas or a dent when a person eating a burger.

Is Sora released now?

Sam Altman announced Sora on X (Twitter) on Feb 16, 2024, but this video generation model hasn't been publicly released yet. According to Sam, the company is starting the red-teaming to understand how Sora will be misused. OpenAI is offering access to a selected range of creators.

Sora Is Granting Access to Artists Now

On March 25, 2024, OpenAI published a blog titled Sora: First Impressions, where they showcase Sora's ability to "make things that are totally surreal". The article features seven videos created by visual artists, filmmakers, designers, and creative directors. You can watch Sora videos here.

According to Paul Trillo, working with Sora has liberated him as the director, allowing him to ideate and experiment in bold and exciting ways without the usual constraints such as time and money.

This opens up a new era of abstract expressionism, enabling artists to expand on stories and concepts that were once thought impossible to visualize.

1. How to Join the Waitlist for Sora

At present, there isn't an open waiting list for broad access to Sora officially. The sole pathway available appears to be participation in the red teaming program through application.

Due to the high demand from the general public, even if you apply for red teaming, you may be in for a lengthy wait, especially since OpenAI is looking for legitimate experts from diverse fields.

Join the OpenAI red teaming: https://openai.com/form/red-teaming-network

2. How to Get Alert When Sora Is Publicly Available

1. Join Sora AI communities on Reddit.

SoraAI: A non-official community, currently the largest devoted to Sora-related content.

OpenAI: With over 1.1 million subscribers, this community can provide you with up-to-the-minute information when Sora is released.

2. Subscribe to AI newsletters.

Below are some popular resources that offer you the latest trends in AI. You won't miss it when Sora AI is ready for the public.

The Rundown AI: Daily AI newsletter that covers the latest AI trends and tools.

Superhuman AI: Daily AI newsletter that delivers up-to-date news and stories.

3. Enable Discord alert.

To get notified when Sora becomes publicly available, follow these steps:

1. Join the unofficial SoraAI Discussion Community on Discord.
2. Head to the "waitlist" channel.
3. Look for the alert icon and respond to it.

This community uses Discord's ping system, which allows members to notify each other instantly when a whitelist is live. By interacting with the alert icon, you'll be kept in the loop and won't miss out on Sora's public release.

4. Follow AI influencers on X (Twitter).

Rather than suggesting specific Sora AI influencers to follow, we'll guide you on how to tailor your Twitter feed intuitively.

1. Follow AI influencers on Twitter, focusing on those who frequently post about Sora, text-to-video, and AI video generation.
2. Search SoraAI and interact with posts with the most views, comments, reposts, and likes.
3. Engage with AI-related content, especially topics about Sora, to signal your interest in Twitter.

By consistently following these steps, Twitter's algorithm will likely prioritize AI and Sora-related content on your timeline, increasing the chances of seeing updates about Sora's public release. Meanwhile, you can learn more about Sora and enjoy showcasing videos.

3. How to Use Sora in Various Scenarios

From what has been disclosed from OpenAI, people can use Sora with text, image and video prompts. It excels at 3D consistency and can maintain coherence across the scenes. Sora AI can simulate real-world people, animals, and landscapes.

Text to Videos

Sora can understand your instructions and generate longer videos.

Prompt: A brown and white border collie stands on a skateboard, wearing sunglasses

Image to Images/Videos

You can turn a still image into images, animations or dynamic videos.

"Travel" Backward or Forward

Sora can extend videos in both forward and backward directions in time – from a segment of the video at your instruction. If you have been surprised previously with AI image outpainting, then this video-extending feature is just mind blowing.

With this feature, you can create interesting loop videos for infinite playback.

Video to Video

• Style transformation: Sora can transform the styles and elements from one video into another. For a car running on a mountain, you can alter it to race in lush jungles or cyberpunk city streets.
• Video merging: Sora can merge two videos into one for an enchanting visual experience.

AI Video to 3D Models

The ability to convert Sora AI video into 3D models hasn't been demonstrated the OpenAI's official presentations or insider clips thus far. However, this potential capability has been eagerly anticipated and positively debated among AI enthusiasts. The excitement stems from the possibility that, if successfully implemented, it could revolutionize the existing workflow in sectors like VFX, 3D modeling, gaming, and beyond.

Currently, some AI enthusiasts are testing Sora-plus-other-tools in the 3D modeling workflow. For instance, you can upload Sora videos into Poly.cam and use the Gaussian Splat tool. Although the rendering is not ideal currently, it opens up the possibilities for the future.

4. Best Prompts for Sora AI Videos

While Sora AI is not yet publicly accessible, insider creations and the official showcase videos have given us a glimpse into its capabilities. Here are some of the best Sora prompts.

Prompt: A red panda and a toucan are best friends taking a stroll through Santorini during the blue hour.

Prompt: In a beautifully rendered papercraft world, a steamboat travels across a vast ocean with wispy clouds in the sky. vast grassy hills lie in the distant background, and some sealife is visible near the papercraft ocean's surface

Prompt: A giant cathedral is completely filled with cats. there are cats everywhere you look. a man enters the cathedral and bows before the giant cat king sitting on a throne.

Prompt: POV footage of an ant navigating the inside of an ant nest.

5. The Technical Aspects: Spacetime Patches, the Diffusion and Transformer Models

OpenAI shares the insight when creating Sora on their research page, and here are some brief explanations.

• Input the raw video into an OpenAI-trained network that can reduce the dimensionality of videos and images.
• The network outputs a temporally and spatially compressed latent space.
• Sora can generate videos from this compressed latent space (Sora is also trained on that latent space).
• Then, there is a decoder model that can "translate" the generated latent (where we can't see) back to pixel space (where we can see).

With the diffusion model, Sora can predict the original "clean" patches out of the input noisy patches. What makes it more scalable is the diffusion transformer. For instance, 32x compute can improve the video quality significantly better than 4x compute, given a fixed seed.

Patches here is Sora's analogy to text tokens in large language models such as ChatGPT. During the training, the spacetime patches are extracted from the compressed input videos and served as tokens for the transformer model.

This patch-based scheme helps Sora to become the general simulator, without being restricted by video resolutions, aspect ratios, and durations, thus voiding errors that other models have due to fixed specs.

In other words, that's why we are seeing those astonishing realistic videos with consistency both spatially and temporally.

Above is a brief introduction of Sora on OpenAI's website. We can infer from it the model's capabilities and underlying goals. It is aiming for something big.

While Sora is primarily framed as a text-to-video generator, it is also intended to serve as a platform for building "world simulators" or, in OpenAI's words, "general purpose simulators of the physical world." This is highlighted in the Sora research paper.

From the technical discussions above, we can see that the use of spacetime patches is essential for fueling the world simulator.