Diffusion Transformers Power OpenAI's Sora and Reshape Generative AI
OpenAI's impressive Sora video generator relies on an older but powerful AI architecture called the diffusion transformer. By replacing older U-Net backbones with transformers, this new approach enables generative models to scale to unprecedented levels.
OpenAI showcases the impressive capabilities of its new Sora model, which generates highly realistic videos and interactive 3D environments using an AI architecture known as the diffusion transformer. This foundational technology originates from a 2022 research project led by NYU professor Saining Xie and William Peebles, who currently co-leads Sora at OpenAI. By successfully merging the concepts of diffusion and transformers, the researchers create a powerful engine that also drives Stability AI's newest image generator, Stable Diffusion 3.0.
Traditional diffusion models rely on a complex backbone called a U-Net to learn how to slowly subtract noise from data in order to generate fresh images or videos. While effective, these U-Net structures contain specialized modules that significantly slow down the generation process. Diffusion transformers solve this efficiency problem by entirely replacing the U-Net backbone with transformer architectures, which deliver a substantial performance and speed boost to the media creation pipeline.
Transformers power complex reasoning models like GPT-4 and Gemini through a unique attention mechanism that evaluates the relevance of all input data simultaneously. When applied to diffusion tasks, this mechanism allows the AI to process image noise much more simply and effectively than previous architectures. This streamlined approach unlocks incredible new scaling potential, proving that diffusion transformers are set to upend the generative AI landscape by enabling models to grow far beyond previous limitations.