EDICT: Putting the Art in Artificial Intelligence
Text-to-image generation has seen enormous innovation in recent years. It is now straightforward to generate an image from a text description. But text-based image editing still has plenty of room to improve. Researchers at Salesforce developed an algorithm called EDICT, Exact Diffusion Inversion via Coupled Transformations, aimed squarely at that gap.
EDICT is a text-to-image diffusion generation algorithm that works with any existing diffusion model. In image generation, diffusion models are generative models that use a diffusion process to produce new images. The process begins from a random image, then iteratively filters it by applying a series of transformations until it reaches a final image resembling the target.
The core idea
To edit an image, EDICT works on obtaining a noisy image that would exactly reproduce the original image when given the original prompt. It is an inverse noising technique. If the original text is then slightly altered, the edited image stays mostly unchanged except for the alterations you asked for.
Consider generating an image of a cat surfing by editing an existing image of a surfing dog. In traditional methods a lot of detail is lost, the waves, the colour of the board, and so on, because noise is simply added to the original image to generate the new one.
In EDICT, reverse generation finds a noisy image that would exactly generate the original. That noisy image then regenerates the actual surfing dog with help from the textual caption. The noise from the generated image is copied to query the model again with the picture without noise. The text is then tweaked, replacing dog with cat, and a comparatively detailed edited image of a surfing cat comes out.
The mechanism is to make two identical copies of an image and alternately improve each one with details from the other, in a reversible manner.
Merits
- Preserves important content of the image by inverting the generation process
- Produces a more detailed and accurate result
- Allows editing in a genuinely reversible way
Demerits
- Still a new algorithm, so its full potential is not yet understood
- May not suit every type of image or editing task
- The complexity can make it difficult to implement in some applications
Overall, EDICT is a promising approach to text-based image editing with real potential to improve accuracy and detail. More research is needed to fully understand its capabilities and limits.