Within AI image generation, speed poses an ongoing challenge. Specifically, the process of creating images with models such as ChatGPT or Stable Diffusion frequently extends over several minutes. Mark Zuckerberg, CEO of Meta, conveyed discontent with image generation speeds during the previous year's Meta Connect event.
Addressing this concern, Hugging Face has unveiled a solution named aMUSEd, designed to create images within mere seconds. This lightweight text-to-image model, based on Google's MUSE, boasts approximately 800 million parameters and holds potential for deployment in on-device applications like mobile phones.
The distinctive speed of aMUSEd is attributed to its Masked Image Model (MIM) architecture, a departure from the latent diffusion used in other image generation models. The Hugging Face team explains that MIM reduces inferencing steps, thereby enhancing both the model's generation speed and interpretability. The model's compact size further contributes to its rapid functionality.
aMUSEd is currently available as a research preview, accompanied by an OpenRAIL license, allowing experimentation and customization while remaining commercially adaptable.
The Hugging Face team acknowledges the potential for enhancing the image quality generated by aMUSEd. They release the model with the intention of encouraging the exploration of non-diffusion frameworks like MIM for image generation.
Notably, aMUSEd exhibits capabilities beyond its swift image creation. It can perform zero-shot image inpainting, a feat not achievable by Stable Diffusion XL, as highlighted by the Hugging Face team.
The MIM methodology in aMUSEd mirrors techniques employed in language modeling, wherein specific data portions are masked, and the model learns to predict these concealed elements. In the case of aMUSEd, this pertains to images instead of text.
During the model's training phase, input images are converted into tokens using the VQGAN (Vector Quantized Generative Adversarial Network) tool. These image tokens undergo partial masking, with the model trained to predict the masked sections. Predictions are made based on unmasked portions and prompts using a text encoder.
In the inferencing process, the text prompt undergoes conversion into a format understood by the model via the same text encoder. aMUSEd initiates with a set of randomly masked tokens, progressively refining the image. At each refinement step, it predicts image sections, retains the confidently predicted parts, and continues refining the rest. After a specified number of steps, the model's predictions pass through the VQGAN decoder, culminating in the final image.
Furthermore, aMUSEd allows fine-tuning on custom datasets. Hugging Face showcased the model fine-tuned with the 8-bit Adam optimizer and float16 precision, utilizing just under 11 GBs of GPU VRAM.
