SDXL is a latent diffusion model where the diffusion process operates within the latent space of a pretrained, learned (and fixed) autoencoder. While most of the semantic synthesis is performed by the latent diffusion model, we can enhance local high-frequency details in generated images by improving the quality of the autoencoder. To achieve this, we train the same autoencoder architecture used in the original Stable Diffusion model with a larger batch size (256 vs. 9) and additionally track weights using Exponential Moving Average (EMA). Results show that the improved autoencoder outperforms the original model across all evaluated reconstruction metrics, as detailed in the table below.

Evaluation SDXL-VAE vs Original kl-f8 VAE vs f8-ft-MSE

COCO 2017 (256x256, val, 5000 images) Model rFID PSNR SSIM PSIM Link Comments SDXL-VAE 4.42 24.7 +/- 3.9 0.73 +/- 0.13 0.88 +/- 0.27 https://huggingface.co/stabilityai/sdxl-vae/blob/main/sdxl_vae.safetensors Used in SDXL Original 4.99 23.4 +/- 3.8 0.69 +/- 0.14 1.01 +/- 0.28 https://ommer-lab.com/files/latent-diffusion/kl-f8.zip Used for SD ft-MSE 4.70 24.5 +/- 3.7 0.71 +/- 0.13 0.92 +/- 0.27 https://huggingface.co/stabilityai/sd-vae-ft-mse-original/resolve/main/vae-ft-mse-840000-ema-pruned.ckpt Resumed EMA from ft-EMA, focusing on MSE (logged loss = MSE + 0.1 * LPIPS), outputs are more