stabilityai/sdxl-vae

SDXL - VAE

How to use with 🧨 diffusers

You can integrate this fine-tuned VAE decoder to your existing diffusers workflows, by including a vae argument to the StableDiffusionPipeline

from diffusers.models import AutoencoderKL
from diffusers import StableDiffusionPipeline

model = "stabilityai/your-stable-diffusion-model"
vae = AutoencoderKL.from_pretrained("stabilityai/sdxl-vae")
pipe = StableDiffusionPipeline.from_pretrained(model, vae=vae)

Model

SDXL is a latent diffusion model, where the diffusion operates in a pretrained, learned (and fixed) latent space of an autoencoder. While the bulk of the semantic composition is done by the latent diffusion model, we can improve local, high-frequency details in generated images by improving the quality of the autoencoder. To this end, we train the same autoencoder architecture used for the original Stable Diffusion at a larger batch-size (256 vs 9) and additionally track the weights with an exponential moving average (EMA). The resulting autoencoder outperforms the original model in all evaluated reconstruction metrics, see the table below.

Evaluation

SDXL-VAE vs original kl-f8 VAE vs f8-ft-MSE

COCO 2017 (256x256, val, 5000 images)

Model	rFID	PSNR	SSIM	PSIM	Link	Comments
SDXL-VAE	4.42	24.7 +/- 3.9	0.73 +/- 0.13	0.88 +/- 0.27	https://huggingface.co/stabilityai/sdxl-vae/blob/main/sdxl_vae.safetensors	as used in SDXL
original	4.99	23.4 +/- 3.8	0.69 +/- 0.14	1.01 +/- 0.28	https://ommer-lab.com/files/latent-diffusion/kl-f8.zip	as used in SD
ft-MSE	4.70	24.5 +/- 3.7	0.71 +/- 0.13	0.92 +/- 0.27	https://huggingface.co/stabilityai/sd-vae-ft-mse-original/resolve/main/vae-ft-mse-840000-ema-pruned.ckpt	resumed with EMA from ft-EMA, emphasis on MSE (rec. loss = MSE + 0.1 * LPIPS), smoother outputs

Changelog

Update config.json

Update config.json

2023-08-04 · by patrickvonplaten · 6f5909a

Revert "update vae weights"

2023-07-27 · by palp · c1b803c

This reverts commit 4eec565eb7167a2252e2a48e0898565b8025661d.

Update config.json

2023-07-26 · by patrickvonplaten · 0cbce97

Update diffusers weights (#8)

2023-07-26 · by rromb, patrickvonplaten · acf5ce6

- update diffusers (3d898e2c0d7cd33cec31b3b9039288818151759b) - correct scaling factor (8987d138a8df8401b9cf382172eb052ec7a06521)

Co-authored-by: Patrick von Platen <[e-mail]>

update vae weights

2023-07-26 · by Robin Rombach · 4eec565

Update scaling_factor for SDXL (#4)

2023-07-02 · by patrickvonplaten, madebyollin · e6e6fee

- Update scaling_factor for SDXL (622c174bdf80e696015d450354fa1c103c417383)

Co-authored-by: madebyollin <[e-mail]>

Add Diffusers weights (#3)

2023-06-30 · by patrickvonplaten · 95f3fcb

- add diffusers weights (c931cf301bb9c9182618768372e9ef343d026407)

Initial commit

2023-06-22 · by rromb, timudk · f85c107

Co-authored-by: Tim Dockhorn <[e-mail]>