ImagenHub is a one-stop library to standardize the inference and evaluation of all the conditional image generation models.
ImagenHub: Standardizing the evaluation of conditional image generation models
ICLR 2024
ImagenHub is a one-stop library to standardize the inference and evaluation of all the conditional image generation models.
Install from PyPI:
pip install imagen-hub
Or build from source:
git clone https://github.com/TIGER-AI-Lab/ImagenHub.git
cd ImagenHub
conda env create -f env_cfg/imagen_environment.yml
conda activate imagen
pip install -e .
For models like Dall-E, DreamEdit, and BLIPDiffusion, please see Extra Setup
For some models (Stable Diffusion, SDXL, CosXL, etc.), you need to login through huggingface-cli.
huggingface-cli login
To reproduce our experiment reported in the paper:
Example for text-guided image generation:
python3 benchmarking.py -cfg benchmark_cfg/ih_t2i.yml
Note that the expected output structure would be:
result_root_folder
└── experiment_basename_folder
├── input (If applicable)
│ └── image_1.jpg ...
├── model1
│ └── image_1.jpg ...
├── model2
│ └── image_1.jpg ...
├── ...
Then after running the experiment, you can run
python3 visualize.py --cfg benchmark_cfg/ih_t2i.yml
to produce a index.html file for visualization.
The file would look like something like this. We hosted our experiment results on Imagen Museum.
import imagen_hub
model = imagen_hub.load("SDXL")
image = model.infer_one_image(prompt="people reading pictures in a museum, watercolor", seed=1)
image
from imagen_hub.metrics import MetricLPIPS
from imagen_hub.utils import load_image, save_pil_image, get_concat_pil_images
def evaluate_one(model, real_image, generated_image):
score = model.evaluate(real_image, generated_image)
print("====> Score : ", score)
image_I = load_image("https://chromaica.github.io/Museum/ImagenHub_Text-Guided_IE/input/sample_102724_1.jpg")
image_O = load_image("https://chromaica.github.io/Museum/ImagenHub_Text-Guided_IE/DiffEdit/sample_102724_1.jpg")
show_image = get_concat_pil_images([image_I, image_O], 'h')
model = MetricLPIPS()
evaluate_one(model, image_I, image_O) # ====> Score : 0.11225218325853348
show_image
The tutorials and API documentation are hosted on imagenhub.readthedocs.io.
By streamlining research and collaboration, ImageHub plays a pivotal role in propelling the field of Image Generation and Editing.
We included more than 30 Models in image synthesis. See the full list here:
| Method | Venue | Type |
|---|---|---|
| Stable Diffusion | - | Text-To-Image Generation |
| Stable Diffusion XL | arXiv'23 | Text-To-Image Generation |
| DeepFloyd-IF | - | Text-To-Image Generation |
| OpenJourney | - | Text-To-Image Generation |
| Dall-E | - | Text-To-Image Generation |
| Kandinsky | - | Text-To-Image Generation |
| MagicBrush | arXiv'23 | Text-guided Image Editing |
| InstructPix2Pix | CVPR'23 | Text-guided Image Editing |
| DiffEdit | ICLR'23 | Text-guided Image Editing |
| Imagic | CVPR'23 | Text-guided Image Editing |
| CycleDiffusion | ICCV'23 | Text-guided Image Editing |
| SDEdit | ICLR'22 | Text-guided Image Editing |
| Prompt-to-Prompt | ICLR'23 | Text-guided Image Editing |
| Text2Live | ECCV'22 | Text-guided Image Editing |
| Pix2PixZero | SIGGRAPH'23 | Text-guided Image Editing |
| GLIDE | ICML'22 | Mask-guided Image Editing |
| Blended Diffusion | CVPR'22 | Mask-guided Image Editing |
| Stable Diffusion Inpainting | - | Mask-guided Image Editing |
| Stable Diffusion XL Inpainting | - | Mask-guided Image Editing |
| TextualInversion | ICLR'23 | Subject-driven Image Generation |
| BLIP-Diffusion | arXiv'23 | Subject-Driven Image Generation |
| DreamBooth(+ LoRA) | CVPR'23 | Subject-Driven Image Generation |
| Photoswap | arXiv'23 | Subject-Driven Image Editing |
| DreamEdit | arXiv'23 | Subject-Driven Image Editing |
| Custom Diffusion | CVPR'23 | Multi-Subject-Driven Generation |
| ControlNet | arXiv'23 | Control-guided Image Generation |
| UniControl | arXiv'23 | Control-guided Image Generation |
Community contributions are encouraged!
ImagenHub is still under development. More models and features are going to be added and we always welcome contributions to help make ImagenHub better. If you would like to contribute, please check out CONTRIBUTING.md.
We believe that everyone can contribute and make a difference. Whether it's writing code 💻, fixing bugs 🐛, or simply sharing feedback 💬, your contributions are definitely welcome and appreciated 🙌
And if you like the project, but just don't have time to contribute, that's fine. There are other easy ways to support the project and show your appreciation, which we would also be very happy about:
- Star the project
- Tweet about it
- Refer this project in your project's readme
- Mention the project at local meetups and tell your friends/colleagues
Q: How can I use your evaluation method for my method?
A: Please Refer to https://imagenhub.readthedocs.io/en/latest/Guidelines/humaneval.html
Q: How can I add my method to ImagenHub codebase?
A: Please Refer to https://imagenhub.readthedocs.io/en/latest/Guidelines/custommodel.html
Q: I want to feature my method on ImagenMuseum!
A: Please Refer to https://imagenhub.readthedocs.io/en/latest/Guidelines/imagenmuseum.html
Please kindly cite our paper if you use our code, data, models or results:
@inproceedings{
ku2024imagenhub,
title={ImagenHub: Standardizing the evaluation of conditional image generation models},
author={Max Ku and Tianle Li and Kai Zhang and Yujie Lu and Xingyu Fu and Wenwen Zhuang and Wenhu Chen},
booktitle={The Twelfth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=OuV9ZrkQlc}
}
@article{ku2023imagenhub,
title={ImagenHub: Standardizing the evaluation of conditional image generation models},
author={Max Ku and Tianle Li and Kai Zhang and Yujie Lu and Xingyu Fu and Wenwen Zhuang and Wenhu Chen},
journal={arXiv preprint arXiv:2310.01596},
year={2023}
}
Please refer to ACKNOWLEDGEMENTS.md
This project is released under the License.
FAQs
ImagenHub is a one-stop library to standardize the inference and evaluation of all the conditional image generation models.
We found that imagen-hub demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
In this segment of the Risky Business podcast, Feross Aboukhadijeh and Patrick Gray discuss the challenges of tracking malware discovered in open source softare.
Research
Security News
A threat actor's playbook for exploiting the npm ecosystem was exposed on the dark web, detailing how to build a blockchain-powered botnet.
Security News
NVD’s backlog surpasses 20,000 CVEs as analysis slows and NIST announces new system updates to address ongoing delays.