A framework to score reasoning capabilities in image generation models at scale. We use datasets from https://github.com/Video-Reason/VMEvalKit. This repo only contains the image generation model support. We make it very convenient to add models, add tasks, run inferences, run scoring. It's permissively open-source, and we welcome everyone to join us and build in public together! 🚀
- Clone the repository
git clone https://github.com/hokindeng/VMEvalKit.git
cd VMEvalKit- Initialize submodules - good for optional open-source models and datasets
git submodule update --init --recursive- Configure environment - Copy the example environment file and add your API keys
cp env.template .env- Set up Python environment – Recommended: use a fresh virtual environment
python -m venv venv
source venv/bin/activateAlternatively, you can use other tools like uv for faster install (uv venv), or conda if your usecase has cross-language dependencies.
- Install dependencies:
pip install -r requirements.txt
pip install -e .Apache 2.0