Preference

VisIT-Bench

VisIT-Bench: 592 real-world image+instruction vision-language queries across ~70 instruction families. Two public per-(model, item) sources are ingested: (1) 5011 human pairwise-preference battles among six vision-language models (MiniGPT-4, LLaMA-Adapter-v2, LLaVA-13B, PandaGPT-13B, mPLUG-Owl, InstructBLIP) -> binary win/loss per (model, item, opponent); (2) human correctness ratings of GPT-4 predictions -> binary correct/incorrect per item for GPT-4. response is binary in {0, 1}; the full model answer text is kept as trace.