Text-to-image AI models have dazzled us with their ability to generate images based on provided textual descriptions. However, evaluating their performance has long been a contentious issue, as current methods have hidden limitations. Here, we discuss Gecko, a groundbreaking new benchmark that offers a more comprehensive assessment.
As an AI researcher, I have witnessed varying capabilities of text-to-image models in specific contexts. The inconsistencies in their performance imply that current evaluation methods may be inadequate. Traditional assessment approaches rely mainly on datasets and automated metrics, which offer a narrow view of a model’s range. These methods’ small sample sizes provide limited insight, and ambiguous prompts introduce uncertainty.
Gecko steps in to tackle these challenges. A comprehensive new benchmark, Gecko offers evaluators a diverse set of 2,000 text prompts that probe a broad spectrum of skills and complexities. By categorizing these prompts into specific sub-skills, Gecko offers a more precise evaluation – enabling evaluators to pinpoint not only which skills are challenging but also at which level of complexity a given skill becomes challenging for a model.
Gecko’s human annotations, consisting of over 100,000 ratings across different templates and four text-to-image models, provide essential context. These extensive ratings provide understanding in teasing apart performance differences, revealing whether disparities arise due to inherent ambiguity in the prompt, inconsistent evaluation methods, or underlying model limitations.
Equipped with these insights, Gecko also introduces a QA-based auto-eval metric – a more reliable method that closely aligns with human judgments compared to existing ones. This improved metric, when combined with the new dataset, uncovers previously undisclosed differences in model strengths and weaknesses.
Gecko offers a significant advancement in text-to-image AI evaluation standards. By addressing current limitations through its comprehensive skills-based benchmarking, extensive human annotations, advanced automatic evaluation metric, and valuable insights into model performance, Gecko paves the way for a more accurate assessment of these intriguing and increasingly common AI models. As we continue exploring the applications of text-to-image AI, Gecko sets the gold standard for evaluating their performance.