Image Editing

The ImageEditingMetric is a multimodal metric designed to evaluate the quality of image edits based on given instructions. It assesses how well the edited image matches the expected output, serving as a proxy for evaluating the performance of image editing models.

Required Arguments

• input: A list containing the edit instructions and the original image. The image is represented as an MLLMImage instance with:
• url: The file path or URL to the image.
• actual_output: A list of MLLMImage instances representing the actual edited images generated by the model. Each MLLMImage requires:
• url: The file path or URL to the image.

Optional Arguments

• threshold: A float representing the minimum passing threshold, defaulted to 0.5.
• model: A string specifying which of OpenAI’s GPT models to use, or any custom LLM model of type DeepEvalBaseLLM. Defaulted to ‘gpt-4o’.
• include_reason: A boolean which, when set to True, includes a reason for its evaluation score. Defaulted to True.
• strict_mode: A boolean which, when set to True, enforces a binary metric score: 1 for perfection, 0 otherwise. It also overrides the current threshold and sets it to 1. Defaulted to False.
• async_mode: A boolean which, when set to True, enables concurrent execution within the measure() method. Defaulted to True.
• verbose_mode: A boolean which, when set to True, prints the intermediate steps used to calculate the metric to the console. Defaulted to False.

Usage Example

import sys
import os

from agensight.eval.metrics import ImageEditingMetric
from agensight.eval.test_case import MLLMTestCase
from agensight.eval.test_case import MLLMImage

input_data = ["Add a hat to the cat in the image.", MLLMImage(url="image-path/cat.jpeg")]
actual_output = [MLLMImage(url="image-path/cat.jpeg")]

metric = ImageEditingMetric(model="gpt-4o", threshold=0.5)
test_case = MLLMTestCase(input=input_data, actual_output=actual_output)

metric.measure(test_case)
print(f"Score: {metric.score}")
print(f"Reason: {metric.reason}")