Property Finder 修复评估偏差提升

Property Finder, a digital real estate application, discovered that 96.8 percent of their evaluation labels were inherited from a single model’s predictions rather than verified by human experts, skewing ground truth toward the incumbent model.

This bias hindered objective model comparison when they migrated their vision AI workload to Amazon Bedrock.

Property Finder processes tens of thousands of property images daily across the Middle East and North Africa (MENA), evaluating 11 quality attributes per image, including watermark detection, room classification, and photo quality.

They needed a more cost-effective option but found that any alternative model was at a disadvantage due to the biased evaluation labels. The challenge was realizing that their evaluation methodology itself was the blocker.

They addressed this by building unbiased ground truth through independent human verification and applying evolutionary prompt optimization on Amazon Bedrock, achieving 86 percent lower cost with higher accuracy.

The process involved an iterative prompt optimization loop, from version v0 to v27, refining prompts based on error analysis and automated optimization, with AI assistants aiding early stages.

This methodology can be applied to other workloads to improve vision model evaluation.

Many teams face similar issues when comparing vision models by using their current model’s outputs as ground truth, creating a circular evaluation that disadvantages alternative models. Property Finder’s evaluation dataset had this problem, with hidden errors in watermark text extraction and photo classification further confirming the bias. To break this cycle, human experts must independently annotate data without referencing previous model outputs, ensuring unbiased ground truth for valid model comparison.