Solving Subjective Quality in AI Generated Design

Our client set out to evaluate their LLM's ability to generate and refine front-end web applications, with one critical requirement: the UIs had to be "visually appealing." This seemingly simple goal revealed three major obstacles:

**Subjective Design Criteria**: With quality assessed by a rotating pool of crowdsourced QA reviewers, "visually appealing" meant different things to different people. This lack of clear standards led to inconsistent feedback, frequent rework, and frustrated annotators.

**Inflated Average Handling Time**: The original workflow had annotators building fully functional React applications, only to have them rejected at the final stage for aesthetic reasons. Tasks that should have taken 4 hours were stretching to 7 hours due to multiple revision cycles.

**Missing Design Alignment**: The lean workflow skipped upfront design validation—a critical oversight. Developers were coding in the dark, guessing at aesthetic requirements that wouldn't be evaluated until the very end of the process.

Revelo recognized that traditional approaches wouldn't work for subjective quality assessment. Instead of just providing talent, we became strategic partners in reimagining the entire workflow:

**Pattern Recognition**: We analyzed QA feedback to identify what was actually getting approved, creating internal guidance documents that decoded the unwritten aesthetic rules.

**Workflow Experimentation**: We ran three different workflows in parallel, testing various approaches to find the optimal balance between efficiency and quality.

**Proactive Design Validation**: We proposed adding a design review layer at the beginning of the process—getting aesthetic approval before any code was written.

**Continuous Calibration**: We built flexible review systems that could adapt in real-time to evolving QA patterns, turning ambiguity into actionable insights.

Working closely with the client, we implemented a complete workflow transformation:

**The Optimized Workflow**:

1. **Designer Sourcing**: Senior designers source and modify high-quality mockups aligned with emerging quality patterns
2. **Client Pre-approval**: Designs are validated before any development begins, eliminating late-stage aesthetic rejections
3. **Focused Development**: Developers build from pre-approved designs, working from a shared repository with clear specifications
4. **Layered Review**: Separate design and functionality reviews ensure both aesthetic and technical quality
5. **Continuous Feedback Loop**: Weekly calibration sessions between Revelo and client teams to refine standards

**Key Innovations**:

- Introduced Figma file support to improve developer efficiency
- Created internal calibration documents for consistent aesthetic standards
- Implemented peer checkpoints to maintain alignment without formal rubrics
- Built a feedback system that helped the client refine their own quality definitions

The impact of our approach was dramatic and measurable:

**Efficiency Gains**:

- **32% reduction in AHT** (from 420 to 231 minutes)
- **4.7x increase in weekly output** (from 75.5 to 356 tasks)
- **2.3x growth in active specialists** (from 70 to 158)

**Quality Improvements**:

- **76% increase in approval rates** (from 54% to 94%)
- **Consistent quality at scale** across 12 weeks of production
- **Sustainable workflow** that the client could replicate for future projects
- **94% final quality score** achieved

**Long-term Impact**: As a result of this success, the client expanded the partnership with Revelo, launching three additional UI generation initiatives. We had transformed a problematic project with subjective quality requirements into a scalable, efficient operation—proving that with the right approach, even the most ambiguous challenges can be systematically solved.