Mercari AI Team’s Research “LLMOps for Eval-Driven Development at Scale” Accepted to FOSSASIA Summit 2025

Overview

We are pleased to announce that the paper "LLMOps for Eval-Driven Development at Scale" by engineers Teofilo Narboneta Zosa and Jehandad Kamal of Mercari’s AI team has been accepted to the international conference FOSSASIA Summit 2025.

The FOSSASIA Summit is an annual event targeting the free and open-source software (FOSS) community in Asia. This summit is established as a place where developers, engineers, community members, corporate leaders, and open-source enthusiasts gather to share and learn about the latest technology trends and open-source projects.

Key points of the presentation

Mercari has invested heavily in DevOps, MLOps, and, recently, LLMOps with significant payoff to developer speed and quality, both in terms of software quality and quality of life. From prompt management, evaluation, and LLM application observability, this talk dives into major LLMOps focus areas and open-source software that were key to the success of our most ambitious projects, which have already delivered unparalleled customer value to over 23 million users of Japan's largest C2C e-commerce marketplace.

Background

Over the past decade, we've seen DevOps transform the landscape of software development and usher in a new generation of robust, maintainable software that is orders of magnitude more reliable and delivered in a fraction of the time. The past few years have seen the same aspirations applied to AI software, first with ML and now with GenAI applications. While the concrete recommendations vary, the goal remains the same: deliver high-quality software with maximum speed and scale.

Summary of paper

Members of Mercari’s AI/LLM team share their validated approach to high-quality GenAI application development at scale. Using Mercari’s AI Listing feature as a motivating example, they demonstrate how Mercari’s AI engineers and subject matter experts effectively work together to utilize evaluation-driven development and tight collaboration loops to supercharge the AI software development lifecycle, in turn delivering features with high customer value at their core.

About the Eliza Team

The Eliza Team at Mercari is a team specialized in AI and LLM (large language models), aiming to utilize generative AI and implement LLMs within Mercari, as well as to build more intelligent services by leveraging existing AI technologies. This team plays a crucial role in enhancing productivity both internally and externally, as well as in the development and deployment of new AI tools.