Blog

<h2>　Overview</h2>
<p>We are pleased to announce that the paper &quot;Zero-Shot Retrieval for Scalable Visual Search in a Two-Sided Marketplace&quot; by Andre Rusli, Shoma Ishimoto, Sho Akiyama, and Aman Kumar Singh of the Mercari AI engineering and product teams has been accepted for presentation at the TSMO 2025 workshop of the international conference KDD 2025 in the field of knowledge discovery and data mining.</p>
<p>TSMO (Two-sided Marketplace Optimization) is a specialized workshop focusing on research related to two-sided marketplace platforms and online ecosystems.</p>
<p>KDD (Knowledge Discovery and Data Mining) is one of the most prestigious conferences in the field of data science and machine learning. The conference will be held in Toronto, Canada from August 3-7, 2025.</p>
<p>The presentation is scheduled for the TSMO 2025 workshop on August 4, 2025.</p>
<h2>Key points of presentation</h2>
<p>This research presents a scalable visual search system deployed in Mercari, addressing the unique challenges of consumer-to-consumer (C2C) platforms where end-users act as both buyers and sellers. The key innovation lies in evaluating recent vision-language models for zero-shot image retrieval and demonstrating their superiority over existing baselines.</p>
<h3>The main contributions include:</h3>
<ul>
<li>Development of a production-ready visual search system that integrates real-time inference and background indexing workflows</li>
<li>Comprehensive evaluation of multilingual SigLIP model showing 13.3% improvement in nDCG@5 over the baseline</li>
<li>Real-world validation through online A/B testing with up to 40.9% increase in transaction rate via image search</li>
<li>Demonstration that zero-shot models can serve as strong practical baselines for production use</li>
</ul>
<h2>Background</h2>
<p>C2C marketplace platforms like Mercari face unique challenges compared to traditional B2C platforms. Most sellers are everyday individuals rather than professional merchants, leading to diverse and often inconsistent product catalogs with varied naming conventions, category assignments, and visual quality. This creates difficulties for customers navigating the catalog, whether for purchasing or researching similar listings before selling.</p>
<p>While text-based search engines remain the default method for product discovery, they often struggle in such environments. Many items are difficult to describe precisely in words or are identified primarily by visual traits (e.g., fashion, character goods, or collectibles). Visual search provides an intuitive alternative, helping end-users locate visually similar items without requiring clean textual metadata.</p>
<p><img src="https://storage.googleapis.com/prd-ai-asset/2025/08/4533db99-image1-1024x549.png" alt="" /><br />
<em>Illustration of a customer need for a visual search feature</em></p>
<p><img src="https://storage.googleapis.com/prd-ai-asset/2025/08/bddb1402-image2-1024x446.png" alt="" /><br />
<em>UX flow of the proposed visual search feature</em></p>
<h2>Summary of paper</h2>
<p>The research evaluates recent vision-language models, specifically multilingual SigLIP, for zero-shot image retrieval in a large-scale marketplace setting. The system architecture combines real-time inference and background indexing workflows, supported by a unified embedding pipeline optimized through dimensionality reduction from 768 to 128 dimensions.</p>
<p><img src="https://storage.googleapis.com/prd-ai-asset/2025/08/4533db99-image1-1024x549.png" alt="" /><br />
<em>System flow of when a user upload a search query image</em></p>
<h3>Key findings include:</h3>
<ul>
<li>Multilingual SigLIP outperformed other models across multiple retrieval metrics in offline evaluation</li>
<li>Online A/B testing confirmed real-world impact with substantial gains in engagement and conversion</li>
<li>The system serves approximately 1.5 million customers per month (around 7% of Mercari's monthly active customers) by the time of writing, and still increasing</li>
<li>Zero-shot approaches can eliminate the need for extensive downstream training while providing robustness to distribution shifts</li>
</ul>
<h3>Comparison of results of different models on the offline evaluation dataset</h3>
<table>
<thead>
<tr>
<th style="text-align: left;">Model</th>
<th style="text-align: left;">nDCG@5</th>
<th style="text-align: left;">Prec@5</th>
<th style="text-align: left;">Recall@5</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: left;">baseline</td>
<td style="text-align: left;">0.510</td>
<td style="text-align: left;">0.190</td>
<td style="text-align: left;">0.551</td>
</tr>
<tr>
<td style="text-align: left;">clip-japanese-base</td>
<td style="text-align: left;">0.472</td>
<td style="text-align: left;">0.175</td>
<td style="text-align: left;">0.516</td>
</tr>
<tr>
<td style="text-align: left;">siglip-base-patch16-256-multilingual</td>
<td style="text-align: left;"><strong>0.578</strong></td>
<td style="text-align: left;"><strong>0.212</strong></td>
<td style="text-align: left;"><strong>0.634</strong></td>
</tr>
<tr>
<td style="text-align: left;">dinov2-large</td>
<td style="text-align: left;">0.526</td>
<td style="text-align: left;">0.190</td>
<td style="text-align: left;">0.565</td>
</tr>
</tbody>
</table>
<p>The study demonstrates that recent zero-shot models can serve as strong and practical baselines for production use, enabling teams to deploy effective visual search systems with minimal overhead while retaining flexibility for future fine-tuning based on domain-specific needs.</p>
<h2>About the Authors' Teams</h2>
<h3>AI/LLM Team:</h3>
<p>Focuses on advancing artificial intelligence and machine learning capabilities across Mercari's products and services. The team conducts experiments in computer vision, natural language processing, and large language models, working closely with various product teams to experiment with and implement the latest AI technologies to enhance customer experiences and solve complex marketplace challenges.</p>
<h3>Search Team:</h3>
<p>Dedicated to building and continuously improving search systems that help customers efficiently find products across Mercari's platform. The team develops and maintains various search modalities including text-based and visual search, ensuring optimal product experiences for millions of customers.</p>
<h3>ML Platform Team:</h3>
<p>Specializes in providing robust, scalable infrastructure and platforms that enable product teams across the organization to deploy and operate machine learning systems effectively. The team focuses on building reliable ML infrastructure, optimizing performance, and ensuring seamless integration of AI capabilities into production environments.</p>
<h2>Acknowledgments</h2>
<p>We would like to express our sincere gratitude to the engineering members of the Search, AI/LLM, and ML Platform teams for their outstanding contributions throughout the development of this work. Their deep technical expertise, collaborative spirit, and relentless focus on scalability and performance were instrumental in building and deploying the system. We also extend our heartfelt thanks to our cross-functional partners, including designers, product managers, and UX researchers, whose thoughtful input ensured a seamless and user-centric experience for our customers. Special appreciation goes to the Business Intelligence (BI) team for their analytical support in shaping our evaluation strategy, to the QA team for their rigorous testing and quality assurance, and to the Marketing team for their efforts in driving awareness and adoption. Their dedication and expertise have collectively played a vital role in bringing this work to life.</p>


Mercari AI’s research paper, “Zero-Shot Retrieval for Scalable Visual Search in a Two-Sided Marketplace” has been accepted to the TSMO 2025 workshop at KDD 2025

<p>Mercari, Inc. (“Mercari”) announces that a paper co-written by Mercari AI Team engineers has been accepted for presentation at the Meeting on Image Recognition and Understanding (MIRU) 2025​. The paper, titled “Visual Recommendation on E-commerce Platforms Via Fine-Tuned Vision-Language Models,” is the work of Mercari engineers Yuki Yada, Sho Akiyama, Ryo Watanabe, Yuta Ueno, Andre Rusli, and Yusuke Shido. Held annually, the MIRU​ is one of the largest symposiums on technology related to image recognition and understanding held in Japan. MIRU will hold its 28th meeting in Kyoto City, from July 29 to August 1, 2025. In 2025, MIRU selected 88 papers for presentation from among the 255 applications it received (acceptance rate: 34.5%).</p>
<p>The presentation is scheduled for Thursday, July 31, 2025, from 3:00 to 4:20 PM.</p>
<h2>Key points of presentation</h2>
<p>The presentation will discuss a project that has improved the image AI model behind Mercari’s recommendations feature, which functions based on items of similar appearance within the platform’s listings pages.</p>
<p>To achieve the improvements, engineers trained an AI model using the extensive record of item image data that Mercari retains. This resulted in a substantial improvement to the e-commerce platform’s visual similarity-based recommendations feature.</p>
<p><img src="https://storage.googleapis.com/prd-ai-asset/2025/07/8ca99930-image1-549x1024.png" alt="" /></p>
<h2>Background of research</h2>
<p>“I really like the design of this clothing item. I’d like to see more clothing designs like this.” “I wonder if Mercari sells items with my favorite cartoon character on it.” These are some of the things that Mercari hears from its users, indicating that there is a certain level of demand for a discovery experience grounded in the image-based characteristics of items.</p>
<p>Mercari has previously released a discovery feature that leverages image-based characteristics including image searches and recommendations. However, the AI model working in the background of the feature had not been trained on Mercari’s item data, and as a result there was an issue where the model could not sufficiently ascertain the visual traits of items.</p>
<h2>Summary of paper</h2>
<p>The presentation will discuss an AI model with a superb understanding of both images and text known as a vision-language model, which was trained using the data of over one million items that Mercari retains.</p>
<p>As a result of building the AI model, the team has seen the following striking improvements:<br />
Increased the click-through rate of the “visually similar items” feature by 50%<br />
Increased the number of purchases made via item page recommendations by 14%<br />
-- The AI model discussed in the presentation has also been used in the platform’s image search feature.</p>
<h2>About Mercari’s AI/LLM Team</h2>
<p>The goal of the AI/LLM Team is to contribute to the improvement of the user experience through product improvements that leverage two cutting-edge AI technologies, LLMs and embedding. The team leads the development of features like image search and AI listing.</p>


Mercari AI Team Research Paper “Visual Recommendation on E-commerce Platforms Via Fine-Tuned Vision-Language Models” Accepted for Presentation at MIRU 2025

<h2>Overview</h2>
<p>We are pleased to announce that the paper &quot;LLMOps for Eval-Driven Development at Scale&quot; by engineers Teofilo Narboneta Zosa and Jehandad Kamal of Mercari’s AI team has been accepted to the international conference FOSSASIA Summit 2025.</p>
<p>The FOSSASIA Summit is an annual event targeting the free and open-source software (FOSS) community in Asia. This summit is established as a place where developers, engineers, community members, corporate leaders, and open-source enthusiasts gather to share and learn about the latest technology trends and open-source projects.</p>
<p><iframe loading="lazy" width="560" height="315" src="https://www.youtube.com/embed/GyHY4g93kPk?si=pue4UE96XKbVUqpv" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe></p>
<h2>Key points of the presentation</h2>
<p>Mercari has invested heavily in DevOps, MLOps, and, recently, LLMOps with significant payoff to developer speed and quality, both in terms of software quality and quality of life. From prompt management, evaluation, and LLM application observability, this talk dives into major LLMOps focus areas and open-source software that were key to the success of our most ambitious projects, which have already delivered unparalleled customer value to over 23 million users of Japan's largest C2C e-commerce marketplace.</p>
<h2>Background</h2>
<p>Over the past decade, we've seen DevOps transform the landscape of software development and usher in a new generation of robust, maintainable software that is orders of magnitude more reliable and delivered in a fraction of the time. The past few years have seen the same aspirations applied to AI software, first with ML and now with GenAI applications. While the concrete recommendations vary, the goal remains the same: deliver high-quality software with maximum speed and scale.</p>
<h2>Summary of paper</h2>
<p>Members of Mercari’s AI/LLM team share their validated approach to high-quality GenAI application development at scale. Using Mercari’s AI Listing feature as a motivating example, they demonstrate how Mercari’s AI engineers and subject matter experts effectively work together to utilize evaluation-driven development and tight collaboration loops to supercharge the AI software development lifecycle, in turn delivering features with high customer value at their core.</p>
<h2>About the Eliza Team</h2>
<p>The Eliza Team at Mercari is a team specialized in AI and LLM (large language models), aiming to utilize generative AI and implement LLMs within Mercari, as well as to build more intelligent services by leveraging existing AI technologies. This team plays a crucial role in enhancing productivity both internally and externally, as well as in the development and deployment of new AI tools.</p>


Mercari AI Team’s Research “LLMOps for Eval-Driven Development at Scale” Accepted to FOSSASIA Summit 2025

<p>Hello, I am <a href="https://www.linkedin.com/in/andre-r-2a401875/">@andre</a>, a machine learning engineer on the AI/LLM team at Mercari.</p>
<p>In <a href="https://ai.mercari.com/en/articles/engineering/20231219-leveraging-llms-in-production-looking-back-going-forward/">a previous article</a>, we discussed how our team utilized commercial LLM APIs to build an initial feature to support our customers and improve the platform&#8217;s selling experience.</p>
<p>This article will describe one of our past experiments in fine-tuning a 2-billion parameter large language model (LLM) using QLoRA, to extract dynamically specified attributes from user-generated content, and compared the performance with GPT-3.5 turbo—a much larger model. Results show that the fine-tuned model outperforms the bigger model in terms of extraction quality while being significantly smaller in size and less costly. We hope this article will provide valuable insights into what it takes to fine-tune an LLM effectively.</p>
<h2>Background</h2>
<p>In a Japanese customer-to-customer (C2C) marketplace, specific details could impact the quality of a listing description. However, understanding the precise details in a user-generated listing description can be tricky. This is due to several challenges, including:</p>
<ul>
<li>Wide variety of user-generated content: Each seller describes their listings differently.</li>
<li>Category specificity: What’s essential varies from one category to another.</li>
<li>Time sensitivity: User-generated content continuously evolves.</li>
</ul>
<p>By accurately extracting existing key attributes from listing descriptions, we can gain a deeper understanding of the contents written by our customers—specifically, in this case, the sellers. Figure 1 below illustrates an example of a listing description and the extracted values. For the purpose of this article, the illustration shows an example of a listing written in English; however, most listings within Mercari are written in Japanese. Such insight can also help us guide our customers to enhance their listings, making them more appealing and effective.</p>
<p><img src="https://storage.googleapis.com/prd-engineering-asset/2024/09/27c3a242-screen-shot-2024-09-11-at-14.00.02-pm.png" alt="Illustration of the extracted attributes from a sample listing description" /><br />
Figure 1. Illustration of the extracted attributes from a sample listing description</p>
<p>Why not just use light-weight, conventional, non-LLM models?</p>
<ul>
<li><strong>Dynamic and varied attributes</strong>: The way attributes are described can change frequently, leading to high maintenance requirements and the need for continuous model re-training. Having a model that could handle dynamically specified attributes could go a long way.</li>
<li><strong>Generalization capability</strong>: Large language models (LLMs) have the potential to generalize far better than conventional ML models with much less training data, even for handling out-of-distribution data.</li>
<li><strong>Multi-linguality</strong>: Most listings in Mercari are written in Japanese, however, with the huge variety of goods being exchanged, there are also listings written in other languages, such as English and Chinese. The multilingual capability of recent LLMs are expected to be able to handle such varieties better than conventional ML models.</li>
</ul>
<p>On the other hand, why not just use existing commercial LLM APIs?</p>
<ul>
<li><strong>Cost of commercial APIs</strong>: Though commercial LLM APIs are becoming more affordable, at the time this article is written, the sheer number of requests in a production environment would still make them prohibitively expensive.</li>
<li><strong>Control over hallucinations</strong>: It’s more difficult to manage and minimize hallucinations purely through prompt engineering with commercial APIs.</li>
</ul>
<p>Given these considerations, we decided to experiment with fine-tuning our own model. For this experiment, we used a single A100 GPU with an 80 GB memory VM instance (<code>a2-ultragpu-1g</code>) from GCP to fine-tune a large language model using QLoRA. Our short-term goal was to see if we can build a model that could achieve similar or even better performance than GPT-3.5 Turbo despite being significantly smaller and cheaper to run in production.</p>
<h2>Dataset and Base Model</h2>
<p>To tackle our task, we first defined the input and output requirements for the model:</p>
<ul>
<li><strong>Input</strong>: A text description of the listing and a list of attribute keys to extract. For example:
<ul>
<li>Listing description: <code>A Mercari T-shirt size M, blue. Used only once and kept in a clean wardrobe after.</code></li>
<li>Attribute keys: <code>size, color, original retail price</code></li>
</ul>
</li>
<li><strong>Output</strong>: The extracted attributes and their values. For example:
<ul>
<li>Size: <code>M</code></li>
<li>Color: <code>Blue</code></li>
<li>Original retail price: <code>NONE</code></li>
</ul>
</li>
</ul>
<p>To build our dataset, we gathered historical descriptions along with their attributes. Since attribute keys can vary across item categories, we started by focusing on the 20 categories with the highest listings on our platform.</p>
<p>We structured the data into inputs and outputs and integrated these pairs with specific prompts, which were then used to fine-tune the LLMs. We experimented with various prompts written in English and Japanese; however, the prompt generally contains the following.</p>
<ul>
<li><strong>An initial prompt sentence</strong>, telling the model that it will receive an instruction below and instructing it to respond accordingly.</li>
<li><strong>The instruction</strong>, mentioning that it will be given a description text in the context of an online marketplace listing, and instructing the model to extract a list of attribute keys from the input text. It also tells the model to respond following a specific format.</li>
<li><strong>The input text</strong>, containing the listing description text from which we want to extract attributes.</li>
<li><strong>The output text</strong>, containing the response text with the attribute keys and the extracted values.</li>
</ul>
<p>Below is an example of the prompt templates we experimented with, written in Japanese:</p>
<pre><code>以下に、あるタスクを説明する指示があり、それに付随する入力が更なる文脈を提供しています。
リクエストを適切に完了するための回答を記述してください。

### 指示:
次の文章はオンラインマーケットプレイスに投稿されているリスティングの情報です。
その文章から{attr_names}の情報を探し出してください。
妥当な情報が存在したら「{attr_name}: &lt;内容&gt;」で応答してください。逆に存在しない場合はかならず「{attr_name}: NONE」で応答してください。

### 入力（文章）:
{input}

### 応答:
{output}</code></pre>
<p>Once the dataset was ready, our next step was identifying potential LLMs for fine-tuning. The <a href="https://wandb.ai/wandb-japan/llm-leaderboard/reports/Nejumi-LLM-Neo--Vmlldzo2MTkyMTU0">Nejumi Leaderboard</a> for Japanese LMs, curated by the Weights and Biases Japan team, was one of our primary resources. It comprehensively evaluates various large language models&#8217; capabilities in handling Japanese text. After testing and experimenting with several models, we decided to move forward with the <em>gemma-2b-it</em> model provided by the team at Google (<a href="https://arxiv.org/abs/2403.08295">paper</a>, <a href="https://huggingface.co/google/gemma-2b-it">HF</a>).</p>
<h2>Parameter efficient fine-tuning with QLoRA</h2>
<p>To embark on our fine-tuning journey, we used QLoRA—a cutting-edge approach known for its efficient fine-tuning. As cited from the original paper, QLoRA significantly reduces memory usage, allowing one to fine-tune a 65B parameter model on a single 48GB GPU while preserving the full 16-bit fine-tuning task performance. The image below illustrates how QLoRA compares to full fine-tuning and LoRA methods.</p>
<p><img src="https://storage.googleapis.com/prd-engineering-asset/2024/09/aaa17652-screen-shot-2024-09-11-at-14.22.09-pm.png" alt="Illustration of how fine-tuning with QLoRA works under the hood" /><br />
Figure 2. Illustration of how fine-tuning with QLoRA works under the hood (adapted from the original figure on <a href="https://arxiv.org/abs/2305.14314">QLoRA: Efficient Finetuning of Quantized LLMs</a>)</p>
<p>Now, let&#8217;s dive into the fine-tuning process!</p>
<p>Initially, we <strong>load the pre-processed dataset</strong> previously stored as W&amp;B artifacts into memory.</p>
<pre><code>...
with wandb.init(entity=ENTITY_NAME, project=PROJECT_NAME, job_type=JOB_TYPE_NAME, tags=[&quot;hf_sft&quot;]):
    artifact = wandb.use_artifact(ENTITY_NAME+&#039;/&#039;+PROJECT_NAME+&#039;/train_test_split:latest&#039;, type=&#039;dataset&#039;)
    artifact_dir = artifact.download()

loaded_dataset = load_dataset(&quot;json&quot;, data_dir=artifact_dir)
train_data = loaded_dataset[&quot;train&quot;]
eval_data  = loaded_dataset[&quot;test&quot;]
...</code></pre>
<p>Then, we define the <strong>LoRA configurations (hyperparameters) and target modules</strong>. One example of the modules and configurations that we experimented with is as follows:</p>
<pre><code>...
target_modules = [&#039;q_proj&#039;,&#039;k_proj&#039;,&#039;v_proj&#039;,&#039;o_proj&#039;,&#039;gate_proj&#039;,&#039;down_proj&#039;,&#039;up_proj&#039;,&#039;lm_head&#039;]

lora_config = LoraConfig(
    r=16,
    lora_alpha=16,
    lora_dropout=0.05,
    bias=&quot;none&quot;,
    target_modules = target_modules,
    task_type=&quot;CAUSAL_LM&quot;,
)
...</code></pre>
<p>And then, the <strong>fine-tuning hyperparameters and quantization configurations</strong>. Following is an example of the configurations that we experimented with:</p>
<pre><code>...
training_args = TrainingArguments(
    output_dir=base_dir,
    report_to=&quot;wandb&quot;,
    save_strategy=&quot;epoch&quot;,
    evaluation_strategy=&quot;epoch&quot;,
    num_train_epochs = 1,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    optim=&#039;adamw_torch&#039;,
    learning_rate=2e-4,
    fp16=True,
    max_grad_norm=0.3,
    warmup_ratio=0.1,
    group_by_length=True,
    lr_scheduler_type=&quot;linear&quot;,
)

nf4_config = BitsAndBytesConfig(
  load_in_4bit=True,
  bnb_4bit_quant_type=&quot;nf4&quot;,
  bnb_4bit_use_double_quant=True,
  bnb_4bit_compute_dtype=torch.bfloat16
)
...</code></pre>
<p>Once the above are set up, we then load the <strong>base model and tokenizer</strong> from HuggingFace:</p>
<pre><code>...
model_path = &quot;google/gemma-2b-it&quot;
tokenizer = AutoTokenizer.from_pretrained(model_path, add_eos_token=True)
model = AutoModelForCausalLM.from_pretrained(
    model_path, device_map=&#039;auto&#039;, quantization_config=nf4_config,
)
...</code></pre>
<p>We then use the SFTTrainer from HuggingFace to <strong>begin fine-tuning</strong>:</p>
<pre><code>...
trainer = SFTTrainer(
    model,
    train_dataset=dataset[&quot;train&quot;],
    eval_dataset=dataset[&quot;test&quot;],
    packing=True,
    max_seq_length=1024,
    args=training_args,
    formatting_func=create_prompt,
)
# Upcast layer norms to float 32 for stability
for name, module in trainer.model.named_modules():
  if &quot;norm&quot; in name:
    module = module.to(torch.float32)

run = wandb.init(entity=ENTITY_NAME, project=PROJECT_NAME, job_type=&quot;start_finetuning&quot;, config=config)
st = time.time()
trainer.train()
elapsed = time.time() - st
run.log({&quot;elapsed_time (seconds)&quot;: elapsed})
run.finish()
...</code></pre>
<p>Finally, we <strong>merge and save</strong> the fine-tuned model:</p>
<pre><code>...
new_model = NEW_MODEL_PATH_AND_NAME
trainer.model.save_pretrained(new_model)
trainer.tokenizer.save_pretrained(new_model)

base_model = AutoModelForCausalLM.from_pretrained(
    model_path,
    torch_dtype=torch.float16,
)
merged_model= PeftModel.from_pretrained(base_model, new_model)
merged_model= merged_model.merge_and_unload()

merged_model.save_pretrained(new_model+&quot;-merged&quot;,safe_serialization=True)
tokenizer.save_pretrained(new_model+&quot;-merged&quot;)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = &quot;right&quot;
...</code></pre>
<h2>Post-training Quantization and Model Evaluation</h2>
<p>Post-training quantization aims to see if we can further shrink the model size while maintaining satisfactory performance. We used the <a href="https://github.com/ggerganov/llama.cpp">llama.cpp</a> library—an open-source tool that enables post-training model quantization and faster inference using LLMs in C/C++.</p>
<p>Here’s an overview of the steps we followed using llama.cpp for model conversion and quantization. Note that some steps might be outdated by the time of publication, so we recommend referring to the llama.cpp repository for the latest information:</p>
<ol>
<li><strong>Clone the Repository</strong>: Clone the llama.cpp GitHub repository and run the build commands using the appropriate settings. Detailed instructions can be found <a href="https://github.com/ggerganov/llama.cpp/blob/master/docs/build.md">here</a>.
<ul>
<li>Note: Since support for Gemma models was added around the end of February 2024, ensure you use the correct version of llama.cpp.</li>
</ul>
</li>
<li><strong>Convert the Model</strong>: Convert the fine-tuned model, previously stored in the HuggingFace format, to a format compatible with llama.cpp.</li>
<li><strong>Select Quantization Method</strong>: Choose the quantization method and start the quantization process. The 4-bit precision method (q4_k_m) worked well for our use case.</li>
<li><strong>Convert and Quantize</strong>: the resulting model is stored in the GGUF format.</li>
</ol>
<p>After the post-training quantization finishes, we evaluated the model in GGUF format and compared its performance. As of our experiment, GPT-4o (including the mini model) was not available. Therefore, considering its cost and latency advantages, we chose GPT-3.5 turbo (specifically, <em>gpt-3.5-turbo-0125</em>) as our baseline model for performance comparison.</p>
<p>Some key metrics for the evaluation:</p>
<ul>
<li><strong>BLEU Score</strong>: This score provided insights into the quality of extracted attribute values compared to the actual values.</li>
<li><strong>Model Size and Latency</strong>: We also checked the resulting model size and latency to assess cost-efficiency and readiness for production use.</li>
</ul>
<p>Here are some key findings from our quick experiment:</p>
<ul>
<li>The final <strong>4-bit precision GGUF model</strong> (q4_k_m) is a QLoRA fine-tuned version of the <em>gemma-2b-it</em> model.</li>
<li>The model is <strong>approximately 95% smaller</strong> than the <em>gemma-2b-it</em> base model downloaded from HuggingFace.</li>
<li>The model achieved a BLEU score slightly <strong>more than five percentage points higher</strong> than <em>gpt-3.5-turbo-0125</em>.</li>
<li>Additionally, an initial rough estimate at the time of the experiment showed that using the fine-tuned model could <strong>reduce the cost by more than 14 times</strong> compared to using <em>gpt-3.5-turbo-0125</em>. However, given the rapidly changing pricing structures of commercial models, this figure should be taken with a grain of salt.</li>
</ul>
<p>In summary, the final model is significantly smaller—approximately 95%—than the original base model from HuggingFace and achieves a BLEU score higher than <em>gpt-3.5-turbo-0125</em>.</p>
<h2>Conclusion</h2>
<p>This experiment demonstrates the practicality of fine-tuning our LLM for attribute value extraction from user-generated content as an effective alternative to commercial LLM APIs. By utilizing QLoRA, we managed to fine-tune the <em>gemma-2b-it</em> model efficiently, reducing its size by around 95% compared to the original base model. Despite this significant size reduction, our fine-tuned model still outperformed <em>gpt-3.5-turbo-0125</em> by achieving a higher BLEU score, thus validating the efficacy of our approach in both performance and resource optimization.</p>
<p>Besides the improvements in performance and cost savings, our hands-on approach provided better control over the model&#8217;s behavior, helping to mitigate issues like hallucinations more effectively than prompt engineering alone. We hope this article offers valuable insights and practical guidance for those looking to fine-tune their models and transition away from expensive and less controllable commercial APIs. By leveraging advancements in large language models and innovative techniques like QLoRA, there are significant opportunities for future development and optimization.</p>


Fine-Tuning an LLM to Extract Dynamically Specified Attributes

<h2>概要</h2>
<p>We are pleased to announce that the paper &quot;<a href="https://drive.google.com/file/d/1gW4ocpkH3To1kgEbg8ciXwcb6V4tRQi0/view">Strategic Coupon Allocation for Increasing Providers' Sales Experiences in Two-sided Marketplaces</a>,&quot; authored by engineers Koya Ohashi, Sho Sekine, Deddy Jobson, and Jie Yang from the Mercari AI team, has been accepted and presented at <a href="https://sites.google.com/view/tsmo2024/home">the Workshop on Two-sided Marketplace Optimization (TSMO 2024)</a> at <a href="https://kdd2024.kdd.org/">KDD2024</a>, an international conference in the field of data mining.</p>
<p>This research was conducted in collaboration with the University of Tsukuba, with support from &quot;<a href="https://r4d.mercari.com/en/">mercari R4D</a>,&quot; the research and development organization of Mercari.</p>
<p>KDD is one of the most prestigious international conferences in the field of data mining, held annually by researchers from around the world. This year, it was held in Barcelona, Spain, from August 25 to 29, 2024.</p>
<h2>Key points of presentation</h2>
<p><a href="https://drive.google.com/file/d/1gW4ocpkH3To1kgEbg8ciXwcb6V4tRQi0/view">Strategic Coupon Allocation for Increasing Providers' Sales Experiences in Two-sided Marketplaces</a></p>
<ul>
<li>In this presentation, we proposed a coupon distribution strategy aimed at providing as many sellers as possible the experience of selling their products.</li>
<li>Through the satisfaction brought about by the selling experience, users are more likely to continue listing their items actively, which is expected to enhance the attractiveness of the platform as well as the quantity and diversity of the products listed.</li>
</ul>
<h2>Background</h2>
<p>In a C2C marketplace like Mercari, where the platform provider does not have a direct supply chain, it is necessary to encourage individual sellers' voluntary listing activities to increase the quantity and variety of listed items. Since listing items involves certain efforts such as selecting products, taking photos, and packaging after sale, strong motivation is essential for sellers to act actively. Although the factors that influence listing motivation vary by individual, we believe that the most fundamental and important experience is the &quot;selling experience&quot; (the experience of having products sold). Therefore, it is crucial to provide as many users as possible with the selling experience.</p>
<p>However, in two-sided markets like Mercari, there is a risk that transactions might be skewed towards a certain group of users without appropriate intervention by the platform provider. To address this, our Data Science team aimed to use coupons, one of the most commonly used marketing tools, to alleviate the concentration of transactions and provide more sellers with the selling experience.</p>
<h2>Summary of paper</h2>
<p>In this paper, we propose a method for optimizing coupon distribution strategies with the number of users having selling experiences as the KPI. Generally, to optimize coupon distribution for a specific KPI, targeted interventions for the users are necessary. In this paper, we propose an efficient coupon distribution strategy using Uplift modeling and mathematical optimization.</p>
<p>By utilizing the proposed method, we confirmed with Mercari's actual data that more sellers could have selling experiences compared to conventional methods. The implementation of this method is expected to enhance the satisfaction of many sellers and promote their voluntary listing activities.</p>
<h2>About Data Science Team</h2>
<p>Mercari's Data Science team leverages techniques such as statistics, machine learning, and mathematical optimization to conduct customer insight analysis, enhance the efficiency of marketing activities, and develop product features.</p>
<h2>About Mercari R4D</h2>
<p><a href="https://r4d.mercari.com/en/">Mercari R4D</a> was established in December 2017 as a research and development organization that aims to implement its findings practically, as part of the world at large. Under its mission of “pioneering the path toward undiscovered value,” R4D promotes a co-innovation approach to solving complex social issues by leveraging the power of science and technology, and also by going beyond the conventional boundaries of industry, academia, and government in order to realize a society where limited resources are circulated and all people can unleash their potential.</p>


The Mercari AI team’s research “Strategic Coupon Allocation for Increasing Providers’ Sales Experiences in Two-sided Marketplaces” has been accepted at the KDD2024 TSMO 2024 Workshop

<h2>Overview</h2>
<p>We are pleased to announce that the paper &quot;<a href="https://dl.acm.org/doi/10.1145/3665939.3665969">LLMs as an Interactive Database Interface for Designing Large Queries</a>&quot; by Engineer Yilin Li and Deddy Jobson, of the Mercaril AI team has been accepted for <a href="https://hilda.io/2024/">HILDA Workshop</a> of the international conference <a href="https://2024.sigmod.org/">SIGMOD 2024</a>  in the field of management of data.</p>
<p>SIGMOD 2024 is one of the most prestigious international conferences in the field of data management, held annually by researchers from around the world. This year's 49th, was held in Santiago, Chile, from June 9 to 14, 2024.</p>
<h2>　Key points of presentation</h2>
<ul>
<li>Current Text2SQL tends to produce inexact queries when scaled to databases the size of what we have in Mercari.</li>
<li>We build a system to incorporate human feedback to iteratively finetune SQL queries.</li>
<li>We plan on leveraging knowledge graphs in the next iteration for better schema linking. </li>
</ul>
<h2>Background</h2>
<p>In large companies, writing SQL queries can be a time-consuming process. The required data can be scattered across a plethora of tables, which constantly get updated, making it difficult for the layperson to create queries for one-shot data analytics tasks. </p>
<h2>Summary of paper</h2>
<p>To improve data democratization in Mercari, we implement a solution using LLMs for text 2 SQL. While a number of existing methods exist, they tend to suffer from errors that are hard to catch. For that reason, instead of treating Text2SQL as a one-shot generation process, we consider it to be an interactive problem with a human in the loop. We build a system that involves validation stages and human feedback to polish the query before returning it to the end user. Our method results in fewer errors and, therefore, more useful queries.</p>
<h2>About Customer Understanding Team</h2>
<p>The Customer Understanding team is dedicated to analyzing the behavior of customers of Mercari Group and proposing strategies to optimize their journey and lifetime value. We employ a variety of technologies like statistics, mathematical optimization, large language models, etc.</p>


mercari AI’s research “LLMs as an Interactive Database Interface for Designing Large Queries” accepted to HILDA Workshop in SIGMOD 2024

Interview

<p>Currently, many ML engineers are part of Mercari's ML team. But those ML engineers aren’t all in the same department—they are scattered throughout the company, working to develop solutions to the challenges faced by their respective teams.</p>
<p>As the Mercari ML group expanded, this decentralization of its engineers created difficulties with sharing knowledge and coordinating between the different teams. To overcome those barriers, the Mercari AI/ML Guild (“ML Guild”) was launched in 2022 under the leadership of on-site engineers. The ML Guild consistently meets at least once a month to achieve the goal of sharing information among ML engineers.</p>
<p>The ML Guild, which started based on ideas from on-site engineers, is now held regularly. We spoke with three engineers who founded and manage the ML Guild—Abhinav Kumar, Paul Willot, and Yusuke Nishioka—about the history of the Guild, and how it has transformed some of the challenges of the organization.</p>
<h2>Profile</h2>
<h3>Abhinav Kumar (@Abhinav)</h3>
<p>Kumar joined Mercari as a new graduate in 2018. He started as an ML Engineer on the Image Search Team and then moved to the Data Science Backend Team where he worked as an Engineering Manager &amp; Tech Lead, focusing on providing a personalized experience to Mercari’s users through ML. Kumar recently joined the Ads Engineering team contributing to developing the Mercari Ads Infrastructure. In addition to his interests in ML and backend development, Kumar also enjoys formulating efficient, agile processes within the engineering teams.</p>
<h3>Paul Willot (@meatboy)</h3>
<p>Paul joined Mercari in 2021 as a Deep Learning specialist. He has experience with the entire ML stack—from modeling and handling large data sets to runtime optimization and cluster deployment—which he leverages to bridge the gap between research and production.<br />
Before coming to Mercari, Paul worked in fintech making real-time forecasting models at scale, and also consulted for major Japanese companies.</p>
<h3>Yusuke Nishioka (@ysk24ok )</h3>
<p>Yusuke had worked on improving ad delivery algorithms and setting up data infrastructure at an internet advertising company, as well as on development of a speech recognition product at an AI startup, before he joined Merpay in 2021. He was transferred to Mercari in 2022 and is currently involved in feature enhancement and maintenance of Mercari’s ML platform.</p>
<p><strong>ーTo start, please introduce yourselves and tell me about your roles at Mercari.</strong></p>
<p><strong>Abhinav：</strong>I joined Mercari as a fresh graduate five and a half years ago. For the first two years, I worked on the ML team attached to Product Image Search. I then moved to the Trust &amp; Safety (TnS) team. After launching the Data Science Backend team together with PMs, I worked as an engineering manager (EM). I am currently in the new Ads Engineering team.</p>
<p><img src="https://storage.googleapis.com/prd-ai-asset/2024/06/a0e7a8ae-mercari-06706-1024x682.jpg" alt="" /></p>
<p><strong>meatboy:</strong> It’s been almost three years since I joined Mercari as an ML engineer. Ever since joining, I’ve been working on projects related to the listing of products. For about a year and a half, I worked with a team that used search engines to make it easier to match with buyers.</p>
<p><strong>ysk24ok: </strong>I joined Merpay in 2021, then transferred to Mercari in 2022. Now, I am part of the ML Platform team, which provides infrastructure and solutions that optimize the work of ML engineers.</p>
<p><strong>ーWhat is the origin of how the Mercari AI/ML Guild (“ML Guild”) was formed?<br />
</strong><br />
Abhinav: The ML Guild started in November 2022. While working as an EM on the Data Science team, we started the ML Guild after speaking with Umezawa (@umechan), an EM on the ML team.</p>
<p>We migrated the organizational systems to a new framework, converting the existing single ML team into units that corresponded to 4–5 different goals, something we dubbed the “Camp System”. ML engineers were then assigned to the various camps, and they remain organizationally dispersed to this day.</p>
<p>Previously, there was a lot of communication between ML members; but as new people joined the company following the introduction of the Camp System, communication among the teams became hard to maintain. In a survey, 80% of respondents believed that “the communication infrastructure for the entire ML team is currently insufficient.” That’s why we came up with the concept of the Mercari AI/ML Guild to build trust and communication between teams. First, we established the goal of getting to know every team and their members.</p>
<p>ML Guild aims to encourage the sharing of knowledge. The time between launching an ML project and actually executing it is very long, so we clearly needed a platform that facilitates the sharing of feedback, what’s going on within key projects, and best practices. At the time, it was unclear to new ML hires where they should seek help and what they should ask.</p>
<p><strong>meatboy:</strong> Even before the ML Guild, we had several groups for discussion. But they were all separate from each other, with no single large group. A lot of projects are focused on very specific topics like ML academic research or search engines, and there wasn’t a place where people could discuss more comprehensive initiatives. The ML Guild provides opportunities to learn the latest news while sharing expertise related to many different areas of machine learning, without being limited to particular topics of interest.</p>
<p><img src="https://storage.googleapis.com/prd-ai-asset/2024/06/d9948a63-mercari-06685-1024x682.jpg" alt="" /></p>
<p><strong>—How is the ML Guild managed?</strong></p>
<p><strong>Abhinav:</strong> Management of the ML Guild is between multiple members, including the three of us here today. But we’ve actually had people participating as volunteers because they want to contribute. Currently, each camp has a representative leader, and those leaders determine the policies governing the ML Guild’s efforts as well as the goals set by the organization.</p>
<p><strong>meatboy: </strong>Right now, all managing members have the same responsibilities. In other words, everyone is a primary organizer. For example, we take turns creating the scheduling for upcoming events. We hold events for team building or just to have conversations.</p>
<p><strong>Abhinav:</strong> Basically, we rotate between three different events each quarter. At the first session, we share the OKR with each team; then in the second session, we share our projects and best practices and make related announcements. For the third event, we work on team building. Also, @meatboy plans a team-building party every three months, creating offline opportunities for people to meet up.</p>
<p><strong>ysk24ok:</strong> For example, what if we introduce a few changes which result in a team having to migrate their work to a different location? We might have to make such announcements, and these sessions function as a way for us to do that.<br />
If we want each team to use the new solutions developed by the Platform team, we must have them migrate from their existing framework. The ML Guild serves as a platform for these announcements.</p>
<p><strong>ー In the years since it was established, has the ML Guild solved any organizational challenges?</strong></p>
<p><strong>meatboy: </strong>We’ve made clear headway in a number of areas. There are still scenarios where we need more rigorous support, and there remain some points that we want to address. But the ML Guild has been really helpful in dealing with some of the issues that we faced at the start. I think it’s fantastic that we can now get regular updates on what’s going on with the other teams.</p>
<p>I personally experienced from learning through the ML Guild that there was a team already working on a similar project, which allowed me to reach out to them and exchange models and knowledge. I’ve had this experience a lot with the ML Guild, building knowledge like that.</p>
<p><strong>ー As ML engineers, have the efforts of the ML Guild changed anything for you?</strong></p>
<p><strong>meatboy:</strong> I regularly communicate with our internal platform team, but many members don’t have the opportunity to do so. As we are constantly developing new services, however, we have to ensure compatibility with existing systems. So it’s very useful to know who has already made certain improvements, or who to ask. I believe this has benefitted many of Mercari’s ML members.</p>
<p>It has also facilitated a lot of direct interaction among our engineers. I was working on projects involving the listing of products when I first joined the company, but I later transferred to the Search department. I had a lot of questions about the size of listed products, but once the ML Guild was created, I was able to speak directly to engineers that had dealt with such matters in the past. That was tremendously helpful.</p>
<p><strong>Abhinav:</strong> This past year, we had an average of over 25 ML engineers participate in each session. And the people who participated once had returned to following sessions as well. Typically, the ongoing attendance rate declines as you continue holding these large-scale sessions. This is just my opinion, but the frequency of ML Guild meetings may be a factor in maintaining these solid attendance numbers. We are only holding them once a month right now, which may allow our busy members to participate without feeling stressed about it.</p>
<p>The ML Guild always meets each month. This creates the awareness that our events are consistently held. But the most crucial thing is that the ML Guild is led not by EMs, but by on-site ML engineers.</p>
<p>When I first joined Mercari, there were similar meetings with opportunities to share information, but it was led almost entirely by EMs. With the ML Guild, however, engineers can share their own insights gleaned from the projects that they themselves are working on. And as they present that information, they’re able to talk to many different people that could become their collaborators.</p>
<p><strong>ysk24ok:</strong> As platform engineers, we sometimes create opportunities to discuss future collaboration. When doing so, there are some difficulties in setting up meetings with a team if you aren’t acquainted with any of its members. Thanks to the ML Guild, we have a chance to get to know the members of all teams.</p>
<p><img src="https://storage.googleapis.com/prd-ai-asset/2024/06/f7f7511c-mercari-06600-1024x682.jpg" alt="" /></p>
<p><strong>ーIs there a future vision for the ML Guild?</strong></p>
<p><strong>Abhinav:</strong> As for short-term goals, we want to maintain our current pace, keep attendance up, and continue working on these initiatives. One of our medium-term goals is for the ML Guild to generate new business projects.</p>
<p><strong>meatboy:</strong> We believe that the ML Guild can help us form cross-team projects, to have a bigger impact on business, and compete externally. We could even aim to become a platform in the future.</p>
<p><strong>ysk24ok:</strong> Right. I don’t know whether this should be a short- or long-term goal, but only certain members have been attending the team-building sessions. And in the ML Guild right now, some members attend the sessions but don’t report on their own projects. I’d like to see how we can improve those areas.</p>
<p><strong>ーWere there any initial difficulties in managing the ML Guild? </strong></p>
<p><strong>Abhinav:</strong> I’m extremely grateful to @meatboy and @ysk24ok for supporting this project from the start. The tricky thing at first was building trust.</p>
<p>“Why did they create these new sessions?” “Why are they necessary?” Since many people didn’t understand this, and everyone was ridiculously busy at their jobs, there wasn’t much participation. But we went to the EMs on each ML team and got their feedback on the ML Guild proposal. This slowly built trust with them, and eventually we got them to ask their team members to attend.</p>
<p><strong>meatboy:</strong> At first, we also had to make a lot of announcements to get members to join. These steady efforts also improved the attendance situation.</p>
<p>One of the things we’ve learned from managing the ML Guild is that we must strike an appropriate balance between team-building events and events for sharing beneficial information.</p>
<p>And I think we always have to ask ourselves, “What is the purpose of this meeting?” Moving forward, I feel that we must continue to provide value that justifies the use of everyone’s time.</p>
<p><strong>Abinav:</strong> I hope for even more people to  join in the future. The ML Guild is independently operated, so I’d like to welcome new members that can rotate in so that we can keep our fresh perspectives while maintaining our current level of skill.</p>
<p><img src="https://storage.googleapis.com/prd-ai-asset/2024/06/9d3a7180-screenshot-2024-06-28-at-23.43.38-1024x682.png" alt="" /></p>


How have the organizational challenges of mercari  ML team been solved? A chat with the engineers who launched the company-wide Guild

<p>Hello, I&#8217;m <a href="https://twitter.com/MLBear2">ML_Bear</a>, an ML Engineer on Mercari&#8217;s Generative AI team.</p>
<p>In a previous article [1], I talked about improving Mercari&#8217;s item recommendations. In this article, I will be presenting a case study involving the categorization of over 3 billion items using large-scale language models (LLMs) and related technologies.</p>
<p>As the LLM boom was sparked by the appearance of ChatGPT, many people became aware that LLMs were used in conversations, but it&#8217;s also true that LLMs can be an extremely useful tool for solving various tasks due to their high level of thinking ability. On the other hand, their slow processing speed and high cost can be a barrier to their implementation in large-scale projects.</p>
<p>This article describes our efforts to overcome these challenges by applying various innovations, maximize the potential of LLMs and its peripheral technologies, and solve the problem of categorizing large-scale item data.</p>
<h2>Challenge</h2>
<p>Let me begin with a brief background of this project and the technical issues involved.<br />
In 2024, Mercari renewed its category structure, revamping its hierarchical structure and significantly increasing the number of item categories. However, when the number of categories and their hierarchical structure are changed, it becomes necessary to change the item data associated with them as well.</p>
<p>Normally, item categorization uses machine learning models or rule-based models. In this case, however, it was not possible to create a classifier using machine learning because the &quot;correct category in the new category structure&quot; for past items was unknowable. In addition, because the number of categories was very large, it was also difficult to construct a rule-based model. This prompted us to see if we could utilize LLMs to address this issue.</p>
<h2>Solution: Prediction algorithm in two-stage configuration with LLM and kNN</h2>
<p>We responded to this issue by constructing a two-stage algorithm as follows.</p>
<ol>
<li>Correctly predict the categories of some past items with ChatGPT 3.5 turbo (OpenAI API[2])</li>
<li>Create a category prediction model for past items using 1. as training data</li>
</ol>
<p>Things would have been simpler if it was possible to predict everything with ChatGPT, but since Mercari&#8217;s past items exceed 3 billion [3], it was impossible to predict everything from the perspective of both processing time and API cost. Therefore, after some trial and error, we settled on this two-stage model configuration. (Classifying all items with ChatGPT 3.5 turbo would have resulted in a cost of approximately 1 million USD and an unrealistic processing time estimate of 1.9 years.)<br />
The following is a brief description of the model. Details will be described in the &quot;Points of Innovation&quot; section, so we will keep the explanations simple here.</p>
<h3>1. Predict some correct categories of past items with ChatGPT 3.5 turbo (OpenAI API)</h3>
<p>First, we sampled several million previously listed items and asked ChatGPT 3.5 turbo to predict the &quot;correct category in the new category structure&quot; for that item. Specifically, we created about 10 candidates for the new category based on each item&#8217;s item name, item description, and original category name, and asked it to provide the correct answer from among those candidates.</p>
<h3>2. Create a category prediction model for past items using 1. as training data</h3>
<p>Next, we created a simple kNN model[4] using the dataset created in 1. as the correct answer data.<br />
Specifically, first the embedding and the correct answer category of the item whose correct answer category was predicted in 1. were stored in a vector database. Then, based on the embedding of the item to be predicted, X similar items were extracted from the vector database, and the most frequent category of those X items was used as the correct category.</p>
<p>Embedding was calculated based on a concatenated string of each item&#8217;s item name, item description, metadata, and original category name. A more complex machine learning model was also considered, but a simple model was adopted because it performed satisfactorily.</p>
<h2>Points of Innovation</h2>
<p>Here are some of the innovations that we devised for this project, applied to the following points which I will explain one by one.</p>
<ul>
<li>Usage of OSS Embedding model</li>
<li>Usage of Multi-GPU with the Sentence Transformers library</li>
<li>Voyager Vector DB for fast neighborhood search on CPU</li>
<li>Accelerated LLM prediction by using max_tokens and CoT</li>
<li>Usage of Numba/cuDF</li>
</ul>
<h3>1. Usage of OSS Embedding model</h3>
<p>The second stage model (kNN) required the computation of the embeddings of items. Although it was possible to build a neural network on our own, it was confirmed that the OpenAI Embeddings API  (<code>text-embedding-ada-002</code>) [5] would provide sufficient accuracy, so we initially decided to use this API.</p>
<p>However, when we made an estimate, we quickly realized that using the OpenAI Embeddings API for all items would be a bit challenging in terms of processing time and cost.<br />
While looking at MTEB[6] and JapaneseEmbeddingEval[7], we noticed that there were many OSS models in languages other than English that were comparable to the OpenAI Embeddings API. We decided to use the OSS models because we found them to be as accurate as the OpenAI Embeddings API when we created our own evaluation dataset and tried them out.<br />
According to the data as of October 2023 in the midst of this project, the following models were evaluated as highly accurate, and we ended up using intfloat/multilingual-e5-base due to its good balance of computational cost and accuracy. (MTEB rankings are constantly changing, so we believe that stronger models may be available as of April 2023.)</p>
<ul>
<li>intfloat/multilingual-e5-large [8]</li>
<li>intfloat/multilingual-e5-base [9]</li>
<li>intfloat/multilingual-e5-small [10]</li>
<li>cl-nagoya/sup-simcse-ja-large [11]</li>
</ul>
<p>Since there are very high-performance embedding models in OSS, we recommend that when doing a project that uses embedding, that you create a simple problem and see if there is a model with sufficient performance in OSS.</p>
<h3>2. Usage of Multi-GPU with Sentence Transformers library</h3>
<p>Although using the OSS model dramatically increased processing speed compared to the OpenAI Embeddings API, more improvements were needed to process billions of items.<br />
Our issues would have been solved much more quickly if we had access to a powerful GPU such as the A100, but it was quite difficult to acquire such a powerful GPU as of November-December 2023 back when the project was launched, possibly due to the global GPU shortage. (It&#8217;s doubtful that the situation has changed much even now.)<br />
We therefore decided to use multiple GPUs such as V100 and L4 in tandem to handle this problem. Fortunately, the Sentence-Transformers[12] library was very helpful because we could easily parallelize multiple GPUs with the following simple code.</p>
<pre><code class="language-python">from sentence_transformers import SentenceTransformer

def embed_multi_process(sentences):
    if &#039;intfloat&#039; in self.model_name:
        sentences = [&quot;query: &quot; + b for b in sentences]
    model = SentenceTransformer(model_name)
    pool = model.start_multi_process_pool()
    embeddings = model.encode_multi_process(sentences, pool)
    model.stop_multi_process_pool(pool)</code></pre>
<p>It would have been ideal if we could use as many powerful GPUs as we needed, but even in situations where this isn&#8217;t possible, we can speed up processing by making use of creative ideas. That’s why it is important to make the most of limited resources by utilizing libraries such as Sentence-Transformers.</p>
<h3>3. Voyager Vector DB for fast neighborhood search on CPU</h3>
<p>A vector database was required when using kNN. Although sampled, the training data held several million items, so it could not fit in the GPU&#8217;s memory. While this may have been solved by using a GPU with a large memory, such as an A100 80GB, the difficulty in obtaining such a powerful GPU hindered us from trying that option.<br />
Around that time, we learned that Spotify&#8217;s Voyager[13] can run at high speed even with a CPU, so we tried it and were able to easily achieve a speed that was sufficient for practical use. Compared to embedding calculations, there was not that much effect on the time required for neighborhood search, so although we did not compare it with other items in the strict sense, we were satisfied at having been able to achieve sufficient speed.<br />
Voyager did not have metadata management capabilities, so we had to write our own client, but we still believe it was a good choice overall.</p>
<h3>4. Accelerated LLM prediction by using max_tokens and CoT</h3>
<p>For this project, ChatGPT 4 was not available due to cost, so we had to use ChatGPT 3.5 turbo. ChatGPT 3.5 turbo is rather clever for the cost, but we were a little concerned about its accuracy. Therefore, we used Chain of Thoughts[14] to improve accuracy by having it generate explanations.<br />
As you may already know, ChatGPT sometimes talks for a long time when asked to provide an explanation, leading to prolonged processing times. Therefore, we tried to shorten the processing time by using the <code>max_tokens</code> parameter to interrupt a long answer midway.</p>
<p>Since the JSON (of Function Calling) is broken when the answer is interrupted, it is necessary to either use <code>llm.stream()</code>of LangChain[15], or restore and parse the JSON yourself, which is a bit time-consuming. Although we have not done an exact comparison, we feel that the method we used strikes a good balance between reducing processing time and improving accuracy.</p>
<p>The following is a sample code for using LangChain&#8217;s <code>llm.stream()</code>.</p>
<pre><code class="language-python">from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate

from typing import Optional
from langchain_core.pydantic_v1 import BaseModel, Field

class ItemCategory(BaseModel):
    item_category_id: int = Field(None, description=&quot;Category ID predicted from product description&quot;)
    reason: Optional[str] = Field(None, description=&quot;Explain in detail why you selected this category ID&quot;)

system_prompt = &quot;&quot;&quot;
Based on the product information given, predict the category of the product.
Please choose a product category from the list of candidates. Explain why you chose it.
&quot;&quot;&quot;
item_info = &quot;(Include product data and potential new categories, etc.) &quot;

llm = ChatOpenAI(
    model_name=&quot;gpt-3.5-turbo&quot;,
    max_tokens=25,
)
structured_llm = llm.with_structured_output(ItemCategory)
prompt = ChatPromptTemplate.from_messages(
    [
        (&quot;system&quot;, system_prompt),
        (&quot;human&quot;, &quot;{item_info}&quot;),
    ]
)
chain = prompt | structured_llm

#  Extract only the last element of streaming
# - Normally, if you terminate the answer with max_tokens, the json is broken and needs to be parsed. 
# - There is no need to parse json when answer is terminated by max_tokens 
#   since it always completes json when termination is executed in langchain stream.
for res in chain.stream({&quot;item_info&quot;: item_info}):
    pass

print(res.json(ensure_ascii=False))  # res: ItemCategory
# {&quot;item_category_id&quot;: 1, &quot;reason&quot;: &quot;The product name contains &#039;stuffed animal&#039; &quot;}</code></pre>
<h3>5. Usage of Numba/cuDF</h3>
<p>Since processing speed is a concern even for minor processes when processing billions of items, all processing was accelerated with cuDF[16] and Numba[17] whenever possible.<br />
Although I am not very good at writing Numba, when I showed the raw Python code to ChatGPT 4, it rewrote it for me, which greatly reduced my coding time.</p>
<h2>Conclusion</h2>
<p>ChatGPT has attracted a lot of attention for its frequent use in a conversational style, and its advanced thinking ability provides effortless solutions to tasks that were previously tedious or deemed impossible. In our project, ChatGPT helped us solve the tedious task of reclassifying a huge amount of item data into new categories within a short period of time.</p>
<p>We were also able to maximize results even with limited time and resources by making use of OSS Embedding models and multiple GPUs, adopting a vector database that enables fast neighborhood search, using ChatGPT to speed up prediction, and using Numba to accelerate processing.<br />
I hope that this case study will demonstrate the potential of ChatGPT and other large-scale language models and will be helpful in future projects. We encourage you to utilize LLMs in a variety of situations and take on challenges that have been difficult to solve in the past.</p>
<h3>Refs</h3>
<ol>
<li><a href="https://engineering.mercari.com/en/blog/entry/20230612-cf-similar-item/">Improving Item Recommendation Accuracy Using Collaborative Filtering and Vector Search Engine</a></li>
<li><a href="https://platform.openai.com/docs/guides/text-generation">OpenAI API</a></li>
<li><a href="https://about.mercari.com/press/news/articles/20221128_threebillion/">Total number of items listed on flea market app &quot;Mercari&quot; surpasses 3 billion (フリマアプリ「メルカリ」累計出品数が30億品を突破)</a></li>
<li><a href="https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm">k-nearest neighbors algorithm</a></li>
<li><a href="https://platform.openai.com/docs/guides/embeddings">OpenAI Embeddings API</a></li>
<li>[Massive Text Embedding Benchmark (MTEB) Leaderboard]()</li>
<li><a href="https://github.com/oshizo/JapaneseEmbeddingEval">JapaneseEmbeddingEval</a></li>
<li><a href="https://huggingface.co/intfloat/multilingual-e5-large">intfloat/multilingual-e5-large</a></li>
<li><a href="https://huggingface.co![](https://storage.googleapis.com/prd-engineering-asset/2024/05/1169138b-mlbear-1024x538.png)/intfloat/multilingual-e5-base">intfloat/multilingual-e5-base</a></li>
<li><a href="https://huggingface.co/intfloat/multilingual-e5-small">intfloat/multilingual-e5-small</a></li>
<li><a href="https://huggingface.co/cl-nagoya/sup-simcse-ja-large">cl-nagoya/sup-simcse-ja-large</a></li>
<li><a href="https://www.sbert.net/">Sentence-Transformers</a></li>
<li><a href="https://github.com/spotify/voyager">Voyager</a></li>
<li><a href="https://arxiv.org/abs/2201.11903">Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (Wei et al. 2022)</a></li>
<li><a href="https://github.com/langchain-ai/langchain">LangChain</a></li>
<li><a href="https://github.com/rapidsai/cudf">rapidsai/cudf</a></li>
<li><a href="https://numba.pydata.org/">Numba: A High Performance Python Compiler</a></li>
</ol>


LLM-based Approach to Large-scale Item Category Classification

Blogs by mercari AI team, delivering information about AI technology and organization

Mercari AI’s research paper, “Zero-Shot Retrieval for Scalable Visual Search in a Two-Sided Marketplace” has been accepted to the TSMO 2025 workshop at KDD 2025

Mercari AI Team Research Paper “Visual Recommendation on E-commerce Platforms Via Fine-Tuned Vision-Language Models” Accepted for Presentation at MIRU 2025

Mercari AI Team’s Research “LLMOps for Eval-Driven Development at Scale” Accepted to FOSSASIA Summit 2025

The Mercari AI team’s research “Strategic Coupon Allocation for Increasing Providers’ Sales Experiences in Two-sided Marketplaces” has been accepted at the KDD2024 TSMO 2024 Workshop

mercari AI’s research “LLMs as an Interactive Database Interface for Designing Large Queries” accepted to HILDA Workshop in SIGMOD 2024

Mercari AI’s research paper, “Zero-Shot Retrieval for Scalable Visual Search in a Two-Sided Marketplace” has been accepted to the TSMO 2025 workshop at KDD 2025

Mercari AI Team Research Paper “Visual Recommendation on E-commerce Platforms Via Fine-Tuned Vision-Language Models” Accepted for Presentation at MIRU 2025

Mercari AI Team’s Research “LLMOps for Eval-Driven Development at Scale” Accepted to FOSSASIA Summit 2025

The Mercari AI team’s research “Strategic Coupon Allocation for Increasing Providers’ Sales Experiences in Two-sided Marketplaces” has been accepted at the KDD2024 TSMO 2024 Workshop

mercari AI’s research “LLMs as an Interactive Database Interface for Designing Large Queries” accepted to HILDA Workshop in SIGMOD 2024

Mercari AI’s research paper, “Zero-Shot Retrieval for Scalable Visual Search in a Two-Sided Marketplace” has been accepted to the TSMO 2025 workshop at KDD 2025

Mercari AI Team Research Paper “Visual Recommendation on E-commerce Platforms Via Fine-Tuned Vision-Language Models” Accepted for Presentation at MIRU 2025

Mercari AI Team’s Research “LLMOps for Eval-Driven Development at Scale” Accepted to FOSSASIA Summit 2025

The Mercari AI team’s research “Strategic Coupon Allocation for Increasing Providers’ Sales Experiences in Two-sided Marketplaces” has been accepted at the KDD2024 TSMO 2024 Workshop

mercari AI’s research “LLMs as an Interactive Database Interface for Designing Large Queries” accepted to HILDA Workshop in SIGMOD 2024

Latest Posts

Mercari AI’s research paper, “Zero-Shot Retrieval for Scalable Visual Search in a Two-Sided Marketplace” has been accepted to the TSMO 2025 workshop at KDD 2025

Mercari AI Team Research Paper “Visual Recommendation on E-commerce Platforms Via Fine-Tuned Vision-Language Models” Accepted for Presentation at MIRU 2025

Mercari AI Team’s Research “LLMOps for Eval-Driven Development at Scale” Accepted to FOSSASIA Summit 2025

Fine-Tuning an LLM to Extract Dynamically Specified Attributes

The Mercari AI team’s research “Strategic Coupon Allocation for Increasing Providers’ Sales Experiences in Two-sided Marketplaces” has been accepted at the KDD2024 TSMO 2024 Workshop

mercari AI’s research “LLMs as an Interactive Database Interface for Designing Large Queries” accepted to HILDA Workshop in SIGMOD 2024

How have the organizational challenges of mercari ML team been solved? A chat with the engineers who launched the company-wide Guild

LLM-based Approach to Large-scale Item Category Classification