Interview

<p>Currently, many ML engineers are part of Mercari's ML team. But those ML engineers aren’t all in the same department—they are scattered throughout the company, working to develop solutions to the challenges faced by their respective teams.</p>
<p>As the Mercari ML group expanded, this decentralization of its engineers created difficulties with sharing knowledge and coordinating between the different teams. To overcome those barriers, the Mercari AI/ML Guild (“ML Guild”) was launched in 2022 under the leadership of on-site engineers. The ML Guild consistently meets at least once a month to achieve the goal of sharing information among ML engineers.</p>
<p>The ML Guild, which started based on ideas from on-site engineers, is now held regularly. We spoke with three engineers who founded and manage the ML Guild—Abhinav Kumar, Paul Willot, and Yusuke Nishioka—about the history of the Guild, and how it has transformed some of the challenges of the organization.</p>
<h2>Profile</h2>
<h3>Abhinav Kumar (@Abhinav)</h3>
<p>Kumar joined Mercari as a new graduate in 2018. He started as an ML Engineer on the Image Search Team and then moved to the Data Science Backend Team where he worked as an Engineering Manager &amp; Tech Lead, focusing on providing a personalized experience to Mercari’s users through ML. Kumar recently joined the Ads Engineering team contributing to developing the Mercari Ads Infrastructure. In addition to his interests in ML and backend development, Kumar also enjoys formulating efficient, agile processes within the engineering teams.</p>
<h3>Paul Willot (@meatboy)</h3>
<p>Paul joined Mercari in 2021 as a Deep Learning specialist. He has experience with the entire ML stack—from modeling and handling large data sets to runtime optimization and cluster deployment—which he leverages to bridge the gap between research and production.<br />
Before coming to Mercari, Paul worked in fintech making real-time forecasting models at scale, and also consulted for major Japanese companies.</p>
<h3>Yusuke Nishioka (@ysk24ok )</h3>
<p>Yusuke had worked on improving ad delivery algorithms and setting up data infrastructure at an internet advertising company, as well as on development of a speech recognition product at an AI startup, before he joined Merpay in 2021. He was transferred to Mercari in 2022 and is currently involved in feature enhancement and maintenance of Mercari’s ML platform.</p>
<p><strong>ーTo start, please introduce yourselves and tell me about your roles at Mercari.</strong></p>
<p><strong>Abhinav：</strong>I joined Mercari as a fresh graduate five and a half years ago. For the first two years, I worked on the ML team attached to Product Image Search. I then moved to the Trust &amp; Safety (TnS) team. After launching the Data Science Backend team together with PMs, I worked as an engineering manager (EM). I am currently in the new Ads Engineering team.</p>
<p><img src="https://storage.googleapis.com/prd-ai-asset/2024/06/a0e7a8ae-mercari-06706-1024x682.jpg" alt="" /></p>
<p><strong>meatboy:</strong> It’s been almost three years since I joined Mercari as an ML engineer. Ever since joining, I’ve been working on projects related to the listing of products. For about a year and a half, I worked with a team that used search engines to make it easier to match with buyers.</p>
<p><strong>ysk24ok: </strong>I joined Merpay in 2021, then transferred to Mercari in 2022. Now, I am part of the ML Platform team, which provides infrastructure and solutions that optimize the work of ML engineers.</p>
<p><strong>ーWhat is the origin of how the Mercari AI/ML Guild (“ML Guild”) was formed?<br />
</strong><br />
Abhinav: The ML Guild started in November 2022. While working as an EM on the Data Science team, we started the ML Guild after speaking with Umezawa (@umechan), an EM on the ML team.</p>
<p>We migrated the organizational systems to a new framework, converting the existing single ML team into units that corresponded to 4–5 different goals, something we dubbed the “Camp System”. ML engineers were then assigned to the various camps, and they remain organizationally dispersed to this day.</p>
<p>Previously, there was a lot of communication between ML members; but as new people joined the company following the introduction of the Camp System, communication among the teams became hard to maintain. In a survey, 80% of respondents believed that “the communication infrastructure for the entire ML team is currently insufficient.” That’s why we came up with the concept of the Mercari AI/ML Guild to build trust and communication between teams. First, we established the goal of getting to know every team and their members.</p>
<p>ML Guild aims to encourage the sharing of knowledge. The time between launching an ML project and actually executing it is very long, so we clearly needed a platform that facilitates the sharing of feedback, what’s going on within key projects, and best practices. At the time, it was unclear to new ML hires where they should seek help and what they should ask.</p>
<p><strong>meatboy:</strong> Even before the ML Guild, we had several groups for discussion. But they were all separate from each other, with no single large group. A lot of projects are focused on very specific topics like ML academic research or search engines, and there wasn’t a place where people could discuss more comprehensive initiatives. The ML Guild provides opportunities to learn the latest news while sharing expertise related to many different areas of machine learning, without being limited to particular topics of interest.</p>
<p><img src="https://storage.googleapis.com/prd-ai-asset/2024/06/d9948a63-mercari-06685-1024x682.jpg" alt="" /></p>
<p><strong>—How is the ML Guild managed?</strong></p>
<p><strong>Abhinav:</strong> Management of the ML Guild is between multiple members, including the three of us here today. But we’ve actually had people participating as volunteers because they want to contribute. Currently, each camp has a representative leader, and those leaders determine the policies governing the ML Guild’s efforts as well as the goals set by the organization.</p>
<p><strong>meatboy: </strong>Right now, all managing members have the same responsibilities. In other words, everyone is a primary organizer. For example, we take turns creating the scheduling for upcoming events. We hold events for team building or just to have conversations.</p>
<p><strong>Abhinav:</strong> Basically, we rotate between three different events each quarter. At the first session, we share the OKR with each team; then in the second session, we share our projects and best practices and make related announcements. For the third event, we work on team building. Also, @meatboy plans a team-building party every three months, creating offline opportunities for people to meet up.</p>
<p><strong>ysk24ok:</strong> For example, what if we introduce a few changes which result in a team having to migrate their work to a different location? We might have to make such announcements, and these sessions function as a way for us to do that.<br />
If we want each team to use the new solutions developed by the Platform team, we must have them migrate from their existing framework. The ML Guild serves as a platform for these announcements.</p>
<p><strong>ー In the years since it was established, has the ML Guild solved any organizational challenges?</strong></p>
<p><strong>meatboy: </strong>We’ve made clear headway in a number of areas. There are still scenarios where we need more rigorous support, and there remain some points that we want to address. But the ML Guild has been really helpful in dealing with some of the issues that we faced at the start. I think it’s fantastic that we can now get regular updates on what’s going on with the other teams.</p>
<p>I personally experienced from learning through the ML Guild that there was a team already working on a similar project, which allowed me to reach out to them and exchange models and knowledge. I’ve had this experience a lot with the ML Guild, building knowledge like that.</p>
<p><strong>ー As ML engineers, have the efforts of the ML Guild changed anything for you?</strong></p>
<p><strong>meatboy:</strong> I regularly communicate with our internal platform team, but many members don’t have the opportunity to do so. As we are constantly developing new services, however, we have to ensure compatibility with existing systems. So it’s very useful to know who has already made certain improvements, or who to ask. I believe this has benefitted many of Mercari’s ML members.</p>
<p>It has also facilitated a lot of direct interaction among our engineers. I was working on projects involving the listing of products when I first joined the company, but I later transferred to the Search department. I had a lot of questions about the size of listed products, but once the ML Guild was created, I was able to speak directly to engineers that had dealt with such matters in the past. That was tremendously helpful.</p>
<p><strong>Abhinav:</strong> This past year, we had an average of over 25 ML engineers participate in each session. And the people who participated once had returned to following sessions as well. Typically, the ongoing attendance rate declines as you continue holding these large-scale sessions. This is just my opinion, but the frequency of ML Guild meetings may be a factor in maintaining these solid attendance numbers. We are only holding them once a month right now, which may allow our busy members to participate without feeling stressed about it.</p>
<p>The ML Guild always meets each month. This creates the awareness that our events are consistently held. But the most crucial thing is that the ML Guild is led not by EMs, but by on-site ML engineers.</p>
<p>When I first joined Mercari, there were similar meetings with opportunities to share information, but it was led almost entirely by EMs. With the ML Guild, however, engineers can share their own insights gleaned from the projects that they themselves are working on. And as they present that information, they’re able to talk to many different people that could become their collaborators.</p>
<p><strong>ysk24ok:</strong> As platform engineers, we sometimes create opportunities to discuss future collaboration. When doing so, there are some difficulties in setting up meetings with a team if you aren’t acquainted with any of its members. Thanks to the ML Guild, we have a chance to get to know the members of all teams.</p>
<p><img src="https://storage.googleapis.com/prd-ai-asset/2024/06/f7f7511c-mercari-06600-1024x682.jpg" alt="" /></p>
<p><strong>ーIs there a future vision for the ML Guild?</strong></p>
<p><strong>Abhinav:</strong> As for short-term goals, we want to maintain our current pace, keep attendance up, and continue working on these initiatives. One of our medium-term goals is for the ML Guild to generate new business projects.</p>
<p><strong>meatboy:</strong> We believe that the ML Guild can help us form cross-team projects, to have a bigger impact on business, and compete externally. We could even aim to become a platform in the future.</p>
<p><strong>ysk24ok:</strong> Right. I don’t know whether this should be a short- or long-term goal, but only certain members have been attending the team-building sessions. And in the ML Guild right now, some members attend the sessions but don’t report on their own projects. I’d like to see how we can improve those areas.</p>
<p><strong>ーWere there any initial difficulties in managing the ML Guild? </strong></p>
<p><strong>Abhinav:</strong> I’m extremely grateful to @meatboy and @ysk24ok for supporting this project from the start. The tricky thing at first was building trust.</p>
<p>“Why did they create these new sessions?” “Why are they necessary?” Since many people didn’t understand this, and everyone was ridiculously busy at their jobs, there wasn’t much participation. But we went to the EMs on each ML team and got their feedback on the ML Guild proposal. This slowly built trust with them, and eventually we got them to ask their team members to attend.</p>
<p><strong>meatboy:</strong> At first, we also had to make a lot of announcements to get members to join. These steady efforts also improved the attendance situation.</p>
<p>One of the things we’ve learned from managing the ML Guild is that we must strike an appropriate balance between team-building events and events for sharing beneficial information.</p>
<p>And I think we always have to ask ourselves, “What is the purpose of this meeting?” Moving forward, I feel that we must continue to provide value that justifies the use of everyone’s time.</p>
<p><strong>Abinav:</strong> I hope for even more people to  join in the future. The ML Guild is independently operated, so I’d like to welcome new members that can rotate in so that we can keep our fresh perspectives while maintaining our current level of skill.</p>
<p><img src="https://storage.googleapis.com/prd-ai-asset/2024/06/9d3a7180-screenshot-2024-06-28-at-23.43.38-1024x682.png" alt="" /></p>


How have the organizational challenges of mercari  ML team been solved? A chat with the engineers who launched the company-wide Guild

<p>Hello, I&#8217;m <a href="https://twitter.com/MLBear2">ML_Bear</a>, an ML Engineer on Mercari&#8217;s Generative AI team.</p>
<p>In a previous article [1], I talked about improving Mercari&#8217;s item recommendations. In this article, I will be presenting a case study involving the categorization of over 3 billion items using large-scale language models (LLMs) and related technologies.</p>
<p>As the LLM boom was sparked by the appearance of ChatGPT, many people became aware that LLMs were used in conversations, but it&#8217;s also true that LLMs can be an extremely useful tool for solving various tasks due to their high level of thinking ability. On the other hand, their slow processing speed and high cost can be a barrier to their implementation in large-scale projects.</p>
<p>This article describes our efforts to overcome these challenges by applying various innovations, maximize the potential of LLMs and its peripheral technologies, and solve the problem of categorizing large-scale item data.</p>
<h2>Challenge</h2>
<p>Let me begin with a brief background of this project and the technical issues involved.<br />
In 2024, Mercari renewed its category structure, revamping its hierarchical structure and significantly increasing the number of item categories. However, when the number of categories and their hierarchical structure are changed, it becomes necessary to change the item data associated with them as well.</p>
<p>Normally, item categorization uses machine learning models or rule-based models. In this case, however, it was not possible to create a classifier using machine learning because the &quot;correct category in the new category structure&quot; for past items was unknowable. In addition, because the number of categories was very large, it was also difficult to construct a rule-based model. This prompted us to see if we could utilize LLMs to address this issue.</p>
<h2>Solution: Prediction algorithm in two-stage configuration with LLM and kNN</h2>
<p>We responded to this issue by constructing a two-stage algorithm as follows.</p>
<ol>
<li>Correctly predict the categories of some past items with ChatGPT 3.5 turbo (OpenAI API[2])</li>
<li>Create a category prediction model for past items using 1. as training data</li>
</ol>
<p>Things would have been simpler if it was possible to predict everything with ChatGPT, but since Mercari&#8217;s past items exceed 3 billion [3], it was impossible to predict everything from the perspective of both processing time and API cost. Therefore, after some trial and error, we settled on this two-stage model configuration. (Classifying all items with ChatGPT 3.5 turbo would have resulted in a cost of approximately 1 million USD and an unrealistic processing time estimate of 1.9 years.)<br />
The following is a brief description of the model. Details will be described in the &quot;Points of Innovation&quot; section, so we will keep the explanations simple here.</p>
<h3>1. Predict some correct categories of past items with ChatGPT 3.5 turbo (OpenAI API)</h3>
<p>First, we sampled several million previously listed items and asked ChatGPT 3.5 turbo to predict the &quot;correct category in the new category structure&quot; for that item. Specifically, we created about 10 candidates for the new category based on each item&#8217;s item name, item description, and original category name, and asked it to provide the correct answer from among those candidates.</p>
<h3>2. Create a category prediction model for past items using 1. as training data</h3>
<p>Next, we created a simple kNN model[4] using the dataset created in 1. as the correct answer data.<br />
Specifically, first the embedding and the correct answer category of the item whose correct answer category was predicted in 1. were stored in a vector database. Then, based on the embedding of the item to be predicted, X similar items were extracted from the vector database, and the most frequent category of those X items was used as the correct category.</p>
<p>Embedding was calculated based on a concatenated string of each item&#8217;s item name, item description, metadata, and original category name. A more complex machine learning model was also considered, but a simple model was adopted because it performed satisfactorily.</p>
<h2>Points of Innovation</h2>
<p>Here are some of the innovations that we devised for this project, applied to the following points which I will explain one by one.</p>
<ul>
<li>Usage of OSS Embedding model</li>
<li>Usage of Multi-GPU with the Sentence Transformers library</li>
<li>Voyager Vector DB for fast neighborhood search on CPU</li>
<li>Accelerated LLM prediction by using max_tokens and CoT</li>
<li>Usage of Numba/cuDF</li>
</ul>
<h3>1. Usage of OSS Embedding model</h3>
<p>The second stage model (kNN) required the computation of the embeddings of items. Although it was possible to build a neural network on our own, it was confirmed that the OpenAI Embeddings API  (<code>text-embedding-ada-002</code>) [5] would provide sufficient accuracy, so we initially decided to use this API.</p>
<p>However, when we made an estimate, we quickly realized that using the OpenAI Embeddings API for all items would be a bit challenging in terms of processing time and cost.<br />
While looking at MTEB[6] and JapaneseEmbeddingEval[7], we noticed that there were many OSS models in languages other than English that were comparable to the OpenAI Embeddings API. We decided to use the OSS models because we found them to be as accurate as the OpenAI Embeddings API when we created our own evaluation dataset and tried them out.<br />
According to the data as of October 2023 in the midst of this project, the following models were evaluated as highly accurate, and we ended up using intfloat/multilingual-e5-base due to its good balance of computational cost and accuracy. (MTEB rankings are constantly changing, so we believe that stronger models may be available as of April 2023.)</p>
<ul>
<li>intfloat/multilingual-e5-large [8]</li>
<li>intfloat/multilingual-e5-base [9]</li>
<li>intfloat/multilingual-e5-small [10]</li>
<li>cl-nagoya/sup-simcse-ja-large [11]</li>
</ul>
<p>Since there are very high-performance embedding models in OSS, we recommend that when doing a project that uses embedding, that you create a simple problem and see if there is a model with sufficient performance in OSS.</p>
<h3>2. Usage of Multi-GPU with Sentence Transformers library</h3>
<p>Although using the OSS model dramatically increased processing speed compared to the OpenAI Embeddings API, more improvements were needed to process billions of items.<br />
Our issues would have been solved much more quickly if we had access to a powerful GPU such as the A100, but it was quite difficult to acquire such a powerful GPU as of November-December 2023 back when the project was launched, possibly due to the global GPU shortage. (It&#8217;s doubtful that the situation has changed much even now.)<br />
We therefore decided to use multiple GPUs such as V100 and L4 in tandem to handle this problem. Fortunately, the Sentence-Transformers[12] library was very helpful because we could easily parallelize multiple GPUs with the following simple code.</p>
<pre><code class="language-python">from sentence_transformers import SentenceTransformer

def embed_multi_process(sentences):
    if &#039;intfloat&#039; in self.model_name:
        sentences = [&quot;query: &quot; + b for b in sentences]
    model = SentenceTransformer(model_name)
    pool = model.start_multi_process_pool()
    embeddings = model.encode_multi_process(sentences, pool)
    model.stop_multi_process_pool(pool)</code></pre>
<p>It would have been ideal if we could use as many powerful GPUs as we needed, but even in situations where this isn&#8217;t possible, we can speed up processing by making use of creative ideas. That’s why it is important to make the most of limited resources by utilizing libraries such as Sentence-Transformers.</p>
<h3>3. Voyager Vector DB for fast neighborhood search on CPU</h3>
<p>A vector database was required when using kNN. Although sampled, the training data held several million items, so it could not fit in the GPU&#8217;s memory. While this may have been solved by using a GPU with a large memory, such as an A100 80GB, the difficulty in obtaining such a powerful GPU hindered us from trying that option.<br />
Around that time, we learned that Spotify&#8217;s Voyager[13] can run at high speed even with a CPU, so we tried it and were able to easily achieve a speed that was sufficient for practical use. Compared to embedding calculations, there was not that much effect on the time required for neighborhood search, so although we did not compare it with other items in the strict sense, we were satisfied at having been able to achieve sufficient speed.<br />
Voyager did not have metadata management capabilities, so we had to write our own client, but we still believe it was a good choice overall.</p>
<h3>4. Accelerated LLM prediction by using max_tokens and CoT</h3>
<p>For this project, ChatGPT 4 was not available due to cost, so we had to use ChatGPT 3.5 turbo. ChatGPT 3.5 turbo is rather clever for the cost, but we were a little concerned about its accuracy. Therefore, we used Chain of Thoughts[14] to improve accuracy by having it generate explanations.<br />
As you may already know, ChatGPT sometimes talks for a long time when asked to provide an explanation, leading to prolonged processing times. Therefore, we tried to shorten the processing time by using the <code>max_tokens</code> parameter to interrupt a long answer midway.</p>
<p>Since the JSON (of Function Calling) is broken when the answer is interrupted, it is necessary to either use <code>llm.stream()</code>of LangChain[15], or restore and parse the JSON yourself, which is a bit time-consuming. Although we have not done an exact comparison, we feel that the method we used strikes a good balance between reducing processing time and improving accuracy.</p>
<p>The following is a sample code for using LangChain&#8217;s <code>llm.stream()</code>.</p>
<pre><code class="language-python">from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate

from typing import Optional
from langchain_core.pydantic_v1 import BaseModel, Field

class ItemCategory(BaseModel):
    item_category_id: int = Field(None, description=&quot;Category ID predicted from product description&quot;)
    reason: Optional[str] = Field(None, description=&quot;Explain in detail why you selected this category ID&quot;)

system_prompt = &quot;&quot;&quot;
Based on the product information given, predict the category of the product.
Please choose a product category from the list of candidates. Explain why you chose it.
&quot;&quot;&quot;
item_info = &quot;(Include product data and potential new categories, etc.) &quot;

llm = ChatOpenAI(
    model_name=&quot;gpt-3.5-turbo&quot;,
    max_tokens=25,
)
structured_llm = llm.with_structured_output(ItemCategory)
prompt = ChatPromptTemplate.from_messages(
    [
        (&quot;system&quot;, system_prompt),
        (&quot;human&quot;, &quot;{item_info}&quot;),
    ]
)
chain = prompt | structured_llm

#  Extract only the last element of streaming
# - Normally, if you terminate the answer with max_tokens, the json is broken and needs to be parsed. 
# - There is no need to parse json when answer is terminated by max_tokens 
#   since it always completes json when termination is executed in langchain stream.
for res in chain.stream({&quot;item_info&quot;: item_info}):
    pass

print(res.json(ensure_ascii=False))  # res: ItemCategory
# {&quot;item_category_id&quot;: 1, &quot;reason&quot;: &quot;The product name contains &#039;stuffed animal&#039; &quot;}</code></pre>
<h3>5. Usage of Numba/cuDF</h3>
<p>Since processing speed is a concern even for minor processes when processing billions of items, all processing was accelerated with cuDF[16] and Numba[17] whenever possible.<br />
Although I am not very good at writing Numba, when I showed the raw Python code to ChatGPT 4, it rewrote it for me, which greatly reduced my coding time.</p>
<h2>Conclusion</h2>
<p>ChatGPT has attracted a lot of attention for its frequent use in a conversational style, and its advanced thinking ability provides effortless solutions to tasks that were previously tedious or deemed impossible. In our project, ChatGPT helped us solve the tedious task of reclassifying a huge amount of item data into new categories within a short period of time.</p>
<p>We were also able to maximize results even with limited time and resources by making use of OSS Embedding models and multiple GPUs, adopting a vector database that enables fast neighborhood search, using ChatGPT to speed up prediction, and using Numba to accelerate processing.<br />
I hope that this case study will demonstrate the potential of ChatGPT and other large-scale language models and will be helpful in future projects. We encourage you to utilize LLMs in a variety of situations and take on challenges that have been difficult to solve in the past.</p>
<h3>Refs</h3>
<ol>
<li><a href="https://engineering.mercari.com/en/blog/entry/20230612-cf-similar-item/">Improving Item Recommendation Accuracy Using Collaborative Filtering and Vector Search Engine</a></li>
<li><a href="https://platform.openai.com/docs/guides/text-generation">OpenAI API</a></li>
<li><a href="https://about.mercari.com/press/news/articles/20221128_threebillion/">Total number of items listed on flea market app &quot;Mercari&quot; surpasses 3 billion (フリマアプリ「メルカリ」累計出品数が30億品を突破)</a></li>
<li><a href="https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm">k-nearest neighbors algorithm</a></li>
<li><a href="https://platform.openai.com/docs/guides/embeddings">OpenAI Embeddings API</a></li>
<li>[Massive Text Embedding Benchmark (MTEB) Leaderboard]()</li>
<li><a href="https://github.com/oshizo/JapaneseEmbeddingEval">JapaneseEmbeddingEval</a></li>
<li><a href="https://huggingface.co/intfloat/multilingual-e5-large">intfloat/multilingual-e5-large</a></li>
<li><a href="https://huggingface.co![](https://storage.googleapis.com/prd-engineering-asset/2024/05/1169138b-mlbear-1024x538.png)/intfloat/multilingual-e5-base">intfloat/multilingual-e5-base</a></li>
<li><a href="https://huggingface.co/intfloat/multilingual-e5-small">intfloat/multilingual-e5-small</a></li>
<li><a href="https://huggingface.co/cl-nagoya/sup-simcse-ja-large">cl-nagoya/sup-simcse-ja-large</a></li>
<li><a href="https://www.sbert.net/">Sentence-Transformers</a></li>
<li><a href="https://github.com/spotify/voyager">Voyager</a></li>
<li><a href="https://arxiv.org/abs/2201.11903">Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (Wei et al. 2022)</a></li>
<li><a href="https://github.com/langchain-ai/langchain">LangChain</a></li>
<li><a href="https://github.com/rapidsai/cudf">rapidsai/cudf</a></li>
<li><a href="https://numba.pydata.org/">Numba: A High Performance Python Compiler</a></li>
</ol>


LLM-based Approach to Large-scale Item Category Classification

Blog

<h2>Overview</h2>
<p>This time, Kanta Suga (from the Graduate School of Advanced Science and Engineering, Waseda University), a former intern of the Mercari AI team, together with Mercari AI team engineers Rintaro Miyamoto, Naoki Katsura, and Keisuke Umezawa, presented their research on the &quot;A related document retrieval system using past queries&quot; at the 30th Annual Conference of the Association for Natural Language Processing (NLP2024).</p>
<p>The Association for Natural Language Processing is the largest domestic conference where researchers and engineers in natural language processing (NLP) gather, and the 30th conference, NLP2024, was held in Kobe, Hyogo Prefecture from March 11 to March 15, 2024.</p>
<h2>Research Background</h2>
<p>At Mercari, we develop in-house tools for customer inquiries and fraud detection. Especially, quickly resolving issues that customers face is very important for improving customer satisfaction.</p>
<p>Although customer issues and their contexts vary, there are similarities among past inquiries, and by leveraging this accumulated information, we aim to reduce resolution time and have proposed this method.</p>
<h2>Research Summary</h2>
<p>Mercari has a contact form within its service, allowing customers to send inquiries when they face issues. With the large number of inquiries received, there is a need for efficient resolution time.</p>
<p>Many inquiries are similar to past ones, and referencing similar past inquiries serves as a starting point for improving operational efficiency.</p>
<p>However, due to changes in support manuals and systems during service operations, referencing past inquiries alone is sometimes insufficient. Therefore, in this research, we propose a method of using past inquiries to search the current help guide, and in our experiments, we report on the results of various conditions changed.</p>
<h2>Key Points of the Presentation</h2>
<p>In this paper, we propose a method whereby when customers face issues on Mercari, the text of their inquiries and data from past conversation histories are used to search for solutions in the existing help guide, associating them based on similarity.</p>
<p>In the proposed method, customer inquiries are vectorized using a document embedding model. Similarly, from vectorized past inquiry histories, inquiries with high similarity are collected, and the help guides showing high similarity to the responses of these inquiries are linked by majority vote, achieving high search accuracy.</p>
<p><img src="https://storage.googleapis.com/prd-ai-asset/2024/05/cf2bad72-image1-724x1024.png" alt="" /></p>
<p><center>The Poster image presented at NLP2024<br />
</center></p>
<h2>Presenter's Comment</h2>
<h3>Kanta Suga | Mercari AI Team, Former Intern</h3>
<p>During this opportunity, I realized the importance of directly exchanging opinions at conferences to find new improvements and to announce the results externally.</p>
<p>Also, while listening to various presentations, I felt the rapid technological advancements in the AI field, and I realized that I need to make more effort to keep up. It was a very valuable experience to have such an opportunity to present even as an intern. Thank you very much.</p>
<h3>Naoki Katsura | ML Engineer</h3>
<p>At this year's annual conference of the Association for Natural Language Processing, we not only presented a poster but also set up a Mercari sponsor booth, where we engaged in discussions with domestic students, researchers, and engineers.</p>
<p>Researchers and engineers in the industry often share common perceptions of the challenges faced with using language models for customer inquiries. This conference was a meaningful platform for input and discussions. We will utilize the insights gained through these discussions in Mercari's operations.</p>
<p><img src="https://storage.googleapis.com/prd-ai-asset/2024/05/af108a4c-image2-1024x576.jpg" alt="" /></p>
<p><center><br />
Members at the site<br />
</center></p>
<p><img src="https://storage.googleapis.com/prd-ai-asset/2024/05/9581bddd-gixoxqxbqaanecv-1-1024x576.jpeg" alt="" /></p>
<p><center><br />
Mercari's sponsor booth at the conference<br />
</center></p>
<h2>About the CRE (Customer Reliability Engineering) ML Team</h2>
<p>The CRE ML Team develops in-house tools and streamlines operations using machine learning in the area of customer support at Mercari.</p>


Mercari AI team presented in NLP2024 “A related document retrieval system using past queries”

<h2>Overview</h2>
<p>We are pleased to announce that Mercari ML Engineers Chingis Oinar and  Teo Narboneta Zosa will have a presentation at <a href="https://eventyay.com/e/55d2a466">FOSSASIA Summit 2024</a>, held from Monday 8th April to Wednesday 10 April.</p>
<p>FOSSASIA is an Asia-based organization that develops open source software applications and open hardware with the global community. The event aims to create connections between developers, vendors and users. With a strong emphasis on OSS, the topics will range from Hardware/AI/Cloud technologies/Databases/Mobile/IT Education and Legal Compliance...etc.</p>
<p>Presentations by Chingis Oinar and  Teo Narboneta Zosa, engineers from the Mercari AI team, will take place as follows.</p>
<ul>
<li>Title :“Unleashing the Potential of AI Through Search Ranking: How Mercari Uses Data &amp; Science to Drive Marketplace Growth”</li>
<li>The presentation Time : Tuesday, April 9, 2024, 11:00 - 11:35 GMT+7</li>
<li>Detail : <a href="https://eventyay.com/e/55d2a466/session/8985">https://eventyay.com/e/55d2a466/session/8985</a></li>
</ul>
<p>Last time Mercari ML Engineers visited FOSSASIA Summit 2023 in Singapore to share their experience of implementing Machine Learning Operations (MLOps) within Mercari, diving into details of the associated integration challenges as well as how they managed to develop solutions that could be efficiently scaled and modified. </p>
<p>This time they are back to present the inner workings of Mercari's ranking system, delving more into details of ML solutions that drive the marketplace growth:</p>
<h2>Background</h2>
<p>In today's competitive e-commerce landscape, effective search ranking systems are no longer a luxury, but a necessity. At Mercari, Japan’s largest C2C e-commerce marketplace, we are leveraging and continuously integrating AI into search to provide the best experience for our millions of users, driving engagement to optimize marketplace transactions and the joy of discovery. </p>
<p>This talk is full of insights from our ongoing journey building and refining Mercari’s search ranking system, geared towards data scientists, machine learning engineers, and anyone interested in developing effective unbiased real-time production AI solutions for e-commerce search ranking.</p>
<h2>Key points of presentation</h2>
<p>The presentation by Chingis and Teo at the FOSSASIA Summit 2024 is an in-depth look into the company’s machine learning ranking system. They will cover four main points needed to develop an efficient ML solution: dataset construction, model tracking and monitoring, model building, and continuous iteration. </p>
<h2>Summary of paper</h2>
<p>They will cover four main points needed to develop an efficient ML solution: dataset construction, model tracking and monitoring, model building, and continuous iteration. </p>
<ul>
<li>Dataset Construction. Problem formulation and understanding the problem at hand forms half of the solution. In the ever-evolving  e-commerce search scene, it’s important to constantly innovate, adapt and refine the training data used. Chingis and Teo will be demonstrating how we build a rich and diverse dataset incorporating user behavior, item attributes, and marketplace dynamics. </li>
<li>Model Tracking &amp; Monitoring. Constant innovation and iteration means recognizing the progress. Therefore, they will aso share how we approach tracking and monitoring our ML ranking models, including key custom metrics we’ve developed for robust evaluation.</li>
<li>Model Building. After formulating the problem and setting up a process to monitor the learning progression, they will delve into the technical details of our model building process, exploring what learning-to-rank is as well as our goals for optimal performance. </li>
<li>De-biasing through Historical Data. Finally, acknowledging existing problems is the first step towards continuous iteration and growth. Chingis and Teo will outline the potential problems inherent in implicit feedback and how we deal with potential biases in our ranking models through counterfactual learning. They will show how biased behavior affects the learning process and what factors can trigger specific user behaviors. </li>
</ul>
<h2>About the Search Quality Team</h2>
<p>The Search Quality Team is committed to enhancing the quality of search results. We focus on continuous improvement on search ranking and relevance to provide our users with a satisfying shopping and selling journey. </p>


Mercari ML Engineers Chingis Oinar and  Teo Narboneta Zosa will give a talk at FOSSASIA Summit2024

<h2>Overview</h2>
<p>We are pleased to announce that Mercari ML Engineers Lirong Zhang and Miao Cao will have a presentation at <a href="https://eventyay.com/e/55d2a466">FOSSASIA Summit 2024</a>, held from Monday 8th April to Wednesday 10 April.</p>
<p>FOSSASIA is an Asia-based organization that develops open source software applications and open hardware with the global community. The event aims to create connections between developers, vendors and users. With a strong emphasis on OSS, the topics will range from Hardware/AI/Cloud technologies/Databases/Mobile/IT Education and Legal Compliance...etc.</p>
<p>Presentations by Lirong Zhang and Miao Cao, engineers from the Mercari AI team, will take place as follows.</p>
<ul>
<li>Title : “Opportunities to Address Complex Challenges in E-Commerce Search Through LLMs”</li>
<li>The presentation Time : Tuesday, April 9, 2024, 14:45 - 15:10 GMT+7</li>
<li>Detail：<a href="https://eventyay.com/e/55d2a466/session/9015">https://eventyay.com/e/55d2a466/session/9015</a></li>
</ul>
<h2>Background</h2>
<p>E-commerce search mechanisms often grapple with multifaceted problems, such as the scarcity and subjectivity of human-annotated datasets and the absence of comprehensive synonym dictionaries, among others. In tackling these challenges, Mercari search team has strategically embraced Large Language Model (LLM)-based approaches across various application scenarios in the search domain.</p>
<h2>Summary of paper</h2>
<p>This talk aims to disseminate valuable insights from our ongoing exploration into LLM-based features within e-commerce search. It is tailored for data scientists, machine learning engineers, and anyone fascinated by the prospects of deploying effective LLM solutions in e-commerce search contexts. Join us as we delve into our journey, highlighting our strategies, successes, and the lessons learned along the way in harnessing the power of LLMs to resolve long-standing and intricate problems in e-commerce search.</p>
<h2>Key points of presentation</h2>
<p>In this presentation, we will share our recent endeavors to leverage LLMs to augment our search system capabilities significantly. This presentation outlines our innovative methods, including:</p>
<ul>
<li>
<p>Offline Evaluation Dataset Construction: We will demonstrate the creation of an offline evaluation system designed to significantly boost development productivity. This system allows for a more streamlined and efficient process of assessing the performance of our search algorithms, ensuring constant improvement and optimization.</p>
</li>
<li>
<p>Synonym Generation: Delving into the mechanisms through which LLMs can enhance our synonym dictionary, making it more comprehensive. This is crucial for understanding and correlating user queries with the vast array of products listed on our platform, particularly when dealing with varying linguistic expressions and regional vernaculars.</p>
</li>
<li>
<p>Image Quality Assessment: Illustrating the application of LLMs in evaluating and categorizing images for image quality model development. This approach aids in improving the quality of search results, providing a more satisfying user experience.</p>
</li>
</ul>
<h2>About Search Quality Team</h2>
<p>The Search Quality Team is committed to enhancing the quality of search results. We focus on continuous improvement on search ranking and relevance to provide our users with a satisfying shopping and selling journey. </p>


Mercari ML Engineers Lirong Zhang and Miao Cao will give a talk at FOSSASIA Summit2024

<h2>■Overview</h2>
<p>We are pleased to announce that &quot;Using Recommendations to balance demand and supply in two-sided marketplaces&quot;, an extended abstract authored by Data Science Team engineer Deddy Jobson, has been accepted as a poster to the Recommendation Ecosystems Workshop (EcoSys2024) of AAAI 2024 (The Conference on Information and Knowledge Management) , an international conference on artificial intelligence.</p>
<p>AAAI  is one of the most prestigious international conferences in the field of artificial intelligence technologies, held annually by researchers from around the world. This year's AAAI, in its 38th year, will be held in Vancouver, Canada, February 20-27, 2024.</p>
<p>The following schedule of poster presentation times is planned</p>
<ul>
<li>Monday, February 26, 2024, 3:00 p.m. - 4:00 p.m. (local time)</li>
</ul>
<h2>■Key points of presentation</h2>
<p>*You can read the entire paper at <a href="https://openreview.net/forum?id=ljtKZdRGu3">this link</a>.</p>
<ul>
<li>We propose the idea of using recommendation systems to balance demand and supply in two-sided marketplaces consisting of buyers and sellers.</li>
<li>We perform experiments on a public dataset to show how the recommendations made by any recommender system can be modified to improve buyer-side metrics or seller-side metrics depending on the business priorities.</li>
</ul>
<h2>■Background</h2>
<p>Current recommender systems are consumer-centric in that their objective is to improve the benefit to consumers. This helps with dealing with the excessive supply of content for consumers.</p>
<p>We introduce the idea of item-centric recommendations that help tackle the complementary problem that can arise in a two-sided marketplace: excess demand. We show using experiments on public datasets that recommendations can be used to prioritize seller experience or buyer experience depending on the business requirements.</p>
<h2>■Summary of paper</h2>
<ul>
<li>We use recommender systems to help balance demand and supply in two-sided marketplaces.</li>
<li>We do so by constructing a lever that one can use to shift the nature of the recommendations towards helping producers or consumers accordingly based on the business objective.</li>
<li>We believe our approach can extend the capabilities of recommender systems to improve the long-term welfare of all participants of the two-sided marketplace.</li>
</ul>
<h2>■About Data Science Team</h2>
<p>The Data Science Team makes use of statistics, machine learning, and mathematical optimization to improve the efficiency of Mercari’s marketing campaigns and core product. We actively learn and adopt the state of the art methods used in the literature to solve technical problems and publish papers along the way.</p>


Mercari DS Teams’s research “Using Recommendations to balance demand and supply in two-sided marketplaces” accepted to EcoSys Workshop at AAAI 2024

Improvement of the Selling Experience Key for Mercari: Six Months of the Item Coupon Project in Retrospect

<p>This post is for Day 19 of <a href="https://engineering.mercari.com/en/blog/entry/20231124-mercari-advent-calendar-2023/">Mercari Advent Calendar 2023</a>, brought to you by <a href="https://www.linkedin.com/in/andre-r-2a401875/">@andre</a> from the Mercari Generative AI/LLM team.</p>
<p>Remember when ChatGPT was first released to the public? It reshaped the boundaries of what was possible and elevated the discourse around artificial intelligence. Yet such innovations were not without their enigmas, presenting as much potential as they did new frontiers to explore.</p>
<p>We have come a long way since then. Earlier this month, for example, many researchers and practitioners shone a light on the capabilities and limitations of current Large Language Model (LLM) technologies at the <a href="https://2023.emnlp.org/">EMNLP 2023 conference</a>, in which Mercari was a sponsor.</p>
<p>In this article, we&#8217;re excited to share the strides our team at Mercari has made in utilizing LLMs to enhance our beloved application. We focus primarily on our initial work with <a href="https://about.mercari.com/en/press/news/articles/20231017_mercariaiassist/">Mercari AI Assist</a> (メルカリAIアシスト), a project at the intersection of innovation and practical application. </p>
<p>We hope that this article will serve as a resource that is not only informative but also offers tangible benefits to readers interested in the practical applications of LLMs.</p>
<p>Some key takeaways</p>
<ul>
<li>Clear and frequent communication across different roles is critical for aligning expectations and making quick progress.</li>
<li>Begin with simple prompt engineering and leverage commercial APIs.</li>
<li>Rigorous pre- and post-processing are required to address LLM output inconsistencies.</li>
<li>Closely following new updates, both from the academic and industrial field, helps us navigate through a rapidly changing field of large language models.</li>
</ul>
<h1>The Team</h1>
<p>The <a href="https://about.mercari.com/en/press/news/articles/20230501_generativeai/">Generative AI/LLM team</a> is on a mission to generate impactful business improvements by integrating LLM technologies into our products and enhance productivity. Generally, our efforts are twofold: building and enabling. Speed is a crucial aspect of our work—on one hand, we strive to improve the user experience for our customers by developing high-quality products; on the other, we also aim to quickly acquire knowledge and expertise to empower more teams to understand and implement LLMs in production environments.</p>
<p>We work in a relatively small team, as close and effective communication between PM, designer, and engineers is crucial to be able to work fast and ship our product. As LLM is a relatively new concept for many people, it is important to maintain a constant dialogue about what is achievable and what lies beyond its current scope.</p>
<p>Additionally, the engineers regularly conduct experiments to assess technical feasibility. With the field of LLMs evolving at a breakneck pace, it&#8217;s imperative to stay abreast of the latest findings and updates. Social media and news outlets are invaluable for acquiring the most immediate updates, while research papers offer a deeper dive, providing a comprehensive understanding and empirical observations of the latest advancements.</p>
<h1>The Product</h1>
<p>Mercari AI Assist is envisioned to be an assistant feature that can guide our customers to use our app effectively depending on their preferences.</p>
<p>There is still a lot of work to be done; however, in the initial version, our focus is on the sellers—Mercari customers who use the platform to list and sell items. Through this feature, we utilize LLMs to assist sellers by offering suggestions to enhance their listing information. Below are illustrations that depict what the Title Suggestion feature looks like within the application.</p>
<p><img src="https://storage.googleapis.com/prd-engineering-asset/2023/12/c7951a2d-aiassist_1.png" alt="" /></p>
<p><img src="https://storage.googleapis.com/prd-engineering-asset/2023/12/966bcc18-aiassist_2.png" alt="" /></p>
<p>You can read more about Mercari AI Assist in the <a href="https://about.mercari.com/en/press/news/articles/20231017_mercariaiassist/">press release article</a>. Meanwhile, this article will focus more on sharing about the technical side of how we use LLMs to bring the two types of suggestions into production.</p>
<h1>Choosing the Right Models and Techniques for Our Case</h1>
<p>Firstly, it&#8217;s important to emphasize that while this article focuses on the use of LLMs, not everything requires the use of LLMs. Some tasks may be more effectively addressed without them, depending on factors such as cost, objectives, and the development team&#8217;s expertise. Knowing when and how to deploy LLMs is crucial.</p>
<p>One of the most challenging tasks in our case is to process and to understand unstructured data from user generated texts. Inside a listing in Mercari, the item’s title and description contain lots of useful information, however, distilling key information and determining how to utilize it has always been difficult. For example, identifying which category had the most listings in the past month might be straightforward, but discerning which factors differentiate listings that sell quickly from those that do not is complex. This is especially true given the varied and unique styles people use to write an item&#8217;s title or description. We believed that, given the breadth of data with which a large language model has been pre-trained, it would be adept at meeting such challenges.</p>
<p>Once we identify tasks that LLMs can address, there are several other things we need to decide. Two of the most commonly considered factors are:</p>
<ol>
<li>Which models to use; e.g. commercially available models or open source models</li>
<li>Fine-tuning or prompt engineering (or training our own LLMs)</li>
</ol>
<p>In general, fine-tuning often yields better results for specialized tasks within a fixed model size, as it allows the entire network to specialize in solving a specific problem. Conversely, prompting or in-context learning (ICL) can be seen as a method to enable a general LLM to perform specialized downstream tasks.</p>
<p>In the case of Mercari AI Assist, we utilized prompt engineering and simple retrieval-augmented generation to enable the use of commercially available LLMs—specifically, OpenAI&#8217;s GPT-4 and GPT-3.5-turbo—for executing a variety of specific tasks. Our objective at the moment is to design an optimal user experience and establish a sustainable and effective workflow for incorporating LLMs into our production environment.</p>
<p>The figure below illustrates the streamlined design of how we implement the Title Suggestion feature within Mercari AI Assist. After experimenting with several methods of leveraging LLMs and taking both cost and performance into account, we determined that this approach best fits our requirements. Generally, the feature is split into two main parts. The first part, highlighted in blue, involves defining “what makes a good title” for a Mercari listing. This is accomplished with assistance from other teams that possess diverse domain expertise. We then collect existing title data aligned with our criteria and utilize GPT-4 to distill the key attributes of an effective title. These key attributes are subsequently stored in a database. The second part of the process, indicated in red, occurs in real-time. We employ GPT-3.5-turbo to identify key attributes (defined by the previous step) from a specific listing as it is created, and then we generate suggestions for refining the listing&#8217;s title as necessary.</p>
<p>Through our experiments, we observed that GPT-4 outperforms GPT-3.5-turbo in terms of quality, but it incurs greater costs and latency. Consequently, we found an optimal balance between quality and cost-efficiency by utilizing GPT-4 exclusively for the initial, offline extraction of key attributes, and employing GPT-3.5-turbo for real-time, online operations.</p>
<p><img src="https://storage.googleapis.com/prd-engineering-asset/2023/12/219563d7-title-suggestion.png" alt="" /></p>
<h1>Continuous Evaluation and Mitigating Unexpected Responses</h1>
<p>We primarily conduct two types of evaluations to ensure that the quality of outputs returned by the models meets our expectations: offline and online evaluations. Both are carried out before the product&#8217;s release and continue thereafter to guarantee that our quality standards are upheld.</p>
<p>Offline evaluation serves several purposes, but it mostly helps us to determine the most effective prompt for the task at hand. We focus on two main aspects: token usage (length) and response quality. Striking the right balance between these two aspects is crucial. Through a combination of manual review and automated evaluation, we ensure that the model&#8217;s responses meet our requirements. This step also allows us to estimate the total cost of deploying the feature to all of our users.</p>
<p>Online evaluation, on the other hand, ensures that the feature performs as expected in a live environment—this is particularly significant because we are dealing with user-generated content and substantial traffic in real-time. We conducted a partial release, only implementing a small segment of Mercari AI Assist that calls the LLM API, to assess performance and confirm that the complete feature is ready for our customer base. In this preliminary online test period, we tasked GPT with extracting a single key attribute from an item&#8217;s description and to respond simply with “YES” if the attribute is present, or “NO” if it is not.</p>
<p>We found that it is very useful for teams who are not familiar with using LLM in production to perform these kinds of partial preliminary releases, especially when using commercially available APIs provided by third-party services.</p>
<p>During the preliminary online test period, we observed that even though we instructed GPT to provide outputs in a straightforward format (YES or NO), the number of inconsistently formatted responses increased along with the number of requests. The table below presents a sampled result from this experiment.</p>
<table>
<thead>
<tr>
<th>LLM Output</th>
<th>Count</th>
</tr>
</thead>
<tbody>
<tr>
<td>NO</td>
<td>311,813</td>
</tr>
<tr>
<td>No</td>
<td>22,948</td>
</tr>
<tr>
<td>Yes</td>
<td>17,236</td>
</tr>
<tr>
<td>&#8230;</td>
<td>&#8230;</td>
</tr>
<tr>
<td><span style="color:red">Sorry, but I can&#8217;t provide the answer you&#8217;re looking for. </span></td>
<td>5</td>
</tr>
<tr>
<td><span style="color:red">Sorry, but I can&#8217;t assist with that request.</span></td>
<td>4</td>
</tr>
<tr>
<td><span style="color:red">The provided text does not contain the information.</span></td>
<td>4</td>
</tr>
<tr>
<td><span style="color:red">NO<br /> NO<br /> YES </span></td>
<td>1</td>
</tr>
<tr>
<td><span style="color:red">NO<br />YES<br />NO<br />NO<br />YES<br />NO<br />NO<br />NO<br />NO</span></td>
<td>1</td>
</tr>
</tbody>
</table>
<p>Being aware of such inconsistencies is crucial for production systems. In the above sampled use case, the wrong format might be non-critical and relatively easy to solve (e.g. with regular expressions). However, as we require more complex outputs from LLMs, detecting inconsistencies—as well as hallucinations, a well-known issue with large language models—becomes increasingly challenging.</p>
<p>It&#8217;s essential to preprocess prompts that contain user-generated content to minimize the likelihood of GPT generating incorrect responses. Additionally, post-processing logic should be implemented to ensure that only the expected output format is relayed to the client application.</p>
<h1>Additional Things to Keep in Mind</h1>
<p>Since we&#8217;re utilizing an LLM provided by a third-party service, it&#8217;s critical to understand how the API functions and what sorts of errors may occur. In addition to common API error types such as authentication and timeout errors, which we might already know how to handle, we need to give special attention to errors more closely related to LLMs. For instance, depending on the API you use, some calls might inadvertently trigger a content violation error. At Mercari, we have our own content moderation system; however, the filtering policy of a third-party API might differ. It is important to be aware of this to accordingly prepare our prompts and to avoid undesired outcomes.</p>
<p>Another consideration is the token count. The number of tokens used can vary depending on the language sent to the model. For instance, <a href="https://aclanthology.org/2023.emnlp-main.614.pdf">an experiment</a> presented at EMNLP 2023 indicated that, using ChatGPT, the average cost of prompt and generated tokens in Japanese can exceed that of English by more than double. This certainly depends on the task at hand and sometimes there&#8217;s no alternative, but it is one thing to keep in mind.</p>
<p>Lastly, in this rapidly evolving field, what is considered the best tool can change in just a short span of time. Libraries are updated constantly—with the occasional breaking change—and many of us are constantly looking for ways to optimally integrate LLMs into production systems. This might sound obvious, but we argue that it is important to closely follow new updates regarding LLM research and best practices.</p>
<h1>Looking Back, Going Forward</h1>
<p>The design and development of Mercari AI Assist has offered us valuable perspectives on working with prompt engineering and integrating commercially available Large Language Models (LLMs) into production. Looking back, I felt that I gained substantial knowledge and experience from the practical aspects of working with LLMs and I am enthusiastic about further advancing my skills alongside the team.</p>
<p>Among the key lessons learned are the significance of cultivating a team equipped with the right mindset and fostering effective communication. I have also experienced and learned about the intricacies of choosing the right model and techniques, finding the right balance between cost and performance, dealing with LLM’s stability when used in a live environment, and addressing unique challenges of LLM, such as hallucination and content moderation. Additionally, I believe it is advisable to have team members with a background in machine learning and natural language processing when working with LLMs. Having the appropriate expertise can speed up various research and experimental processes. For instance, it can enable the team to swiftly determine the suitability of LLMs for a specific task and also decide the most suitable evaluation metrics.</p>
<p>Going forward, we are focusing on improvements such as LLM operations and the development of automated workflows. We are also exploring the use of LLMs for more complex and specialized tasks, which may require the adoption of parameter-efficient fine-tuning techniques. With a rapidly growing field, our team is continuously experimenting and learning, and we understand that our implementation is far from perfect. As with many other practitioners in the field, we are constantly following updates from the field, sharing, listening, and looking for best practices most suitable for our use cases.</p>
<p>I look forward to yet another exciting year filled with obstacles and successes, and to share in these experiences with the incredible members at Mercari.</p>
<p>Tomorrow&#8217;s article will be by @ayaneko. Look forward to it!</p>


Leveraging LLMs in Production: Looking Back, Going Forward

<p>Mercari announced <a href="https://about.mercari.com/press/news/articles/20230501_generativeai/">the creation of a new dedicated generative AI/LLM</a> in May 2023. Yuki Ishikawa (<a href="(https://twitter.com/maze_rapid)">@maze</a>), who has led the Mercari Shops business as Representative Director and CEO of Souzoh, launched the team and leads it. Up until now, Mercari has used AI in a wide range of areas, including marketing and customer support, as well as UI/UX. Ishikawa proposed the launch of the Generative AI/LLM Team at a managerial meeting, believing it would become one of the pillars of Mercari's AI utilization. We asked Ishikawa about his thoughts on this technology, what he has seen in the past three months of work, and the team's longer-term prospects.</p>
<h3>Profile</h3>
<p><strong>Yuki Ishikawa | Mercari VP of Generative AI / LLM<br />
</strong><br />
After graduating from the University of Tokyo, was hired by Nintendo Co., Ltd in 2012. Joined Moi Corp (TwitCasting) in 2014, where he was involved in various development and product launches. Hired by the former company Souzoh, Inc. of the Mercari Group in June 2017. After moving to Mercari, Inc., became VP of Product at Merpay, Inc. in July 2020. Appointed Representative Director and CEO of Souzoh, Inc. in January 2021. Received concurrent appointment as VP at Mercari, Inc. in July 2022.</p>
<h2>Ishikawa felt that &quot;this is an area Mercari definitely has to move into.&quot;</h2>
<p><img src="https://storage.googleapis.com/prd-ai-asset/2023/10/4a8052b6-maze_interview3-1024x683.jpg" alt="" /></p>
<p><center><br />
Yuki Ishikawa | Mercari VP of Generative AI / LLM<br />
</center></p>
<p><strong>ーーMercari's dedicated Generative AI/LLM Team was created in May 2023. Tell us about the background and purpose behind it.</strong></p>
<p><strong>Ishikawa：</strong>Although we officially launched the dedicated Generative AI/LLM Team in May, we had been working on the concept since February 2023, and I had personally been paying attention to it following the emergence of <a href="https://stablediffusionweb.com/">Stable Diffusion</a> in the summer of 2022 and <a href="https://chat.openai.com/auth/login">ChatGPT</a> at the end of 2022. I toyed with various ideas and came up with some that we could use in Mercari's business and internal operations, so I made a proposal at a managerial meeting. I said, &quot;This is an area Mercari definitely has to move into.&quot; That was what led to the team's creation. In the spring of this year, there was a lot of uncertainty about generative AI, so we started by creating a small, dedicated team that could act quickly and flexibly.</p>
<p>The Generative AI/LLM Team's mission is twofold: 1) maximize business impact by creating new customer experiences using generative AI technology, and 2) dramatically improve company-wide productivity.</p>
<p><img src="https://storage.googleapis.com/prd-ai-asset/2023/10/36b404ab-maze_interview1-1024x570.png" alt="" /></p>
<p><center><br />
The  presentation at MS Build Japan 2023<br />
</center></p>
<p>Initially, we had full-stack engineers who had been involved in ML projects join us. Currently, the team consists of engineers, designers, and BizOps members. We also continue to hire engineers.</p>
<h2>Introduction of GitHub Copilot to improve the development experience for engineers</h2>
<p><strong>ーーA few months have passed since the AI/LLM team was created. What are some specific things you've done?<br />
</strong></p>
<p><strong>Ishikawa：</strong>First, in connection with our mission to dramatically improve company-wide productivity, we introduced <a href="https://github.com/features/copilot">GitHub Copilot</a> for our engineers' development experience. It's a programming support tool that uses AI to suggest the optimal code. It generates code from comments and completes code in progress to improve the development experience. GitHub Copilot makes development more productive. After introducing it, some engineers, according to official statements, finish tasks twice as fast. In the future, I think we'll see a big difference between companies that use GitHub Copilot and those that don't, and we have a lot of engineers, so that's why we started using it. The Generative AI/LLM Team spearheaded the project, the Corporate Engineer Team took the lead, and in June, the tool was made available to engineers who want to use it.</p>
<p>We have identified three risks of concern when using GitHub Copilot. The first is legal risk, including copyright infringement and violation of open-source licenses. The second is reputational risk linked to legal risk. And the last is the risk of data breaches containing internal company code through GitHub Copilot.</p>
<p>After making various verifications, we have concluded that by setting certain restrictions, there are no problems with using GitHub Copilot, and now any engineer at Mercari can use it. Although we haven't yet completed full verification since its introduction, we have received feedback from senior engineers that &quot;it has made things a lot easier,&quot; and from those with relatively short engineering experience that &quot;it's like a textbook I can use when I'm having trouble.&quot; Even those who aren't purely programmers can write reasonably clean code, so I feel that the tool is beneficial to a wide range of members involved in development.</p>
<p><img src="https://storage.googleapis.com/prd-ai-asset/2023/10/bf386527-maze_interview5-1024x683.jpg" alt="" /></p>
<h2>A new experience for customers allowing them to search for items with natural language</h2>
<p><strong>ーー<a href="https://about.mercari.com/press/news/articles/20230724_chatgpt/">A ChatGPT plugin</a> was announced in July 2023. </strong></p>
<p><strong>Ishikawa：</strong>E-commerce sites, including Mercari, generally focus on keyword searches when customers have some idea of what they want and displaying recommended items based on search history.</p>
<p>However, with the introduction of this plug-in, customers can now search for items by interacting with ChatGPT. This functionality is different from previous customer experiences because it allows users to search for items through natural conversation, even if you're not thinking of a specific product. For example, you can say, &quot;Tell me what I need to go camping for the first time,&quot; or &quot;What gift would make a five-year-old girl happy?&quot; Depending on the results of our upcoming verifications, we are considering implementing this feature in the Mercari app, too.</p>
<p>After we introduced the plug-in, my honest impression is that customers are using it more than I originally expected. Currently, the ChatGPT plug-in is only available to users of the paid version, ChatGPT Plus. These are mostly men in their 30s and 40s, and they often use it to discuss gifts for their daughters or partners, or to narrow down itens for hobbies related to electronics and cameras. It's certainly very useful to be able to use natural language to narrow down items in terms of complex features in gadgets and home appliances. ChatGPT's suggestions also contribute to the influx of users to Mercari. We'll continue to track the data.</p>
<p><img src="https://storage.googleapis.com/prd-ai-asset/2023/10/2a5fa68e-maze_interview2-1024x683.jpg" alt="" /></p>
<h2>Utilizing generative AI creatives for hiring and marketing</h2>
<p><strong>ーーOn X (formerly Twitter), you have uploaded hiring creatives utilizing generative AI. Looking ahead, how do you see its use for creatives?<br />
</strong><br />
<img src="https://storage.googleapis.com/prd-ai-asset/2023/10/cb5c6c13-images_maze.png" alt="" /></p>
<p><center><br />
Creatives to recruit engineers<br />
</center></p>
<p><strong>Ishikawa：</strong>First, as a test, I generated creatives to hire Generative AI/LLM Team members and posted them on my X account for a few days. These images are creatives that generative AI made, with text added by a designer to match the background. Right now, people are involved a bit, but I think that in the future, we'll be able to generate whatever creatives we want just with text instructions.</p>
<p>Looking ahead, we'll consider using generative AI for campaigns and ad creatives.</p>
<p>Specifically, we're working on generating ad creatives based on data from previously well-received Mercari advertisements to see if we can use them in digital and OOH (out-of-home) ads. Currently (as of October 3, 2023), we're running a Halloween commercial for Mercari outside around Shibuya Station. The creatives for this promotional video were actually produced by generative AI.</p>
<p><center><br />
<iframe loading="lazy" width="560" height="315" src="https://www.youtube.com/embed/IWgwWYrhaMs?si=IvUJHo_QU7DOznVL" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe><br />
Mercari commercial airing at the Shibuya scramble crossing.<br />
</center></p>
<p>We are also working on possible ways to use this technology to generate trading card creatives. Although we can't use images of actual cards in advertisements due to copyrights, we're considering the possibility of using AI to generate creatives that look like trading cards and that we can use in advertisements and elsewhere. We're also working on the legal aspects, and we hope to release this soon.</p>
<h2>This next half year will be a phase with many challenges and a mix of successes and failures</h2>
<p><strong>ーーWhat else does the Generative AI/LLM Team want to try and do?<br />
</strong></p>
<p><strong>Ishikawa：</strong>During the next six months, we will continue to experiment in all areas to see what kind of experiences we can create with generative AI and what impact we can achieve.</p>
<p>During these six months, it will be important to take on as many challenges as we can in as broad a range as possible, with the idea that it's OK to fail half the time. As a result, we'll probably see where generative AI/LLM will succeed and where it won't. I think we need to plan some action and ascertain if and how we need an evaluation layer, a platform layer, and an ML Ops-like organization.</p>
<p>While the introduction of the technology into our services and its business impact are of course important indicators, we also consider it a very important mission to promote the understanding of the technology within Mercari. A year from now, generative AI/LLM should be a more core technology in Mercari's development experience. We are aiming for a state in which generative AI/LLM is not only in the UI/UX (for example, to streamline transactions), but also in back-end algorithms.</p>
<p><strong>ーーWhat do you think the AI/LLM team's composition will look like moving forward?</strong></p>
<p><strong>Ishikawa：</strong>While we will continue to promote agile technology exploration and product development with a dedicated team, and in the longer term, we hope to advance our work so that every member of Mercari can make use of generative AI/LLM.</p>
<p>In the future, we are also considering whether to develop our own generative AI model inside Mercari. Beyond the company, we want to also look at widespread possibilities for partnerships outside the company.</p>
<p><img src="https://storage.googleapis.com/prd-ai-asset/2023/10/5596b7e9-maze_interview6-1024x683.jpg" alt="" /></p>
<h2>Mercari's appeal is that we can enable direct feedback while utilizing a wealth of data</h2>
<p><strong>ーーWhat is Mercari's appeal as a place to explore generative AI/LLM possibilities?</strong></p>
<p><strong>Ishikawa：</strong>There are two appealing reasons to work on AI/LLM at Mercari. First, our company has a wealth of data. AI products are dependent on the amount of data they can use. On that point, we can handle a large amount of data we've accumulated at Mercari. The second reason is that we get a lot of direct feedback as a result of the distribution of the systems we've created. I feel that these two factors are strong reasons for us to develop AI-powered services. </p>
<p>There are many things we can do with generative AI in areas related to our customers. I think we will see major changes in the way we work with regard to improving productivity at the company. Mercari currently has over 2,000 employees, all of whom are highly IT literate, and we are in an environment where we can experiment with a variety of approaches. At Mercari, I believe we can study business design and organizational design using generative AI/LLM, so I want to move fast on PoC.</p>
<p>In fact, the theme that we will dramatically improve companywide productivity was added to the mission at my request. The environment within the company is conducive to technological trial and error, making it easy to take on new challenges. I also believe that changes to our employees' workplace environment will make us more familiar with generative AI/LLM and further the technology's spread.</p>
<p><img src="https://storage.googleapis.com/prd-ai-asset/2023/10/d23a319e-maze_interview4-1024x683.jpg" alt="" /></p>
<h2>The Generative AI/LLM Team is recruiting new members to work with us!</h2>
<p>The Generative AI/LLM Team is recruiting new members to work with us! We hope you’ll find it interesting and a good challenge to work in Mercari’s Generative AI/LLM Team. Click here for more details.<br />
<a href="https://apply.workable.com/mercari/j/183899D2BA/">Software Engineer, Full Stack (LLM/GenAI) - Mercari</a> </p>
<ul>
<li>Photo : Shun Nakayama (<a href="https://shunnakayama.myportfolio.com/">Web</a> / <a href="https://twitter.com/shunnakayama_">X</a>)</li>
<li>Text : Ayumi Iga</li>
</ul>


How does Mercari leverage generative AI and large language models? The stated mission: “create new customer experiences and business impact” and “dramatically improve companywide productivity.”

<h2>Overview</h2>
<p>We are pleased to announce that the paper &quot;Layout Optimizer for Personalized Home Screen based on Contextual Multi-Armed Bandit in C2C Marketplace&quot; has been accepted to the <a href="https://sites.google.com/view/consequences2023/home">CONSEQUENCES '23 workshop</a> within the international conference &quot;<a href="https://recsys.acm.org/recsys23/">RecSys 2023(ACM Conference on Recommender Systems)</a>&quot;.</p>
<p>RecSys is one of the most prestigious international conferences in the field of recommendation systems, held annually by researchers from around the world. This year's RecSys, in its 17th year, was held in Singapore from September 18 to 22, 2023.</p>
<h2>Key points of presentation</h2>
<p>In this presentation, based on how customers use Mercari, we propose a method to display more attractive content on the home screen with priority among other content, including product recommendations. We reported the results of our experiments.</p>
<p>In the proposed method, an exploratory screen display and machine learning are performed to collect data, without giving prior knowledge of what kind of content is attractive to which users. As a result, the screen display is automatically optimized to improve overall satisfaction.</p>
<h2>Background</h2>
<p>At Mercari app, several teams, including the Recommendation Team, are developing content on the home screen.There are various types of content on the home screen, including item recommendations and campaign announcements. Furthermore, even if the same item is being recommended, there are multiple methods of recommendation and display. </p>
<p>Not only do the items to be recommended vary from user to user, but also the type of recommendation method and how effective the campaign announcement is depends on each user.</p>
<h2>Summary of paper</h2>
<p>In e-commerce services, including Mercari, the home screen is an important function that attracts many views. However, the amount of content that can be displayed is limited, and it is common practice to place attractive content at the top of the home screen as a priority.</p>
<p>However, since different users have different perspectives on what kind of content is attractive, it is necessary to create a personalized screen structure.</p>
<p>In this study, we proposed a method that applies the contextual multi-armed bandit problem algorithm and reported the details and results of our experiments.</p>
<p><img src="https://storage.googleapis.com/prd-ai-asset/2023/09/6d29996b-recsys.png" alt="" /></p>
<h2>Comments from author</h2>
<p><strong>ML Engineer / Backend Engineer | Yusuke Shido<br />
</strong><br />
Since this year's event was held offline, and in the form of a poster presentation, we were able to take the time to have direct discussions with researchers and engineers from various countries. In particular, I received a lot of interest and questions from developers and researchers belonging to companies, and I felt that the content of our paper was an issue common to many different companies. The five days at RecSys, including the workshop, were an opportunity for very meaningful input. We hope to apply the knowledge gained to Mercari's business in the future and continue to disseminate the results.</p>
<p><img src="https://storage.googleapis.com/prd-ai-asset/2023/09/cdf923d7-recsys23_1-1024x552.jpg" alt="" /></p>
<p><img src="https://storage.googleapis.com/prd-ai-asset/2023/09/046bfffa-recsys23_3-1024x576.jpg" alt="" /></p>
<p><center><br />
Members making a poster presentation in Singapore<br />
</center></p>
<h2>About recommendation team</h2>
<p>Mercari's recommendation team is working to improve the quality of product recommendations on each screen of the Mercari App, as well as the mechanisms involved in screen configurations such as the one in this study.</p>


Mercari AI’s research “Layout Optimizer for Personalized Home Screen based on Contextual Multi-Armed Bandit in C2C Marketplace” accepted to RecSys 2023’s Workshop

At Mercari, we have been using AI to improve our UI/UX and also to provide ways for our service to be used safely and securely. We now aim to further advance our use of AI to realize a circular economy.

AI to enable many different purchases<br />across the world to be done simply, quickly, and safely

The specialized team driving our AI technology

We are the engineering organization developing AI at Mercari. Team members from across the world work together to solve problems.

Utilize AI to change
billions of lives.

Latest Updates

How have the organizational challenges of mercari ML team been solved? A chat with the engineers wh…

LLM-based Approach to Large-scale Item Category Classification

Mercari AI team presented in NLP2024 “A related document retrieval system using past queries”

Mercari ML Engineers Chingis Oinar and Teo Narboneta Zosa will give a talk at FOSSASIA Summit2024

Mercari ML Engineers Lirong Zhang and Miao Cao will give a talk at FOSSASIA Summit2024

Mercari DS Teams’s research “Using Recommendations to balance demand and supply in two-sided marke…

Improvement of the Selling Experience Key for Mercari: Six Months of the Item Coupon Project in Retr…

Leveraging LLMs in Production: Looking Back, Going Forward

How does Mercari leverage generative AI and large language models? The stated mission: “create new …

Mercari AI’s research “Layout Optimizer for Personalized Home Screen based on Contextual Multi-Arm…

About

People

Utilize AI to changebillions of lives.

Latest Updates

How have the organizational challenges of mercari ML team been solved? A chat with the engineers wh…

LLM-based Approach to Large-scale Item Category Classification

Mercari AI team presented in NLP2024 “A related document retrieval system using past queries”

Mercari ML Engineers Chingis Oinar and Teo Narboneta Zosa will give a talk at FOSSASIA Summit2024

Mercari ML Engineers Lirong Zhang and Miao Cao will give a talk at FOSSASIA Summit2024

Mercari DS Teams’s research “Using Recommendations to balance demand and supply in two-sided marke…

Improvement of the Selling Experience Key for Mercari: Six Months of the Item Coupon Project in Retr…

Leveraging LLMs in Production: Looking Back, Going Forward

How does Mercari leverage generative AI and large language models? The stated mission: “create new …

Mercari AI’s research “Layout Optimizer for Personalized Home Screen based on Contextual Multi-Arm…

About

People

Utilize AI to change
billions of lives.