Blog

<h2>■Overview</h2>
<p>We are pleased to announce that &quot;Using Recommendations to balance demand and supply in two-sided marketplaces&quot;, an extended abstract authored by Data Science Team engineer Deddy Jobson, has been accepted as a poster to the Recommendation Ecosystems Workshop (EcoSys2024) of AAAI 2024 (The Conference on Information and Knowledge Management) , an international conference on artificial intelligence.</p>
<p>AAAI  is one of the most prestigious international conferences in the field of artificial intelligence technologies, held annually by researchers from around the world. This year's AAAI, in its 38th year, will be held in Vancouver, Canada, February 20-27, 2024.</p>
<p>The following schedule of poster presentation times is planned</p>
<ul>
<li>Monday, February 26, 2024, 3:00 p.m. - 4:00 p.m. (local time)</li>
</ul>
<h2>■Key points of presentation</h2>
<p>*You can read the entire paper at <a href="https://openreview.net/forum?id=ljtKZdRGu3">this link</a>.</p>
<ul>
<li>We propose the idea of using recommendation systems to balance demand and supply in two-sided marketplaces consisting of buyers and sellers.</li>
<li>We perform experiments on a public dataset to show how the recommendations made by any recommender system can be modified to improve buyer-side metrics or seller-side metrics depending on the business priorities.</li>
</ul>
<h2>■Background</h2>
<p>Current recommender systems are consumer-centric in that their objective is to improve the benefit to consumers. This helps with dealing with the excessive supply of content for consumers.</p>
<p>We introduce the idea of item-centric recommendations that help tackle the complementary problem that can arise in a two-sided marketplace: excess demand. We show using experiments on public datasets that recommendations can be used to prioritize seller experience or buyer experience depending on the business requirements.</p>
<h2>■Summary of paper</h2>
<ul>
<li>We use recommender systems to help balance demand and supply in two-sided marketplaces.</li>
<li>We do so by constructing a lever that one can use to shift the nature of the recommendations towards helping producers or consumers accordingly based on the business objective.</li>
<li>We believe our approach can extend the capabilities of recommender systems to improve the long-term welfare of all participants of the two-sided marketplace.</li>
</ul>
<h2>■About Data Science Team</h2>
<p>The Data Science Team makes use of statistics, machine learning, and mathematical optimization to improve the efficiency of Mercari’s marketing campaigns and core product. We actively learn and adopt the state of the art methods used in the literature to solve technical problems and publish papers along the way.</p>


Mercari DS Teams’s research “Using Recommendations to balance demand and supply in two-sided marketplaces” accepted to EcoSys Workshop at AAAI 2024

Interview

Improvement of the Selling Experience Key for Mercari: Six Months of the Item Coupon Project in Retrospect

<p>This post is for Day 19 of <a href="https://engineering.mercari.com/en/blog/entry/20231124-mercari-advent-calendar-2023/">Mercari Advent Calendar 2023</a>, brought to you by <a href="https://www.linkedin.com/in/andre-r-2a401875/">@andre</a> from the Mercari Generative AI/LLM team.</p>
<p>Remember when ChatGPT was first released to the public? It reshaped the boundaries of what was possible and elevated the discourse around artificial intelligence. Yet such innovations were not without their enigmas, presenting as much potential as they did new frontiers to explore.</p>
<p>We have come a long way since then. Earlier this month, for example, many researchers and practitioners shone a light on the capabilities and limitations of current Large Language Model (LLM) technologies at the <a href="https://2023.emnlp.org/">EMNLP 2023 conference</a>, in which Mercari was a sponsor.</p>
<p>In this article, we&#8217;re excited to share the strides our team at Mercari has made in utilizing LLMs to enhance our beloved application. We focus primarily on our initial work with <a href="https://about.mercari.com/en/press/news/articles/20231017_mercariaiassist/">Mercari AI Assist</a> (メルカリAIアシスト), a project at the intersection of innovation and practical application. </p>
<p>We hope that this article will serve as a resource that is not only informative but also offers tangible benefits to readers interested in the practical applications of LLMs.</p>
<p>Some key takeaways</p>
<ul>
<li>Clear and frequent communication across different roles is critical for aligning expectations and making quick progress.</li>
<li>Begin with simple prompt engineering and leverage commercial APIs.</li>
<li>Rigorous pre- and post-processing are required to address LLM output inconsistencies.</li>
<li>Closely following new updates, both from the academic and industrial field, helps us navigate through a rapidly changing field of large language models.</li>
</ul>
<h1>The Team</h1>
<p>The <a href="https://about.mercari.com/en/press/news/articles/20230501_generativeai/">Generative AI/LLM team</a> is on a mission to generate impactful business improvements by integrating LLM technologies into our products and enhance productivity. Generally, our efforts are twofold: building and enabling. Speed is a crucial aspect of our work—on one hand, we strive to improve the user experience for our customers by developing high-quality products; on the other, we also aim to quickly acquire knowledge and expertise to empower more teams to understand and implement LLMs in production environments.</p>
<p>We work in a relatively small team, as close and effective communication between PM, designer, and engineers is crucial to be able to work fast and ship our product. As LLM is a relatively new concept for many people, it is important to maintain a constant dialogue about what is achievable and what lies beyond its current scope.</p>
<p>Additionally, the engineers regularly conduct experiments to assess technical feasibility. With the field of LLMs evolving at a breakneck pace, it&#8217;s imperative to stay abreast of the latest findings and updates. Social media and news outlets are invaluable for acquiring the most immediate updates, while research papers offer a deeper dive, providing a comprehensive understanding and empirical observations of the latest advancements.</p>
<h1>The Product</h1>
<p>Mercari AI Assist is envisioned to be an assistant feature that can guide our customers to use our app effectively depending on their preferences.</p>
<p>There is still a lot of work to be done; however, in the initial version, our focus is on the sellers—Mercari customers who use the platform to list and sell items. Through this feature, we utilize LLMs to assist sellers by offering suggestions to enhance their listing information. Below are illustrations that depict what the Title Suggestion feature looks like within the application.</p>
<p><img src="https://storage.googleapis.com/prd-engineering-asset/2023/12/c7951a2d-aiassist_1.png" alt="" /></p>
<p><img src="https://storage.googleapis.com/prd-engineering-asset/2023/12/966bcc18-aiassist_2.png" alt="" /></p>
<p>You can read more about Mercari AI Assist in the <a href="https://about.mercari.com/en/press/news/articles/20231017_mercariaiassist/">press release article</a>. Meanwhile, this article will focus more on sharing about the technical side of how we use LLMs to bring the two types of suggestions into production.</p>
<h1>Choosing the Right Models and Techniques for Our Case</h1>
<p>Firstly, it&#8217;s important to emphasize that while this article focuses on the use of LLMs, not everything requires the use of LLMs. Some tasks may be more effectively addressed without them, depending on factors such as cost, objectives, and the development team&#8217;s expertise. Knowing when and how to deploy LLMs is crucial.</p>
<p>One of the most challenging tasks in our case is to process and to understand unstructured data from user generated texts. Inside a listing in Mercari, the item’s title and description contain lots of useful information, however, distilling key information and determining how to utilize it has always been difficult. For example, identifying which category had the most listings in the past month might be straightforward, but discerning which factors differentiate listings that sell quickly from those that do not is complex. This is especially true given the varied and unique styles people use to write an item&#8217;s title or description. We believed that, given the breadth of data with which a large language model has been pre-trained, it would be adept at meeting such challenges.</p>
<p>Once we identify tasks that LLMs can address, there are several other things we need to decide. Two of the most commonly considered factors are:</p>
<ol>
<li>Which models to use; e.g. commercially available models or open source models</li>
<li>Fine-tuning or prompt engineering (or training our own LLMs)</li>
</ol>
<p>In general, fine-tuning often yields better results for specialized tasks within a fixed model size, as it allows the entire network to specialize in solving a specific problem. Conversely, prompting or in-context learning (ICL) can be seen as a method to enable a general LLM to perform specialized downstream tasks.</p>
<p>In the case of Mercari AI Assist, we utilized prompt engineering and simple retrieval-augmented generation to enable the use of commercially available LLMs—specifically, OpenAI&#8217;s GPT-4 and GPT-3.5-turbo—for executing a variety of specific tasks. Our objective at the moment is to design an optimal user experience and establish a sustainable and effective workflow for incorporating LLMs into our production environment.</p>
<p>The figure below illustrates the streamlined design of how we implement the Title Suggestion feature within Mercari AI Assist. After experimenting with several methods of leveraging LLMs and taking both cost and performance into account, we determined that this approach best fits our requirements. Generally, the feature is split into two main parts. The first part, highlighted in blue, involves defining “what makes a good title” for a Mercari listing. This is accomplished with assistance from other teams that possess diverse domain expertise. We then collect existing title data aligned with our criteria and utilize GPT-4 to distill the key attributes of an effective title. These key attributes are subsequently stored in a database. The second part of the process, indicated in red, occurs in real-time. We employ GPT-3.5-turbo to identify key attributes (defined by the previous step) from a specific listing as it is created, and then we generate suggestions for refining the listing&#8217;s title as necessary.</p>
<p>Through our experiments, we observed that GPT-4 outperforms GPT-3.5-turbo in terms of quality, but it incurs greater costs and latency. Consequently, we found an optimal balance between quality and cost-efficiency by utilizing GPT-4 exclusively for the initial, offline extraction of key attributes, and employing GPT-3.5-turbo for real-time, online operations.</p>
<p><img src="https://storage.googleapis.com/prd-engineering-asset/2023/12/219563d7-title-suggestion.png" alt="" /></p>
<h1>Continuous Evaluation and Mitigating Unexpected Responses</h1>
<p>We primarily conduct two types of evaluations to ensure that the quality of outputs returned by the models meets our expectations: offline and online evaluations. Both are carried out before the product&#8217;s release and continue thereafter to guarantee that our quality standards are upheld.</p>
<p>Offline evaluation serves several purposes, but it mostly helps us to determine the most effective prompt for the task at hand. We focus on two main aspects: token usage (length) and response quality. Striking the right balance between these two aspects is crucial. Through a combination of manual review and automated evaluation, we ensure that the model&#8217;s responses meet our requirements. This step also allows us to estimate the total cost of deploying the feature to all of our users.</p>
<p>Online evaluation, on the other hand, ensures that the feature performs as expected in a live environment—this is particularly significant because we are dealing with user-generated content and substantial traffic in real-time. We conducted a partial release, only implementing a small segment of Mercari AI Assist that calls the LLM API, to assess performance and confirm that the complete feature is ready for our customer base. In this preliminary online test period, we tasked GPT with extracting a single key attribute from an item&#8217;s description and to respond simply with “YES” if the attribute is present, or “NO” if it is not.</p>
<p>We found that it is very useful for teams who are not familiar with using LLM in production to perform these kinds of partial preliminary releases, especially when using commercially available APIs provided by third-party services.</p>
<p>During the preliminary online test period, we observed that even though we instructed GPT to provide outputs in a straightforward format (YES or NO), the number of inconsistently formatted responses increased along with the number of requests. The table below presents a sampled result from this experiment.</p>
<table>
<thead>
<tr>
<th>LLM Output</th>
<th>Count</th>
</tr>
</thead>
<tbody>
<tr>
<td>NO</td>
<td>311,813</td>
</tr>
<tr>
<td>No</td>
<td>22,948</td>
</tr>
<tr>
<td>Yes</td>
<td>17,236</td>
</tr>
<tr>
<td>&#8230;</td>
<td>&#8230;</td>
</tr>
<tr>
<td><span style="color:red">Sorry, but I can&#8217;t provide the answer you&#8217;re looking for. </span></td>
<td>5</td>
</tr>
<tr>
<td><span style="color:red">Sorry, but I can&#8217;t assist with that request.</span></td>
<td>4</td>
</tr>
<tr>
<td><span style="color:red">The provided text does not contain the information.</span></td>
<td>4</td>
</tr>
<tr>
<td><span style="color:red">NO<br /> NO<br /> YES </span></td>
<td>1</td>
</tr>
<tr>
<td><span style="color:red">NO<br />YES<br />NO<br />NO<br />YES<br />NO<br />NO<br />NO<br />NO</span></td>
<td>1</td>
</tr>
</tbody>
</table>
<p>Being aware of such inconsistencies is crucial for production systems. In the above sampled use case, the wrong format might be non-critical and relatively easy to solve (e.g. with regular expressions). However, as we require more complex outputs from LLMs, detecting inconsistencies—as well as hallucinations, a well-known issue with large language models—becomes increasingly challenging.</p>
<p>It&#8217;s essential to preprocess prompts that contain user-generated content to minimize the likelihood of GPT generating incorrect responses. Additionally, post-processing logic should be implemented to ensure that only the expected output format is relayed to the client application.</p>
<h1>Additional Things to Keep in Mind</h1>
<p>Since we&#8217;re utilizing an LLM provided by a third-party service, it&#8217;s critical to understand how the API functions and what sorts of errors may occur. In addition to common API error types such as authentication and timeout errors, which we might already know how to handle, we need to give special attention to errors more closely related to LLMs. For instance, depending on the API you use, some calls might inadvertently trigger a content violation error. At Mercari, we have our own content moderation system; however, the filtering policy of a third-party API might differ. It is important to be aware of this to accordingly prepare our prompts and to avoid undesired outcomes.</p>
<p>Another consideration is the token count. The number of tokens used can vary depending on the language sent to the model. For instance, <a href="https://aclanthology.org/2023.emnlp-main.614.pdf">an experiment</a> presented at EMNLP 2023 indicated that, using ChatGPT, the average cost of prompt and generated tokens in Japanese can exceed that of English by more than double. This certainly depends on the task at hand and sometimes there&#8217;s no alternative, but it is one thing to keep in mind.</p>
<p>Lastly, in this rapidly evolving field, what is considered the best tool can change in just a short span of time. Libraries are updated constantly—with the occasional breaking change—and many of us are constantly looking for ways to optimally integrate LLMs into production systems. This might sound obvious, but we argue that it is important to closely follow new updates regarding LLM research and best practices.</p>
<h1>Looking Back, Going Forward</h1>
<p>The design and development of Mercari AI Assist has offered us valuable perspectives on working with prompt engineering and integrating commercially available Large Language Models (LLMs) into production. Looking back, I felt that I gained substantial knowledge and experience from the practical aspects of working with LLMs and I am enthusiastic about further advancing my skills alongside the team.</p>
<p>Among the key lessons learned are the significance of cultivating a team equipped with the right mindset and fostering effective communication. I have also experienced and learned about the intricacies of choosing the right model and techniques, finding the right balance between cost and performance, dealing with LLM’s stability when used in a live environment, and addressing unique challenges of LLM, such as hallucination and content moderation. Additionally, I believe it is advisable to have team members with a background in machine learning and natural language processing when working with LLMs. Having the appropriate expertise can speed up various research and experimental processes. For instance, it can enable the team to swiftly determine the suitability of LLMs for a specific task and also decide the most suitable evaluation metrics.</p>
<p>Going forward, we are focusing on improvements such as LLM operations and the development of automated workflows. We are also exploring the use of LLMs for more complex and specialized tasks, which may require the adoption of parameter-efficient fine-tuning techniques. With a rapidly growing field, our team is continuously experimenting and learning, and we understand that our implementation is far from perfect. As with many other practitioners in the field, we are constantly following updates from the field, sharing, listening, and looking for best practices most suitable for our use cases.</p>
<p>I look forward to yet another exciting year filled with obstacles and successes, and to share in these experiences with the incredible members at Mercari.</p>
<p>Tomorrow&#8217;s article will be by @ayaneko. Look forward to it!</p>


Leveraging LLMs in Production: Looking Back, Going Forward

<p>Mercari announced <a href="https://about.mercari.com/press/news/articles/20230501_generativeai/">the creation of a new dedicated generative AI/LLM</a> in May 2023. Yuki Ishikawa (<a href="(https://twitter.com/maze_rapid)">@maze</a>), who has led the Mercari Shops business as Representative Director and CEO of Souzoh, launched the team and leads it. Up until now, Mercari has used AI in a wide range of areas, including marketing and customer support, as well as UI/UX. Ishikawa proposed the launch of the Generative AI/LLM Team at a managerial meeting, believing it would become one of the pillars of Mercari's AI utilization. We asked Ishikawa about his thoughts on this technology, what he has seen in the past three months of work, and the team's longer-term prospects.</p>
<h3>Profile</h3>
<p><strong>Yuki Ishikawa | Mercari VP of Generative AI / LLM<br />
</strong><br />
After graduating from the University of Tokyo, was hired by Nintendo Co., Ltd in 2012. Joined Moi Corp (TwitCasting) in 2014, where he was involved in various development and product launches. Hired by the former company Souzoh, Inc. of the Mercari Group in June 2017. After moving to Mercari, Inc., became VP of Product at Merpay, Inc. in July 2020. Appointed Representative Director and CEO of Souzoh, Inc. in January 2021. Received concurrent appointment as VP at Mercari, Inc. in July 2022.</p>
<h2>Ishikawa felt that &quot;this is an area Mercari definitely has to move into.&quot;</h2>
<p><img src="https://storage.googleapis.com/prd-ai-asset/2023/10/4a8052b6-maze_interview3-1024x683.jpg" alt="" /></p>
<p><center><br />
Yuki Ishikawa | Mercari VP of Generative AI / LLM<br />
</center></p>
<p><strong>ーーMercari's dedicated Generative AI/LLM Team was created in May 2023. Tell us about the background and purpose behind it.</strong></p>
<p><strong>Ishikawa：</strong>Although we officially launched the dedicated Generative AI/LLM Team in May, we had been working on the concept since February 2023, and I had personally been paying attention to it following the emergence of <a href="https://stablediffusionweb.com/">Stable Diffusion</a> in the summer of 2022 and <a href="https://chat.openai.com/auth/login">ChatGPT</a> at the end of 2022. I toyed with various ideas and came up with some that we could use in Mercari's business and internal operations, so I made a proposal at a managerial meeting. I said, &quot;This is an area Mercari definitely has to move into.&quot; That was what led to the team's creation. In the spring of this year, there was a lot of uncertainty about generative AI, so we started by creating a small, dedicated team that could act quickly and flexibly.</p>
<p>The Generative AI/LLM Team's mission is twofold: 1) maximize business impact by creating new customer experiences using generative AI technology, and 2) dramatically improve company-wide productivity.</p>
<p><img src="https://storage.googleapis.com/prd-ai-asset/2023/10/36b404ab-maze_interview1-1024x570.png" alt="" /></p>
<p><center><br />
The  presentation at MS Build Japan 2023<br />
</center></p>
<p>Initially, we had full-stack engineers who had been involved in ML projects join us. Currently, the team consists of engineers, designers, and BizOps members. We also continue to hire engineers.</p>
<h2>Introduction of GitHub Copilot to improve the development experience for engineers</h2>
<p><strong>ーーA few months have passed since the AI/LLM team was created. What are some specific things you've done?<br />
</strong></p>
<p><strong>Ishikawa：</strong>First, in connection with our mission to dramatically improve company-wide productivity, we introduced <a href="https://github.com/features/copilot">GitHub Copilot</a> for our engineers' development experience. It's a programming support tool that uses AI to suggest the optimal code. It generates code from comments and completes code in progress to improve the development experience. GitHub Copilot makes development more productive. After introducing it, some engineers, according to official statements, finish tasks twice as fast. In the future, I think we'll see a big difference between companies that use GitHub Copilot and those that don't, and we have a lot of engineers, so that's why we started using it. The Generative AI/LLM Team spearheaded the project, the Corporate Engineer Team took the lead, and in June, the tool was made available to engineers who want to use it.</p>
<p>We have identified three risks of concern when using GitHub Copilot. The first is legal risk, including copyright infringement and violation of open-source licenses. The second is reputational risk linked to legal risk. And the last is the risk of data breaches containing internal company code through GitHub Copilot.</p>
<p>After making various verifications, we have concluded that by setting certain restrictions, there are no problems with using GitHub Copilot, and now any engineer at Mercari can use it. Although we haven't yet completed full verification since its introduction, we have received feedback from senior engineers that &quot;it has made things a lot easier,&quot; and from those with relatively short engineering experience that &quot;it's like a textbook I can use when I'm having trouble.&quot; Even those who aren't purely programmers can write reasonably clean code, so I feel that the tool is beneficial to a wide range of members involved in development.</p>
<p><img src="https://storage.googleapis.com/prd-ai-asset/2023/10/bf386527-maze_interview5-1024x683.jpg" alt="" /></p>
<h2>A new experience for customers allowing them to search for items with natural language</h2>
<p><strong>ーー<a href="https://about.mercari.com/press/news/articles/20230724_chatgpt/">A ChatGPT plugin</a> was announced in July 2023. </strong></p>
<p><strong>Ishikawa：</strong>E-commerce sites, including Mercari, generally focus on keyword searches when customers have some idea of what they want and displaying recommended items based on search history.</p>
<p>However, with the introduction of this plug-in, customers can now search for items by interacting with ChatGPT. This functionality is different from previous customer experiences because it allows users to search for items through natural conversation, even if you're not thinking of a specific product. For example, you can say, &quot;Tell me what I need to go camping for the first time,&quot; or &quot;What gift would make a five-year-old girl happy?&quot; Depending on the results of our upcoming verifications, we are considering implementing this feature in the Mercari app, too.</p>
<p>After we introduced the plug-in, my honest impression is that customers are using it more than I originally expected. Currently, the ChatGPT plug-in is only available to users of the paid version, ChatGPT Plus. These are mostly men in their 30s and 40s, and they often use it to discuss gifts for their daughters or partners, or to narrow down itens for hobbies related to electronics and cameras. It's certainly very useful to be able to use natural language to narrow down items in terms of complex features in gadgets and home appliances. ChatGPT's suggestions also contribute to the influx of users to Mercari. We'll continue to track the data.</p>
<p><img src="https://storage.googleapis.com/prd-ai-asset/2023/10/2a5fa68e-maze_interview2-1024x683.jpg" alt="" /></p>
<h2>Utilizing generative AI creatives for hiring and marketing</h2>
<p><strong>ーーOn X (formerly Twitter), you have uploaded hiring creatives utilizing generative AI. Looking ahead, how do you see its use for creatives?<br />
</strong><br />
<img src="https://storage.googleapis.com/prd-ai-asset/2023/10/cb5c6c13-images_maze.png" alt="" /></p>
<p><center><br />
Creatives to recruit engineers<br />
</center></p>
<p><strong>Ishikawa：</strong>First, as a test, I generated creatives to hire Generative AI/LLM Team members and posted them on my X account for a few days. These images are creatives that generative AI made, with text added by a designer to match the background. Right now, people are involved a bit, but I think that in the future, we'll be able to generate whatever creatives we want just with text instructions.</p>
<p>Looking ahead, we'll consider using generative AI for campaigns and ad creatives.</p>
<p>Specifically, we're working on generating ad creatives based on data from previously well-received Mercari advertisements to see if we can use them in digital and OOH (out-of-home) ads. Currently (as of October 3, 2023), we're running a Halloween commercial for Mercari outside around Shibuya Station. The creatives for this promotional video were actually produced by generative AI.</p>
<p><center><br />
<iframe loading="lazy" width="560" height="315" src="https://www.youtube.com/embed/IWgwWYrhaMs?si=IvUJHo_QU7DOznVL" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe><br />
Mercari commercial airing at the Shibuya scramble crossing.<br />
</center></p>
<p>We are also working on possible ways to use this technology to generate trading card creatives. Although we can't use images of actual cards in advertisements due to copyrights, we're considering the possibility of using AI to generate creatives that look like trading cards and that we can use in advertisements and elsewhere. We're also working on the legal aspects, and we hope to release this soon.</p>
<h2>This next half year will be a phase with many challenges and a mix of successes and failures</h2>
<p><strong>ーーWhat else does the Generative AI/LLM Team want to try and do?<br />
</strong></p>
<p><strong>Ishikawa：</strong>During the next six months, we will continue to experiment in all areas to see what kind of experiences we can create with generative AI and what impact we can achieve.</p>
<p>During these six months, it will be important to take on as many challenges as we can in as broad a range as possible, with the idea that it's OK to fail half the time. As a result, we'll probably see where generative AI/LLM will succeed and where it won't. I think we need to plan some action and ascertain if and how we need an evaluation layer, a platform layer, and an ML Ops-like organization.</p>
<p>While the introduction of the technology into our services and its business impact are of course important indicators, we also consider it a very important mission to promote the understanding of the technology within Mercari. A year from now, generative AI/LLM should be a more core technology in Mercari's development experience. We are aiming for a state in which generative AI/LLM is not only in the UI/UX (for example, to streamline transactions), but also in back-end algorithms.</p>
<p><strong>ーーWhat do you think the AI/LLM team's composition will look like moving forward?</strong></p>
<p><strong>Ishikawa：</strong>While we will continue to promote agile technology exploration and product development with a dedicated team, and in the longer term, we hope to advance our work so that every member of Mercari can make use of generative AI/LLM.</p>
<p>In the future, we are also considering whether to develop our own generative AI model inside Mercari. Beyond the company, we want to also look at widespread possibilities for partnerships outside the company.</p>
<p><img src="https://storage.googleapis.com/prd-ai-asset/2023/10/5596b7e9-maze_interview6-1024x683.jpg" alt="" /></p>
<h2>Mercari's appeal is that we can enable direct feedback while utilizing a wealth of data</h2>
<p><strong>ーーWhat is Mercari's appeal as a place to explore generative AI/LLM possibilities?</strong></p>
<p><strong>Ishikawa：</strong>There are two appealing reasons to work on AI/LLM at Mercari. First, our company has a wealth of data. AI products are dependent on the amount of data they can use. On that point, we can handle a large amount of data we've accumulated at Mercari. The second reason is that we get a lot of direct feedback as a result of the distribution of the systems we've created. I feel that these two factors are strong reasons for us to develop AI-powered services. </p>
<p>There are many things we can do with generative AI in areas related to our customers. I think we will see major changes in the way we work with regard to improving productivity at the company. Mercari currently has over 2,000 employees, all of whom are highly IT literate, and we are in an environment where we can experiment with a variety of approaches. At Mercari, I believe we can study business design and organizational design using generative AI/LLM, so I want to move fast on PoC.</p>
<p>In fact, the theme that we will dramatically improve companywide productivity was added to the mission at my request. The environment within the company is conducive to technological trial and error, making it easy to take on new challenges. I also believe that changes to our employees' workplace environment will make us more familiar with generative AI/LLM and further the technology's spread.</p>
<p><img src="https://storage.googleapis.com/prd-ai-asset/2023/10/d23a319e-maze_interview4-1024x683.jpg" alt="" /></p>
<h2>The Generative AI/LLM Team is recruiting new members to work with us!</h2>
<p>The Generative AI/LLM Team is recruiting new members to work with us! We hope you’ll find it interesting and a good challenge to work in Mercari’s Generative AI/LLM Team. Click here for more details.<br />
<a href="https://apply.workable.com/mercari/j/183899D2BA/">Software Engineer, Full Stack (LLM/GenAI) - Mercari</a> </p>
<ul>
<li>Photo : Shun Nakayama (<a href="https://shunnakayama.myportfolio.com/">Web</a> / <a href="https://twitter.com/shunnakayama_">X</a>)</li>
<li>Text : Ayumi Iga</li>
</ul>


How does Mercari leverage generative AI and large language models? The stated mission: “create new customer experiences and business impact” and “dramatically improve companywide productivity.”

<h2>Overview</h2>
<p>We are pleased to announce that the paper &quot;Layout Optimizer for Personalized Home Screen based on Contextual Multi-Armed Bandit in C2C Marketplace&quot; has been accepted to the <a href="https://sites.google.com/view/consequences2023/home">CONSEQUENCES '23 workshop</a> within the international conference &quot;<a href="https://recsys.acm.org/recsys23/">RecSys 2023(ACM Conference on Recommender Systems)</a>&quot;.</p>
<p>RecSys is one of the most prestigious international conferences in the field of recommendation systems, held annually by researchers from around the world. This year's RecSys, in its 17th year, was held in Singapore from September 18 to 22, 2023.</p>
<h2>Key points of presentation</h2>
<p>In this presentation, based on how customers use Mercari, we propose a method to display more attractive content on the home screen with priority among other content, including product recommendations. We reported the results of our experiments.</p>
<p>In the proposed method, an exploratory screen display and machine learning are performed to collect data, without giving prior knowledge of what kind of content is attractive to which users. As a result, the screen display is automatically optimized to improve overall satisfaction.</p>
<h2>Background</h2>
<p>At Mercari app, several teams, including the Recommendation Team, are developing content on the home screen.There are various types of content on the home screen, including item recommendations and campaign announcements. Furthermore, even if the same item is being recommended, there are multiple methods of recommendation and display. </p>
<p>Not only do the items to be recommended vary from user to user, but also the type of recommendation method and how effective the campaign announcement is depends on each user.</p>
<h2>Summary of paper</h2>
<p>In e-commerce services, including Mercari, the home screen is an important function that attracts many views. However, the amount of content that can be displayed is limited, and it is common practice to place attractive content at the top of the home screen as a priority.</p>
<p>However, since different users have different perspectives on what kind of content is attractive, it is necessary to create a personalized screen structure.</p>
<p>In this study, we proposed a method that applies the contextual multi-armed bandit problem algorithm and reported the details and results of our experiments.</p>
<p><img src="https://storage.googleapis.com/prd-ai-asset/2023/09/6d29996b-recsys.png" alt="" /></p>
<h2>Comments from author</h2>
<p><strong>ML Engineer / Backend Engineer | Yusuke Shido<br />
</strong><br />
Since this year's event was held offline, and in the form of a poster presentation, we were able to take the time to have direct discussions with researchers and engineers from various countries. In particular, I received a lot of interest and questions from developers and researchers belonging to companies, and I felt that the content of our paper was an issue common to many different companies. The five days at RecSys, including the workshop, were an opportunity for very meaningful input. We hope to apply the knowledge gained to Mercari's business in the future and continue to disseminate the results.</p>
<p><img src="https://storage.googleapis.com/prd-ai-asset/2023/09/cdf923d7-recsys23_1-1024x552.jpg" alt="" /></p>
<p><img src="https://storage.googleapis.com/prd-ai-asset/2023/09/046bfffa-recsys23_3-1024x576.jpg" alt="" /></p>
<p><center><br />
Members making a poster presentation in Singapore<br />
</center></p>
<h2>About recommendation team</h2>
<p>Mercari's recommendation team is working to improve the quality of product recommendations on each screen of the Mercari App, as well as the mechanisms involved in screen configurations such as the one in this study.</p>


Mercari AI’s research “Layout Optimizer for Personalized Home Screen based on Contextual Multi-Armed Bandit in C2C Marketplace” accepted to RecSys 2023’s Workshop

<p>Hello, I am <a href="https://twitter.com/MLBear2">ML_Bear</a>, an ML Engineer in the Mercari Recommend Team. <a href="https://engineering.mercari.com/en/blog/entry/20221004-attempt-to-improve-item-recommendation-accuracy-using-item2vec/">In a previous article</a> [1], I talked about improving Mercari&#8217;s home screen recommendations using item2vec and item metadata. This time, I will talk about improving recommendations on the item details screen. An example of the item details screen can be seen in figure 1. This screen is displayed to the user every time they want to see a detailed description of an item, which makes it a natural touch point to recommend similar items.</p>
<p>To make these improvements, we did the following:</p>
<ul>
<li>Implemented a vector search-based recommendation algorithm in one of Japan&#8217;s largest EC services, significantly improving recommendation accuracy.</li>
<li>Successfully utilized user browsing history by constructing a recommendation algorithm using collaborative filtering and NN, avoiding the cold start problem.</li>
<li>Accelerated extensive behavior user-browsing-log calculations using the Python implicit library and GPU during collaborative filtering learning.</li>
<li>Created a lightweight NN model, partially referencing solutions from Kaggle competitions.</li>
<li>Ensured continuous improvement by conducting offline evaluations using user-browsing-log in modeling.</li>
<li>Adopted the VertexAI Matching Engine for our vector search engine, enabling efficient vector searches with a small team.</li>
<li>Actual A/B testing led to the discovery of important features overlooked during NN modeling. After the initial test failure, we quickly corrected it and completed a powerful recommendation model contributing to the actual business.</li>
</ul>
<p>Next, Iwill talk about some of the challenges we faced.</p>
<p>Figure 1. Target of this story: &quot;Recommended for those who are viewing this item&quot; (この商品を見ている人におすすめ)<br />
<img src="https://storage.googleapis.com/prd-engineering-asset/2023/06/c145c4fc--2023-06-09-18.12.58-348x1024.png" alt="" /></p>
<h2>Utilizing a vector search engine in Mercari</h2>
<p>As introduced in <a href="https://engineering.mercari.com/blog/entry/20220224-similar-search-using-matching-engine/">the article</a> [2] written by <a href="https://twitter.com/WakanaNogami">wakanapo</a> last year, Mercari Group is trying to improve recommendation accuracy using vector search engines. The previous article was about improving recommendations for Mercari Shops items, but in this article, I will introduce an attempt to improve recommendations for all items listed on Mercari.</p>
<p>We adopted the Vertex AI Matching Engine [3] (the Matching Engine) for the vector search engine. We chose it because other teams have already been using it (allowing us to reuse some of their code and their operational know-how), and because it can withstand high access loads, as we will discuss later.</p>
<h2>Item recommendation using vector search engine</h2>
<p>The item recommendation system we built this time recommends items using the following flow. I will explain the details later, but here’s the basic idea:</p>
<p>(The numbers in parentheses correspond to the system architecture overview.)</p>
<ul>
<li>Indexing
<ul>
<li>Calculate the item vector by some method (i, ii, iii)</li>
<li>Store the item vector in the following two GCP services (iv)
<ul>
<li>Bigtable [4]: Save all item vectors</li>
<li>Matching Engine: Save vectors of items for sale</li>
</ul>
</li>
</ul>
</li>
<li>Recommendation
<ul>
<li>When a user views an item (1), recommendations are made using the following flow.
<ul>
<li>Retrieve the vector of the item being viewed from Bigtable (2, 3)</li>
<li>Use Matching Engine to search for items for sale with similar vectors, using approximate nearest neighbor search. (4, 5)</li>
<li>Display the search results of Matching Engine in &quot;Recommended for those who are viewing this item&quot; (この商品を見ている人におすすめ) (6)</li>
</ul>
</li>
</ul>
</li>
</ul>
<p>Figure 2. System Architecture Overview<br />
<img src="https://storage.googleapis.com/prd-engineering-asset/2023/06/afa9fae6--2023-06-09-18.18.31.png" alt="" /></p>
<p>Initially, Matching Engine could only accept vectors as queries and return similar item IDs. This is why the Item index mapping in Bigtable (2, 3) is needed. Later improvements to Matching Engine removed the need for the mapping, as Matching Engine could now accept vector id directly.</p>
<p>We also adopted Streaming Update [5] for creating the Matching Engine index. I will not go into the details here, but with this method we can instantly reflect the addition of newly listed items to the index and the removal of sold-out items from the index. This was a very convenient feature for Mercari, where item inventory changes at an incredible pace.</p>
<h2>The first A/B test targeting the toys category</h2>
<p>The inventory on Mercari is huge with hundreds of millions of items for sale, and over 3 billion items listed in total [6]. Since the &quot;Recommended for those who are viewing this item&quot; part needs to be displayed even for sold-out items, we need to calculate vector embeddings for sold-out items as well.</p>
<p>To validate our hypothesis more quickly, we decided to focus on a specific subset of our inventory. If initial experiments work well, we could then  expand it to the full item inventory. In this specific case, we decided to start with the toys category.</p>
<p>We selected the toys category first for the following reasons:</p>
<ul>
<li>The trends for items in this category changes very quickly, which results in our current recommendation logic not working very well. For example, when new characters were introduced in a TV show, unrelated items were being recommended when searching for the new characters. This was because we could not keep up with these additions, as well as new items related to them.</li>
<li>The toys category contained several sub-categories with high sales volumes such as trading cards. We could expect improvements in recommendations to contribute to sales.</li>
</ul>
<h2>Utilizing collaborative filtering</h2>
<p>For the modeling, I decided to use word2vec [7] for the baseline mode, as it was also used during work for improving Mercari Shops items. However, the metrics of offline evaluation (MRR: Mean Reciprocal Rank) did not perform well with word2vec when dealing with a very large number of items, and the recommended results did not look very good to our eyes either. Specifically, it seemed like subtle differences between items were being ignored as the number of items became  large.</p>
<p>I also tried using a word2vec trained using our own dataset, but the accuracy did not improve as much as I expected. After some trial and error, I decided to use a more  classical collaborative filtering.</p>
<p>Specifically, I used the Python “implicit” library [8] to calculate the factors of items from user browsing logs. The “implicit” library can accelerate calculations using GPUs, so it can complete calculations in a realistic time even if you put in billions of rows of data. In addition, it supports differential updates, so you can update to more sophisticated vectors as the item browsing history accumulates.</p>
<p>This library turned out to be extremely beneficial for Mercari, which has a huge amount of user log data and item data, but there were two problems.</p>
<ul>
<li>The handling of logs in the implicit library is very complicated
<ul>
<li>Due to the constraints of the library, data must be handled with IDs starting from 0, and it requires a conversion table between implicit IDs and item IDs.</li>
</ul>
</li>
<li>Cold start problem
<ul>
<li>Due to the nature of the marketplace app service, new items tend to attract more views. If the &quot;Recommended for those who are viewing this item&quot; does not work well for new items, it will negatively affect the user experience.</li>
<li>(However, this is a problem with the collaborative filtering method itself, not just the implicit library.)</li>
</ul>
</li>
</ul>
<p>To solve these, just before the A/B test, we tested the following model changes.</p>
<ul>
<li>Calculate the vector (factor) of items with sufficient item browsing counts using collaborative filtering</li>
<li>Train a neural network model (NN model) that reproduces the vector using item information such as title and item description</li>
<li>Calculate the vector for all items in the toys category using the NN model and use it as the item vector.</li>
</ul>
<p>I will skip over the details of the implementation of the NN model because it would make this post too long, but basically, I built a simple model with the following configuration. (We did not use heavy models such as BERT in the first test because we had to process tens of millions of items.)</p>
<p>Figure 3. NN Model Architecture (simplified)<br />
<img src="https://storage.googleapis.com/prd-engineering-asset/2023/06/0cea7378--2023-06-09-18.20.57.png" alt="" /></p>
<p>During string processing, we referred to some of the solutions of the Mercari Kaggle competition [13] (like processing the item title and category name together).</p>
<p>After a lot of work, I ended up having to approximate the collaborative filtering factors with a neural network. Other models such as two-tower models may have been more effective. We plan on trying this another time.</p>
<h2>Recommending new items as much as possible</h2>
<p>The first A/B test did not work out too well. Fortunately, we were able to quickly identify the cause of the failure and conduct a second A/B test, which was successful, so we were able to avoid further problems.</p>
<p>The reason for the failure was that we were recommending too many items that had been left unsold for a long time after being listed.</p>
<p>As I mentioned earlier, the item vectors were mainly generated using item information such as titles and descriptions, without considering when the items were listed (freshness). We later realized that when conducting offline evaluations, we only used data from specific periods, so the lack of consideration for freshness did not manifest itself as a problem during the modeling. As a result, we did not notice that we needed to consider the freshness of the items until the first A/B test.</p>
<p>After modifying the recommendation logic to consider the freshness of the items, the purchase rate of the recommended items improved significantly, leading to the overwhelming numerical improvements which I will talk about at the end of this article.</p>
<h2>Other challenges</h2>
<p>This was our first use of the Matching Engine at this scale. We encountered several issues while deploying in production. Some highlights:</p>
<ul>
<li>We were not able to find everything we needed in some of the documentation (how to use the SDK, how to configure the public endpoint, etc.), though the Google Cloud team was quick to answer our questions.</li>
<li>There were occasional times when we could not get a GPU at all with GKE&#8217;s node auto-provisioning (NAP) [12], possibly due to a shortage of GPU resources in the Tokyo Region. In the end, we gave up on NAP and set up an instance to always keep a GPU. (I wonder if this is due to the rise of image generation AI&#8230;)</li>
</ul>
<h2>Improvement results: Item recommendation tap rate tripled</h2>
<p>Now, as a result of the modeling described so far, we have achieved the ability to make the following recommendations. Previously, we were not effectively recommending related items when users were browsing items related to new characters. However, by adopting the approach presented this time, we were able to overcome this weakness.</p>
<p>Figure 4. Successfully making recommendations (The numbers in parentheses indicate the recommended order.)</p>
<p>Browsing item: <code>ちいかわ ワクワクゆうえんち Pouch</code></p>
<p>Before improvement (many unrelated to “ちいかわ”):</p>
<pre><code>[1] ハイキュー Art Coaster Bulk Pack
[2] 呪術廻戦0 TOHO-Lottery H-Prize Sticker...
[3] ちいかわ Mascot with Dialogue ハチワレ Prize Item
[4] 美少女戦士セーラームーンR S カードダス アマダ
[5] プロメア ガロ＆リオ SGTver. Special Box PROMARE
[6] 宇宙戦艦ヤマト 2205 新たなる旅立ち Keychain Set
[7] Doraemon Wallet with Strap and Pass Case
[8] [New &amp; Not for Sale] 日本食研 バンコ Plush Toy
[9] Pocket Monsters メイ EP-0137 Bath Towel Size...
[10]ちいかわワクワクゆうえんち Limited Edition Towel Set</code></pre>
<p>After improvement (recognizing “ちいかわ ワクワクゆうえんち”)</p>
<pre><code>[1] ちいかわ ワクワクゆうえんち 2-Piece Set Pouch Jet Co...
[2] ちいかわ ワクワクゆうえんち Pouch
[3] ちいかわワクワクゆうえんち Limited Edition Towel Set
[4] Anonymous Delivery ちいかわワクワクゆうえんち Gacha Ax…
[5] ちいかわ ワクワクゆうえんち Mug Cup
[6] Anonymous Delivery Unopened New ちいかわ ワクワクゆうえんち Mascot...
[7] ちいかわ ワクワク ゆうえんち Side Plate
[8] ちいかわ ワクワクゆうえんち Mini Frame Art ハチワレ
[9] ちいかわ ワクワクゆうえんち Mascot Set
[10] ちいかわ ワクワクゆうえんち 2-Piece Set Pouch</code></pre>
<p>(For copyright reasons, I will not upload images here, but you can see the results yourself by looking up items on the app.)</p>
<p>As a result of the A/B test, we were able to achieve the following great results.</p>
<ol>
<li>The tap rate for items under &quot;Recommended for people viewing this item&quot; tripled</li>
<li>Purchases from &quot;Recommended for people viewing this item&quot; increased by 20%</li>
<li>As a result, the Mercari app&#8217;s overall sales increased significantly.</li>
</ol>
<p>Of course, it&#8217;s great that the business metrics have improved, but more importantly, we were able to properly recommend items more strongly related to the items users are viewing, and we were proud of that as a team.</p>
<h2>There’s still room for improvement</h2>
<p>I had a lot of information to present, and the detailed explanations for each section ended up being very concise, but I hope this served as a helpful reference for you.</p>
<p>This time the model design itself was very simple, as it was the first test of vector search item recommendation for all of Mercari. We have not considered images yet, and we have not used advanced features of the Matching Engine (such as the Crowding Option [14] for diversity).</p>
<p>In addition, we have not  yet applied this model to categories other than toys, so there is still room for improvement. We will continue to make improvements and evolve the service to be better.</p>
<p>Let me know if you have any opinions or impressions on Twitter or elsewhere.</p>
<p>See you again!</p>
<h2>References</h2>
<p>[1] <a href="https://engineering.mercari.com/en/blog/entry/20221004-attempt-to-improve-item-recommendation-accuracy-using-item2vec/">Attempt to improve item recommendation accuracy using Item2vec | Mercari Engineering</a><br />
[2] <a href="https://engineering.mercari.com/blog/entry/20220224-similar-search-using-matching-engine/">Vertex AI Matching Engineをつかった類似商品検索APIの開発 | メルカリエンジニアリング</a><br />
[3] <a href="https://cloud.google.com/vertex-ai/docs/matching-engine/overview">Vertex AI Matching Engine overview | Google Cloud</a><br />
[4] <a href="https://cloud.google.com/bigtable">Cloud Bigtable: HBase-compatible NoSQL database</a><br />
[5] <a href="https://cloud.google.com/vertex-ai/docs/matching-engine/update-rebuild-index#update_an_index_using_streaming_updates">Update and rebuild an active index | Vertex AI | Google Cloud</a><br />
[6] <a href="https://about.mercari.com/press/news/articles/20221128_threebillion/">フリマアプリ「メルカリ」累計出品数が30億品を突破</a><br />
[7] <a href="https://arxiv.org/abs/1301.3781">[1301.3781] Efficient Estimation of Word Representations in Vector Space</a><br />
[8] <a href="https://github.com/benfred/implicit">GitHub &#8211; benfred/implicit: Fast Python Collaborative Filtering for Implicit Feedback Datasets</a><br />
[9] <a href="https://arxiv.org/abs/1408.5882">[1408.5882] Convolutional Neural Networks for Sentence Classification</a><br />
[10] <a href="https://arxiv.org/abs/1805.09843">[1805.09843] Baseline Needs More Love: On Simple Word-Embedding-Based Models and Associated Pooling Mechanisms</a><br />
[11] <a href="https://taku910.github.io/mecab/">MeCab: Yet Another Part-of-Speech and Morphological Analyzer</a><br />
[12] <a href="https://www.kaggle.com/code/lopuhin/mercari-golf-0-3875-cv-in-75-loc-1900-s/script">Mercari Golf: 0.3875 CV in 75 LOC, 1900 s | Kaggle</a><br />
[13] <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/node-auto-provisioning">Use node auto-provisioning | Google Kubernetes Engine（GKE）</a><br />
[14] <a href="https://cloud.google.com/vertex-ai/docs/matching-engine/update-rebuild-index#upsert_with_crowding">Update and rebuild an active index | Vertex AI | Google Cloud</a></p>


Improving Item Recommendation Accuracy Using Collaborative Filtering and Vector Search Engine

<h2>■  Presentation video also available on Youtube!</h2>
<p><strong> (Added: June 22, 2023)<br />
Check out the presentation at Berlin Buzzwords 2023 here.<br />
</strong></p>
<p><iframe loading="lazy" width="560" height="315" src="https://www.youtube.com/embed/11xxPUSJTss" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe></p>
<h2>■ Overview</h2>
<p>We are pleased to announce that Mercari ML Engineers Teo Narboneta Zosa and Ryan Ginstrom will give a talk at Berlin Buzzwords 2023.<br />
<a href="https://program.berlinbuzzwords.de/berlin-buzzwords-2023/talk/YLCZP8/">Building MLOps Infrastructure at Japan's Largest C2C E-Commerce Site</a></p>
<ul>
<li>Local time: Tuesday, June 20, 2023, 10:00 - 10:40 (GMT+2)</li>
<li>Japan Time: Tuesday, June 20, 2023, 17:00 - 17:40</li>
</ul>
<p>Berlin Buzzwords is Europe’s leading conference for modern data infrastructure, search and machine learning, and will take place on June 18th – 20th, 2023. The conference is focused on open source software projects in the field of data analysis, machine learning, scalability, storage and searchability.<br />
<a href="https://2023.berlinbuzzwords.de/about/">About – Berlin Buzzwords</a></p>
<h2>■ Talk contents</h2>
<p><a href="https://program.berlinbuzzwords.de/berlin-buzzwords-2023/talk/YLCZP8/">Building MLOps Infrastructure at Japan's Largest C2C E-Commerce Site</a></p>
<p>We describe the system we built to support ML in search at Mercari, Japan’s largest C2C e-commerce platform. </p>
<p>We start by describing the journey to enable the use of ML in a “traditional” term-based search infrastructure with high throughput and strict latency requirements. We also discuss the mixed blessing of rushing a successful proof of concept into production and the technical challenges this posed on the infrastructure side. Next, we discuss the nuts and bolts of data engineering, ETLs, training pipelines, and serving/monitoring our ML model in production. We also discuss some of the weaknesses of our initial homegrown system, including A/B testing and model monitoring. Finally, we discuss our efforts to evolve our homegrown system into a more modern MLOps infrastructure using an A/B testing framework and Seldon for traffic routing and model serving.</p>
<h2>■ About the Search Item Understanding Team</h2>
<p>The Search Item Understanding Team is in charge of bringing advanced ML techniques to Search and Discovery at Mercari.</p>


Mercari ML Engineers Teo Narboneta Zosa and Ryan Ginstrom will give a talk at Berlin Buzzwords 2023

<p>Hi everyone! I am Rakesh from the Mercari’s Seller Engagement team. Recently I had the opportunity to mentor an Intern at Mercari. His name is <a href="https://github.com/Priyansh-Kedia">Priyansh</a> and this article summarizes part of his work on using Firebase for client side machine learning models.</p>
<h2>Introduction</h2>
<p>The availability of huge data, advances in powerful processing and storage, and the increasing demand for automation and data-driven decision making has led to the widespread popularity of machine learning today.</p>
<p>Many companies want to use machine learning but it can be very expensive, the cost associated with developing, training and deploying can be significant, as well as the cost of using specialized hardware like GPUs, cloud computing resources, storage and networking.</p>
<p>Traditionally machine learning models are deployed on the server side but the popularity to enable client side machine learning is also increasing. There are many advantages to using AI on the edge.</p>
<ul>
<li>No internet connection required.</li>
<li>Do not need to send data to the server.</li>
<li>Real time inference, with extremely low latency.</li>
<li>Little to no server cost.<br />
<img src="https://storage.googleapis.com/prd-engineering-asset/2023/04/144f3720-image3-1024x348.png" alt="" /></li>
</ul>
<h2>Why we wanted to use ML on client side</h2>
<p>We at mercari are always trying to increase the value in our marketplace by building many innovative features. Barcode listing is one such powerful feature which makes sellers list easily by scanning the barcode of certain categories of items.</p>
<p><img src="https://storage.googleapis.com/prd-engineering-asset/2023/04/c31b2d22-image2-1024x889.png" alt="" /></p>
<p>Sometimes users, especially new users are not fully aware of all our features and barcode listing is such a feature. To address this and to improve the customer experience, we wanted to use Edge AI. So, we built a new feature called listing dispatcher, which uses an ML model on client side to predict if the category of item being listed by the seller supports barcode listing or not using the item’s first photo.</p>
<p><img src="https://storage.googleapis.com/prd-engineering-asset/2023/04/634182fc-image1.gif" alt="" /></p>
<h2>How to use ML on client devices?</h2>
<p>We can use one of the popular open source mobile libraries called <a href="https://www.tensorflow.org/lite">TensorFlow Lite</a> (in short also called TFLite) for client side machine learning inferences. It has many key features like</p>
<ul>
<li>Optimized for on-device machine learning.</li>
<li>Supports multiple platforms.</li>
<li>Support across many languages like Java, Swift, etc.</li>
<li>High performance.</li>
<li>Support many common machine learning tasks like image classification, object detection, etc.</li>
</ul>
<p><img src="https://storage.googleapis.com/prd-engineering-asset/2023/04/268480bd-image4.png" alt="" /></p>
<p>We can simply embed the TensorFlow Lite model to an android or iOS app while packaging it to apk or ipa. This is one of the easiest ways to use ML on the client side but it has certain limitations. </p>
<ul>
<li>ML models are usually huge and packaging it with the app will increase the app size and can lead to drop in installations on iOS app store or Android play store.</li>
<li>An additional overhead to optimize the ML model for compression, for which we may need to compromise with model accuracy.</li>
<li>It can also affect the development flow of the client side.</li>
</ul>
<p><img src="https://storage.googleapis.com/prd-engineering-asset/2023/04/a60a0475-image5.png" alt="" /></p>
<p><center><br />
Embedding the tflite model on Android by adding it to app assets<br />
</center></p>
<p>Is there a better way?&#8230;</p>
<h2>Firebase Machine Learning Custom Model</h2>
<p>It is one of the services of <a href="https://firebase.google.com/docs/ml/use-custom-models">Firebase</a> which enables you to deploy your own custom ML model on edge devices. We don&#8217;t need to package the app with ML model, but instead the client will download the ML model after installation in background on demand from remote only once and reuse it for subsequent inferences. </p>
<p><img src="https://storage.googleapis.com/prd-engineering-asset/2023/04/be38b0cc-image7-972x1024.png" alt="" /></p>
<p>There are many advantages to using this method</p>
<ul>
<li>ML models are hosted on Firebase.</li>
<li>Helps in version control of ML models and ML model management.</li>
<li>Decoupling of client side development flow and ML development flow.</li>
<li>Ensuring that the app always uses the latest version of ML model.</li>
<li>Can configure conditions on when to download the latest model.</li>
<li><a href="https://firebase.google.com/docs/ml/android/ab-test-models">A/B test two versions of a model</a></li>
</ul>
<p><img src="https://storage.googleapis.com/prd-engineering-asset/2023/04/66d12291-image6-1024x631.png" alt="" /></p>
<p><center><br />
Firebase console to use custom machine learning<br />
</center></p>
<p>Even though using Firebase dynamic model loading has many advantages, We need to be mindful of the following</p>
<ul>
<li>Error handling &#8211; Fallback to default model or disable ML feature, if a new version of ML model has errors.</li>
<li>When should we download the model? For example download only when the user is connected to a wifi etc.</li>
</ul>
<h2>Conclusion</h2>
<p>ML is very useful in a wide range of use cases like object detection, barcode scanning etc. There are various challenges to using ML on the client side but Firebase Custom Model has made it so easy with APIs to fetch ML models from remote, model management, faster experimentation and ability for developers to customize configurations.</p>


Model management for client side ML powered by Firebase

The Mercari AI team’s research “Strategic Coupon Allocation for Increasing Providers’ Sales Experiences in Two-sided Marketplaces” has been accepted at the KDD2024 TSMO 2024 Workshop

mercari AI’s research “LLMs as an Interactive Database Interface for Designing Large Queries” accepted to HILDA Workshop in SIGMOD 2024

How have the organizational challenges of mercari  ML team been solved? A chat with the engineers who launched the company-wide Guild

Mercari AI team presented in NLP2024 “A related document retrieval system using past queries”

Mercari ML Engineers Chingis Oinar and  Teo Narboneta Zosa will give a talk at FOSSASIA Summit2024

Blogs by mercari AI team, delivering information about AI technology and organization

The Mercari AI team’s research “Strategic Coupon Allocation for Increasing Providers’ Sales Experiences in Two-sided Marketplaces” has been accepted at the KDD2024 TSMO 2024 Workshop

mercari AI’s research “LLMs as an Interactive Database Interface for Designing Large Queries” accepted to HILDA Workshop in SIGMOD 2024

How have the organizational challenges of mercari ML team been solved? A chat with the engineers who launched the company-wide Guild

Mercari AI team presented in NLP2024 “A related document retrieval system using past queries”

Mercari ML Engineers Chingis Oinar and Teo Narboneta Zosa will give a talk at FOSSASIA Summit2024

The Mercari AI team’s research “Strategic Coupon Allocation for Increasing Providers’ Sales Experiences in Two-sided Marketplaces” has been accepted at the KDD2024 TSMO 2024 Workshop

mercari AI’s research “LLMs as an Interactive Database Interface for Designing Large Queries” accepted to HILDA Workshop in SIGMOD 2024

How have the organizational challenges of mercari ML team been solved? A chat with the engineers who launched the company-wide Guild

Mercari AI team presented in NLP2024 “A related document retrieval system using past queries”

Mercari ML Engineers Chingis Oinar and Teo Narboneta Zosa will give a talk at FOSSASIA Summit2024

The Mercari AI team’s research “Strategic Coupon Allocation for Increasing Providers’ Sales Experiences in Two-sided Marketplaces” has been accepted at the KDD2024 TSMO 2024 Workshop

mercari AI’s research “LLMs as an Interactive Database Interface for Designing Large Queries” accepted to HILDA Workshop in SIGMOD 2024

How have the organizational challenges of mercari ML team been solved? A chat with the engineers who launched the company-wide Guild

Mercari AI team presented in NLP2024 “A related document retrieval system using past queries”

Mercari ML Engineers Chingis Oinar and Teo Narboneta Zosa will give a talk at FOSSASIA Summit2024

Articles

Mercari DS Teams’s research “Using Recommendations to balance demand and supply in two-sided marketplaces” accepted to EcoSys Workshop at AAAI 2024

Improvement of the Selling Experience Key for Mercari: Six Months of the Item Coupon Project in Retrospect

Leveraging LLMs in Production: Looking Back, Going Forward

How does Mercari leverage generative AI and large language models? The stated mission: “create new customer experiences and business impact” and “dramatically improve companywide productivity.”

Mercari AI’s research “Layout Optimizer for Personalized Home Screen based on Contextual Multi-Armed Bandit in C2C Marketplace” accepted to RecSys 2023’s Workshop

Improving Item Recommendation Accuracy Using Collaborative Filtering and Vector Search Engine

Mercari ML Engineers Teo Narboneta Zosa and Ryan Ginstrom will give a talk at Berlin Buzzwords 2023

Model management for client side ML powered by Firebase