<p>Hello, I am <a href="https://twitter.com/MLBear2">ML_Bear</a>, an ML Engineer in the Mercari Recommend Team. <a href="https://engineering.mercari.com/en/blog/entry/20221004-attempt-to-improve-item-recommendation-accuracy-using-item2vec/">In a previous article</a> [1], I talked about improving Mercari&#8217;s home screen recommendations using item2vec and item metadata. This time, I will talk about improving recommendations on the item details screen. An example of the item details screen can be seen in figure 1. This screen is displayed to the user every time they want to see a detailed description of an item, which makes it a natural touch point to recommend similar items.</p>
<p>To make these improvements, we did the following:</p>
<ul>
<li>Implemented a vector search-based recommendation algorithm in one of Japan&#8217;s largest EC services, significantly improving recommendation accuracy.</li>
<li>Successfully utilized user browsing history by constructing a recommendation algorithm using collaborative filtering and NN, avoiding the cold start problem.</li>
<li>Accelerated extensive behavior user-browsing-log calculations using the Python implicit library and GPU during collaborative filtering learning.</li>
<li>Created a lightweight NN model, partially referencing solutions from Kaggle competitions.</li>
<li>Ensured continuous improvement by conducting offline evaluations using user-browsing-log in modeling.</li>
<li>Adopted the VertexAI Matching Engine for our vector search engine, enabling efficient vector searches with a small team.</li>
<li>Actual A/B testing led to the discovery of important features overlooked during NN modeling. After the initial test failure, we quickly corrected it and completed a powerful recommendation model contributing to the actual business.</li>
</ul>
<p>Next, Iwill talk about some of the challenges we faced.</p>
<p>Figure 1. Target of this story: &quot;Recommended for those who are viewing this item&quot; (この商品を見ている人におすすめ)<br />
<img src="https://storage.googleapis.com/prd-engineering-asset/2023/06/c145c4fc--2023-06-09-18.12.58-348x1024.png" alt="" /></p>
<h2>Utilizing a vector search engine in Mercari</h2>
<p>As introduced in <a href="https://engineering.mercari.com/blog/entry/20220224-similar-search-using-matching-engine/">the article</a> [2] written by <a href="https://twitter.com/WakanaNogami">wakanapo</a> last year, Mercari Group is trying to improve recommendation accuracy using vector search engines. The previous article was about improving recommendations for Mercari Shops items, but in this article, I will introduce an attempt to improve recommendations for all items listed on Mercari.</p>
<p>We adopted the Vertex AI Matching Engine [3] (the Matching Engine) for the vector search engine. We chose it because other teams have already been using it (allowing us to reuse some of their code and their operational know-how), and because it can withstand high access loads, as we will discuss later.</p>
<h2>Item recommendation using vector search engine</h2>
<p>The item recommendation system we built this time recommends items using the following flow. I will explain the details later, but here’s the basic idea:</p>
<p>(The numbers in parentheses correspond to the system architecture overview.)</p>
<ul>
<li>Indexing
<ul>
<li>Calculate the item vector by some method (i, ii, iii)</li>
<li>Store the item vector in the following two GCP services (iv)
<ul>
<li>Bigtable [4]: Save all item vectors</li>
<li>Matching Engine: Save vectors of items for sale</li>
</ul>
</li>
</ul>
</li>
<li>Recommendation
<ul>
<li>When a user views an item (1), recommendations are made using the following flow.
<ul>
<li>Retrieve the vector of the item being viewed from Bigtable (2, 3)</li>
<li>Use Matching Engine to search for items for sale with similar vectors, using approximate nearest neighbor search. (4, 5)</li>
<li>Display the search results of Matching Engine in &quot;Recommended for those who are viewing this item&quot; (この商品を見ている人におすすめ) (6)</li>
</ul>
</li>
</ul>
</li>
</ul>
<p>Figure 2. System Architecture Overview<br />
<img src="https://storage.googleapis.com/prd-engineering-asset/2023/06/afa9fae6--2023-06-09-18.18.31.png" alt="" /></p>
<p>Initially, Matching Engine could only accept vectors as queries and return similar item IDs. This is why the Item index mapping in Bigtable (2, 3) is needed. Later improvements to Matching Engine removed the need for the mapping, as Matching Engine could now accept vector id directly.</p>
<p>We also adopted Streaming Update [5] for creating the Matching Engine index. I will not go into the details here, but with this method we can instantly reflect the addition of newly listed items to the index and the removal of sold-out items from the index. This was a very convenient feature for Mercari, where item inventory changes at an incredible pace.</p>
<h2>The first A/B test targeting the toys category</h2>
<p>The inventory on Mercari is huge with hundreds of millions of items for sale, and over 3 billion items listed in total [6]. Since the &quot;Recommended for those who are viewing this item&quot; part needs to be displayed even for sold-out items, we need to calculate vector embeddings for sold-out items as well.</p>
<p>To validate our hypothesis more quickly, we decided to focus on a specific subset of our inventory. If initial experiments work well, we could then  expand it to the full item inventory. In this specific case, we decided to start with the toys category.</p>
<p>We selected the toys category first for the following reasons:</p>
<ul>
<li>The trends for items in this category changes very quickly, which results in our current recommendation logic not working very well. For example, when new characters were introduced in a TV show, unrelated items were being recommended when searching for the new characters. This was because we could not keep up with these additions, as well as new items related to them.</li>
<li>The toys category contained several sub-categories with high sales volumes such as trading cards. We could expect improvements in recommendations to contribute to sales.</li>
</ul>
<h2>Utilizing collaborative filtering</h2>
<p>For the modeling, I decided to use word2vec [7] for the baseline mode, as it was also used during work for improving Mercari Shops items. However, the metrics of offline evaluation (MRR: Mean Reciprocal Rank) did not perform well with word2vec when dealing with a very large number of items, and the recommended results did not look very good to our eyes either. Specifically, it seemed like subtle differences between items were being ignored as the number of items became  large.</p>
<p>I also tried using a word2vec trained using our own dataset, but the accuracy did not improve as much as I expected. After some trial and error, I decided to use a more  classical collaborative filtering.</p>
<p>Specifically, I used the Python “implicit” library [8] to calculate the factors of items from user browsing logs. The “implicit” library can accelerate calculations using GPUs, so it can complete calculations in a realistic time even if you put in billions of rows of data. In addition, it supports differential updates, so you can update to more sophisticated vectors as the item browsing history accumulates.</p>
<p>This library turned out to be extremely beneficial for Mercari, which has a huge amount of user log data and item data, but there were two problems.</p>
<ul>
<li>The handling of logs in the implicit library is very complicated
<ul>
<li>Due to the constraints of the library, data must be handled with IDs starting from 0, and it requires a conversion table between implicit IDs and item IDs.</li>
</ul>
</li>
<li>Cold start problem
<ul>
<li>Due to the nature of the marketplace app service, new items tend to attract more views. If the &quot;Recommended for those who are viewing this item&quot; does not work well for new items, it will negatively affect the user experience.</li>
<li>(However, this is a problem with the collaborative filtering method itself, not just the implicit library.)</li>
</ul>
</li>
</ul>
<p>To solve these, just before the A/B test, we tested the following model changes.</p>
<ul>
<li>Calculate the vector (factor) of items with sufficient item browsing counts using collaborative filtering</li>
<li>Train a neural network model (NN model) that reproduces the vector using item information such as title and item description</li>
<li>Calculate the vector for all items in the toys category using the NN model and use it as the item vector.</li>
</ul>
<p>I will skip over the details of the implementation of the NN model because it would make this post too long, but basically, I built a simple model with the following configuration. (We did not use heavy models such as BERT in the first test because we had to process tens of millions of items.)</p>
<p>Figure 3. NN Model Architecture (simplified)<br />
<img src="https://storage.googleapis.com/prd-engineering-asset/2023/06/0cea7378--2023-06-09-18.20.57.png" alt="" /></p>
<p>During string processing, we referred to some of the solutions of the Mercari Kaggle competition [13] (like processing the item title and category name together).</p>
<p>After a lot of work, I ended up having to approximate the collaborative filtering factors with a neural network. Other models such as two-tower models may have been more effective. We plan on trying this another time.</p>
<h2>Recommending new items as much as possible</h2>
<p>The first A/B test did not work out too well. Fortunately, we were able to quickly identify the cause of the failure and conduct a second A/B test, which was successful, so we were able to avoid further problems.</p>
<p>The reason for the failure was that we were recommending too many items that had been left unsold for a long time after being listed.</p>
<p>As I mentioned earlier, the item vectors were mainly generated using item information such as titles and descriptions, without considering when the items were listed (freshness). We later realized that when conducting offline evaluations, we only used data from specific periods, so the lack of consideration for freshness did not manifest itself as a problem during the modeling. As a result, we did not notice that we needed to consider the freshness of the items until the first A/B test.</p>
<p>After modifying the recommendation logic to consider the freshness of the items, the purchase rate of the recommended items improved significantly, leading to the overwhelming numerical improvements which I will talk about at the end of this article.</p>
<h2>Other challenges</h2>
<p>This was our first use of the Matching Engine at this scale. We encountered several issues while deploying in production. Some highlights:</p>
<ul>
<li>We were not able to find everything we needed in some of the documentation (how to use the SDK, how to configure the public endpoint, etc.), though the Google Cloud team was quick to answer our questions.</li>
<li>There were occasional times when we could not get a GPU at all with GKE&#8217;s node auto-provisioning (NAP) [12], possibly due to a shortage of GPU resources in the Tokyo Region. In the end, we gave up on NAP and set up an instance to always keep a GPU. (I wonder if this is due to the rise of image generation AI&#8230;)</li>
</ul>
<h2>Improvement results: Item recommendation tap rate tripled</h2>
<p>Now, as a result of the modeling described so far, we have achieved the ability to make the following recommendations. Previously, we were not effectively recommending related items when users were browsing items related to new characters. However, by adopting the approach presented this time, we were able to overcome this weakness.</p>
<p>Figure 4. Successfully making recommendations (The numbers in parentheses indicate the recommended order.)</p>
<p>Browsing item: <code>ちいかわ ワクワクゆうえんち Pouch</code></p>
<p>Before improvement (many unrelated to “ちいかわ”):</p>
<pre><code>[1] ハイキュー Art Coaster Bulk Pack
[2] 呪術廻戦0 TOHO-Lottery H-Prize Sticker...
[3] ちいかわ Mascot with Dialogue ハチワレ Prize Item
[4] 美少女戦士セーラームーンR S カードダス アマダ
[5] プロメア ガロ＆リオ SGTver. Special Box PROMARE
[6] 宇宙戦艦ヤマト 2205 新たなる旅立ち Keychain Set
[7] Doraemon Wallet with Strap and Pass Case
[8] [New &amp; Not for Sale] 日本食研 バンコ Plush Toy
[9] Pocket Monsters メイ EP-0137 Bath Towel Size...
[10]ちいかわワクワクゆうえんち Limited Edition Towel Set</code></pre>
<p>After improvement (recognizing “ちいかわ ワクワクゆうえんち”)</p>
<pre><code>[1] ちいかわ ワクワクゆうえんち 2-Piece Set Pouch Jet Co...
[2] ちいかわ ワクワクゆうえんち Pouch
[3] ちいかわワクワクゆうえんち Limited Edition Towel Set
[4] Anonymous Delivery ちいかわワクワクゆうえんち Gacha Ax…
[5] ちいかわ ワクワクゆうえんち Mug Cup
[6] Anonymous Delivery Unopened New ちいかわ ワクワクゆうえんち Mascot...
[7] ちいかわ ワクワク ゆうえんち Side Plate
[8] ちいかわ ワクワクゆうえんち Mini Frame Art ハチワレ
[9] ちいかわ ワクワクゆうえんち Mascot Set
[10] ちいかわ ワクワクゆうえんち 2-Piece Set Pouch</code></pre>
<p>(For copyright reasons, I will not upload images here, but you can see the results yourself by looking up items on the app.)</p>
<p>As a result of the A/B test, we were able to achieve the following great results.</p>
<ol>
<li>The tap rate for items under &quot;Recommended for people viewing this item&quot; tripled</li>
<li>Purchases from &quot;Recommended for people viewing this item&quot; increased by 20%</li>
<li>As a result, the Mercari app&#8217;s overall sales increased significantly.</li>
</ol>
<p>Of course, it&#8217;s great that the business metrics have improved, but more importantly, we were able to properly recommend items more strongly related to the items users are viewing, and we were proud of that as a team.</p>
<h2>There’s still room for improvement</h2>
<p>I had a lot of information to present, and the detailed explanations for each section ended up being very concise, but I hope this served as a helpful reference for you.</p>
<p>This time the model design itself was very simple, as it was the first test of vector search item recommendation for all of Mercari. We have not considered images yet, and we have not used advanced features of the Matching Engine (such as the Crowding Option [14] for diversity).</p>
<p>In addition, we have not  yet applied this model to categories other than toys, so there is still room for improvement. We will continue to make improvements and evolve the service to be better.</p>
<p>Let me know if you have any opinions or impressions on Twitter or elsewhere.</p>
<p>See you again!</p>
<h2>References</h2>
<p>[1] <a href="https://engineering.mercari.com/en/blog/entry/20221004-attempt-to-improve-item-recommendation-accuracy-using-item2vec/">Attempt to improve item recommendation accuracy using Item2vec | Mercari Engineering</a><br />
[2] <a href="https://engineering.mercari.com/blog/entry/20220224-similar-search-using-matching-engine/">Vertex AI Matching Engineをつかった類似商品検索APIの開発 | メルカリエンジニアリング</a><br />
[3] <a href="https://cloud.google.com/vertex-ai/docs/matching-engine/overview">Vertex AI Matching Engine overview | Google Cloud</a><br />
[4] <a href="https://cloud.google.com/bigtable">Cloud Bigtable: HBase-compatible NoSQL database</a><br />
[5] <a href="https://cloud.google.com/vertex-ai/docs/matching-engine/update-rebuild-index#update_an_index_using_streaming_updates">Update and rebuild an active index | Vertex AI | Google Cloud</a><br />
[6] <a href="https://about.mercari.com/press/news/articles/20221128_threebillion/">フリマアプリ「メルカリ」累計出品数が30億品を突破</a><br />
[7] <a href="https://arxiv.org/abs/1301.3781">[1301.3781] Efficient Estimation of Word Representations in Vector Space</a><br />
[8] <a href="https://github.com/benfred/implicit">GitHub &#8211; benfred/implicit: Fast Python Collaborative Filtering for Implicit Feedback Datasets</a><br />
[9] <a href="https://arxiv.org/abs/1408.5882">[1408.5882] Convolutional Neural Networks for Sentence Classification</a><br />
[10] <a href="https://arxiv.org/abs/1805.09843">[1805.09843] Baseline Needs More Love: On Simple Word-Embedding-Based Models and Associated Pooling Mechanisms</a><br />
[11] <a href="https://taku910.github.io/mecab/">MeCab: Yet Another Part-of-Speech and Morphological Analyzer</a><br />
[12] <a href="https://www.kaggle.com/code/lopuhin/mercari-golf-0-3875-cv-in-75-loc-1900-s/script">Mercari Golf: 0.3875 CV in 75 LOC, 1900 s | Kaggle</a><br />
[13] <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/node-auto-provisioning">Use node auto-provisioning | Google Kubernetes Engine（GKE）</a><br />
[14] <a href="https://cloud.google.com/vertex-ai/docs/matching-engine/update-rebuild-index#upsert_with_crowding">Update and rebuild an active index | Vertex AI | Google Cloud</a></p>


Improving Item Recommendation Accuracy Using Collaborative Filtering and Vector Search Engine

Blog

<h2>■  Presentation also available on Youtube!</h2>
<p><strong> (Added: June 22, 2023)<br />
Check out the presentation at Berlin Buzzwords 2023 here.<br />
</strong></p>
<p><iframe loading="lazy" width="560" height="315" src="https://www.youtube.com/embed/11xxPUSJTss" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe></p>
<h2>■ Overview</h2>
<p>We are pleased to announce that Mercari ML Engineers Teo Narboneta Zosa and Ryan Ginstrom will give a talk at Berlin Buzzwords 2023.<br />
<a href="https://program.berlinbuzzwords.de/berlin-buzzwords-2023/talk/YLCZP8/">Building MLOps Infrastructure at Japan's Largest C2C E-Commerce Site</a></p>
<ul>
<li>Local time: Tuesday, June 20, 2023, 10:00 - 10:40 (GMT+2)</li>
<li>Japan Time: Tuesday, June 20, 2023, 17:00 - 17:40</li>
</ul>
<p>Berlin Buzzwords is Europe’s leading conference for modern data infrastructure, search and machine learning, and will take place on June 18th – 20th, 2023. The conference is focused on open source software projects in the field of data analysis, machine learning, scalability, storage and searchability.<br />
<a href="https://2023.berlinbuzzwords.de/about/">About – Berlin Buzzwords</a></p>
<h2>■ Talk contents</h2>
<p><a href="https://program.berlinbuzzwords.de/berlin-buzzwords-2023/talk/YLCZP8/">Building MLOps Infrastructure at Japan's Largest C2C E-Commerce Site</a></p>
<p>We describe the system we built to support ML in search at Mercari, Japan’s largest C2C e-commerce platform. </p>
<p>We start by describing the journey to enable the use of ML in a “traditional” term-based search infrastructure with high throughput and strict latency requirements. We also discuss the mixed blessing of rushing a successful proof of concept into production and the technical challenges this posed on the infrastructure side. Next, we discuss the nuts and bolts of data engineering, ETLs, training pipelines, and serving/monitoring our ML model in production. We also discuss some of the weaknesses of our initial homegrown system, including A/B testing and model monitoring. Finally, we discuss our efforts to evolve our homegrown system into a more modern MLOps infrastructure using an A/B testing framework and Seldon for traffic routing and model serving.</p>
<h2>■ About the Search Item Understanding Team</h2>
<p>The Search Item Understanding Team is in charge of bringing advanced ML techniques to Search and Discovery at Mercari.</p>


Mercari ML Engineers Teo Narboneta Zosa and Ryan Ginstrom will give a talk at Berlin Buzzwords 2023

<p>Hi everyone! I am Rakesh from the Mercari’s Seller Engagement team. Recently I had the opportunity to mentor an Intern at Mercari. His name is <a href="https://github.com/Priyansh-Kedia">Priyansh</a> and this article summarizes part of his work on using Firebase for client side machine learning models.</p>
<h2>Introduction</h2>
<p>The availability of huge data, advances in powerful processing and storage, and the increasing demand for automation and data-driven decision making has led to the widespread popularity of machine learning today.</p>
<p>Many companies want to use machine learning but it can be very expensive, the cost associated with developing, training and deploying can be significant, as well as the cost of using specialized hardware like GPUs, cloud computing resources, storage and networking.</p>
<p>Traditionally machine learning models are deployed on the server side but the popularity to enable client side machine learning is also increasing. There are many advantages to using AI on the edge.</p>
<ul>
<li>No internet connection required.</li>
<li>Do not need to send data to the server.</li>
<li>Real time inference, with extremely low latency.</li>
<li>Little to no server cost.<br />
<img src="https://storage.googleapis.com/prd-engineering-asset/2023/04/144f3720-image3-1024x348.png" alt="" /></li>
</ul>
<h2>Why we wanted to use ML on client side</h2>
<p>We at mercari are always trying to increase the value in our marketplace by building many innovative features. Barcode listing is one such powerful feature which makes sellers list easily by scanning the barcode of certain categories of items.</p>
<p><img src="https://storage.googleapis.com/prd-engineering-asset/2023/04/c31b2d22-image2-1024x889.png" alt="" /></p>
<p>Sometimes users, especially new users are not fully aware of all our features and barcode listing is such a feature. To address this and to improve the customer experience, we wanted to use Edge AI. So, we built a new feature called listing dispatcher, which uses an ML model on client side to predict if the category of item being listed by the seller supports barcode listing or not using the item’s first photo.</p>
<p><img src="https://storage.googleapis.com/prd-engineering-asset/2023/04/634182fc-image1.gif" alt="" /></p>
<h2>How to use ML on client devices?</h2>
<p>We can use one of the popular open source mobile libraries called <a href="https://www.tensorflow.org/lite">TensorFlow Lite</a> (in short also called TFLite) for client side machine learning inferences. It has many key features like</p>
<ul>
<li>Optimized for on-device machine learning.</li>
<li>Supports multiple platforms.</li>
<li>Support across many languages like Java, Swift, etc.</li>
<li>High performance.</li>
<li>Support many common machine learning tasks like image classification, object detection, etc.</li>
</ul>
<p><img src="https://storage.googleapis.com/prd-engineering-asset/2023/04/268480bd-image4.png" alt="" /></p>
<p>We can simply embed the TensorFlow Lite model to an android or iOS app while packaging it to apk or ipa. This is one of the easiest ways to use ML on the client side but it has certain limitations. </p>
<ul>
<li>ML models are usually huge and packaging it with the app will increase the app size and can lead to drop in installations on iOS app store or Android play store.</li>
<li>An additional overhead to optimize the ML model for compression, for which we may need to compromise with model accuracy.</li>
<li>It can also affect the development flow of the client side.</li>
</ul>
<p><img src="https://storage.googleapis.com/prd-engineering-asset/2023/04/a60a0475-image5.png" alt="" /></p>
<p><center><br />
Embedding the tflite model on Android by adding it to app assets<br />
</center></p>
<p>Is there a better way?&#8230;</p>
<h2>Firebase Machine Learning Custom Model</h2>
<p>It is one of the services of <a href="https://firebase.google.com/docs/ml/use-custom-models">Firebase</a> which enables you to deploy your own custom ML model on edge devices. We don&#8217;t need to package the app with ML model, but instead the client will download the ML model after installation in background on demand from remote only once and reuse it for subsequent inferences. </p>
<p><img src="https://storage.googleapis.com/prd-engineering-asset/2023/04/be38b0cc-image7-972x1024.png" alt="" /></p>
<p>There are many advantages to using this method</p>
<ul>
<li>ML models are hosted on Firebase.</li>
<li>Helps in version control of ML models and ML model management.</li>
<li>Decoupling of client side development flow and ML development flow.</li>
<li>Ensuring that the app always uses the latest version of ML model.</li>
<li>Can configure conditions on when to download the latest model.</li>
<li><a href="https://firebase.google.com/docs/ml/android/ab-test-models">A/B test two versions of a model</a></li>
</ul>
<p><img src="https://storage.googleapis.com/prd-engineering-asset/2023/04/66d12291-image6-1024x631.png" alt="" /></p>
<p><center><br />
Firebase console to use custom machine learning<br />
</center></p>
<p>Even though using Firebase dynamic model loading has many advantages, We need to be mindful of the following</p>
<ul>
<li>Error handling &#8211; Fallback to default model or disable ML feature, if a new version of ML model has errors.</li>
<li>When should we download the model? For example download only when the user is connected to a wifi etc.</li>
</ul>
<h2>Conclusion</h2>
<p>ML is very useful in a wide range of use cases like object detection, barcode scanning etc. There are various challenges to using ML on the client side but Firebase Custom Model has made it so easy with APIs to fetch ML models from remote, model management, faster experimentation and ability for developers to customize configurations.</p>


Model management for client side ML powered by Firebase

<h2>■Overview</h2>
<p>We are pleased to announce that the &quot;<a href="https://ceur-ws.org/Vol-3318/short20.pdf">Want robust explanations? Get smoother predictions first.</a>&quot; by Engineer Deddy Jobson of the Mercaril AI team has been accepted for the <a href="https://project.inria.fr/aimlai/">AIMLAI workshop</a> of the international conference &quot;<a href="https://www.cikm2022.org/">CIKM2022</a>&quot;.<br />
CIKM (The Conference on Information and Knowledge Management) is one of the most prestigious international conferences in the field of data mining and information retrieval, and is held annually by researchers from around the world.</p>
<p>The accepted papers were presented at the AIMLAI workshop of &quot;CIKM 2022&quot;  held at Atlanta, Georgia, USA in October 2022.</p>
<h2>■Key points of presentation</h2>
<ul>
<li>We improve the robustness of model-agnostic ML interpretability methods like LIME and SHAP by smoothening the predictions of the model before passing it through the explainer. </li>
<li>We find an improvement in the robustness of the explanations as measured by a variant of the Lipshitz score we introduce in our paper. </li>
</ul>
<h2>■Background</h2>
<ul>
<li>Model-agnostic interpretability methods like LIME and SHAP help explain the predictions made by complex ML algorithms increasing their trustworthiness. </li>
<li>However, they suffer from a lack of robustness where small perturbations to the input can greatly change the explanations provided by these methods even when the prediction of the output does not change. </li>
<li>This undermines trust in the explanations and therefore hinders adoption of the ML model. </li>
</ul>
<h2>■Summary of paper</h2>
<ul>
<li>We propose a simple approach to improve the robustness of the explanations. </li>
<li>For each data point, we add gaussian noise to the input, pass it through the model, and get the output. We do this multiple times and take the average for each data point. </li>
<li>The resulting predictions are smoother making the explanations provided by LIME and SHAP more robust and consequently trustworthier.</li>
</ul>
<h2>■About MDS Team</h2>
<p>The Marketing Data Science Team makes use of statistics, machine learning, and mathematical optimization to improve the efficiency of marketing campaigns. We actively learn and adopt the state of the art methods used in the literature to solve technical problems and publish papers along the way. </p>


Mercari AI’s research “Want robust explanations? Get smoother predictions first.” accepted to CIKM 2022’s workshop

<h2>■Overview</h2>
<p>We are pleased to announce that the &quot;Estimating the impact of coupon non-usage&quot; by Engineer Deddy Jobson of the Mercaril AI team has been accepted for the <a href="https://upliftworkshop.ipipan.waw.pl/">Uplift Modeling Tutorial &amp; Workshop</a> collocated with the international conference &quot;<a href="https://2022.ecmlpkdd.org/">ECML PKDD 2022</a>&quot;.<br />
ECML PKDD (“European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases”) is one of the top conferences in the field of data mining and machine learning, and is held annually by researchers from around the world.</p>
<p>The accepted papers were presented at the workshop of &quot;ECML PKDD 2022&quot; held at Grenoble, France in September 2022.</p>
<h2>■Key points of presentation</h2>
<p>We look into the phenomenon of users not redeeming discount coupons.<br />
We use causal analysis to quantify the negative business impact of the lack of usability of marketing coupons. </p>
<h2>■Background / 研究の背景</h2>
<p>Mercari runs many marketing campaigns in the form of coupons which can be redeemed by users when making purchases.<br />
While the marketing campaigns are very successful, we observe that a substantial number of users who have access to the coupon fail to make use of it.<br />
While we can measure the coupon redemption rate and find the number of users who bought items without coupons, we can’t directly estimate the counterfactual number of users who didn’t buy a coupon but would have been able to use the coupon they were given. </p>
<h2>■Summary of paper / 研究概要</h2>
<p>We created a causal diagram and used a bayesian model to estimate the business impact of the lack of usage of the coupon.<br />
We show through our experiments that campaigns can greatly benefit from an increase in usability of the coupons we distribute. </p>
<h2>■About MDS Team</h2>
<p>The Marketing Data Science Team makes use of statistics, machine learning, and mathematical optimization to improve the efficiency of marketing campaigns. We actively learn and adopt the state of the art methods used in the literature to solve technical problems and publish papers along the way.</p>


Mercari AI’s research “Estimating the impact of coupon non-usage” accepted to ECML/PKDD 2022’s workshop

<h2>Overview</h2>
<p>We are pleased to announce that &quot;Counterfactual Learning with General Data-generating Policies,&quot; a paper co-authored by Team AI engineer Akihiro Shimizu and Yusuke Narita, founder of Hanjuku Kaso, Inc. and assistant professor at Yale University, has been accepted to the plenary session of AAAI 2023, an international conference on artificial intelligence.</p>
<p>AAAI (Association for the Advancement of Artificial Intelligence) is one of the most prestigious conferences in the field of artificial intelligence. This is the 37th year of the conference, and 1,721 papers have been accepted out of 8,777 submissions (acceptance rate of 19.6%).</p>
<p>The following schedule of presentation times is planned</p>
<p><strong><br />
Sunday, February 12, 2023, 2:00 PM - 3:15 PM (GMT-5)<br />
</strong></p>
<h2>Key points of presentation</h2>
<p><a href="https://arxiv.org/abs/2212.01925">Counterfactual Learning with General Data-generating Policies</a></p>
<ul>
<li>Proposed a method for Off-policy Evaluation that can be applied to a wide class of existing measures, including non-probabilistic measures.</li>
<li>Developed a method for improving algorithms using data naturally generated by the algorithm.</li>
<li>Analyzed coupon distribution policies in Mercari using the proposed method. The effectiveness of coupon measures can now be measured by using data from Mercari.</li>
</ul>
<h2>Background</h2>
<p>An important part of algorithmic decision making is to predict the performance of new decision-making algorithms (also called measures) that have not yet been used. However, A/B testing, in which old algorithms are randomly assigned to users and compared with new algorithms, can be time-consuming to develop and can lead to dissatisfaction due to unfairness to some users who could not use the new functionality.</p>
<p>Therefore, it is important to solve this problem by using &quot;Off-policy Evaluation,&quot; which attempts to estimate the performance of a new algorithm using only the data naturally generated by the old algorithm. However, it is known that current Off-policy Evaluation techniques are difficult to use for non-probabilistic measures such as coupon distribution algorithms.</p>
<h2>Summary of paper</h2>
<p>Therefore, we have proposed a new method of ex-post evaluation that can be applied to a wide range of existing measures, including non-probabilistic measures such as coupon distribution algorithms.</p>
<p>The proposed method is developed based on the observation that when an algorithm makes a decision, the data generated from it will almost always include natural experiments. For example, in the case of supervised learning used in coupon distribution algorithms, etc., decisions are often made based on whether or not some predicted variable exceeds a certain criterion value, such as whether or not to distribute coupons, etc. However, even though the situation before and after the criterion value is almost identical In our proposed method, the decision-making process is based on such a coincidence. Our proposed method considers these differences in decision making caused by chance factors as local natural experiments, and uses them to compare the performance of new measures.</p>
<p>By using data from Mercari, we have verified that our proposed method can solve the problems of real data and A/B testing in a real-world problem. Using the proposed method, it is possible to estimate the performance of a new coupon and its business impact through ex-post evaluation rather than A/B testing.</p>
<h2>About MDS Team</h2>
<p>The MDS (Marketing Data Science) team uses statistics, machine learning, and mathematical optimization to improve the efficiency of marketing campaigns.</p>
<h2>About Hanjuku Kaso, Inc.</h2>
<p>A startup that combines data, algorithms, mathematics, and thought to design the future of business, policy, and society.<br />
The company's strengths are &quot;market design,&quot; &quot;counterfactual machine learning,&quot; and &quot;causal inference,&quot; and it conducts joint projects and research with numerous companies as well as its own basic research and software development.<br />
The collaboration involves Yusuke Narita, the company's founder and an assistant professor at Yale University; Kyohei Okumura, a PhD student in the Department of Economics at Northwestern University; and Kohei Yata, an assistant professor at the University of Wisconsin-Madison.</p>


Mercari AI’s collaborative research paper “Counterfactual Learning with General Data-generating Policies” accepted to AAAI 2023

The Journey to Machine-Learned Re-ranking

Interview

<p>Tomohiro Furusawa (<a href="https://twitter.com/rilmayer_jp">@furufuru</a>), a member of Mercari’s AI Team, has held a variety of positions in his three years at Mercari, including PM, engineer, and scrum master. Despite shifting among these roles, he has always been involved in improving the discovery experience at Mercari. We sat down with him to talk about how Mercari’s Recommendation Team has developed since its establishment, and some of the things he wants to work on in the future.</p>
<h2>Profile</h2>
<p>Tomohiro Furusawa (<a href="https://twitter.com/rilmayer_jp">@furufuru</a>) | Product Manager<br />
Tomohiro discovered the joys of data science and software engineering, and before he knew it, he made the leap to the world of IT. After finishing graduate school, he used his background in library and information science to develop products for information retrieval and recommendations. He joined Mercari Inc. in 2019 to work on search evaluations as a member of the Data Science Team. He then established the Recommendation Team and is currently handling product improvement as a Product Manager.</p>
<h2>“Since my days as a student, I’ve always been figuring out how to organize information and make it easier to use.”</h2>
<p><strong> ──Can you tell us a little about what you’re working on right now? </strong></p>
<p>Right now, I’m on a team focused on improving the core Mercari experience, which is the buying and selling experience. My own particular focus on that team is improving the buying experience.</p>
<p>When you think about customers purchasing items, and how exactly they find an item they want, the most common way is that they search for it. Searching is a process where a customer comes up with an idea of something they want, so they look for the item on Mercari, find it, and then they buy it.</p>
<p>Beyond searches, though, we’re also working to improve the “discovery experience.” This involves finding new purchasing needs or finding items based on purchasing needs without making any special effort to find it—in other words, without needing to search. You might open Mercari, and items you’re interested in or thinking about buying are right there. Or, while you’re searching for one item, you find another item you’re interested in, and you buy that one. That sort of thing is how we define the “discovery experience.”</p>
<p>Recommendations are a powerful tool in improving the discovery experience.<br />
Mercari currently gives customers recommendations by controlling three units. One is “content,” which is about introducing items of interest. Then there are “topics,” AKA search criteria, which are item groupings that can be interpreted as such. Finally, there are “items,” which are the items customers actually purchase.</p>
<p><img src="https://storage.googleapis.com/prd-ai-asset/2022/12/219d7b80-image_1-1024x683.jpg" alt="" /></p>
<p><center></p>
<p>Tomohiro Furusawa (@furufuru)<br />
Furusawa has consistently been involved in recommendations since joining Mercari in 2019</p>
<p></center></p>
<p>When I was in college, I worked on library and information science. Through academics I studied how to collect and store large amounts of knowledge and information and how to make it easily accessible to a wide range of people. Looking back at my career since my days as a student, I feel like I’ve always been involved in figuring out how to organize large amounts of information and make it easier to use.</p>
<p>When I was a student, I thought about a career as a librarian, but when I started interning as an engineer and data scientist, I found myself really interested in manufacturing and data usage. So, after I graduated, I decided to become a data scientist at Recruit. Recruit, which is where I worked before Mercari, has a lot of contact points with customers through all sorts of life events, so it was interesting just to look at the data. That said, I was also almost painfully aware that data is meaningless just by itself, and that its real significance lies in how it’s used. That’s when I started digging deeper into the technology of searches and recommendations.</p>
<p><strong> What’s the difference between a “search” and a “recommendation?”</strong></p>
<p>When we want a certain type of information, we use keywords and other things to ask questions through some sort of app, right? Then we look at all the results and get the information we want. That series of actions is what we call a “search,” and a search system is the mechanism that makes that experience possible.<br />
However, if we utilize customer data we already have, we can provide information from the service side without customers needing to enter any kind of keywords or other information. A system that can provide information without a customer having to actively ask questions is a recommendation system. A “recommendation” is different from a “search” in that it refers not to an experience, but rather to a means used by the service provider.<br />
Building good search and recommendation systems makes the service easier to use, and I think it ultimately bring us closer to the world I want to create, which is a world where people from all walks of life can access knowledge and information with ease.</p>
<h2>Improving the recommendation system while shifting from position to position</h2>
<p><strong>──Why were you interested in a job at Mercari?</strong></p>
<p>It started when a former colleague of mine was working on improving the search experience at Mercari and invited me to join. Mercari’s user base was growing fast, and the company had a lot of momentum, so I thought changing jobs would give me the chance to dig deeper into search technologies that use a wide range of data.</p>
<p>After joining Mercari, I started by evaluating the existing search system. All services—not just search functions—of course need to be built first, but they also then need to be improved through evaluation cycles.</p>
<p>With our service, a search is considered to have gone well when a series of experiences is born—for example, when you enter a keyword for something you want, you find the perfect item, and then make your purchase. On the flip side, a search doesn’t go well when, say, a customer is looking for an item but can’t find it using keywords, or a lot of items with little relevance show up in the search results.</p>
<p>Another issue unique to Mercari is that the search results sometimes show only items in good condition that can be purchased right away, which means the customer can’t compare and consider other items as well. The Search Team was using trial and error, repeatedly defining and evaluating what makes a search a success or a failure, to figure how to improve the search process.</p>
<p><img src="https://storage.googleapis.com/prd-ai-asset/2022/12/c184be5d-image_2-1024x683.jpg" alt="" /></p>
<p>We established Mercari’s Recommendation Team after that. I’d actually been quietly working on a prototype of the recommendation system before the team was formed, but the company also decided to focus on the recommendation system, so I ended up supporting the establishment of the team as a PM. After a while, Aki Saarinen (@akis) joined the team as PM and I shifted back to being an engineer.</p>
<p>Moving through the development process as an engineer, I found that a good development system hadn’t been created at the team level, so I acted as scrum master to fix the group’s development system while handling my engineer duties as well. After a while, things came together as an organization and the Recommendation Team started producing results, but there was also a shortage of PMs at the time, so I went back to being a PM again. So, although I’ve always been involved in the recommendation system, my job title has gone from PM, to engineer, to scrum master, and then back again.</p>
<h2>Thoughts on getting recommendation features to contribute to GMV</h2>
<p><strong>──Have you hit any walls in your career so far?</strong></p>
<p>Actually, it took some time before the Recommendation Team was able to successfully contribute to sales. Before the current system, items on the home screen were recommended to customers based on their past activity, but appearance wise, it was just a timeline listing one item after another. From there, we increased the types of components in the recommendation system to display recommended keywords and categories based on a history of customers’ recent activity. That was when we first saw a big boost in sales. Instead of just recommending items, we also display why an item is recommended and define functions for components that allow for deeper and easier exploration, and I think that has made things easier to understand for our customers.</p>
<p><img src="https://storage.googleapis.com/prd-ai-asset/2022/12/5fb7f5d4-image_3-1024x679.png" alt="" /></p>
<p><center><br />
メルカリのホーム画面の変遷<br />
</center></p>
<p><strong>Related Article：<a href="https://mercan.mercari.com/articles/27899/">No Two Home Screens are Alike – The Reasons Behind Mercari’s “Uniquely Difficult” Personalization | mercan ＊This article is only available in Japanese</a> </strong></p>
<p>The home screen went through big changes in terms of UI appearance during this period, but we also revamped background systems at the same time. To handle the labor-intensive process of developing home screen functions in a short period of time, we reworked most of the content displayed by the client into a form we can manipulate on the back end. We had always thought it would be great if the system could be modularized so that we could just change the back end to switch between home screens, but it took until around 2020 before we could actually develop and deploy it.</p>
<p><strong>──Are there any other things you want to try or challenge yourself with at Mercari in the future?</strong></p>
<p>The recommendation system is still using only relatively simple logic, so I’d like to try applying a broader range of technologies to solve advanced problems. We still need to address a lot of things, like developing data utilization infrastructure for the recommendation system. If you’re someone who loves approaching things from a variety of logical perspectives, definitely get in touch with us!</p>
<p>Mercari is a high-traffic service used by a huge number of customers, so someone with experience in operating a large-scale service will probably have an easier time grasping the essential points intuitively. That said, even if there are some things they don’t know, we love working with people who enjoy meeting challenges head-on!</p>
<p><img src="https://storage.googleapis.com/prd-ai-asset/2022/12/3e6f9193-image_4-1024x683.jpg" alt="" /></p>
<p><a href="https://ai.mercari.com/en/articles/ai/furufuru_interview_2/">In the second part of the interview</a>, Tomohiro talks about the things he focused on while working at Mercari, and Mercari’s unique organizational culture, as someone with experience in a range of positions, including PM, engineer, and scrum master.</p>
<ul>
<li>Photo : Shun Nakayama (<a href="https://shunnakayama.myportfolio.com/">Web</a>/<a href="https://twitter.com/shunnakayama_">Twitter</a>)</li>
<li>Text : Ayumi Iga (<a href="https://twitter.com/aymmm_i">Twitter</a>)</li>
</ul>


The Past and Future of Mercari’s Discovery Experience, from the PM who Launched the Recommendation Team

Blogs by mercari AI team, delivering information about AI technology and organization

Mercari ML Engineers Teo Narboneta Zosa and Ryan Ginstrom will give a talk at Berlin Buzzwords 2023

Mercari AI’s research “Want robust explanations? Get smoother predictions first.” accepted to CIKM 2022’s workshop

Mercari AI’s research “Estimating the impact of coupon non-usage” accepted to ECML/PKDD 2022’s workshop

Mercari AI’s collaborative research paper “Counterfactual Learning with General Data-generating Policies” accepted to AAAI 2023

The Past and Future of Mercari’s Discovery Experience, from the PM who Launched the Recommendation Team

Mercari ML Engineers Teo Narboneta Zosa and Ryan Ginstrom will give a talk at Berlin Buzzwords 2023

Mercari AI’s research “Want robust explanations? Get smoother predictions first.” accepted to CIKM 2022’s workshop

Mercari AI’s research “Estimating the impact of coupon non-usage” accepted to ECML/PKDD 2022’s workshop

Mercari AI’s collaborative research paper “Counterfactual Learning with General Data-generating Policies” accepted to AAAI 2023

The Past and Future of Mercari’s Discovery Experience, from the PM who Launched the Recommendation Team

Mercari ML Engineers Teo Narboneta Zosa and Ryan Ginstrom will give a talk at Berlin Buzzwords 2023

Mercari AI’s research “Want robust explanations? Get smoother predictions first.” accepted to CIKM 2022’s workshop

Mercari AI’s research “Estimating the impact of coupon non-usage” accepted to ECML/PKDD 2022’s workshop

Mercari AI’s collaborative research paper “Counterfactual Learning with General Data-generating Policies” accepted to AAAI 2023

The Past and Future of Mercari’s Discovery Experience, from the PM who Launched the Recommendation Team

Latest Posts

Improving Item Recommendation Accuracy Using Collaborative Filtering and Vector Search Engine

Mercari ML Engineers Teo Narboneta Zosa and Ryan Ginstrom will give a talk at Berlin Buzzwords 2023

Model management for client side ML powered by Firebase

Mercari AI’s research “Want robust explanations? Get smoother predictions first.” accepted to CIKM 2022’s workshop

Mercari AI’s research “Estimating the impact of coupon non-usage” accepted to ECML/PKDD 2022’s workshop

Mercari AI’s collaborative research paper “Counterfactual Learning with General Data-generating Policies” accepted to AAAI 2023

The Journey to Machine-Learned Re-ranking

The Past and Future of Mercari’s Discovery Experience, from the PM who Launched the Recommendation Team