Interview

<p>In <a href="https://ai.mercari.com/en/articles/ai/suadd_kimuras_1/">AI Listings, Image Search… CEO Shintaro Yamada and Director Shunya Kimura Look Back at Mercari’s AI Journey</a>, the two discussed the background and challenges that prompted Mercari to actively pursue the use of AI, which led to the release, starting in 2017, of a variety of features powered by AI technology, including AI listings, personalized timelines, and image search. In this article, the two discuss AI ethics and a concrete path forward to achieving Mercari's vision of a recycling-oriented society.</p>
<h2>How Mercari arrived at &quot;recycling-oriented society” as its watchword</h2>
<p>Shunya Kimura: When did Mercari start using &quot;recycling-oriented society&quot; as a watchword?</p>
<p>Shintaro Yamada: Probably starting around September 2019. Previously, we had a sense of the issue as an organization, but we started our internal discussion on ESG in earnest around January 2019. I felt that a recycling-oriented society was an issue that Mercari should confront head-on. First, we identified the materiality of Mercari to identify the issues that needed to be addressed, and then we published our first sustainability report.</p>
<p>Mercari currently has more than 20 million customers per month. In the future, we hope that our service will be used by 30 million people, 40 million people, and so on. To get there, I believe we should move beyond just &quot;convenient to have&quot; to become an indispensable part of society. In order to achieve this goal, we’ve had discussions with experts since last year to define our platform philosophy and the basic principles of our marketplace. I can now see a path forward for Mercari to fulfill its responsibility to contribute to a recycling-oriented society, which will also lead to our business growth.</p>
<p><img src="https://storage.googleapis.com/prd-ai-asset/2022/02/4d2363c1-image1-1024x683.jpg" alt="" /></p>
<p>Over the last few years, interest in the natural environment has grown rapidly, partly due to the influence of the coronavirus pandemic. Global warming and massive generation of waste are also major problems. Our current wasteful use of resources is a problem that should be dealt with by society as a whole. I want to see a world where goods are recycled and used for as long as they are usable. We’re always thinking about how Mercari can become an indispensable part of such a recycling-oriented society. Thinking about how Mercari's services can deliver value to society today, I think “contributing to a recycling-oriented society” is a particularly important slogan.</p>
<h2>What friction needs to be eliminated to close the 145 trillion yen gap?</h2>
<p>Kimura: In terms of accelerating the reuse of goods, I think that Mercari is contributing to some extent. What do you think?</p>
<p>Yamada: We're still a long way off. Total gross sales in Japan, including non-reusable goods such as food, are valued at about 146 trillion yen per year in retail alone. The total transaction value of Mercari is 1 trillion yen, so we have not even reached 1%. As this difference shows, I believe there are still a lot of goods that are discarded or stashed without being listed on the second-hand market.</p>
<p><img src="https://storage.googleapis.com/prd-ai-asset/2022/02/1b4220f5-image5-1024x683.jpg" alt="" /></p>
<p>Yamada: For example, if everything you have at home has been converted into data and you can complete your listing in an instant, it will be even easier to sell on Mercari. Right now, you still need to think a lot and put in a lot of effort to make a listing. There are still complicated aspects such as shipping methods. If we can remove that friction from the current situation, we’ll get more listings. Merpay makes the exchange of money even more seamless. I think we can make it a service that further circulates goods, where people think, &quot;I'm going to buy this, so I might as well try listing it.”</p>
<p>Right now, we have to actively buy and sell, but I think this is where AI can be very powerful. It can make suggestions like, “Since you’re buying this, why not put this up for sale?” If we can promote the effective use of resources through Mercari, we can accelerate progress towards a more recycling-oriented society.</p>
<h2>The second-hand market alone is not the answer: partnering with retailers of new merchandise</h2>
<p>Kimura: It’s quite interesting how AI can contribute to a recycling-oriented society. Mercari customers search for goods they want, but they don't actively look for goods they might potentially need, so Mercari uses AI to suggest such goods to them. Similarly, by showing the demand for goods the customer already has, you can create opportunities for listings. Mercari is CtoC but is expanding its domain to include BtoC with the launch of its online store service, Mercari Shops. It is now becoming a huge marketplace, more than just a flea market app between individuals. The cycling of goods will break down unless both the selling side and the buying side accelerate together. Matching up the numbers is also important in our business strategy. In order to contribute to a recycling-oriented society, we also need the cooperation of retailers of new merchandise.</p>
<p><img src="https://storage.googleapis.com/prd-ai-asset/2022/02/569af45c-image3-1024x683.jpg" alt="" /></p>
<p>Yamada: I agree. To meet this challenge going forward, I believe it’s not enough to simply develop the second-hand market. We’re also working on an approach that communicates data on what’s tenable in the second-hand market to retailers of new merchandise. I believe that circulating data as well as goods can gradually move us closer to a world where manufacturers produce the products they need in the quantities they need and customers get as much use out of them as possible.</p>
<p>Currently, we’re gradually starting to establish data links with manufacturers. We’re not quite there yet, but we’re starting to see some areas where we can get some value. If you restrict yourself to the second-hand market only, the task of converting all the goods around you into data becomes much more difficult. Instead of trying to do everything in our own domain, we at Mercari need to connect more with retailers of new merchandise, including catalog (SKU) linking, while keeping data privacy in mind.</p>
<h2>What are the AI ethics? Ethical issues at Mercari</h2>
<p>Kimura: We already discussed the importance of AI as a technology for the future development of Mercari. However, I believe it’s equally important to discuss the topic of how to handle AI in an ethical manner. For example, with regard to the recommendation function, you shouldn't handle data in a way that violates customers’ privacy or make recommendations they find objectionable. Recommend what they need, but don't force unwanted information on them.</p>
<p>Yamada: Yes, it's important not to force unwanted information on them. Another thing I would like to be even more conscious of in the future is whether what we are doing will benefit humanity. Even for goods the customer needs, Mercari should have higher ethical standards because Mercari also handles the money part with Merpay. That’s why, for example, you shouldn’t be in a situation where you have to go into debt to purchase a product beyond your ability to repay. Mercari is a platform for buying and selling, so you might think that ethical issues would be less likely to arise than in advertising or social media. However, we want to avoid creating a system that makes people buy more and more for the sake of business growth. In that sense, AI development in Mercari is a place that offers an interesting experience. You’re likely to encounter ethical challenges that no one has ever thought of in the past.</p>
<p><img src="https://storage.googleapis.com/prd-ai-asset/2022/02/f72b46c6-image2-1024x683.jpg" alt="" /></p>
<p>Kimura: Recently, when we were having an internal discussion about the ethics of AI, we talked about the possibility of sensitive issues that could cause discomfort and mistrust among customers when we predict products they may want to buy next. For example, they may have hobbies they don't want others to know about, or they may feel the recommendations interfere with their privacy.</p>
<p>Yamada: Yes, you need a balance between serendipity and recommendations, otherwise it can feel unpleasant. Because your activities on Mercari are not the only thing you do, it’s possible that you go camping on a daily basis, but don’t buy or sell camping goods on Mercari. We’re required to have a sense of ethics as to what kinds of products we recommend to which customers. That's the challenge we face going forward.</p>
<h2>Data can reveal the intrinsic value: What's interesting about Mercari's AI and the path forward</h2>
<p>Kimura: In order to achieve a recycling-oriented society, we’re working toward the goal of using AI to estimate supply and demand and deliver what is needed, in the amount needed. But to be honest, we still have a ways to go. We need to accelerate our AI modeling and personalization experiments, which we will continue to work on going forward.</p>
<p>Yamada: There's a lot to do, and there's a lot to try. We don't have as many AI/ML engineers as Big Tech. On the other hand, the data handled by Mercari includes a lot of information that is close to the customer, such as customer-generated product images, captions, and prices. Since there are few companies and services that have this kind of data, I believe there is potential for innovation.</p>
<p>Kimura: The type and nature of data is different from regular e-commerce. Understanding the intrinsic value of things is a very interesting part of the AI at Mercari.</p>
<p><img src="https://storage.googleapis.com/prd-ai-asset/2022/02/a8b8d5d0-image4-1024x683.jpg" alt="" /></p>
<p>Yamada: And it also includes data from Merpay and Mercari Shops. Now that the data infrastructure has been established, we can say that the timing is right for a variety of interesting challenges. We need to think creatively about what we can do with the Mercari data and Mercari services.</p>
<p>At the same time, we’re an organization that takes a medium- to long-term view and can tolerate failure. For example, we can work over long spans of time to regenerate the data and build a data infrastructure. And in terms of human resources, we’ve hired many new graduates from around the world, including the Indian Institute of Technology (IIT). I think this kind of environment is a natural climate for nurturing ML engineers. What these things have in common is they don’t produce immediate results. It's about thinking in the medium to long term. That's why I want to hunker down and get to work on all of these issues, including AI ethics.</p>
<p>Mercari's use of AI began in 2017. In addition to improving the UI/UX, AI has contributed to the development of features that make Mercari safer and more secure.<br />
Related article: <a href="https://ai.mercari.com/en/articles/ai/suadd_kimuras_1/">AI Listings, Image Search… CEO Shintaro Yamada and Director Shunya Kimura Look Back at Mercari's AI Journey</a></p>
<p>In the future, we will continue to seek new innovations with the aim of contributing to a recycling-oriented society through AI. Mercari's AI team will continue to take on new challenges.</p>
<p>Photo : Shun Nakayama (<a href="https://shunnakayama.myportfolio.com/">Web</a>/<a href="https://twitter.com/shunnakayama_">Twitter</a>)<br />
Text : Ayumi Iga (<a href="https://twitter.com/aymmm_i">Twitter</a>)</p>


Contributing to a Circular Economy : CEO Shintaro Yamada and AI Director Shunya Kimura Discuss the Challenges that Mercari’s AI Technology Needs to Overcome

<p>During 2017-2018, Mercari released a variety of features powered by AI technology, including AI listings, personalized timelines, and image search. These are still core features of Mercari. What was the process leading up to that? Shintaro Yamada, Representative Director &amp; CEO, and Shunya Kimura, Director of AI &amp; Search, look back at the history of Mercari's active use of AI, the challenges they faced along the way, and the organizational transformation that resulted.</p>
<h2>Mercari has always had a data-driven decision-making culture</h2>
<p><img src="https://storage.googleapis.com/prd-ai-asset/2022/01/6dd58f3a-image1-1024x683.jpg" alt="" /></p>
<div style="text-align: center;">Shunya Kimura, Director of AI & Search </div>
<p>Shunya Kimura: At Mercari, we’ve been focusing on the use of AI for some time now. When did you decide to make AI the focus?</p>
<p>Shintaro Yamada: Mercari was originally a company with many employees who had worked in game development. I was also at a company called Zynga, and when I founded Mercari, we had many employees who joined us from DeNA and mixi in particular. From the days when we were creating game services for feature phones through the transition to smartphone culture, we all felt the importance of continuing to be data-driven. That's why we've been doing A/B testing and data-driven decision-making and product development.</p>
<p>Kimura: Was that inspired by your own first-hand experience?</p>
<p>Yamada: When I was at Zynga, we actively used A/B testing to select button colors and try out new game systems in the course of system development. By implementing the PDCA (Plan-Do-Check-Act) cycle, we were able to see the impact on our business, such as improved customer retention and increased sales. When we released Mercari in 2013, we didn't even have BigQuery yet, but we were doing data enhancement from the very beginning. Utilizing those systems, we saw ways to utilize more value-added data, including AI and machine learning.</p>
<p>Kimura: When did that happen?</p>
<p>Yamada: I remember it was around 2015-2016 that the world began to take notice of the active use of deep learning in AI. We asked ourselves if we might be able to use AI to develop a variety of features. We started development, and in 2017, we started developing AI listings and other products that incorporate machine learning.</p>
<h2>The catalyst for the use of AI was witnessing the use of data by Chinese companies</h2>
<p>Kimura: What inspired you to actively promote the use of AI? Was it your own sense the power of AI?</p>
<p>Yamada: Looking back at the developments within the company, the flow of events was as I just described, but personally, I was really struck by how the various services in China were using AI. At the time, Merpay was in its start-up phase, and we began getting many reports within the company from our employees who went to research FinTech services not only in China but also other countries. In the summer of 2017, I visited China with Aoyagi (Naoki Aoyagi, CEO of Merpay and Mercoin) before he joined the company.</p>
<p><img src="https://storage.googleapis.com/prd-ai-asset/2022/01/1581af58-image5-1024x683.jpg" alt="" /><br />
<strong></p>
<div style="text-align: center;">Shintaro Yamada, Representative Director &amp; CEO of Mercari</div>
<p></strong></p>
<p>ByteDance, which would later become famous in Japan for TikTok, developed a personalized news service using deep learning called TouTiao. It's a service similar to SmartNews in Japan. Meituan developed Dianping, a gourmet food site similar to Japan's Tabelog site, and then expanded into food delivery and ride sharing. In each case, they progressively applied AI. Baidu's search engine was also amazing.</p>
<p>What they had in common was they collected data from users, placed that data at the center of their business strategy, and developed their services by thinking about how they could use it and what they could do with it. In the case of Meituan, they had just entered the ridesharing market using their data logged from food delivery bikes and other data.</p>
<p>At the time, seeing this company that was gaining momentum in China that was a data company making good use of data, it made me realize that it was possible to develop a range of businesses in this way. And looking at WeChatPay, Alipay, and so on, I could see that there was a new form of finance ahead.</p>
<p>Kimura: Indeed, the use of data by Chinese companies back in 2017 was quite a shock.</p>
<h2>Start of implementation of core Mercari features “personalized timeline” and “AI listings”</h2>
<p>Kimura: The first step in the use of AI at Mercari was the development of the &quot;AI listing&quot; feature. This is a feature that, when you take a picture of a product, suggests information about it and even estimates the price. It made an impact on the world, and some people still find this feature so useful that they say it’s one of the best things about Mercari. How did you feel about it at the time?</p>
<p>Yamada: The first thing that struck me was simply how far AI had come. It was a way to take a picture of a product with your phone, identify the products and brand, and even estimate the price. When I heard an overview of the feature, I honestly thought that we would not be able to achieve that level of accuracy. However, we took suggestions from the product side and proceeded with the implementation. When I tried out the actual feature, I was really surprised because the accuracy was higher than I expected. The image recognition technology is implemented correctly and incorporated into the feature in a way that is consistent with the Mercari experience. In fact, the completion rate from the time the listing screen is displayed to the time the listing is completed has also clearly improved. After this series of experiences, I felt the need to invest more in technology in the future.</p>
<p><iframe loading="lazy" width="560" height="315" src="https://www.youtube.com/embed/NHxc2m3SEL8?controls=0" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe></p>
<p><strong></p>
<div style="text-align: center;">Video introducing the AI listing feature</div>
<p></strong></p>
<p>After that, we implemented a “personalized timeline” feature that personalizes the order of products displayed on your timeline. At the time, Mercari just displayed all the product categories in order of recency, but we released a feature that estimates categories of interest from the customer's browsing history and puts them on the timeline. This feature was very effective and had a significant impact in terms of total transaction value.</p>
<p>Kimura: The great thing about AI is that it can make suggestions that are one step ahead of, or at an angle to, what you might want to buy.</p>
<p>Yamada: I think what's nice about AI is that you don't need to manually construct a rule-based display order. The system is able to handle and process a wide range of data and automatically derive the most relevant output. How do you change the numbers in a particular product category for a particular customer segment? How do you adjust when the response is not good? The beauty of AI is that it can do that kind of complicated automation. I'm also inspired by the personalized services that TouTiao and Meituan have come up with.</p>
<p><img src="https://storage.googleapis.com/prd-ai-asset/2022/01/2f7d3cdf-image4-1024x683.jpg" alt="" /></p>
<h2>In a rapidly expanding organization, don’t try to march in lockstep, just create synergies where you need them</h2>
<p>Yamada: When you first joined the company, what was the organizational situation like?</p>
<p>Kimura: In July 2017, when I joined the company, there weren’t as many engineers as there are now. We didn't even have an AI team yet. AI-driven development is now done by ML engineers, but back then it was done by back-end engineers. I remember that the AI team was formed in early 2018. We assigned AI engineers to various projects such as violation detection, AI listing, photo search, price suggestion, and for the US version of Mercari, weight estimation and shipping method recommendation.</p>
<p>At the time, we had created Merpay and were growing as an organization, and we kept the projects separate. Our ML engineers were split into different organizations such as Merpay and US version of Mercari. Under this arrangement, we didn’t try to align our AI development efforts, but proceeded with development in each individual organization. Rather than working too closely together, which slows down the development process, each organization works independently, loosely connected to the others. We emphasize speed while creating synergies where we need them. In this context, where Mercari's services are diversifying and globalizing, each organization is doing its own development using AI. I find that really interesting.</p>
<p><img src="https://storage.googleapis.com/prd-ai-asset/2022/01/3caec9fc-image3-1024x683.jpg" alt="" /></p>
<h2>The data management project that became a major task for advancing the use of AI</h2>
<p>Yamada: The headcount has increased considerably since then. We now have several hundred engineers. There was a time when the A/B testing framework was full and we started to have conflicts. From that time on, I felt that automation of A/B tests was essential. I also felt it was extremely important to have the data acquisition system organized before reaching that phase.</p>
<p>At the time in 2019, Mercari had already grown into a service used by many customers, but we had challenges such as a lack of consistency in how data is obtained and where data is stored. After that, we started a project to revamp how we obtain data.</p>
<p>Kimura: We first tackled the development of a proper data infrastructure around 2019. As background, amid the acceleration of data-driven decision-making and rising expectations for AI development, the need to improve the accuracy and timeliness of data was growing across the company. In order to strengthen AI utilization and data-driven decision-making, it became necessary to reorganize the structure of the data Mercari was obtaining and the ways we were obtaining it.</p>
<p><img src="https://storage.googleapis.com/prd-ai-asset/2022/01/222f69c7-image6-1024x683.jpg" alt="" /></p>
<p>At the time, the system was designed to make data storage quite free and easy, because we were focusing on flexibility in analysis. As a result, we ended up with large amounts of data stored in databases, making data maintenance and management increasingly difficult. There is also a debate going on about data management, with each stakeholder having their own opinions from a legal, engineering, and analytics perspective. Meanwhile, the development of Merpay started, and the situation became quite complicated.</p>
<p>This is still a problem that needs to be tackled company-wide, and there are a large number of stakeholders. Jeff, the incoming CEO of Mercari Japan, organized the situation, moved forward with the implementation, and gained company-wide consensus. Since then, we’ve unified how data is obtained and where data is stored, and we now have a more robust data platform that can store data in real time. Furthermore, a project to improve data governance is currently underway company-wide.</p>
<p><img src="https://storage.googleapis.com/prd-ai-asset/2022/01/a7895e3d-image2-1024x683.jpg" alt="" /></p>
<p>Related article:<br />
<a href="https://ai.mercari.com/en/articles/ai/suadd_kimuras_2/">Contributing to a Circular Economy : CEO Shintaro Yamada and AI Director Shunya Kimura Discuss the Challenges that Mercari’s AI Technology Needs to Overcome</a></p>
<p>In this article, the two discuss Mercari's vision of AI ethics and a concrete path forward to achieving a recycling-oriented society.</p>
<p>Photo : Shun Nakayama (<a href="https://shunnakayama.myportfolio.com/">Web</a>/<a href="https://twitter.com/shunnakayama_">Twitter</a>)<br />
Text : Ayumi Iga (<a href="https://twitter.com/aymmm_i">Twitter</a>)</p>


AI Listings, Image Search… CEO Shintaro Yamada and Director Shunya Kimura Look Back at Mercari’s AI Journey

Blog

<p>The &quot;MERCARI GEARS Lecture Series&quot; is a video series for engineers that introduces the direction that Mercari Inc. and other companies in the Mercari Group are heading and the technical challenges they will tackle in the future.</p>
<p>In this edition of MERCARI GEARS, Mercari's Customer Relationship Management (CRM) ML team* introduces how machine learning and causal inference can be used to provide a better marketing experience and improve campaign effectiveness.</p>
<p>*Now renamed the Marketing Data Science team</p>
<p><center><br />
<iframe loading="lazy" width="560" height="315" src="https://www.youtube.com/embed/g1r67WFVKUU" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe></iframe><br />
Optimizing Marketing Campaigns using Artificial Intelligence [AI@Mercari #1]</center></p>
<p>The efficiency of Mercari's coupon distribution was greatly improved by incorporating machine learning models into the segmentation definition of coupon distribution. Mercari's AI team is putting AI into practice to create a better customer experience.</p>
<p>Related projects:.<br />
<a href="https://ai.mercari.com/en/projects/eaf0a7f389/">Applying Uplift Modeling to Increase Customer Satisfaction and Streamline Marketing Efforts</a></p>


AI Application to CRM – Providing Better Marketing Experience and Improving Campaign Effectiveness

<p>MERCARI GEARS Lecture Series&quot; is a video series for engineers that introduces the future direction and technical challenges that Mercari Inc. and other companies in the Mercari Group will tackle in the future.</p>
<p>In this edition of MERCARI GEARS, we focus on the moderation system developed by Mercari's TnS team (Trust and Safety team).</p>
<p><center><br />
<iframe loading="lazy" width="560" height="315" src="https://www.youtube.com/embed/STmIHmDCnfI" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe></iframe><br />
Violation Detection Using Machine Learning at Mercari [AI@Mercari].<br />
</center></p>
<p>The moderation system is a system used by the Customer Success (CS) team to provide a trust and safety marketplace, and uses machine learning to detect items that may violate market policies.</p>
<p>To achieve a safer marketplace, we have devised a system architecture for the moderation system, separate models for each market policy, and request processing in coordination with the CS, which is explained in the video.</p>
<p>Related projects :<br />
<a href="https://ai.mercari.com/en/projects/auto-content-moderation-for-trust-safety/" title="Auto Content Moderation for Trust &amp; Safety">Auto Content Moderation for Trust &amp; Safety</a></p>


For a Trust and Safety Marketplace, Machine Learning for Violation Detection

<p>Creating labels for machine learning is hard and time consuming, wouldn&#8217;t it be nice to train a model without any? We can with self-supervised learning!</p>
<p>I&#8217;m <a href="https://www.linkedin.com/in/paul-willot-903a6b5b/">Paul Willot</a> from the AI/ML team and in this post I&#8217;ll introduce an experiment we did to get rid of labels by leveraging the structure of Mercari&#8217;s data.</p>
<p>Let&#8217;s dive in!</p>
<h2>Self-supervised learning</h2>
<p>Self-supervised learning, or SSL for short, is a training method free of external supervision. Instead labels are generated from a <b>proxy task</b>. Solving this task will require the model to <b>learn a representation, usable for downstream applications</b>.</p>
<p>The difficulty is of course to define that proxy task! But before showing how to create one let&#8217;s see why SSL might be useful in the first place.</p>
<h4>Why SSL?</h4>
<p>In short: <b>collecting good labels is hard</b>, but raw data is plentiful. SSL allows us to use a large dataset to bootstrap a model, then fine-tune on only a few high quality samples.</p>
<p>Let&#8217;s make a concrete example.</p>
<p>As a Mercari user, when selling an item I can select the category that best describes it, making my item easier to find for buyers. However, picking the correct category is cumbersome.  Machine Learning (ML) can help by suggesting likely categories directly from the item pictures. We can define this problem as a simple classification objective, a common ML task. Good!</p>
<p>However the existing categories, our labels, might have a few issues. Notably </b>class imbalance</b>, <b>label noise</b> (picking among many is hard) and <b>ambiguity</b> as some items straddle categories. The model will also learn a <b>task specific representation</b>, which might be hard to transfer to other tasks.</p>
<p>Those issues come from the need for labels, so not needing any would be great. That&#8217;s where self-supervised learning can help!</p>
<h2>Defining a proxy task</h2>
<p>Various proxy tasks have been crafted for SSL, like solving <a href="https://arxiv.org/abs/1603.09246">jigsaw puzzles</a> or <a href="https://arxiv.org/abs/1803.07728">predicting an image rotation</a>. Recent state of the art methods have focused on simpler proxy, like randomly augmented images, in which the task is defined as obtaining consistent representations despite visual deformations.<br />
For example, in the image of a golden retriever below, after alterations each transformed image (called a view) still contains a recognizable part of the dog.</p>
<p><img src="https://storage.googleapis.com/prd-engineering-asset/2021/10/0ca9bd87-augmented_views.png" alt="augmented views" /></p>
<p style="text-align: center;font-style: italic">Augmented views</p>
</p>
<p>Choosing which augmentations to apply is not easy however. We need to make it hard enough for the model to learn a useful representation, but not too hard that it becomes impossible. Augmentations add design choices and picking the right ones is an <a href="https://link.springer.com/article/10.1186/s40537-019-0197-0?code=a6ae644c-3bfc-43d9-b292-82d77d5890d5">ongoing research topic</a>.</p>
<p>Ideally, we would like to use existing views.</p>
<p><img src="https://storage.googleapis.com/prd-engineering-asset/2021/10/bdb3912f-natural_views.png" alt="natural views" /></p>
<p style="text-align: center;font-style: italic">Natural views</p>
<p>Luckily we have billions of those at Mercari!</p>
<p>When selling, uploading pictures from multiple angles improves buyers&#8217; appreciation, so the large majority of listings have multiple views. We call them <i>natural views</i>, as they are taken to show multiple aspects of an item in a natural context.<br />
We can note that between views, more than just the camera angle might change, for example the shoes above are positioned differently on the box. This is good! We want the model to learn a common representation that is not tied to a particular pose.</p>
<p>Some sets of views might be harder to associate, for example an item and it&#8217;s enclosing box photographed separately. In practice this was not an issue for us as those are relatively rare.</p>
<h2>Contrastive learning</h2>
<p><img src="https://storage.googleapis.com/prd-engineering-asset/2021/10/58783b48-contrastive_example.png" alt="Contrastive learning" /></p>
<p style="text-align: center;font-style: italic">Contrastive learning on a pair of images (credit: <a href="https://ai.googleblog.com/2020/04/advancing-self-supervised-and-semi.html">SimCLR</a>)</p>
<p>Now with those views, we can define a learning objective through <b>contrastive learning</b>, that is: the <b>same views should produce representations that are more similar</b>, and <b>different views should produce representations that are dissimilar</b>. Visually, we can imagine representations of views of the same item being pushed closer together, and of different items being pushed apart, as the figure above illustrates.</p>
<p>Note that &quot;embedding&quot; and &quot;representation&quot; are used interchangeably in the litterature, both stand for a 1 dimensional scalar vector in a SSL context.</p>
<p>Contrastive learning is related to <a href="https://proceedings.neurips.cc/paper/1993/file/288cc0ff022877bd3df94bc9360b9c5d-Paper.pdf">siamese networks</a>, an older ML technique, but has regained popularity in the last two years with <a href="https://arxiv.org/abs/1911.05722">numerous</a> <a href="https://arxiv.org/abs/2002.05709">methods</a> narrowing the gap with supervised training on the ImageNet benchmark. This <a href="https://ai.googleblog.com/2020/04/advancing-self-supervised-and-semi.html">SimCLR blog post</a> is a good place to go deeper.</p>
<p>Introduced by the <a href="https://arxiv.org/abs/2006.07733">Bootstrap your own latent</a> paper, momentum based methods have also appeared recently. They simplify the implementation by using only positive pairs and address the issue of large batches. I recommend the <a href="https://arxiv.org/abs/2011.10566">SimSiam paper</a> to understand momentum learning.</p>
<p><img src="https://storage.googleapis.com/prd-engineering-asset/2021/10/8d036056-ssl_moby_structure.png" alt="MoBY self-supervised method" /></p>
<p style="text-align: center;font-style: italic">SSL with contrastive loss and momentum, aka MoBY (credit: <a href="https://arxiv.org/abs/2105.04553">Self-Supervised Learning with Swin Transformers</a>)</p>
<p>Latest research even combines the two, like <a href="https://arxiv.org/abs/2105.04553">MoBY</a> that we used in our experiment.</p>
<p>The key point in these methods is that we can create models that obtain a <b>consistent representation of the subject across multiple views</b>.</p>
<p><img src="https://storage.googleapis.com/prd-engineering-asset/2021/10/afdedcb6-similarity_listings_ssl.png" alt="Distance of representations within and between listings" /></p>
<p style="text-align: center;font-style: italic">Distance between representations of views within and outside listings.</br>Each quadrant is one listing, the innermost links are between listings </br>green = similar, red = dissimilar</p>
<p>We were able to confirm this in our dataset, as shown above. After training, items within a listing are more similar than across listings, even with large changes in poses or a tight zoom on an object detail.</p>
<p>Regarding training, a few tips that helped during our experiment:</p>
<ul>
<li>Large datasets are necessary for any non-trivial task; think millions of samples</li>
<li>For training, with pre-trained weights you can get by with a couple of GPU-days, otherwise a couple of weeks</li>
<li>Large models are easier to train, ResNet-50 is still a solid baseline in 2021</li>
<li>A simple method goes a long way, <a href="https://github.com/lucidrains/byol-pytorch#byol--simsiam">start with SimSiam</a></li>
<li>SSL is neat but if you already have labels, use them! It will be more accurate and faster to train</li>
</ul>
<h2>Using the learned representation</h2>
<p>We have a few ways to develop downstream applications using the representations obtained from these images</p>
<p>Using labels, we can train a </b>specialized model on top of the learned embeddings</b>, which will require fewer samples as the input space is smaller and simpler (compared to raw images).</p>
<p>We can also do a <a href="https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm">nearest neighbor search</a> by looking at the closest representations to infer the properties of a sample. The advantage is that we don&#8217;t need further training, and we can leverage any attributes of our neighbors even if poorly structured.<br />
We could compute the cosine similarity (or other <a href="https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence">divergence measure</a>) between each sample pairs directly, but at Mercari&#8217;s scale it quickly become computationally intractable. Instead we can leverage existing vector database search like <a href="https://github.com/facebookresearch/faiss">FAISS</a> or <a href="https://github.com/google-research/google-research/tree/master/scann">ScaNN</a> to search millions of samples in a few milliseconds.</p>
<h4>Conclusion</h4>
<p>Even without labels we can train a useful model by leveraging the existing structure of our data.</p>
<p>At mercari we leverage ML across the board, from <a href="https://engineering.mercari.com/en/blog/entry/pricing-guidance-system/">pricing recommendation</a>, <a href="https://engineering.mercari.com/en/blog/entry/2019-09-12-130000/">fraud detection</a>, <a href="https://engineering.mercari.com/en/blog/entry/20210702-develop-general-customer-profiling-platform-to-host-machine-learning-models/">customer profile customization</a>, <a href="https://engineering.mercari.com/blog/entry/20201202-mediapipe/">object detection at the edge</a>… and much more. Obtaining a general representation directly from images and without costly labels is a helpful tool to address those diverse tasks.</p>
<p>Thank you for reading!</p>


Natural views for self-supervised learning

<h1>Introduction</h1>
<p>Hello! My name is <a href="http://twitter.com/kentandx" title="@kentan">@kentan</a>, and I work as an Engineering Manager here at Mercari.</p>
<p>Our company’s Customer Reliability Engineering Team has developed what we call the Moderation System, which uses mechanisms like machine learning and rules to detect listings that violate Mercari’s rules. The Moderation System is systematically connected to the Mercari marketplace and has the traits of providing a web-based UI designed for use by Mercari Customer Support in their work cracking down on illegal activity. The system includes elements of frontend, backend, and machine learning technology and has features whose sophistication continues to grow.</p>
<p>Historically, at Mercari the backend, frontend, and machine learning engineering teams have worked separately according to their team and job function. This is why, when we started development on the Moderation System, we had to work together on development as a project team composed of several teams and comprising multiple job functions.</p>
<p>From the get-go, the existing four teams took part in the development of the Moderation System. We did things this way because the technical specialties of each team involved were indispensable to the Moderation System.</p>
<ul>
<li>Backend A Team</li>
<li>Backend B Team</li>
<li>Machine Learning A Team</li>
<li>Machine Learning B Team</li>
</ul>
<p><img src="https://storage.googleapis.com/prd-engineering-asset/2021/10/f6e25fcf-journey-to-a-feature-oriented-team.jpg" alt="" /><img src="https://storage.googleapis.com/prd-engineering-asset/2021/10/0193fdbe-ogp_b.jpeg" alt="" /><br />
From the time we first broke ground on the project, we easily imagined the difficulties of getting four teams to communicate. This is why, in order to keep the members participating in the project on the same page, we created cross-team milestones, and we laid out the format for communication between teams. This worked very effectively, but as one would expect, we could not ignore the problems involved in developing a feature of this size relative to the large number of teams involved. In addition, due to things like some members leaving the company, one of the teams also faced a shortage of engineers and had to restructure their team.</p>
<p>What’s more, as other members were either transferred to different teams or resigned from Mercari, the product owner roles for three teams were consolidated in the hands of a single product manager (PdM). It became necessary to reduce their workload because this one PdM was taking part in the scrum events of three teams.</p>
<h1>Team structure and ownership of microservices</h1>
<p>To solve the problems we encountered, we considered restructuring the teams, and at the same time considered keeping the same team structure and re-examining who held ownership of microservices. This is because we expected that, even if we kept the existing structure of the four teams, we would be able to solve or at least reduce our problems by keeping the development of microservices for Moderation Systems to two or three teams.</p>
<p>Through our discussions, we concluded that we should first look at restructuring our teams rather than transferring ownership. There were two reasons for this:</p>
<ul>
<li>Each team only had a few members, and therefore they could not be expected to be in charge of developing any more microservices than they already were.</li>
<li>Because the teams were divided by job function, they could not inherit a different team’s microservices (i.e., backend teams could not inherit the microservices of machine learning teams).</li>
</ul>
<p>Therefore, we decided not to touch the ownership of microservices and instead looked at restructuring our teams.</p>
<h1>Restructuring the teams</h1>
<p>In restructuring our teams, we first got to work on the areas that we thought would benefit the most from being shuffled. As a result, we ended up merging the Backend A and Machine Learning A teams. Generally speaking, we chose to merge these two teams for the following reasons:</p>
<ul>
<li>Backend A Team was in charge of features used for such things as registering, changing, and applying rules related to detecting listings that violate Mercari’s rules. At the same time, Machine Learning A Team developed and operated machine learning models that identified items that our rules stipulate cannot be listed. Although the two fields of expertise differ from an engineering standpoint, they are extremely close as business domains, and we therefore had very big expectations for the synergy generated by merging the two teams together.</li>
<li>Engineers from Machine Learning A Team had expressed their interest in expanding their areas of expertise into backend engineering. You could say that software development relies almost entirely on the ability and motivation of the talent involved, so we anticipated that our members’ motivation would be conducive to having them work effectively.</li>
<li>Backend A Team and Machine Learning A Team shared the same PdM, and we anticipated that the effect of merging the two teams would also be tremendous from the perspective of the PdM’s job.</li>
</ul>
<h2>How we merged the teams</h2>
<h3>Phase 0: Checking the motivation of our members</h3>
<p>We explained to each team member the purpose behind merging the teams. Then, when we asked them about their intentions, every last member said they could handle the challenges of taking on a new field. You could tell that they were driven. For the machine learning engineers, this would allow them to expand their expertise into the field of backend engineering, and backend engineers could also expand their expertise to include machine learning.</p>
<p>We merged the teams in four stages.</p>
<h3>Phase 1: EM learns domain knowledge</h3>
<p>As the Engineering Manager of Backend A Team, I was put in charge of the merged team as the EM. Personally, my experience was exclusively as a full-stack engineer working mainly on backend engineering, and I had absolutely no job experience working on machine learning. I didn’t have any domain knowledge regarding the development that Machine Learning A Team worked on either. For these reasons, acquiring this knowledge was a top priority.</p>
<h4>Goals</h4>
<ul>
<li>The EM understands the projects of Machine Learning A Team.</li>
<li>The EM understands Machine Learning A Team’s development processes and scrum practices.</li>
<li>The EM knows things about the Machine Learning A Team members like their personalities, skill sets, strengths and areas for improvement, motivations, and career plans.</li>
<li>The EM learns the basics of machine learning so that they can communicate with their team members.</li>
</ul>
<h4>Non-Goal</h4>
<ul>
<li>There were no goals set regarding team members.</li>
</ul>
<h4>Actions</h4>
<ul>
<li>Participated in scrum events such as sprint planning sessions and retrospectives</li>
<li>Participated in the EM meetings for the division to which ML engineers belong</li>
<li>Read past design docs and pull requests</li>
<li>Looked through the past year of Slack posts to get a general sense of what has been discussed</li>
<li>Read the onboarding documents of Machine Learning A Team (Because some materials were lacking content or out of date, we created and updated documents as needed.)</li>
<li>Created pull requests for the projects that Machine Learning A Team developed</li>
<li>Read the reports about experiments conducted when newly creating and improving ML models</li>
<li>Worked to build trust relationships through 1-on-1 sessions with team members</li>
<li>Also learned knowledge about machine learning through books, online courses, and other such sources</li>
</ul>
<h3>Phase 2: Members learn domain knowledge</h3>
<p>Next, team members of each team learned domain knowledge.</p>
<h4>Goals</h4>
<ul>
<li>Backend engineers understand the projects of Machine Translation A Team.</li>
<li>Backend engineers understand Machine Learning A Team’s development processes and scrum practices. They also learn Python, the programming language that the Machine Learning A Team uses.</li>
<li>Machine learning engineers understand the projects of Backend A Team.</li>
<li>Machine learning engineers understand Backend A Team’s development processes and scrum practices. They also learn Go, the programming language that the Backend A Team uses.</li>
</ul>
<h4>Non-Goal</h4>
<ul>
<li>Contributing to development was not one of the goals. (It was okay for the engineers to just focus on learning.)</li>
</ul>
<h4>Actions</h4>
<ul>
<li>Members created an open channel on Slack and used it for Q&amp;A and for studying machine learning and backend engineering.</li>
<li>The two teams held a joint study group and studied domain knowledge.</li>
<li>The Tech Lead of the machine learning team took part in the backend team’s scrum sprint and executed some development tasks.</li>
</ul>
<h3>Phase 3: Unify scrums</h3>
<p>After each and every member had acquired domain knowledge, the groups unified their scrum processes. The teams had brought together their sprint refinement and planning, and so the merged team operated like a single development team.</p>
<h4>Goals</h4>
<ul>
<li>Bring together the separate scrum process of the two teams.</li>
<li>Backend engineers perform development on ML projects.</li>
<li>Machine learning engineers perform development on backend projects.</li>
</ul>
<h4>Non-Goal</h4>
<ul>
<li>Backend engineers create and improve machine learning models.</li>
</ul>
<h4>Actions</h4>
<ul>
<li>As the Scrum Master, the EM led the scrum operations of the unified scrum team. At the same time, in order for other members to take over the role of Scrum Master, the EM also defined the scrum process.</li>
<li>We held a study session in order for members to understand the background of JIRA development tickets. Because JIRA development tickets simply must be high context, it is hard for new employees who join the company to look at a ticket and understand all of its contents. To help with this, we gave the details of and the reasons why a certain development ticket was necessary, and we also worked to explain the business-related reasons behind development.</li>
</ul>
<h3>Phase 4: Unify on-call rotation systems</h3>
<p>Last but not least, we adopted a single on-call system for the entire team. What this meant for our members is that they were on call half as often, which drastically reduced their workload. What’s more, allowances associated with on-call duties were cut in half. For the company, this led to improved cost efficiency on development.</p>
<h4>Goals</h4>
<ul>
<li>Backend Team members can be on call for Machine Learning Team projects.</li>
<li>Machine Learning Team members can be on call for Backend Team projects.</li>
</ul>
<h4>Non-Goal</h4>
<ul>
<li>None (on account of Phase 4 being the final phase)</li>
</ul>
<h4>Actions</h4>
<ul>
<li>With the Tech Lead providing coordination, we discussed what we needed in order for all members to feel at ease when on call. We then discussed what we needed to do for that to happen and compiled our ideas in a strategy document.</li>
<li>Members held drills that role-played situations in which given incidents occurred.</li>
<li>Members created on-call handbooks for each microservice.</li>
</ul>
<h1>Future Outlook</h1>
<p>We have started frontend development as the newly formed Feature Team. Customer support work involves a very complex group of features, and the UI is also within the purview of these duties. The percentage of frontend work that comprises Moderation System development work is substantial, and when we examined this work next to the Feature Team’s concept of executing all of the necessary development within their team, we saw a goal that we wanted them to achieve.</p>
<p>What’s more, for us to work on full-scale frontend development, we naturally must have frontend engineers with robust skills. To this end, we are hiring. Interested? If the answer is yes, be sure to <a href="http://twitter.com/kentandx" title="contact me">contact me</a>.</p>
<h1>Conclusion</h1>
<p>It’s of course no easy task to merge two teams into one when the members’ skills differ as much as the skills of backend engineers and machine learning engineers. I think one reason stood out above all other as to why it was possible for us to succeed at merging teams under these circumstances and without any issues: Our team members worked proactively to merge the teams. With regard to the unifying of on-call duties in Phase 4, all steps, from concrete planning to implementation, were centered on the Tech Lead and were completed with me having almost no involvement in the process as the EM.</p>
<p>Although generally classed as being out of the norm, I think that having an EM fill the role of scrum master was effective. To put it bluntly, when it comes to things where almost any format will work, like day-to-day standup meetings and the writing of tickets, having the EM make decisions about these things skips over a lot of fuss. For the time being, I make a decision, we put it to work, and then continue with scrum development.</p>
<p>In the process, for anything that is no longer a good fit, we discuss making improvements at the sprint retrospective as topics surface.</p>
<p>One issue that we are facing is that engineers with a background in backend engineering, myself included, have not yet learned all of the expert knowledge needed for machine learning, and therefore we cannot create machine learning models yet. Currently, we are in charge of developing systems and applications for machine learning projects, but for us to be able to create machine learning models will require us to look at applying a few more tricks.</p>


How the Machine Learning Team and Backend Team Joined Forces to Create the Feature Team

<p>Hi, I’m Yusuke Shibui, a member of the Image Search and Edge AI team in Mercari Japan. I publicized design patterns for implementing a machine learning model into a production environment. The patterns are available in GitHub as OSS, and I welcome you to take a look if you are interested!<br />
The repository is open for anybody to raise a PR or issue. If you have your own machine learning system pattern, please feel free to share your ideas with us!</p>
<ul>
<li>
<p>GitHub: <a href="https://github.com/mercari/ml-system-design-pattern">https://github.com/mercari/ml-system-design-pattern</a></p>
</li>
<li>
<p>GitHub Pages: <a href="https://mercari.github.io/ml-system-design-pattern/">https://mercari.github.io/ml-system-design-pattern/</a></p>
</li>
</ul>
<h2>Why do we need design patterns for machine learning systems?</h2>
<p>For machine learning models to demonstrate their true value, they must be implemented and used in a production service or internal system. In order to do so, we design a workflow and operation for machine learning as well as related systems to develop an integrated system, but this also uses skills other than machine learning. For instance, while data retrieval, preprocessing, inference, postprocessing, and training pipeline are used from a machine learning perspective, we also need UI/UX design, backend engineering, quality assurance, monitoring and alerts from a software engineering and DevOps perspective.<br />
Even if a machine learning engineer or researcher takes great effort to develop a model to achieve the highest accuracy on the ImageNet dataset, they probably cannot acquire software engineering skills. Some of them may not need those skills in their career. On the other hand, software engineers, including application developers, backend engineers and SREs, might not be familiar with machine learning. Those having both machine learning and software engineering skills or experience are rare.</p>
<p><figure class="figure-image figure-image-fotolife" title="Example of overall system design of `Cat or Dog` application"><span itemscope itemtype="http://schema.org/Photograph"><img src="https://storage.googleapis.com/prd-hatenaa-engineering-asset/20200501174859.png" alt="f:id:shibui-yusuke:20200501174859p:plain" title="f:id:shibui-yusuke:20200501174859p:plain" class="hatena-fotolife" itemprop="image"></span><figcaption>Example of overall system design of `Cat or Dog` application</figcaption></figure>
</p>
<p>The machine learning design patterns aim to fill in the gaps between machine learning engineers and software engineers to integrate the machine learning lifecycle and software engineering manners. One of the goals of developing a system with machine learning is to make a business impact with the model in production. This requires understanding of machine learning specific technologies and implementing them into a business system in an operable structure. The aim of the machine learning design patterns is to organize patterns of practices for “making a machine learning model and workflow into a production system”.</p>
<p>There is a famous diagram in a paper, <a href="https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf">“Hidden Technical Debt in Machine Learning Systems”</a>, that depicts components required to run a core machine learning model. The core is located in the center as a small black square, and is surrounded by various systems.</p>
<p><figure class="figure-image figure-image-fotolife" title="Hidden Technical Debt in Machine Learning Systems"><span itemscope itemtype="http://schema.org/Photograph"><img src="https://storage.googleapis.com/prd-hatenaa-engineering-asset/20200422172657.png" alt="f:id:shibui-yusuke:20200422172657p:plain" title="f:id:shibui-yusuke:20200422172657p:plain" class="hatena-fotolife" itemprop="image"></span><figcaption>Hidden Technical Debt in Machine Learning Systems</figcaption></figure>
</p>
<p>The paper emphasizes that in order to use machine learning effectively in production, we need to develop and operate non-machine learning components. It does not say machine learning is unimportant, but to make machine learning productive in a system, we need many other resources. We have to implement machine learning logic into a web service or system in order to make our business succeed with machine learning. Note that all the systems require business evaluation and system operation. For a plain system, we certainly utilize an infrastructure and resource management, and may optionally use monitoring, alerts and backups. For a machine learning system, we need a prediction server as well as an interface and connection to it. In addition, we can have rather advanced DevOps with a data logging, analytics, visualization, and training pipeline. We do not need to make all of them at once, but to make a machine learning model into our production system, we have to design and develop resource utilization with consideration of cost effectiveness and related systems.</p>
<h2>Structure of the machine learning design patterns</h2>
<p>I wrote the patterns in the following categories:</p>
<ul>
<li>
<p>Training patterns: ways to design a training pipeline</p>
</li>
<li>
<p>QA patterns: test and quality assurance</p>
</li>
<li>
<p>Serving patterns: designs to develop a model into a prediction system</p>
</li>
<li>
<p>Operation patterns: model and system operations</p>
</li>
<li>
<p>Lifecycle patterns: compositions of other patterns</p>
</li>
</ul>
<p>Each category consists of explanations to realize a machine learning lifecycle, from training, QA, and serving to operation, in a production system. The lifecycle pattern is aimed to design an overall system integrating training, QA, serving and operation.</p>
<p>Each pattern includes:</p>
<ul>
<li>
<p>Use case: when to use the pattern</p>
</li>
<li>
<p>Architecture: what issue the pattern is trying to solve and how the system is structured</p>
</li>
<li>
<p>Diagram: overview of the system</p>
</li>
<li>
<p>Pros: advantages of the pattern</p>
</li>
<li>
<p>Cons: disadvantages of the pattern</p>
</li>
<li>
<p>Consideration: things to watch out for when using the pattern</p>
</li>
</ul>
<p>I also wrote some anti-patterns to explain what should not be made.</p>
<h2>What I want the patterns to become</h2>
<p>I tried to make the initial format of the design patterns not be so dependent on a certain platform, language or library. They are written to be able to be deployed on major cloud services and Kubernetes clusters. Choosing which cloud or Kubernetes cluster to use depends on each company’s or system’s scheme, and I made the pattern quite abstract so that they are applicable to as many systems as possible.</p>
<p>I joined Mercari JP in July 2018, as a MLOps engineer, formerly SysML engineer, and had chances to develop and operate various machine learning systems. I guess I have taken part, in some ways, in most of the machine learning systems running in production. Some projects I worked on from their initial stage, some for model development, and some after release. There are some that I joined from production troubleshooting :)>. This machine learning design pattern documentation is the result of my experience with those projects. Since Mercari is a web service based business and uses cloud and Kubernetes, I assume the patterns might be biased to those conditions. I would like to enhance the patterns to compose in major cloud services, such as AWS, GCP and Microsoft Azure, and make workflows based on KubeFlow, to achieve more general and practical patterns and have more fun with machine learning!</p>


Machine learning system in patterns

<p><i>This article is the 14th entry in the <a href="https://tech.mercari.com/entry/2019/08/26/085000">Mercari Bold Challenge Month</a></i>.</p>
<p>Hello everyone, I’m Kengo <a href="https://twitter.com/karolis_ml">(@karolis_ml)</a> and I’m with Mercari this summer as a software engineering intern in the AI Engineering team in Tokyo. In this blog post, I’d like to present the experimental results on information fusion techniques in multimodal modelling in the context of prohibited items detection at Mercari JP.</p>
<h4>TL;DR</h4>
<ul>
<li>Machine learning is essential for a scalable prohibited items detection system</li>
<li>Simple concatenation of feature vectors still a strong baseline for information fusion</li>
<li>Max Pooling demonstrated to be effective for feature selection across modalities</li>
<li>Gated Multimodal Units (GMU), an attention-based mechanism for assigning weights to different modalities, outperforms baseline and provides explainability</li>
</ul>
<p></p>
<h4>Managing Risks at Scale</h4>
<p>A platform as sizeable as Mercari inevitably attracts a small but certain fraction of bad agents on both sides of the marketplace in buyers and sellers, and one such example is prohibited items being listed by malicious sellers including drugs, counterfeit goods, cash, medical devices, and many more. A comprehensive list of prohibited items can be found <a href="https://www.mercari.com/jp/help_center/getting_started/prohibited_items/">here</a> (JP) and <a href="https://www.mercari.com/us/help_center/article/318">here</a> (EN).</p>
<p>Not only does this behavior present legal risks to the users and the platform, it undermines the ecosystem as a whole when it comes to the sense of trust and safety once these items are seen as tolerated by the platform. Ideally, these prohibited items should not be allowed to enter the marketplace in the first place or taken off of the marketplace as quickly as possible, but the sheer volume of listings with over 13 million MAU makes this process extremely cumbersome and costly to manually deal with.</p>
<p>Here comes in the power of machine learning. Automatic detection of these prohibited items with machine learning helps identify previously undetected items with rule-based systems as well as reducing the cost of customer support. As the cost of false positives where a legitimate item is mistakenly considered as illegitimate is quite high in terms of user experience and customer support, our key metric to focus on is precision. This is to ensure that the items the model predicts to be prohibited are actually prohibited.</p>
<p>In solving this problem of binary classification, we first note that listings at Mercari consist of several components including an image of the item, title, description, price, and item condition and so on. As we build a neural network that takes all these factors into account, we need to somehow represent these features as an input vector that can be fed into machine learning models, and information fusion techniques are primarily concerned with how we combine these different modalities in such a way that we can extract information from them more effectively. With this in mind, I will first outline key methods and then results and implications from experiments for one specific category of prohibited items at Mercari.</p>
<h4>Preliminaries</h4>
<p>In the literature of multimodal learning, a common baseline for information fusion is to simply concatenate feature vectors across different modalities such as images and texts, which is surprisingly performant in various multimodal tasks. Concatenation was adopted as the baseline in previous experiments at Mercari also in similar settings (see References at the bottom of this post). Other simple yet powerful methods are element-wise linear sum/multiplication for each unit in a fully connected layer of common size (e.g. 512). Nonetheless, these element-wise operations are often limited in expressiveness as some noisy modalities could clutter the entire feature vectors.</p>
<p>Our main methods of interest are max pooling and Gated Multimodal Units (GMU) as illustrated in Figure 1. <a href="https://tech.mercari.com/entry/2018/04/24/164919">One of our previous experiments</a> demonstrated that max pooling helps identify informative units among multiple layers with different modalities, as empirically shown by <a href="https://arxiv.org/pdf/1708.02099.pdf">Duong et al. (2017)</a>. GMU proposed by <a href="https://arxiv.org/abs/1702.01992">Arevalo et al. (2017)</a>, on the other hand, learns weights for each unit in a layer of given modality, in a manner that resembles the attention mechanism. This can be considered as a continuous version of max pooling with units being averaged over with continuous weights, instead of simply assigning a weight of 1.0 to one modality over the other. In Figure 1, weights for inputs in the first modality shown in the upper half are represented by <b>z</b>, and, in this bimodal case, the other modality is assigned weights of <b>1-z</b>.</p>
<p><figure class="figure-image figure-image-fotolife" title="Figure 1: Max Pooling and Gated Bimodal Units"><span itemscope itemtype="http://schema.org/Photograph"><img src="https://storage.googleapis.com/prd-hatenaa-engineering-asset/20190828124233.png" alt="f:id:karolis_ml:20190828124233p:plain" title="f:id:karolis_ml:20190828124233p:plain" class="hatena-fotolife" itemprop="image"></span><figcaption>Figure 1: Max Pooling and Gated Bimodal Units</figcaption></figure>
</p>
<p>Before diving into our own data, we first replicated the results of the paper by reimplementing GMU in PyTorch based on authors&#8217; <a href="https://github.com/johnarevalo/gmu-mmimdb">original implementation</a> in Theano using their preprocessed multimodal dataset from <a href="https://www.imdb.com/">IMDb.com</a>, solving a task of movie genre classification with posters as visual input and titles and descriptions as text input.</p>
<p>Besides the GMU&#8217;s competitive performance against the baseline (concatenate) and other existing methods, attention for the feature vector of common size was calculated using the hyperbolic tangent function. In this setting, <b>z</b> indicates the weights of units from the visual input, and Figure 2 from the paper shows the percentage of units that are activated with <b>z</b> more than 0.5 for visual input and less than or equal to 0.5 for text input. Crucially, we confirmed in our environment that genres &#8220;animation&#8221;, &#8220;family&#8221;, and &#8220;musical&#8221; do have a higher percentage of activations than other genres for the visual input, which is sensible in that these movies, especially animation, tend to have a distinctive format or designs in their posters. This provides additional model explainability as we humans have an intuitive and often better understanding of what factors differentiate one class from the other regardless of modalities.</p>
<p><figure class="figure-image figure-image-fotolife" title="Figure 2: Gate Activations for Each Genre in the Muldimodal-IMDb dataset(Arevalo et al., 2017)"><span itemscope itemtype="http://schema.org/Photograph"><img src="https://storage.googleapis.com/prd-hatenaa-engineering-asset/20190829141409.png" alt="f:id:karolis_ml:20190829141409p:plain" title="f:id:karolis_ml:20190829141409p:plain" class="hatena-fotolife" itemprop="image"></span><figcaption>Figure 2: Gate Activations for Each Genre (Arevalo et al., 2017)</figcaption></figure>
</p>
<h4>Experiments</h4>
<p>Figure 3 illustrates an overview of multimodal modelling with three modalities in our setting at Mercari for prohibited items detection; image, text, and meta information (auxiliary). Features are extracted with sklearn&#8217;s TF-IDF Vectorizer for text and pre-trained InceptionV3 for images. Note that while concatenation can be performed on vectors with different lengths (e.g. (n, 4096) for image and (n, 300) for text to create (n, 4396)), other information fusion methods first need to set up a linear layer to align the dimensions (e.g. 512). Here, we pay particular attention to the bimodal fusion mechanism between image and text as indicated in red in Figure 3, where auxiliary features are simply added to the features via concatenation after the first information fusion. This is simply due to the fact that auxiliary features are rather small in dimensions and known to be less informative compared to the other two.</p>
<p>As per the implementation by <a href="https://arxiv.org/abs/1702.01992">Arevalo et al. (2017)</a>, we added batch normalization and maxout for GMU and max pooling. Since the effect of data normalization was demonstrated in our previous experiment linked above, we trained a different model without batch normalization to reexamine this effect. Additionally, we also included LinearSum (i.e. element-wise sum) as another method for a linear information fusion mechanism, following the experiments by the authors.</p>
<p>Due to the imbalance between prohibited items (positive class) and legitimate items (negative class) in Mercari&#8217;s listings, we downsampled the negative pool to match the size of the positive class. Models are written in TensorFlow to fit into our ML platform, and training the neural networks took 5-6 hours for each model on NVIDIA Tesla V100 with the best model chosen based on the validation loss.</p>
<p><figure class="figure-image figure-image-fotolife" title="Figure 3: Feature Fusion Flow"><span itemscope itemtype="http://schema.org/Photograph"><img src="https://storage.googleapis.com/prd-hatenaa-engineering-asset/20190910121233.png" alt="f:id:karolis_ml:20190910121233p:plain" title="f:id:karolis_ml:20190910121233p:plain" class="hatena-fotolife" itemprop="image"></span><figcaption>Figure 3: Feature Fusion Flow</figcaption></figure>
</p>
<h4>Results</h4>
<p>The following table summarizes model performance with average precision scores evaluated over an unseen 7-day testing data, as well as precision and recall scores. Negative class was downsampled such that the ratio between positive and negative is 1:3 for each day.</p>
<p>In terms of performance, max pooling trumps other models with average precision of 0.8634 followed by GMU at 0.8631, while concatenation still proving to be a strong baseline at 0.8239. This confirms <a href="https://arxiv.org/abs/1802.02892">Kiela et al. (2018)</a>&#8216;s additional experiments on multimodal-IMDb dataset which was originally provided by Arevalo et al. (2017), where max pooling outperforms GMU by a small margin in the task of movie genre classification with a weighted F-score of 0.6220 against the GMU&#8217;s 0.6180.</p>
<p>Despite its simplicity, LinearSum also appears to fare well at 0.8420, and this is also in line with Arevalo et al (2017)&#8217;s results where LinearSum trails the weighted F-score of GMU by a bit while exceeding the baseline, which we replicated in our environment.</p>
<p>Interestingly, whereas GMU with batch normalization is almost as performant as max pooling, the same model without batch normalization lags behind LinearSum in terms of average precision. This seems to show that, as our past experiment has similarly illuminated, normalization of inputs plays a vital role in information fusion.</p>
<p><figure class="figure-image figure-image-fotolife" title="Table 1: Model Performance"><span itemscope itemtype="http://schema.org/Photograph"><img src="https://storage.googleapis.com/prd-hatenaa-engineering-asset/20190911031417.png" alt="f:id:karolis_ml:20190911031417p:plain" title="f:id:karolis_ml:20190911031417p:plain" class="hatena-fotolife" itemprop="image"></span><figcaption>Table 1: Model Performance</figcaption></figure>
</p>
<p>Notably, we also extracted weights for each modality in the testing dataset, and confirmed that text has much higher weights than images in this specific category of prohibited items, with the split between the two at around 0.92 for text and 0.08 for images. As the authors demonstrated in Figure 2, we also examined the difference in weights by conditioning on a certain binary attribute X in the testing dataset. We know for a fact that this attribute X has a differential effect on how certain images are presented in this category of specific items, and, as it turns out, images are given higher weights if this attribute takes a value of 1. This indicates that the GMU is properly adjusting weights for certain items where features from one modality are more informative.</p>
<h4>Moving Forward</h4>
<p>The main implication from the findings above is that the weighting mechanism provides a better understanding of the model on top of its superiority over the baseline method of concatenation, which leads to insights as to how to improve performance. One example is that more sophisticated preprocessing and feature extraction methods for the text features will likely improve the overall performance, given that the weight assigned to text is quite high and that we have a rather simple feature extraction technique with tokenization, normalization, and TF-IDF. In fact, our <a href="https://tech.mercari.com/entry/2019/04/26/163000">previous experiment</a> illustrates the superiority of the DARTS-based network with the character-level embeddings for the text input, and an approach like this would definitely be a good start.</p>
<p>Another important takeaway is that having weights for different modalities is also useful in selecting an appropriate number of modalities to use as inputs, especially when some modalities such as images and videos are computationally costly. For instance, while using a visual input might contribute to the model’s performance by an infinitesimal margin, it might not make business sense to deploy a model that is significantly larger and hence harder to maintain.</p>
<p>On top of the superiority in performance compared to the common baseline of concatenation, the explainability provided by an information fusion mechanism like GMU hopefully helps with the exacting task of iterating through different model specifications and preprocessing techniques in other multimodal problems at Mercari and beyond.</p>
<p>Thank you for reading this blog post. If you&#8217;re keen on this topic, please take a look at our past experiments and relevant papers shown below, and stay tuned for more updates from the AI Engineering team!</p>
<h4>References</h4>
<ul>
<li>Arevalo, John; Solorio, Thamar; Montes-y-Gómez, Manuel; González, Fabio A. (2017) Gated Multimodal Units for Information Fusion, <a href="https://arxiv.org/abs/1702.01992">ArXiv:1702.01992</a></li>
<li>Thang Duong, Chi; Lebret, Remi; Aberer, Karl (2017) Multimodal Classification for Analysing Social Media,  <a href="https://arxiv.org/abs/1708.02099">arXiv:1708.02099</a></li>
<li>Kiela, D.; Grave, E.; Joulin, A.; Mikolov, T. (2018) Efficient Large-Scale Multi-Modal Classification <a href="https://arxiv.org/abs/1802.02892">arXiv:1708.02099</a></li>
<li><a href="https://tech.mercari.com/entry/2018/04/24/164919">&#x30DE;&#x30EB;&#x30C1;&#x30E2;&#x30FC;&#x30C0;&#x30EB;&#x30E2;&#x30C7;&#x30EB;&#x306B;&#x3088;&#x308B;&#x4E0D;&#x6B63;&#x51FA;&#x54C1;&#x306E;&#x691C;&#x77E5; &#8211; Mercari Engineering Blog</a></li>
<li><a href="https://tech.mercari.com/entry/2019/04/26/163000">Using neural architecture search to automate multimodal modeling for prohibited item detection &#8211; Mercari Engineering Blog</a></li>
</ul>

Multimodal Information Fusion for Prohibited Items Detection

Mercari AI Team Research Paper “Improving Visual Recommendation on E-commerce Platforms Using Vision-Language Models” Accepted for Presentation at RecSys 2025

Mercari AI’s research paper, “Zero-Shot Retrieval for Scalable Visual Search in a Two-Sided Marketplace” has been accepted to the TSMO 2025 workshop at KDD 2025

Mercari AI Team Research Paper “Visual Recommendation on E-commerce Platforms Via Fine-Tuned Vision-Language Models” Accepted for Presentation at MIRU 2025

Mercari AI Team’s Research “LLMOps for Eval-Driven Development at Scale” Accepted to FOSSASIA Summit 2025

The Mercari AI team’s research “Strategic Coupon Allocation for Increasing Providers’ Sales Experiences in Two-sided Marketplaces” has been accepted at the KDD2024 TSMO 2024 Workshop

Blogs by mercari AI team, delivering information about AI technology and organization

Mercari AI Team Research Paper “Improving Visual Recommendation on E-commerce Platforms Using Vision-Language Models” Accepted for Presentation at RecSys 2025

Mercari AI’s research paper, “Zero-Shot Retrieval for Scalable Visual Search in a Two-Sided Marketplace” has been accepted to the TSMO 2025 workshop at KDD 2025

Mercari AI Team Research Paper “Visual Recommendation on E-commerce Platforms Via Fine-Tuned Vision-Language Models” Accepted for Presentation at MIRU 2025

Mercari AI Team’s Research “LLMOps for Eval-Driven Development at Scale” Accepted to FOSSASIA Summit 2025

The Mercari AI team’s research “Strategic Coupon Allocation for Increasing Providers’ Sales Experiences in Two-sided Marketplaces” has been accepted at the KDD2024 TSMO 2024 Workshop

Mercari AI Team Research Paper “Improving Visual Recommendation on E-commerce Platforms Using Vision-Language Models” Accepted for Presentation at RecSys 2025

Mercari AI’s research paper, “Zero-Shot Retrieval for Scalable Visual Search in a Two-Sided Marketplace” has been accepted to the TSMO 2025 workshop at KDD 2025

Mercari AI Team Research Paper “Visual Recommendation on E-commerce Platforms Via Fine-Tuned Vision-Language Models” Accepted for Presentation at MIRU 2025

Mercari AI Team’s Research “LLMOps for Eval-Driven Development at Scale” Accepted to FOSSASIA Summit 2025

The Mercari AI team’s research “Strategic Coupon Allocation for Increasing Providers’ Sales Experiences in Two-sided Marketplaces” has been accepted at the KDD2024 TSMO 2024 Workshop

Mercari AI Team Research Paper “Improving Visual Recommendation on E-commerce Platforms Using Vision-Language Models” Accepted for Presentation at RecSys 2025

Mercari AI’s research paper, “Zero-Shot Retrieval for Scalable Visual Search in a Two-Sided Marketplace” has been accepted to the TSMO 2025 workshop at KDD 2025

Mercari AI Team Research Paper “Visual Recommendation on E-commerce Platforms Via Fine-Tuned Vision-Language Models” Accepted for Presentation at MIRU 2025

Mercari AI Team’s Research “LLMOps for Eval-Driven Development at Scale” Accepted to FOSSASIA Summit 2025

The Mercari AI team’s research “Strategic Coupon Allocation for Increasing Providers’ Sales Experiences in Two-sided Marketplaces” has been accepted at the KDD2024 TSMO 2024 Workshop

Articles

Contributing to a Circular Economy : CEO Shintaro Yamada and AI Director Shunya Kimura Discuss the Challenges that Mercari’s AI Technology Needs to Overcome

AI Listings, Image Search… CEO Shintaro Yamada and Director Shunya Kimura Look Back at Mercari’s AI Journey

AI Application to CRM – Providing Better Marketing Experience and Improving Campaign Effectiveness

For a Trust and Safety Marketplace, Machine Learning for Violation Detection

Natural views for self-supervised learning

How the Machine Learning Team and Backend Team Joined Forces to Create the Feature Team

Machine learning system in patterns

Multimodal Information Fusion for Prohibited Items Detection