This article is the 14th entry in the <a href="https://tech.mercari.com/entry/2019/08/26/085000">Mercari Bold Challenge Month</a>.
Hello everyone, I’m Kengo <a href="https://twitter.com/karolis_ml">(@karolis_ml)</a> and I’m with Mercari this summer as a software engineering intern in the AI Engineering team in Tokyo. In this blog post, I’d like to present the experimental results on information fusion techniques in multimodal modelling in the context of prohibited items detection at Mercari JP.
<h4>TL;DR</h4>
<ul>
<li>Machine learning is essential for a scalable prohibited items detection system</li>
<li>Simple concatenation of feature vectors still a strong baseline for information fusion</li>
<li>Max Pooling demonstrated to be effective for feature selection across modalities</li>
<li>Gated Multimodal Units (GMU), an attention-based mechanism for assigning weights to different modalities, outperforms baseline and provides explainability</li>
</ul>

<h4>Managing Risks at Scale</h4>
A platform as sizeable as Mercari inevitably attracts a small but certain fraction of bad agents on both sides of the marketplace in buyers and sellers, and one such example is prohibited items being listed by malicious sellers including drugs, counterfeit goods, cash, medical devices, and many more. A comprehensive list of prohibited items can be found <a href="https://www.mercari.com/jp/help_center/getting_started/prohibited_items/">here</a> (JP) and <a href="https://www.mercari.com/us/help_center/article/318">here</a> (EN).
Not only does this behavior present legal risks to the users and the platform, it undermines the ecosystem as a whole when it comes to the sense of trust and safety once these items are seen as tolerated by the platform. Ideally, these prohibited items should not be allowed to enter the marketplace in the first place or taken off of the marketplace as quickly as possible, but the sheer volume of listings with over 13 million MAU makes this process extremely cumbersome and costly to manually deal with.
Here comes in the power of machine learning. Automatic detection of these prohibited items with machine learning helps identify previously undetected items with rule-based systems as well as reducing the cost of customer support. As the cost of false positives where a legitimate item is mistakenly considered as illegitimate is quite high in terms of user experience and customer support, our key metric to focus on is precision. This is to ensure that the items the model predicts to be prohibited are actually prohibited.
In solving this problem of binary classification, we first note that listings at Mercari consist of several components including an image of the item, title, description, price, and item condition and so on. As we build a neural network that takes all these factors into account, we need to somehow represent these features as an input vector that can be fed into machine learning models, and information fusion techniques are primarily concerned with how we combine these different modalities in such a way that we can extract information from them more effectively. With this in mind, I will first outline key methods and then results and implications from experiments for one specific category of prohibited items at Mercari.
<h4>Preliminaries</h4>
In the literature of multimodal learning, a common baseline for information fusion is to simply concatenate feature vectors across different modalities such as images and texts, which is surprisingly performant in various multimodal tasks. Concatenation was adopted as the baseline in previous experiments at Mercari also in similar settings (see References at the bottom of this post). Other simple yet powerful methods are element-wise linear sum/multiplication for each unit in a fully connected layer of common size (e.g. 512). Nonetheless, these element-wise operations are often limited in expressiveness as some noisy modalities could clutter the entire feature vectors.
Our main methods of interest are max pooling and Gated Multimodal Units (GMU) as illustrated in Figure 1. <a href="https://tech.mercari.com/entry/2018/04/24/164919">One of our previous experiments</a> demonstrated that max pooling helps identify informative units among multiple layers with different modalities, as empirically shown by <a href="https://arxiv.org/pdf/1708.02099.pdf">Duong et al. (2017)</a>. GMU proposed by <a href="https://arxiv.org/abs/1702.01992">Arevalo et al. (2017)</a>, on the other hand, learns weights for each unit in a layer of given modality, in a manner that resembles the attention mechanism. This can be considered as a continuous version of max pooling with units being averaged over with continuous weights, instead of simply assigning a weight of 1.0 to one modality over the other. In Figure 1, weights for inputs in the first modality shown in the upper half are represented by z, and, in this bimodal case, the other modality is assigned weights of 1-z.
<figure class="figure-image figure-image-fotolife" title="Figure 1: Max Pooling and Gated Bimodal Units"><img src="https://storage.googleapis.com/prd-hatenaa-engineering-asset/20190828124233.png" alt="f:id:karolis_ml:20190828124233p:plain" title="f:id:karolis_ml:20190828124233p:plain" class="hatena-fotolife" itemprop="image"><figcaption>Figure 1: Max Pooling and Gated Bimodal Units</figcaption></figure>

Before diving into our own data, we first replicated the results of the paper by reimplementing GMU in PyTorch based on authors&#8217; <a href="https://github.com/johnarevalo/gmu-mmimdb">original implementation</a> in Theano using their preprocessed multimodal dataset from <a href="https://www.imdb.com/">IMDb.com</a>, solving a task of movie genre classification with posters as visual input and titles and descriptions as text input.
Besides the GMU&#8217;s competitive performance against the baseline (concatenate) and other existing methods, attention for the feature vector of common size was calculated using the hyperbolic tangent function. In this setting, z indicates the weights of units from the visual input, and Figure 2 from the paper shows the percentage of units that are activated with z more than 0.5 for visual input and less than or equal to 0.5 for text input. Crucially, we confirmed in our environment that genres &#8220;animation&#8221;, &#8220;family&#8221;, and &#8220;musical&#8221; do have a higher percentage of activations than other genres for the visual input, which is sensible in that these movies, especially animation, tend to have a distinctive format or designs in their posters. This provides additional model explainability as we humans have an intuitive and often better understanding of what factors differentiate one class from the other regardless of modalities.
<figure class="figure-image figure-image-fotolife" title="Figure 2: Gate Activations for Each Genre in the Muldimodal-IMDb dataset(Arevalo et al., 2017)"><img src="https://storage.googleapis.com/prd-hatenaa-engineering-asset/20190829141409.png" alt="f:id:karolis_ml:20190829141409p:plain" title="f:id:karolis_ml:20190829141409p:plain" class="hatena-fotolife" itemprop="image"><figcaption>Figure 2: Gate Activations for Each Genre (Arevalo et al., 2017)</figcaption></figure>

<h4>Experiments</h4>
Figure 3 illustrates an overview of multimodal modelling with three modalities in our setting at Mercari for prohibited items detection; image, text, and meta information (auxiliary). Features are extracted with sklearn&#8217;s TF-IDF Vectorizer for text and pre-trained InceptionV3 for images. Note that while concatenation can be performed on vectors with different lengths (e.g. (n, 4096) for image and (n, 300) for text to create (n, 4396)), other information fusion methods first need to set up a linear layer to align the dimensions (e.g. 512). Here, we pay particular attention to the bimodal fusion mechanism between image and text as indicated in red in Figure 3, where auxiliary features are simply added to the features via concatenation after the first information fusion. This is simply due to the fact that auxiliary features are rather small in dimensions and known to be less informative compared to the other two.
As per the implementation by <a href="https://arxiv.org/abs/1702.01992">Arevalo et al. (2017)</a>, we added batch normalization and maxout for GMU and max pooling. Since the effect of data normalization was demonstrated in our previous experiment linked above, we trained a different model without batch normalization to reexamine this effect. Additionally, we also included LinearSum (i.e. element-wise sum) as another method for a linear information fusion mechanism, following the experiments by the authors.
Due to the imbalance between prohibited items (positive class) and legitimate items (negative class) in Mercari&#8217;s listings, we downsampled the negative pool to match the size of the positive class. Models are written in TensorFlow to fit into our ML platform, and training the neural networks took 5-6 hours for each model on NVIDIA Tesla V100 with the best model chosen based on the validation loss.
<figure class="figure-image figure-image-fotolife" title="Figure 3: Feature Fusion Flow"><img src="https://storage.googleapis.com/prd-hatenaa-engineering-asset/20190910121233.png" alt="f:id:karolis_ml:20190910121233p:plain" title="f:id:karolis_ml:20190910121233p:plain" class="hatena-fotolife" itemprop="image"><figcaption>Figure 3: Feature Fusion Flow</figcaption></figure>

<h4>Results</h4>
The following table summarizes model performance with average precision scores evaluated over an unseen 7-day testing data, as well as precision and recall scores. Negative class was downsampled such that the ratio between positive and negative is 1:3 for each day.
In terms of performance, max pooling trumps other models with average precision of 0.8634 followed by GMU at 0.8631, while concatenation still proving to be a strong baseline at 0.8239. This confirms <a href="https://arxiv.org/abs/1802.02892">Kiela et al. (2018)</a>&#8216;s additional experiments on multimodal-IMDb dataset which was originally provided by Arevalo et al. (2017), where max pooling outperforms GMU by a small margin in the task of movie genre classification with a weighted F-score of 0.6220 against the GMU&#8217;s 0.6180.
Despite its simplicity, LinearSum also appears to fare well at 0.8420, and this is also in line with Arevalo et al (2017)&#8217;s results where LinearSum trails the weighted F-score of GMU by a bit while exceeding the baseline, which we replicated in our environment.
Interestingly, whereas GMU with batch normalization is almost as performant as max pooling, the same model without batch normalization lags behind LinearSum in terms of average precision. This seems to show that, as our past experiment has similarly illuminated, normalization of inputs plays a vital role in information fusion.
<figure class="figure-image figure-image-fotolife" title="Table 1: Model Performance"><img src="https://storage.googleapis.com/prd-hatenaa-engineering-asset/20190911031417.png" alt="f:id:karolis_ml:20190911031417p:plain" title="f:id:karolis_ml:20190911031417p:plain" class="hatena-fotolife" itemprop="image"><figcaption>Table 1: Model Performance</figcaption></figure>

Notably, we also extracted weights for each modality in the testing dataset, and confirmed that text has much higher weights than images in this specific category of prohibited items, with the split between the two at around 0.92 for text and 0.08 for images. As the authors demonstrated in Figure 2, we also examined the difference in weights by conditioning on a certain binary attribute X in the testing dataset. We know for a fact that this attribute X has a differential effect on how certain images are presented in this category of specific items, and, as it turns out, images are given higher weights if this attribute takes a value of 1. This indicates that the GMU is properly adjusting weights for certain items where features from one modality are more informative.
<h4>Moving Forward</h4>
The main implication from the findings above is that the weighting mechanism provides a better understanding of the model on top of its superiority over the baseline method of concatenation, which leads to insights as to how to improve performance. One example is that more sophisticated preprocessing and feature extraction methods for the text features will likely improve the overall performance, given that the weight assigned to text is quite high and that we have a rather simple feature extraction technique with tokenization, normalization, and TF-IDF. In fact, our <a href="https://tech.mercari.com/entry/2019/04/26/163000">previous experiment</a> illustrates the superiority of the DARTS-based network with the character-level embeddings for the text input, and an approach like this would definitely be a good start.
Another important takeaway is that having weights for different modalities is also useful in selecting an appropriate number of modalities to use as inputs, especially when some modalities such as images and videos are computationally costly. For instance, while using a visual input might contribute to the model’s performance by an infinitesimal margin, it might not make business sense to deploy a model that is significantly larger and hence harder to maintain.
On top of the superiority in performance compared to the common baseline of concatenation, the explainability provided by an information fusion mechanism like GMU hopefully helps with the exacting task of iterating through different model specifications and preprocessing techniques in other multimodal problems at Mercari and beyond.
Thank you for reading this blog post. If you&#8217;re keen on this topic, please take a look at our past experiments and relevant papers shown below, and stay tuned for more updates from the AI Engineering team!
<h4>References</h4>
<ul>
<li>Arevalo, John; Solorio, Thamar; Montes-y-Gómez, Manuel; González, Fabio A. (2017) Gated Multimodal Units for Information Fusion, <a href="https://arxiv.org/abs/1702.01992">ArXiv:1702.01992</a></li>
<li>Thang Duong, Chi; Lebret, Remi; Aberer, Karl (2017) Multimodal Classification for Analysing Social Media, <a href="https://arxiv.org/abs/1708.02099">arXiv:1708.02099</a></li>
<li>Kiela, D.; Grave, E.; Joulin, A.; Mikolov, T. (2018) Efficient Large-Scale Multi-Modal Classification <a href="https://arxiv.org/abs/1802.02892">arXiv:1708.02099</a></li>
<li><a href="https://tech.mercari.com/entry/2018/04/24/164919">&#x30DE;&#x30EB;&#x30C1;&#x30E2;&#x30FC;&#x30C0;&#x30EB;&#x30E2;&#x30C7;&#x30EB;&#x306B;&#x3088;&#x308B;&#x4E0D;&#x6B63;&#x51FA;&#x54C1;&#x306E;&#x691C;&#x77E5; &#8211; Mercari Engineering Blog</a></li>
<li><a href="https://tech.mercari.com/entry/2019/04/26/163000">Using neural architecture search to automate multimodal modeling for prohibited item detection &#8211; Mercari Engineering Blog</a></li>
</ul>

Multimodal Information Fusion for Prohibited Items Detection

Hi everyone, I’m <a href="https://twitter.com/hurutoriya">@hurutoriya</a>, a Machine Learning Engineer.
On May 9th, 2018, we held the closing ceremony for the Kaggle competition hosted by Mercari: the Mercari Price Suggestion Challenge.
In the Mercari Price Suggestion Challenge, participants competed to improve Mercari’s price suggestion algorithm. For this, we published the product data of items sold through our US app.
<a href="https://mercari.connpass.com/event/84299/">Mercari Kaggle Competition: Closing Ceremony 2018 &#8211; connpass</a>
<img src="https://storage.googleapis.com/prd-hatenaa-engineering-asset/20181109005757.png" alt="f:id:hurutoriya:20181109005757p:plain" title="f:id:hurutoriya:20181109005757p:plain" class="hatena-fotolife" itemprop="image">

For the closing ceremony, we invited the top three contestants from Russia, Poland, and China, as well as top-ranking contestants from Japan. In a panel-discussion format, we asked each contestant to explain their solution and what they think about Kaggle. 
This challenge also had a Kernel-only (constraints on computer resources and calculation time) rule, which made for a very interesting competition.
We uploaded videos of the panel discussion, so read on to check it out!
<h3>Part 1</h3>
<iframe loading="lazy" width="480" height="270" src="https://www.youtube.com/embed/1ZIFPu0T1iM?feature=oembed" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe><cite class="hatena-citation"><a href="https://www.youtube.com/watch?v=1ZIFPu0T1iM">www.youtube.com</a></cite>
<h2>Price suggestion at Mercari</h2>
<h3>The importance of price suggestion at Mercari</h3>
First, let me explain why price suggestion is important here at Mercari. 
On the Mercari app, users can choose whatever price they want when listing an item.
Here are some examples which show why this becomes a problem: 
-Some items on Mercari cannot be sold because their listing prices are too high compared to the market price. 
-Conversely, if the listing price is lower than the market price, customers lose out.
As a solution,
<ul>
<li>users can search Mercari for an item they plan to list, effectively performing market research.</li>
</ul>
However,
<ul>
<li>this is a lot of effort for the user, and</li>
<li>this idea may not occur to new Mercari users.</li>
</ul>
Therefore, listing becomes easier if we <code>automatically display a suitable price</code> for users when they list an item.
<h3>Explanation regarding price suggestion</h3>
Currently, <a href="https://twitter.com/lain_m21">@lain_m21</a>, an intern in the ML team, is helping us with the price suggestion system at Mercari JP.
Tasks involving price suggestion are evaluated by the Root Mean Squared Logarithmic Error (RMSLE) metric.
This is defined as:
<img src="https://storage.googleapis.com/prd-hatenaa-engineering-asset/20181114122755.png" alt="f:id:hurutoriya:20181114122755p:plain" title="f:id:hurutoriya:20181114122755p:plain" class="hatena-fotolife" itemprop="image">
<ul>
<li><img src="https://chart.apis.google.com/chart?cht=tx&#038;chl=%20%5Cepsilon" alt=" epsilon"/>: RMSLE score</li>
<li><img src="https://chart.apis.google.com/chart?cht=tx&#038;chl=%20n" alt=" n"/>: Number of predicted data cases</li>
<li><img src="https://chart.apis.google.com/chart?cht=tx&#038;chl=%20p_i" alt=" p_i"/>: Predicted price</li>
<li><img src="https://chart.apis.google.com/chart?cht=tx&#038;chl=%20a_i" alt=" a_i"/>: Real price</li>
<li><img src="https://chart.apis.google.com/chart?cht=tx&#038;chl=%20%5Clog%28x%29" alt=" log(x)"/>: Natural logarithm of x</li>
</ul>
The lower the score is, the higher the accuracy of the price suggestion function.
RMSLE is innately difficult to understand, so for the purpose of this explanation let’s try visualizing it.
Below is a diagram showing the margin of error for RMSLE scores corresponding to a price of 3,000 JPY. 
<img src="https://storage.googleapis.com/prd-hatenaa-engineering-asset/20181115113703.png" alt="f:id:hurutoriya:20181115113703p:plain" title="f:id:hurutoriya:20181115113703p:plain" class="hatena-fotolife" itemprop="image">
The horizontal axis represents RMSLE values. 
The vertical axis represents the error range of RMSLE values. 
This is a logarithmic metric, so for a RMSLE value of 1.0: 
&#8211; Real price: 3,000 JPY 
&#8211; Range of estimated prices: 1,103 JPY ~ 8,156 JPY
As you can see, this isn’t quite accurate enough to use in an actual product.
The top contestant of the Kaggle Competition was able to achieve an RMSLE value of 0.3875 with their model. (<a href="https://www.kaggle.com/lopuhin/mercari-golf-0-3875-cv-in-75-loc-1900-s">75 line kernel</a>)
Results:
<ul>
<li>Real price: 3000 JPY</li>
<li>Range of estimated prices: 2051 JPY ~ 4387 JPY</li>
</ul>
Results with these levels of accuracy could be incorporated into the actual product.
The machine learning team was very impressed with the concise, polished nature of the Code Golf written in under 100 lines by the first-place team.
<a href="https://www.kaggle.com/lopuhin/mercari-golf-0-3875-cv-in-75-loc-1900-s">Mercari Golf: 0.3875 CV in 75 LOC, 1900 s | Kaggle</a>
Even though the code is under 100 lines, it is a very sophisticated piece of Python code and operationalizes the MLP model. We found that even when using the newest product data at Mercari US, this solution significantly enhanced the price suggestion function.
Their final solution has also been published on GitHub.
<iframe src="https://hatenablog-parts.com/embed?url=https%3A%2F%2Fgithub.com%2Fpjankiewicz%2Fmercari-solution" title="pjankiewicz/mercari-solution" class="embed-card embed-webcard" scrolling="no" frameborder="0" style="display: block; width: 100%; height: 155px; max-width: 500px; margin: 10px 0px;"></iframe><cite class="hatena-citation"><a href="https://github.com/pjankiewicz/mercari-solution">github.com</a></cite>
Explanations regarding the solution have also been published on GitHub.
<a href="https://github.com/pjankiewicz/mercari-solution/blob/master/presentation/build/yandex.pdf">mercari-solution/yandex.pdf at master &middot; pjankiewicz/mercari-solution &middot; GitHub</a>
<h3>Conclusions</h3>
Our team is focusing on the price suggestion function for the <code>Japanese version</code> of Mercari. As we provided an English dataset on Kaggle, many problems arise when we try to apply this to the Japanese version of Mercari. Thus, even the model provided by the winning team was unable to achieve the same score in the Japanese version of the app.
Nonetheless, the Mercari Price Suggestion Challenge provided an opportunity for rich debate by many Kagglers discussing various methods and ideas, which will contribute greatly to improving the Mercari JP price suggestion function. 
As the price suggestion function improves, we plan to update our readers with the technical details right here on the engineering blog.
Please stay tuned!

Report of Mercari Price Suggestion Challenge

How does Mercari leverage generative AI and large language models? The stated mission: “create new customer experiences and business impact” and “dramatically improve companywide productivity.”

Mercari AI’s research “Layout Optimizer for Personalized Home Screen based on Contextual Multi-Armed Bandit in C2C Marketplace” accepted to RecSys 2023’s Workshop

Mercari ML Engineers Teo Narboneta Zosa and Ryan Ginstrom will give a talk at Berlin Buzzwords 2023

Mercari AI’s research “Want robust explanations? Get smoother predictions first.” accepted to CIKM 2022’s workshop

Mercari AI’s research “Estimating the impact of coupon non-usage” accepted to ECML/PKDD 2022’s workshop

Blogs by mercari AI team, delivering information about AI technology and organization

How does Mercari leverage generative AI and large language models? The stated mission: “create new customer experiences and business impact” and “dramatically improve companywide productivity.”

Mercari AI’s research “Layout Optimizer for Personalized Home Screen based on Contextual Multi-Armed Bandit in C2C Marketplace” accepted to RecSys 2023’s Workshop

Mercari ML Engineers Teo Narboneta Zosa and Ryan Ginstrom will give a talk at Berlin Buzzwords 2023

Mercari AI’s research “Want robust explanations? Get smoother predictions first.” accepted to CIKM 2022’s workshop

Mercari AI’s research “Estimating the impact of coupon non-usage” accepted to ECML/PKDD 2022’s workshop

How does Mercari leverage generative AI and large language models? The stated mission: “create new customer experiences and business impact” and “dramatically improve companywide productivity.”

Mercari AI’s research “Layout Optimizer for Personalized Home Screen based on Contextual Multi-Armed Bandit in C2C Marketplace” accepted to RecSys 2023’s Workshop

Mercari ML Engineers Teo Narboneta Zosa and Ryan Ginstrom will give a talk at Berlin Buzzwords 2023

Mercari AI’s research “Want robust explanations? Get smoother predictions first.” accepted to CIKM 2022’s workshop

Mercari AI’s research “Estimating the impact of coupon non-usage” accepted to ECML/PKDD 2022’s workshop

How does Mercari leverage generative AI and large language models? The stated mission: “create new customer experiences and business impact” and “dramatically improve companywide productivity.”

Mercari AI’s research “Layout Optimizer for Personalized Home Screen based on Contextual Multi-Armed Bandit in C2C Marketplace” accepted to RecSys 2023’s Workshop

Mercari ML Engineers Teo Narboneta Zosa and Ryan Ginstrom will give a talk at Berlin Buzzwords 2023

Mercari AI’s research “Want robust explanations? Get smoother predictions first.” accepted to CIKM 2022’s workshop

Mercari AI’s research “Estimating the impact of coupon non-usage” accepted to ECML/PKDD 2022’s workshop

Articles

Multimodal Information Fusion for Prohibited Items Detection

Report of Mercari Price Suggestion Challenge