Mercari AI’s collaborative research paper “Counterfactual Learning with General Data-generating Policies” accepted to AAAI 2023

Overview

We are pleased to announce that "Counterfactual Learning with General Data-generating Policies," a paper co-authored by Team AI engineer Akihiro Shimizu and Yusuke Narita, founder of Hanjuku Kaso, Inc. and assistant professor at Yale University, has been accepted to the plenary session of AAAI 2023, an international conference on artificial intelligence.

AAAI (Association for the Advancement of Artificial Intelligence) is one of the most prestigious conferences in the field of artificial intelligence. This is the 37th year of the conference, and 1,721 papers have been accepted out of 8,777 submissions (acceptance rate of 19.6%).

The following schedule of presentation times is planned

Sunday, February 12, 2023, 2:00 PM - 3:15 PM (GMT-5)

Key points of presentation

Counterfactual Learning with General Data-generating Policies

• Proposed a method for Off-policy Evaluation that can be applied to a wide class of existing measures, including non-probabilistic measures.
• Developed a method for improving algorithms using data naturally generated by the algorithm.
• Analyzed coupon distribution policies in Mercari using the proposed method. The effectiveness of coupon measures can now be measured by using data from Mercari.

Background

An important part of algorithmic decision making is to predict the performance of new decision-making algorithms (also called measures) that have not yet been used. However, A/B testing, in which old algorithms are randomly assigned to users and compared with new algorithms, can be time-consuming to develop and can lead to dissatisfaction due to unfairness to some users who could not use the new functionality.

Therefore, it is important to solve this problem by using "Off-policy Evaluation," which attempts to estimate the performance of a new algorithm using only the data naturally generated by the old algorithm. However, it is known that current Off-policy Evaluation techniques are difficult to use for non-probabilistic measures such as coupon distribution algorithms.

Summary of paper

Therefore, we have proposed a new method of ex-post evaluation that can be applied to a wide range of existing measures, including non-probabilistic measures such as coupon distribution algorithms.

The proposed method is developed based on the observation that when an algorithm makes a decision, the data generated from it will almost always include natural experiments. For example, in the case of supervised learning used in coupon distribution algorithms, etc., decisions are often made based on whether or not some predicted variable exceeds a certain criterion value, such as whether or not to distribute coupons, etc. However, even though the situation before and after the criterion value is almost identical In our proposed method, the decision-making process is based on such a coincidence. Our proposed method considers these differences in decision making caused by chance factors as local natural experiments, and uses them to compare the performance of new measures.

By using data from Mercari, we have verified that our proposed method can solve the problems of real data and A/B testing in a real-world problem. Using the proposed method, it is possible to estimate the performance of a new coupon and its business impact through ex-post evaluation rather than A/B testing.

About MDS Team

The MDS (Marketing Data Science) team uses statistics, machine learning, and mathematical optimization to improve the efficiency of marketing campaigns.

About Hanjuku Kaso, Inc.

A startup that combines data, algorithms, mathematics, and thought to design the future of business, policy, and society.
The company's strengths are "market design," "counterfactual machine learning," and "causal inference," and it conducts joint projects and research with numerous companies as well as its own basic research and software development.
The collaboration involves Yusuke Narita, the company's founder and an assistant professor at Yale University; Kyohei Okumura, a PhD student in the Department of Economics at Northwestern University; and Kohei Yata, an assistant professor at the University of Wisconsin-Madison.