22.6 million people (*monthly active users, MAU) currently use Mercari to buy and sell a variety of items. However, until recently there were some items that just would not sell. To encourage the purchase of these kinds of items, Mercari started the internal Item Coupon Project, which assigns coupons to those items and sellers found to be a good fit with this type of promotion using uplift modeling. In just six months the project has not only improved sellers’ experiences but also contributed to a dramatic increase in lifetime value (LTV).
In this interview we spoke with Sho Sekine (@sho) and Koya Ohashi (@koya), Data Science Team members responsible for developing the project’s models, about the project’s background, its achievements and future issues.
Profiles
Sho Sekine (@sho) | Applied Scientist
@sho came to Mercari in 2023 after working as a consultant and data scientist at various companies. As an applied scientist he is responsible for such tasks as statistical modeling, system development, project management and strategic planning.
Koya Ohashi (@koya) | Applied Scientist
@koya joined Mercari in 2022 after working as an analytics consultant. As a marketing data scientist at Mercari, he analyzes marketing measures using machine learning and statistical models. He is currently working on initiatives related to improving customer satisfaction and is responsible for a wide range of tasks from project design to implementation and evaluation.
--Would you introduce yourselves and briefly talk about what you are currently working on at Mercari?
Sho：I joined Mercari’s Data Science Team in April 2023 after working as a consultant and data scientist at several companies and have been working on the Item Coupon Project ever since.
Koya：Having majored in mathematical statistics in graduate school, I decided to work as a data scientist at consulting firms prior to coming here. After analyzing data from numerous companies as part of those jobs, I realized I enjoyed directly seeing feedback in the data of toC businesses, and this led me to join Mercari.
Since starting here in May 2022, I’ve been involved with improving seller engagement (the experience Mercari users have as sellers). After working on improving ad ranking algorithms, I joined up with the same team @sho was on to work on the Item Coupon Project.
--What was behind the launching of this project to improve seller engagement?
Koya：Mercari’s engagement strategies are divided between buyers and sellers, the two major categories service users, and the aim of these is to improve the experiences of both groups. Seller engagement aims to enhance the experience of selling items on Mercari, so it is not just improvements to quantifiable revenue metrics like gross merchandise value (GMV) that are within its sc
Sho：Another reason for this project comes from past research done by another team. When they asked users whether they identified Mercari as a place for buying or selling things, an overwhelming majority answered the latter. As everyday consumers, we buy things all the time, but we rarely go about listing goods for sale. So, we see it as extremely important that the people who have taken time out to list their items on Mercari experience them selling successfully.
Our theory is that seller engagement will improve by not allowing sellers to experience their items being unsellable, and by transforming their experience into that of successful selling by intervening with item coupons and other measures to help move sellers’ stagnant stock.
We believe that seller engagement that starts with successful selling will lead to both an increased desire by sellers to list and an improvement in the quality of the items they list, and in turn result in growth of the marketplace as a whole.
The team started from the theory that they could encourage sellers to use Mercari even more through successful selling experiences
--Would you go into more detail about the reason for the launching of the Item Coupon Project as part of your seller engagement initiatives?
Koya：Well, it was started as part of our previously mentioned efforts to improve seller engagement.
Sho：We started it because, when you give buyers normal coupons, they are the ones who decide which coupon to use and what to purchase with it: That is difficult to connect to improving the experience of sellers. In comparison, item coupons, which let Mercari specify the sellers and items they are assigned to, are a unique incentive method at our disposal for intervening in the selling experience.
--So, you are encouraging successful selling experiences using this special approach of item coupons. What measures have you implemented as part of this in the last six months?
Sho：First we ran an evaluation experiment on the revenue produced by successful selling experiences. To assess our theory, we started with an A/B test. We randomly divided sellers into control and treatment groups and assigned item coupons to the items listed by sellers in the treatment group. The result was that the number of sellers experiencing their items selling successfully grew in that group. We then measured the retention rate in terms of new purchases, listing new items and new payments using Merpay over the long-term following this experience.
In particular, we monitored the total of both sales from marketplace buying and selling and from use of Merpay inside (inner pay) and outside (outer pay) of Mercari. We then took the difference between the total sales for the treatment group, where we implemented this measure, and those of the control group to get the amount of sales the successful selling experiences produced.
Next, we set about using uplift modeling to estimate and improve item/seller selection return on investment (ROI).
Which combinations of items and coupons are best is worked out after calculating the lift value, done by subtracting the probability of an item selling with a coupon from the probability of it selling without one
Sho：For Mercari, designing incentives involving handing out coupons is a form of investment. With a set budget for this project, there is a limit on how much we can invest, so we cannot help every seller with support like this for the long term.
Besides that, there are sellers who are selling their items without any incentive intervention at all, and there are others whose LTV would not improve even if we were to support them through something like the item coupons.
Due to this, we have to solve a mathematical problem that involves figuring out which sellers we should assist, which of their items to invest in, and how much to invest in each of those items within our limited budget so we can keep this measure going while also maintaining ROI efficiency.
To do this we first need to line up each seller’s currently listed items and predict the probability of each of these selling without intervention (probability of organic sale). With those numbers, we then do probability calculations to work out the chance that not a single one of the items will sell.
Next, we predict how much the probability of each item selling will be lifted when assigned a specific coupon and how much total sales will be. Then we make our decision. We solve these sorts of optimization problems with Python.
Item coupons are automatically assigned to items in the app like in the screenshot above
--How do buyers use the item coupons?
Sho：We use push notifications as the specific touchpoint for recommending discounted items to buyers, and we are considering expanding out to other touchpoints. For example, we are currently testing out these recommendations as a home screen component.
Item coupons themselves are set independently of buyers. As far as the notifications, we give thought to which buyers to send them to, as we are trying to create an experience where buyers are able to organically purchase the things they wanted at a bargain while they browse the marketplace.
Item coupon shown in a push notification
Item coupons shown on a home screen component
Koya：The Mercari home screen components usually show items that will grab a user’s interest. When they scroll to the “For You” page, items with coupons are suggested under “Bargains.” Unique to this is the need to optimize these items not just from the buyer’s standpoint but also from seller’s standpoint. When optimizing our recommendations to both buyers and sellers, we not only suggest items in response to the needs of the buyers, but also in ways that will maximize the profits of sellers.
While an extreme example, if we were to completely ignore the seller’s experience in this project, there is the possibility that some sellers whose items never sell, because buyers do not notice them, would stay that way. That is why this project is centered around the creation of successful selling experiences to improve seller engagement, and why it puts so much importance on the experiences of both buyers and sellers.
Sho：So, while we have established the effectiveness of item coupons, touchpoints will continue to be a challenge.
--When exactly are item coupons assigned?
Sho：They are currently assigned automatically to eligible items twice a week. We think that carrying out the process of identifying sellers that need help based on our predictions and assigning coupons to their items so often enables us to intervene on their behalf at just the right time.
We have automated the entire process end-to-end, including not just prediction, optimization, coupon assignment and notifications, but A/B testing, modeling of effect, cost and ROI, and other analyses, too. Since we are investing in the form of incentives, we need to look at their cost-effectiveness and scale them as appropriate.
Koya：Since Mercari is a flea market app, there is basically just one of each item listed available, and sellers have no surplus stock. Once an item sells, it is gone. If a seller’s other listed goods do not sell after that, then there is the risk their engagement will drop. This means we must constantly check on the current state of sellers and offer support at the right time, which makes the monitoring, predictions and automation to do this a necessity.
--Would you talk more about the Data Science Team? What synergies, if any, have you gained with the current members?
Sho：We have people from a variety of fields on the team, including engineering, data science, marketing and product data management.
Koya：A unique aspect of this team is that it is composed of analysts and developers all pursuing the same KPIs. While each group may have its assigned roles, the line dividing them is blurred, meaning everyone from one group does some of what the other group does as they carry out their tasks.
Sho：Since analysis and development have some overlap, everyone on the entire team knows our past successes and failures, the system’s current capabilities, and where the data they need is located. We can also move quickly: after deciding whether analysis or tests are needed, tests can be up and running in a week or two. I feel that a big plus with the way the team is currently structured is that we can carry out validations quickly while coordinating individual members’ priorities. We also have many engineers from outside the team who work with us.
--How do you feel about the results you have obtained from this implementation? Did you gain any insights?
Sho：The improvement in LTV produced by successful selling experiences was more than we had expected. After taking about four to eight months to watch seller engagement for a specific period, we found both sales and GMV had increased significantly.
Our team’s initial theory was that the number of items sellers listed would increase after they had successful selling experiences. However, there was a certain number of sellers who bought more items on Mercari as their sales grew. A direct factor in this was likely the money coming in from the completed sale leading straight into their next purchase. However, a possible indirect factor was that the user’s positive experience of their item selling on Mercari increased their engagement, which led smoothly into their next purchase. This is something we will need to continue to explore further.
Koya：At the start of the project, we did not predict that a successful selling experience would trigger the mutual increase of engagement for both buyers and sellers. That was a very interesting result: Encouraging the right sales with item coupons encourages buying by sellers, too. With this project we have learned that Mercari, being a C2C flea market app, is structured such that effects mutually diffuse into both buying and selling engagement. One unique aspect of Mercari is that we can grow LTV while going back and forth between encouraging buying and selling: focusing on just one does not have the same effect.
--Are there any other areas where there have been gaps between your initial expectations and the results you achieved?
Sho：That would have to be that sales resulting from the project exceeded our expectations. We had to review the numbers multiple times because we could not believe what they were telling us.
Something else we did not expect was the diversity of sellers’ responses that were triggered by successful selling experiences: for some only either their sales or purchases grew, for others both their sales and purchases grew, while still others showed no change at all. The results also differed depending on when sellers had their successful selling experience. In the end, the project has brought home to us that we still do not have a very clear understanding of our users.
Koya：In terms of having a diverse group of users, there was also a wide range between the amounts of sales they made.
--Is taking on the challenge of uplift modeling at Mercari fun or fulfilling?
Sho：This project is essentially us with our theories groping around every day in the dark. Yet, Mercari’s outstanding culture has enabled us to repeatedly test and invest despite the high level of uncertainty surrounding what we are doing. We have come this far thanks to the direction of @osari (Backend Team) and @shuichi (Data Analytics & Science Director), and @mappy, the Data Science Team’s manager, who is cultivating the project for the long run even when the results are not what we expect.
Koya：Data scientists and engineers are free to try things out here at Mercari. Even for theory-based trials, the company’s culture encourages you to “Go Bold,” do things without fear of failure, and use the insights gained in analysis to take the next step.
--Are there other things you want to pursue with this project in the future?
Sho：Since it is the selling experience that is at the heart of Mercari, I want to continue working on improving engagement centered on selling and holistically broaden efforts for doing so.
In the short term, I want to improve efficiency and optimization by deepening our understanding of our users and enhancing the precision of incentive assignment. Long-term, I’d like to broaden our scope and also try offering item recommendations using Mercari’s official LINE account in addition to our current home screen component touchpoint.
Koya：In the last six months, I think we have been able to create a selling experience that meets or exceeds what we set out to do through this project in the short term. However, many issues remain when looking long term.
Since it will be difficult for Mercari to perpetually assist every user with selling their items with incentives, in the medium to long term I would also like to work on providing sellers with recommendations about when and by how much to lower their prices, as well as offering suggestions related to improving how they take pictures of their items.
Sho：There are many users who do not know what to do to improve their photos or revise their items’ descriptions, or just cannot make the time to do so.
Koya：Among those items listed on Mercari that are not selling, there are some where lowering the asking price to match the average sold price, changing how the pictures are taken, or increasing the quantity offered would lead to an immediate sale. Using item coupons to encourage sales of these kinds of items will not fundamentally solve the reason they are not selling.
While from a motivation standpoint it is vital that users have successful selling experiences so they will use Mercari more, our long-term goal is to enable them to sell their items without the need for item coupons. In both the short and long term, I would like to create even more positive experiences for our users.
What do both of you see as further issues with the Mercari selling experience that need to be addressed?
Sho：We are aware of issues with things users put importance on in addition to successful selling experiences that we cannot solve using incentives.
We learned from a survey we carried out a while back that there are also sellers who value their interactions with buyers and communicating with others within the flea market app. I have children, and sometimes I will intentionally list their outgrown clothes at a low price in the hopes that some other child will continue to wear and enjoy them. When someone buys items listed in this sort of context, it makes the seller happy. This and other kinds of positive interactions between buyers and sellers can also serve as a motivation to use Mercari. Nothing would make me happier than to help encourage more of this.
Koya：And Mercari surely has value beyond just being a place where people buy and sell.
Mercari AI Team is recruiting new members to work with us!
We hope you’ll find it interesting and a good challenge to work in Mercari AI Team. Click here for more details.
Job Openings in the AI Field
