How have the organizational challenges of mercari ML team been solved? A chat with the engineers who launched the company-wide Guild

Currently, many ML engineers are part of Mercari's ML team. But those ML engineers aren’t all in the same department—they are scattered throughout the company, working to develop solutions to the challenges faced by their respective teams.

As the Mercari ML group expanded, this decentralization of its engineers created difficulties with sharing knowledge and coordinating between the different teams. To overcome those barriers, the Mercari AI/ML Guild (“ML Guild”) was launched in 2022 under the leadership of on-site engineers. The ML Guild consistently meets at least once a month to achieve the goal of sharing information among ML engineers.

The ML Guild, which started based on ideas from on-site engineers, is now held regularly. We spoke with three engineers who founded and manage the ML Guild—Abhinav Kumar, Paul Willot, and Yusuke Nishioka—about the history of the Guild, and how it has transformed some of the challenges of the organization.

Profile

Abhinav Kumar (@Abhinav)

Kumar joined Mercari as a new graduate in 2018. He started as an ML Engineer on the Image Search Team and then moved to the Data Science Backend Team where he worked as an Engineering Manager & Tech Lead, focusing on providing a personalized experience to Mercari’s users through ML. Kumar recently joined the Ads Engineering team contributing to developing the Mercari Ads Infrastructure. In addition to his interests in ML and backend development, Kumar also enjoys formulating efficient, agile processes within the engineering teams.

Paul Willot (@meatboy)

Paul joined Mercari in 2021 as a Deep Learning specialist. He has experience with the entire ML stack—from modeling and handling large data sets to runtime optimization and cluster deployment—which he leverages to bridge the gap between research and production.
Before coming to Mercari, Paul worked in fintech making real-time forecasting models at scale, and also consulted for major Japanese companies.

Yusuke Nishioka (@ysk24ok )

Yusuke had worked on improving ad delivery algorithms and setting up data infrastructure at an internet advertising company, as well as on development of a speech recognition product at an AI startup, before he joined Merpay in 2021. He was transferred to Mercari in 2022 and is currently involved in feature enhancement and maintenance of Mercari’s ML platform.

ーTo start, please introduce yourselves and tell me about your roles at Mercari.

Abhinav：I joined Mercari as a fresh graduate five and a half years ago. For the first two years, I worked on the ML team attached to Product Image Search. I then moved to the Trust & Safety (TnS) team. After launching the Data Science Backend team together with PMs, I worked as an engineering manager (EM). I am currently in the new Ads Engineering team.

meatboy: It’s been almost three years since I joined Mercari as an ML engineer. Ever since joining, I’ve been working on projects related to the listing of products. For about a year and a half, I worked with a team that used search engines to make it easier to match with buyers.

ysk24ok: I joined Merpay in 2021, then transferred to Mercari in 2022. Now, I am part of the ML Platform team, which provides infrastructure and solutions that optimize the work of ML engineers.

ーWhat is the origin of how the Mercari AI/ML Guild (“ML Guild”) was formed?

Abhinav: The ML Guild started in November 2022. While working as an EM on the Data Science team, we started the ML Guild after speaking with Umezawa (@umechan), an EM on the ML team.

We migrated the organizational systems to a new framework, converting the existing single ML team into units that corresponded to 4–5 different goals, something we dubbed the “Camp System”. ML engineers were then assigned to the various camps, and they remain organizationally dispersed to this day.

Previously, there was a lot of communication between ML members; but as new people joined the company following the introduction of the Camp System, communication among the teams became hard to maintain. In a survey, 80% of respondents believed that “the communication infrastructure for the entire ML team is currently insufficient.” That’s why we came up with the concept of the Mercari AI/ML Guild to build trust and communication between teams. First, we established the goal of getting to know every team and their members.

ML Guild aims to encourage the sharing of knowledge. The time between launching an ML project and actually executing it is very long, so we clearly needed a platform that facilitates the sharing of feedback, what’s going on within key projects, and best practices. At the time, it was unclear to new ML hires where they should seek help and what they should ask.

meatboy: Even before the ML Guild, we had several groups for discussion. But they were all separate from each other, with no single large group. A lot of projects are focused on very specific topics like ML academic research or search engines, and there wasn’t a place where people could discuss more comprehensive initiatives. The ML Guild provides opportunities to learn the latest news while sharing expertise related to many different areas of machine learning, without being limited to particular topics of interest.

—How is the ML Guild managed?

Abhinav: Management of the ML Guild is between multiple members, including the three of us here today. But we’ve actually had people participating as volunteers because they want to contribute. Currently, each camp has a representative leader, and those leaders determine the policies governing the ML Guild’s efforts as well as the goals set by the organization.

meatboy: Right now, all managing members have the same responsibilities. In other words, everyone is a primary organizer. For example, we take turns creating the scheduling for upcoming events. We hold events for team building or just to have conversations.

Abhinav: Basically, we rotate between three different events each quarter. At the first session, we share the OKR with each team; then in the second session, we share our projects and best practices and make related announcements. For the third event, we work on team building. Also, @meatboy plans a team-building party every three months, creating offline opportunities for people to meet up.

ysk24ok: For example, what if we introduce a few changes which result in a team having to migrate their work to a different location? We might have to make such announcements, and these sessions function as a way for us to do that.
If we want each team to use the new solutions developed by the Platform team, we must have them migrate from their existing framework. The ML Guild serves as a platform for these announcements.

ー In the years since it was established, has the ML Guild solved any organizational challenges?

meatboy: We’ve made clear headway in a number of areas. There are still scenarios where we need more rigorous support, and there remain some points that we want to address. But the ML Guild has been really helpful in dealing with some of the issues that we faced at the start. I think it’s fantastic that we can now get regular updates on what’s going on with the other teams.

I personally experienced from learning through the ML Guild that there was a team already working on a similar project, which allowed me to reach out to them and exchange models and knowledge. I’ve had this experience a lot with the ML Guild, building knowledge like that.

ー As ML engineers, have the efforts of the ML Guild changed anything for you?

meatboy: I regularly communicate with our internal platform team, but many members don’t have the opportunity to do so. As we are constantly developing new services, however, we have to ensure compatibility with existing systems. So it’s very useful to know who has already made certain improvements, or who to ask. I believe this has benefitted many of Mercari’s ML members.

It has also facilitated a lot of direct interaction among our engineers. I was working on projects involving the listing of products when I first joined the company, but I later transferred to the Search department. I had a lot of questions about the size of listed products, but once the ML Guild was created, I was able to speak directly to engineers that had dealt with such matters in the past. That was tremendously helpful.

Abhinav: This past year, we had an average of over 25 ML engineers participate in each session. And the people who participated once had returned to following sessions as well. Typically, the ongoing attendance rate declines as you continue holding these large-scale sessions. This is just my opinion, but the frequency of ML Guild meetings may be a factor in maintaining these solid attendance numbers. We are only holding them once a month right now, which may allow our busy members to participate without feeling stressed about it.

The ML Guild always meets each month. This creates the awareness that our events are consistently held. But the most crucial thing is that the ML Guild is led not by EMs, but by on-site ML engineers.

When I first joined Mercari, there were similar meetings with opportunities to share information, but it was led almost entirely by EMs. With the ML Guild, however, engineers can share their own insights gleaned from the projects that they themselves are working on. And as they present that information, they’re able to talk to many different people that could become their collaborators.

ysk24ok: As platform engineers, we sometimes create opportunities to discuss future collaboration. When doing so, there are some difficulties in setting up meetings with a team if you aren’t acquainted with any of its members. Thanks to the ML Guild, we have a chance to get to know the members of all teams.

ーIs there a future vision for the ML Guild?

Abhinav: As for short-term goals, we want to maintain our current pace, keep attendance up, and continue working on these initiatives. One of our medium-term goals is for the ML Guild to generate new business projects.

meatboy: We believe that the ML Guild can help us form cross-team projects, to have a bigger impact on business, and compete externally. We could even aim to become a platform in the future.

ysk24ok: Right. I don’t know whether this should be a short- or long-term goal, but only certain members have been attending the team-building sessions. And in the ML Guild right now, some members attend the sessions but don’t report on their own projects. I’d like to see how we can improve those areas.

ーWere there any initial difficulties in managing the ML Guild?

Abhinav: I’m extremely grateful to @meatboy and @ysk24ok for supporting this project from the start. The tricky thing at first was building trust.

“Why did they create these new sessions?” “Why are they necessary?” Since many people didn’t understand this, and everyone was ridiculously busy at their jobs, there wasn’t much participation. But we went to the EMs on each ML team and got their feedback on the ML Guild proposal. This slowly built trust with them, and eventually we got them to ask their team members to attend.

meatboy: At first, we also had to make a lot of announcements to get members to join. These steady efforts also improved the attendance situation.

One of the things we’ve learned from managing the ML Guild is that we must strike an appropriate balance between team-building events and events for sharing beneficial information.

And I think we always have to ask ourselves, “What is the purpose of this meeting?” Moving forward, I feel that we must continue to provide value that justifies the use of everyone’s time.

Abinav: I hope for even more people to join in the future. The ML Guild is independently operated, so I’d like to welcome new members that can rotate in so that we can keep our fresh perspectives while maintaining our current level of skill.