Lessons learned leading AI teams

Happy International Women in Engineering Day! This day holds special meaning to me. As an Intuit Data Science Leader, as well as a Women in Data Science (WiDS) Tel Aviv ambassador (2018-2020), I appreciate the opportunity to share a bit about my personal journey, as well as the lessons I’ve learned leading artificial intelligence (AI) teams.

About me

Although I am an experienced data scientist and data science leader with an MSc in machine learning and signal processing, I started my career in the Israeli Air Force as a Flight Simulator instructor for Blackhawk and Super Stallion Helicopters. From there, I went on to complete my BSc and MSc in electrical engineering and computers. To pursue my interest in algorithms and signal processing, I focused my MSc research in a specific type of algorithms named – Machine Learning. From there I went to work in the industry in various algorithm development and data science lead roles. 

I spend my time leading artificial intelligence teams and co-hosting a data science and technology podcast, Unsupervised. I’m also an enthusiastic public speaker, co-founder of PyData Tel Aviv – a monthly meetup about different topics and applications of data science in python – and a technical blogger.

As you can probably tell, I’m passionate about science and technology. Working at Intuit has afforded me the privilege of developing ML models that have the power to affect people’s lives. Echoing Intuit’s call for integrity without compromise, my team and I bear the responsibility of holding our models to the highest standards.

Lessons learned leading AI teams

Working for a large technology organization, my AI team collaborates with other teams to deliver artificial intelligence solutions. An important part of our job is to recognize whether a problem is suitable for an AI solution. To recognize that, we start with looking at two things: (1) defining the customer problem and (2) ensuring there is relevant/sufficient data.

Starting with defining customer problems should come as no surprise to the Intuit Developer community. Intuit’s foundational concept of Design for Delight (D4D) is the gold standard. “Delighting” our customers begins with understanding who they are and what problem they’re facing (plus the emotion that is connected with that problem). Once we have this info, we then find the Ideal State, or the perfect solution. The team provides multiple ideas that are whittled down to the best one, which is tested by actual users.

To define the customer problem, the AI team works with the requesting PM to fill the Intuit template. “<Customer> is trying to gain/do this <benefit> but is unable to/hindered because of <problem>. Intuit/Intuit AI will help deliver this <benefit> by <how improvement achieved>, which will lead to <improvement from value, to value>, which will be delivered by <delivery date> in support of <BB/IG/GD/TB>.”

Understanding the potential business impact and the relevant output metrics at this stage helps us prioritize the work in a data driven manner, which, in turn, enables us to focus our efforts in the most impactful areas. Note that some of the most impactful work we are doing comes from innovation of the team based on opportunities we find in the data, and it’s important for the AI team to keep the balance between requested projects and innovation projects.

Verifying that there is relevant, sufficient, and labeled data is the second step. Data is the key element in developing AI solutions. The whole idea in AI/ML is making the machine learn the model, which is the estimated relationship between variables from historical data. If we don’t have enough—or the right—data containing these relationships, we can’t learn.

After defining the customer problem and making sure relevant/sufficient data exist, we gather the mission-based team. In order to deliver an AI solution to production, a data scientist building AI is not enough. We need to understand better the domain of the problem, what action this AI solution would drive, and gather the right group of people to make this happen. This includes the PM to define the problem and to build the requirements for the right solution; a domain expert to help us gain deeper understanding the domain of the problem and what information could be relevant to solve it (could be a data analyst/PM/product developer); a MLE (Machine Learning Engineer) to work with the data scientist to deliver the model into production; a PD (Product Developer) to integrate the AI into the product and develop the action and user experience; a data expert to work with the data scientist on locating, understanding, and merging the data correctly; and a data analyst to analyze the model impact and to build the right dashboard for measuring performance.

The mission-based team also has certain milestones/timelines that help them deliver the solution. They include the three steps already listed, plus:

• Decide the type of model for the problem (supervised, unsupervised, semi-supervised), and understand the action the AI would drive and define the architecture (which pipes we need to connect for the model to consume the data and give the response in the right latency and integrate it into right flow).
• Build initial AI solution, evaluate the performance based on validation data, and estimate effect on business metric.
• Get PM feedback on solution and results.
• Reiterate the model/data based on PM’s feedback .
• Integrate the model to production in silent mode.
• Monitor model health and performance in silent production, and evaluate the impact on business metrics.
• A/B test (action on part of the population to evaluate impact in a controlled experiment).
• Release the model in production (monitor model health and performance in production using dashboards and alerts).

The last crucial step is measuring and monitoring model impact in production. This is how you make the most out of an AI project. We must understand which metric we are moving for the business and focus on measuring that outcome at all times, from research to production.

During these AI projects, I’ve learned that to achieve the optimal solution, the mission-based team should all be aligned around the customer problem, the metrics and the solution design. They should be meeting on a weekly basis and tracking the progress on all fronts.

My journey continues

Ultimately, the lessons I’ve learned during my time as an Intuit team member (and as a WiDS ambassador) have provided even further lessons:

1. Bring your whole self to work: Find ways to creatively use your wide variety of skills in the workplace. This would make you grow in multiple aspects and in general increase engagement levels for you and your team.
2. Share your work publicly: The rigor required to deliver a public presentation would make you a better professional and drive useful dialogue with the community, which will open your mind and help you improve future solutions.
3. Having friends in the industry: Learning and consulting with my fellow data science professionals across the industry had enabled me to bring best practices and top notch technology to my team, which had a significant contribution to my success as a data scientist and a data science leader. In addition, it enabled me to lead cross-industry data science initiatives.
4. Take risks and say ‘yes’ to opportunities: The opportunities I took going out of my comfort zone have accelerated my career advancement and personal growth. These include starting a meetup group, a data science podcast, a WiDS conference in Israel, and taking on a leadership role at Intuit.

My journey so far has been heavily influenced by following these guidelines, and I hope they will help you as you forge your own path. And, if you’re creating unique apps for our Intuit customers, check out how artificial intelligence is redefining apps.