Evolving the ML Platform organisation at Netflix: a case study

Please log in to watch this conference skillscast.

Do you wish there was a Machine Learning model to tell you how to structure your ML teams? So do I! While we're waiting for that, I'll share the story of how the ML Platform organisation evolved at Netflix. Although this story is specific to our own journey to expand Netflix ML investments, there are a few lessons learned along the way that you'll be able to relate to. There are several factors going into org structure that we'll discuss, including: the specialty and skillsets of ML practitioners, the variety and depth of ML use cases, who's responsible for the data, the ownership model as ML projects go to production, and how the underlying Platforms are situated. I look forward to sharing and hearing your own thoughts afterward!

Q&A

Question: Do AEs at Netflix prototype ML models themselves or just validate/deploy models prepared by others e.g. data scientists? Considering the differences of skills and responsibilities, do AEs normally earn more than DSs? (Please ignore this Q if confidential)

Answer: AEs do research, prototyping in addition to productionising their models!

It's hard to compare salaries apples-to-apples because different folks are valued for different skills. So I wouldn't make a generalisation that one makes more than the other!

Question: How long did it take for Netflix to understand the differences between algorithm engineers and data scientists?

Answer: I would say that the key difference is where they fall on the software engineering spectrum and that folks were explicitly hired with a desired level of this skill. I would also say that AEs working in Personalisation were more likely to have a background in search & recommender systems.

Question: How important do you think having a product research team composed of Data Scientists/ ML Engineers is to a business? It seems like a lot of organisations have the ML hammer but are not always clear on which nail to hit. What’s your advice for setting up teams to identify potential use cases apart from capitalising on already existing customer data?

Answer: There are a lot of methods that are much easier to apply than ML, so I definitely don't think ML is right for every problem! It really depends on the business and the data.

Depending on the problem setup basic heuristics, mathematical models or even really basic linear models would be a starting point. I'd want to establish a baseline that needs to be improved upon before applying ML.

You also have to be able to define an objective to optimise and collect a dataset that can be labeled reliably with "correct" answers for the model to learn on.

Bottom line, see if you can come up with the least fancy way to solve the problem first, see how far you can get and when you've exhausted that option, start to consider ML, but not before. The size of the prize also needs to be big enough. If you have $1B in revenue but the opportunity is to save costs or increase revenue by $100K, it is likely not worth the cost to pay the Data Scientist to do the work!

Download Slides as PDF

YOU MAY ALSO LIKE:

• How Metaflow became Netflix’s beloved Data Science framework (SkillsCast recorded in December 2022)
• How to Scale: Rethinking the Product Design Career Ladder (SkillsCast recorded in November 2022)
• How to Scale: Rethinking the Product Design Career Ladder (SkillsCast recorded in November 2022)