The Emergence of Product Analytics: An Under-appreciated Yet Critical Part of AI Development
What Would You Say … Ya Do Here?
The term data science has always been nebulous. When the term began gaining currency in the early 2010s, the tech industry relied more on a diagram than words to define it. In practice, things were not easily delineated, either. Early data scientists tended to have broad, shifting job descriptions. They could work on a machine-learning (ML) model in the morning before switching to an ANOVA in the afternoon.
More than a decade later, I’m pretty sure the role of a data scientist is even less clear. At some companies, people with this title serve as machine-learning engineers (MLEs), building and deploying models in service of the buzzword that has supplanted data science: AI. (For the record, machine learning is AI, and these disciplines are not new.) Data scientists in these positions are people with deep knowledge of ML, from model conception to deployment. They are also skilled in ongoing monitoring and maintenance.
In other environments, data scientists (including me) focus on product analytics. Workers in this field may be called data analysts, product scientists, or something similar. Product analytics emphasizes the use of statistical methods to extract insights from data (often capturing customer activity), which in turn inform business decisions. It’s not all A/B tests and Kaplan-Meier Curves, though. A product analyst might also build an ML model to explain a predictive relationship among features. The emphasis would be on isolating and reporting on the relationship, though, rather than putting the relevant model into production.
Product Analytics as AI Support
To be sure, the MLE role aligns more intuitively with AI development. But that doesn’t mean that product science has nothing to contribute to the field. In fact, product analytics offers several key complements to ML. For one, it often informs opportunity identification, since product analysts are typically skilled data detectives who can spot relevant trends in data before any model selection begins. In this early analysis, product analysts identify features with predictive value, messy data that needs cleaning, and necessary assumptions that must be satisfied before proceeding. These steps also significantly reduce risk in the modeling process.
Beyond risk mitigation, product analytics also supports AI development through prototype modeling. Prototypes offer useful proofs of concept (or better yet, proofs of value) by providing evidence that AI methods can solve stakeholders’ problems and add business value. Explanatory models like these drive model development by showing which features have the most predictive value, for instance.
Consider an example: a monthly doorknob (Change it every month to keep your front door fresh!) delivery service wants an AI tool to identify customers at risk of canceling their service. Rather than training and deploying a model at the outset and then evaluating results, they could have product analysts build a prototype. Exploratory analysis would show the team which features correlate the most with canceled service, and a quick baseline model would ensure that an effective model can be built (a legitimate question, given the potential paucity of customers for this company). Such a model would also unearth information around the explanatory value of different variables, as well as illustrating the business value (or lack thereof) of the planned AI tool.
Finally, on the tail end of model development, it’s critical to gauge model performance, effects on customers, and overall viability of the output. Enter experimentation — another specialty of product analysts. This work ensures that the model does in fact bolster customer retention without unforeseen downside. Experimentation also enables and powers other analytical methods, such as customer segmentation, tracking the effectiveness of discounts, and more.
Interpreter of Models
Despite its wonkish, technical origins, product analytics is largely rooted in effective communication. Practitioners need to clearly explain technical concepts to non-technical stakeholders — elucidating the inner workings of a model and why it makes certain predictions, for example. Their work might also entail talking through experimentation options with product managers to ensure they achieve their desired goals. Regardless of the task, effective product analysts must excel at telling data’s story.
Macro-level communication about data and AI is essential, too. As the use (and societal discussion) of AI hastily grows, so too does the importance of explaining it. (Is it too much to think of it as a public service at this point?) If you’ve ever wondered why LLMs hallucinate, or why recommendation engines are slow to understand your interest in Robert Altman, a product analyst can often help. Through thoughtful visualizations and written reports, product analysts establish and explain the interaction between business technology and everyday consumers.
One last communication skill that I’ve always highly valued in other product analysts is healthy skepticism. I believe it’s important to start every project with a question in need of an answer, as well as a valid reason for answering it, lest project teams encounter the dreaded solution in search of a problem. To paraphrase a former manager of mine, the answer to the question “Can we build it?” is typically yes; the important question to answer is “Should we build it?” Put differently, every data project should start with a clear, defensible raison d’etre, ideally rooted in existing data and available customer insight. Product analysts are essential here, as they specialize in customer insight and can help product teams make informed decisions about whether an AI tool will be effective and — critically — ethical.
Thanks for reading! If you’d like to read more of my work, or just enable my own shameless self-promotion, please follow my publication here.
