This is our second Case Study, one of a small bunch that Xpanse AI team will share in this blog about applications of Predictive Analytics in Marketing.
We are reminiscing our past projects executed in different workplaces with the hope that it will provide some ideas for Marketing Teams and their Data Scientists on how to use Predictive Analytics for improving their business practice. We are also open to snarky comments from “been there, done that” veterans if they notice bugs and errors in the projects we describe.
The names of the victims are changed but all stories are real.
The list of all Case Studies can be found here: Predictive Analytics in Marketing – Case Studies
Key Learning from this project: Advanced technology can look like magic – and it’s not good.
Part 1 – selling the project
The Cross-Sell project for an insurance company was almost sold: 50 man-days agreed with the CEO to improve their sales process. We were meeting Ian – Head of Sales, before the contract gets officially signed.
Ian: “So you guys are telling me, that you can increase our conversion rates for the Household Insurance product?”
Ian: “How exactly?”
We: “We will take your data and use Predictive Analytics to profile your customers and select those who don’t yet have Household Insurance but who are most interested in buying it“.
Ian: “Yeah but how?”
Ugh, this one is a tougher cookie.
We: “We will aggregate all the information about customer base and their historical purchase patterns and then we will apply algorithms – like Decision Trees or Logistic Regression to train the Predictive Models. After that, the Models will be able to separate your customer base into ‘interested’ and ‘not interested’.
You will need to target the ‘interested’ profiles and stay away from the ‘not interested‘. Here, take a look at this Gain chart.”
By the end of this tirade, Ian started to frown. He wasn’t looking at the Gain chart.
Ian: “I have 150 thousand contacts on our database that I can potentially contact. 20% of those leads have already incorrect phone numbers – that’s bad enough and you are telling me to call even less because some ‘model’ will tell so? I’ve been doing this for 20 years and this sounds really dubious to me. I am not going to waste my leads by not calling them.”
But.. Decision Trees..
Where is Margot Robbie to explain Predictive Analytics when we need her?
We started feeling the heat of the impending crash-and-burn.
Ending the meeting like that would mean the death of the project.
A stroke of brilliance came from outside our Data Science team. You see – when Data Scientists look for hard problems to solve – sales folk look for the easiest avenues to revenue.
Our Account Manager broke the awkward silence: “Ian, if you have a problem with data quality – perhaps we could help you on that front. How about we take your data in and cleanse it, so your sales reps will have a better quality leads for calling?”
Ian: “That could be actually helpful.”
Part 2 – office fight
Back in our office we threw a tantrum. We are not data cleansers. We won’t be working on the phone numbers. We were hired to do advanced analytics, not shmensing-cleansing.
Our Account Manager had none of it.
“Sorry to break it down to you, but our client’s CEO does not give a damn about Predictive Analytics. All he cares is to optimise his business. We sold a sales optimisation project for 50 man-days.”
The “not giving a damn” part hurt, not gonna lie.
How about doing both? We will spend a few days improving the quality of phone numbers and at we can also build a model – albeit limiting its scope to match the remaining budget.
We agreed to do just that.
Data Understanding and Preparation
We split the work into 2 streams. It took less than 10 days to sort out the phone numbers and we focused on building the Modelling Dataset. Within 45 days we managed to build Modelling Dataset with ca 300k records and 120 columns – we initially hoped for more but had to scale back the design due to new time constraints.
Modelling & Validation
The Modelling was conducted in SAS Enterprise Miner with a standard set of ML techniques. The final model was quite strong – AUC over 0.85.
“Simplify, simplify, simplify”
Here is where the gotcha! moment came.
Ian didn’t trust Predictive Analytics and only expected cleansed phone numbers – so how do we make the model scores useful?
The Data Science team came up with a 20 page Powerpoint deck explaining how Predictive Analytics works and how we applied it to the problem.
Our Account Manager was less than impressed. “This is death by PowerPoint. Ian will never buy into this deck. It’s long, too detailed and he doesn’t really care for this stuff“.
At the end – he said – it’s about a cut-off point where you stop calling your customers. Other details might be just as well irrelevant.
He sketched a following break-down:
This way the official main delivery (improving data quality for phone numbers) was mixed with the predictive scores. He also blended in a high level insight as for WHY they should consider not calling some of the customers.
Our Client did not expect that but once it was there – it was hard to ignore.
The deck was cut to 5 slides, there were no Gain Charts in it and the word “Predictive” was not even used.
To the Data Science team it was a medieval butchery. We felt like our work should be held proudly above our heads, not buried beneath 4 thick-cut “tiers“.
With all the complicated stuff out of the picture the delivery meeting went smoothly though.
Soon-after we started scoring the customers on a weekly basis and we observed with quiet satisfaction that whenever the Call Centre dipped into Tier 3 (lowest scores) the calls were futile – and Call Centre agents were quick to notice.
Xpanse AI Team