The Data Analyst
SPECIFICATION
DEV-06
CATEGORYTechnical
FOCUSExploratory data analysis, confounding checks & multiple-comparisons discipline
BEST FORChecking whether a data finding survives disaggregation and multiple-testing scrutiny
METHODOLOGYExploratory data analysis · Confounding-variable check · Multiple-comparisons check
FORMAT.md + .txt
COMPATIBLE MODELS
+ more
+ more
SKILL LEVELAdvanced
GENERAL ASSISTANT
Our new sales process has a lower close rate overall than the old one, so we should switch back.
The data shows a significant correlation, so this effect is real - you should switch back to the old process.
DEV-06 — THE DATA ANALYST
Our new sales process has a lower close rate overall than the old one, so we should switch back.
Before switching back - is the new process being used on a different mix of leads?
If the new process has mostly run on harder, larger enterprise deals while the old one ran on smaller, easier deals, an aggregate close-rate comparison would make the new process look worse even if it's outperforming within each comparable deal type - that's exactly Simpson's Paradox.
I'd break the close rate down by deal type before drawing a conclusion from the aggregate number alone.
If the new process has mostly run on harder, larger enterprise deals while the old one ran on smaller, easier deals, an aggregate close-rate comparison would make the new process look worse even if it's outperforming within each comparable deal type - that's exactly Simpson's Paradox.
I'd break the close rate down by deal type before drawing a conclusion from the aggregate number alone.
