OperationsFramework / modelModelAccessible

Data mining

How can data mining support strategic choice or positioning?

AccessibleStrategicIndividual2 min read

Contents

Data mining is often used as a buzzword of generic description applied to any form of large-scale information processing, but this is not very accurate.

Data mining is often used loosely to mean any large-scale data processing, but its purpose is more specific: discovering useful, previously unrecognised patterns and relationships. The name can be misleading because the objective is not to extract more data; it is to turn data into validated knowledge that improves a decision.

When to use it

Use data mining when a substantial dataset may contain patterns capable of improving prediction, classification, detection or prioritisation. The resulting insight can reduce cost, support planning, change strategy and lower decision risk.

A discovered association does not explain its own cause. The method can expose a pattern, anomaly or dependency, but additional analysis or experimentation may be needed to establish why it occurs and whether acting on it will work.

Typical questions include:

Which characteristics are shared by the most profitable customers?

How can customers in the smart-watch market be grouped?

Which signals recur in fraudulent transactions?

What navigation paths do website users follow?

Origins

Data mining emerged from the convergence of statistics, database systems, artificial intelligence and machine learning. The knowledge-discovery-in-databases community formalised the field through workshops beginning in the late nineteen eighties. Usama Fayyad, Gregory Piatetsky-Shapiro and Padhraic Smyth later distinguished data mining—the algorithmic pattern-discovery step—from the broader process of selecting, preparing, interpreting and deploying useful knowledge.

What it is

Data mining is the automatic or semi-automatic exploration of data, often very large business datasets, to find patterns, anomalies and dependencies that are both previously unknown and potentially useful. It combines computational techniques with domain judgement: a pattern must survive validation and lead to a relevant action before it becomes business insight.

How to use it

The work has three stages:

initial exploration and preparation;

model building and validation;

deployment.

Stage 1: Initial exploration. Define the decision and outcome, then clean the data, transform variables, select appropriate records and reduce an unmanageably large feature set. Explore distributions, missingness, leakage and bias before modelling. Depending on the problem, this may range from choosing straightforward predictors for a regression to broad exploratory analysis that determines the model family and complexity.

Stage 2: Model building and validation. Train credible candidate models and compare them on data not used to fit them. Competitive evaluation applies several approaches to the same task and selects according to out-of-sample performance, interpretability, operating cost and business constraints. Common ensemble techniques include bagging, boosting, stacking and meta-learning; the Statsoft reference below provides additional detail.

Stage 3: Deployment. Apply the selected model to new data to generate predictions or estimates, integrate the output into a real workflow and monitor data drift, performance and unintended effects. Buying software is not a substitute for a clear decision, sound data or accountable ownership.

Practical example

As discussed in Correlation Analysis, Walmart reportedly found that Pop-Tarts sales rose when hurricane warnings were issued.

The relationship was unexpected—flashlight demand was easier to anticipate—but operational action did not require a causal theory. Walmart could place Pop-Tarts near the front of the store when severe weather was forecast and test whether availability and placement increased sales. The example shows the value of a discoverable association while also illustrating why a pattern should be validated before broad deployment.

Top practical tip

Use data transparently and for a defined customer benefit. Explain what is collected and why, obtain the required permission, minimise the data used and test whether model errors disadvantage particular groups. Ethical design is part of model quality.

Top pitfall

Anonymise or aggregate records when individual identity is unnecessary, but do not assume anonymisation is irreversible. Protect raw data, derived features and model outputs throughout their lifecycle. Walmart needed the weather-related sales pattern, not the identity of every Pop-Tarts buyer.