Decoding the Data Science Terrain: Part 7
In the six previous posts in this series, we talked about a framework for mapping the data science terrain (part 1), discussed ways to monetize and extract value from information (use cases, part 2), look at where datasets reside (part 3), and how data is extracted from sources (part 4), key considerations for storing and processing data (part 5) and learning from data (part 6).
In the last part of this series, we look at how organizations practice better decision making with data-driven insights. The focus of this part is on decision making, primarily by humans, but increasingly, in the emerging world of Artificial Intelligence, by machines.
It’s highly likely that over the next few years, Artificial Intelligence-driven machines will begin to make decisions related to a wide range of actions, especially those that are considered to be routine decisions.
The table below provides an overview of the type of decisions/actions that machines will excel at:
| Type of Decision | Description | Examples |
|---|---|---|
| Classification | Classify as good or bad | Potential borrower likely to charge-off, customer likely to be lost, object in the image likely to be a cat, etc. |
| Grouping | Identify natural grouping of customers | Segment customers into clusters |
| Association | Discover meaningful correlations between entities | Recommend next best product or offer based on other similar customers |
| Prediction | Forecast the future value of a continuous quantity | Forecast sales, forecast value of customers over lifetime |
| Natural Language Processing | Understand structure, recognize entities, extract meaning | Translation, generate summary from details, identify entities and their relationships |
| Visual Perception | Identify objects in an image or video, generate 3D world | Understand edges, texture, objects. Reconstruct 3D world, detect anomalies |
| Robotics | Manipulate objects in the real world | Mobile manipulators such as self-driving cars, assembly line part picking robots |
While robots and machines help a lot in decision making, they may not understand the full context in which the decisions are made. Today’s Artificial Intelligence is still brittle in comparison to the depth and breadth of human intelligence. While this is likely to improve, it’s still a long way off. This means that humans will still be involved in making critical decisions one way or the other.
However, the nature of human involvement in decision making is likely to undergo a radical change. Rather than making routine operational decisions themselves, humans will be more involved in making higher order decisions rather than operational decisions.
These actions are likely to be in directing machines do the right things, such as:
- Define the “problem” to solve or the goal to optimize
- Provide the machines with knowledge of their environment
- Formulate performance measures that defines some criteria of success for every decision
- Define the range of actions that the machines can perform
- Develop the algorithms that help the machines carry out these decisions
- Evaluate if the machines doing the right things, and understand why things go wrong
- Improve the capacity of the machines over time
Over time, proponents of AI claim that even these things will be taken over by machines.
However, until such time arrives, humans are firmly in control of decision making. When it comes to human decision making, there are a few things we need to consider:
Not everything is captured in data: A lot of managers seem to have taken to heart Edward Deming’s mantra of “In God we trust, all others bring data”. I’m a little skeptical that everything that matters to the organization can be quantified. For one, there’s way too much digital noise in the world out there. By the time the noise becomes a trend, it’s probably too late to do anything. As Peter Drucker writes in his classic book “The Effective Executive”, “the truly important events… are not trends… but changes in trends”. There’s a natural bias / error built into any data, and managers need to be aware of these biases before they bet everything on being data driven.
Intuition vs. being data-driven: While using data-driven approaches will certainly improve the quality of decisions, the key insight from behavioral economics is that humans are more emotional creatures rather than rational beings. Data cannot capture every nuance of a situation like intuition does. Moreover, intuition is seen as a short-cut to decision making, especially when there is little time or little data. With a lot of experience, intuition can lead to better decisions. Decision makers need not understand all root causes of every phenomena that impacts their organization, since doing so will only lead to analysis paralysis. There will always be a case for making quicker decisions based on intuition.
Information disfluency: Another important insight from behavioral economics is this: excessive information leads to cognitive overload on humans. Cognitive research has indicated that, contrary to popular belief, making it very easy to consume information may sometimes produce shallow levels of learning. Contrarily, making material harder to learn (disfluency) can improve long-term memory and retention, due to higher engagement need to understand and assimilate the material. We need to strike the right balance between being data-driven versus being overloaded with too much information.
To summarize, AI-powered machines are creeping into the domain of human decision making. Routine activities (such as driving a car) are being taken over by machines, and humans will focus on higher order decisions (such as optimizing these machines). We are far away from a world where everything is fully AI-driven. Until such time, humans need to know the art of making right decisions. My view is that not everything that matters can be captured in data, that intuition cannot be wished away, and that information disfluency is not always a bad thing.
Read part 1 of our blog series on ‘Decoding the Data Science terrain’ and learn more about the Data Science and Analytics framework. Click here to read part 2 of our blog series on ‘Decoding the Data Science terrain’! Find out how to evaluate the quality of data in part 3 of our blog series on ‘Decoding the Data Science terrain.’ Understand the steps in the data collection process in part 4 of this series. Find out the key considerations for storing and processing data (part 5) and learning from data (part 6).
