In Moneyball to Machine Intelligence, I explored an idea that professional sports understood long before many enterprises did:
Data matters because it changes decisions.
Baseball statistics don’t win games.
Telemetry doesn’t drive a Formula 1 car.
And machine-learning predictions don’t make business decisions.
People—or increasingly intelligent systems—must decide what to do with those predictions.
That led me to another question.
How does a machine know when it has enough information to act?
That question takes us from Moneyball into decision science.
Start With 100 Baseball Players
Imagine that I run a baseball organization with one open roster position.
There are 100 prospects.
I can evaluate them one at a time.
Once I reject a player, however, I can’t go backward.
My objective is to select the best candidate.
Should I take the first impressive player?
Probably not.
Should I evaluate all 100?
I can’t. By then every opportunity has disappeared.
This is a version of one of the classic problems in mathematics: the secretary problem.
Under its strict assumptions, mathematics provides an unusual strategy.
Observe approximately the first:
of the candidates.
Don’t select anyone.
Use them to establish what “good” looks like.
Then begin searching for the first subsequent candidate who exceeds everything seen in that initial sample.
For 100 candidates, that means learning from approximately 37 and then becoming willing to act.
The number 37 gets most of the attention.
I think the more important insight is this:
There is a time to learn and a time to act.
Prediction Is Not Decision
This distinction matters enormously in AI.
Suppose a machine-learning model tells me:
Player 44 has an 86% probability of becoming a high-value player.
That’s prediction.
The model has estimated:
But management has a different question:
Should we sign Player 44?
That decision may depend on salary, roster availability, alternatives, injury risk, the cost of waiting and what we have already learned about the talent pool.
The prediction is one input into the decision.
It is not the decision itself.
Building the Experiment in Microsoft Fabric
I decided to model this idea as a small Microsoft Fabric experiment.
The system begins with synthetic historical baseball data containing statistics such as:
- on-base percentage;
- slugging percentage;
- walk and strikeout rates;
- exit velocity;
- defensive contribution;
- age;
- and projected salary.
A machine-learning model learns the historical relationship between these measurements and future player value.
We then introduce 100 new prospects.
The model calculates a probability for each candidate.
But we add another layer above the model.
A decision policy.
For the first 37% of candidates, the policy says:
LEARN.
After the observation period, candidates are compared against the strongest prospect observed during that window.
The system can then return:
KEEP SEARCHING
or:
SELECT
The result is something I find much more interesting than a traditional prediction demonstration.
The machine isn’t simply saying:
“This prospect has an 86% probability of success.”
It is saying:
“Based on what we have observed so far, the available alternatives, our decision threshold and this candidate’s expected value, this is the point where acting is preferable to continuing the search.”
That’s decision intelligence.
Give an Agent the Same Problem
Now the experiment becomes even more interesting.
The scored results are written back into Microsoft Fabric.
A data agent can query those governed results and answer a question in plain language:
We have one roster position. Should we select the current player or keep searching?
The agent can examine:
- where we are in the candidate sequence;
- the reference threshold;
- the model’s predicted probability;
- salary;
- and predicted value relative to cost.
This is an important distinction in thinking about agentic AI.
A retrieval system asks:
What do we know?
A predictive model asks:
What is likely to happen?
Decision science asks:
Which option produces the greatest expected value?
An agent must eventually ask:
Do I act now, gather more information, or wait?
That final question changes everything.
37% Is Not the AI Rule
There is an important warning here.
Businesses should not suddenly begin making decisions after examining exactly 37% of every dataset.
The secretary problem contains assumptions that rarely survive contact with reality.
Real candidates may return.
Their quality may change.
The number of future opportunities may be unknown.
Decisions may have different costs.
Multiple options may be acceptable.
Information itself may have a price.
That is precisely why the simple 37% experiment is useful.
It gives us a starting point.
Modern decision systems replace the fixed rule with techniques such as:
- optimal stopping;
- expected utility;
- Bayesian decision theory;
- contextual bandits;
- reinforcement learning;
- dynamic programming;
- and regret minimization.
The question evolves from:
When should I stop searching?
to:
What is the expected value of learning one more thing before acting?
This is the key.
Jackie Robinson
source:britannica.com
Moneyball Was Really Decision Science
This is where I return to Moneyball.
The revolution wasn’t simply that baseball discovered statistics.
Baseball already had statistics.
The innovation was deciding that certain measurements deserved more weight in decisions about scarce resources.
Which player gets drafted?
Who gets traded?
Who gets the roster position?
How much should the organization pay?
Billy Beane’s advantage was not possessing infinitely more information.
It was developing a different decision policy from competitors looking at much of the same world.
That should sound familiar to anyone building AI systems today.
Competitive advantage may not come from possessing the largest model.
It may come from designing the better decision system around the model.
From Machine Learning to Machine Judgment
For decades, computing has increasingly answered:
What happened?
Business intelligence helped answer:
Why did it happen?
Machine learning increasingly answers:
What is likely to happen next?
Agentic AI introduces another frontier:
What should happen next?
And eventually:
Should I act now?
This is why decision science may become one of the most important disciplines in the agentic-AI era.
An agent capable of accessing thousands of documents but incapable of deciding when it has gathered enough information isn’t particularly autonomous.
An agent that predicts outcomes without understanding costs isn’t making a business decision.
And an agent that always acts on the highest probability without considering uncertainty, alternatives or consequences isn’t displaying intelligence.
It is optimizing a number.
Knowing More Versus Knowing When
There is a temptation in AI to associate intelligence with possessing more information.
More parameters.
More context.
More data.
More compute.
But decision science suggests another definition.
Perhaps intelligence is partly the ability to recognize when additional information no longer changes the decision enough to justify waiting.
Moneyball showed us the value of seeing something other organizations overlooked.
Machine learning gave us increasingly powerful ways of predicting what might happen.
Agentic AI is forcing us to confront the next question:
When should a machine stop analyzing and begin acting?
The 37% rule doesn’t answer that question for every situation.
It does something more useful.
It teaches us that waiting has a cost, acting has a cost, and intelligence lives somewhere in the mathematics between them.
Project source and references:
Ferguson, Thomas S. “Who Solved the Secretary Problem?” Statistical Science 4, no. 3 (1989): 282–289. A useful historical and mathematical treatment of the secretary problem and optimal stopping.
Gilbert, John P., and Frederick Mosteller. “Recognizing the Maximum of a Sequence.” Journal of the American Statistical Association 61, no. 313 (1966): 35–73. One of the foundational treatments behind the optimal-stopping result associated with the approximately 37% rule.
Robbins, Herbert. “Some Aspects of the Sequential Design of Experiments.” Bulletin of the American Mathematical Society 58, no. 5 (1952): 527–535. Foundational work underlying the multi-armed bandit problem and the exploration-versus-exploitation tradeoff.
