Aditya Chaubey — AI Daily

What a Matmul Actually Costs

Fri, 30 Oct 2026 03:30:00 GMT

Take a 4,096 by 4,096 matrix of weights. Push 128 token vectors through it in half precision, on an A100.

The arithmetic is easy to count. Every output value is a dot product of length 4,096, and there are 128 by 4,096 of them. So the card has 4.3 billion multiply-adds to chew through. Spread over its rated 312 trillion operations a second, that is 13.8 microseconds.

Then you run it, and it takes about 17.5 microseconds.

For one afternoon the gap looks like launch overhead, or the operating system, or whatever your profiler labels as noise. Then you run the same multiply on a shape half a percent larger, and it takes twice as long. The floating point count did not move nearly enough to explain that.

Two bills arrive at once here. Neither of them is the arithmetic you just counted.

The first bill

A multiply of an m by k matrix with a k by n matrix has a cost you can write down. It is 2 × m × n × k floating point operations.

The 2 is honest. Each output value is a dot product of length k, which takes k multiply-adds, and hardware fuses each pair into one instruction. NVIDIA’s performance guide counts the fused operation as two, because the multiply and the add are both real work on real numbers.

Sit with what that formula can see. It takes three integers. It cannot see the values inside the arrays.

Multiply matrices full of random noise, and you get that bill. Multiply matrices full of the number one, and you get the same bill. Multiply matrices that are ninety percent zeros, and you get the same bill again.

That last case makes people stumble. A sparse matrix is not cheaper to multiply unless the chip is built to notice the zeros. NVIDIA’s A100 datasheet lists 312 TFLOPS for half precision, and 624 with an asterisk for sparsity. The second number describes a card that can prove half the weights are zeros, and then decline to multiply them.

Every output entry is a dot product of length k, and there are m times n of them. That is where 2 × m × n × k comes from. Image: User1042, CC BY-SA 4.0, via Wikimedia Commons.

The second bill

The arithmetic is only half of it. Every number the chip multiplies has to arrive from memory first, and memory is a slower road than the arithmetic units beside it.

Count the arrivals for one multiply. The inputs are m by k and k by n. The output is m by n. If nothing is reused from cache, that is m×k + k×n + m×n values read or written, whatever their size in bytes.

Divide the two and you get a ratio with a name. Arithmetic intensity for a matmul works out to m×n×k / (m×k + k×n + m×n). NVIDIA prints that same expression in their guide, and works two examples. Their 8192 by 128 by 8192 case scores 124. Their 8192 by 8192 by 8192 case scores 2,730.

Now you need something to compare it against, and that number belongs to the card rather than the problem. Take peak arithmetic and divide by peak bandwidth. The A100 with 80 gigabytes runs at 312 TFLOPS and 2,039 gigabytes a second, so 312e12 / 2.039e12 gives 153.

That 153 is flops per byte. Above it, arithmetic limits you. Below it, memory does.

The idea is old and has a good name. Samuel Williams, Andrew Waterman and David Patterson published it as the roofline model in 2009. The flat part of the roof is the arithmetic you cannot exceed. The sloped part is the memory you cannot feed any faster.

Our 128-token multiply lands at 120 flops per byte. That is under 153, so memory sets the pace. Count the bytes and you get 35.7 megabytes moving, which at 2,039 gigabytes a second takes 17.5 microseconds. That is the number the stopwatch gave us. The arithmetic was never the constraint.

The same 4,096-wide layer, pushed harder. Attainable throughput climbs almost linearly until the batch clears about 165 tokens, then the card is full however many tokens you add.

There is a clean consequence. A square half-precision matmul needs a side of at least 459, since 3 × 153 = 459, before arithmetic becomes the limit. Anything smaller cannot reach peak, however good the code.

And a matrix times a single vector, the case where n is 1, has an intensity below 1. That is a model generating one token at a time. Its ceiling on this card works out to about 2 TFLOPS out of 312, because the processors spend nearly all their time waiting for weights to arrive. This is the entire reason batching exists.

The cliff

Now the shape that breaks the stopwatch, and intensity has nothing to do with it.

An A100 has 108 streaming multiprocessors. cuBLAS cuts the output into tiles, and its preferred tile is 256 by 128. Fix m at 2,304. That divides into nine tiles down the columns.

Set n to 1,536. Across the rows you get twelve tiles, and 9 × 12 = 108. One wave of tiles, every processor busy, nothing left over.

Set n to 1,544, which is eight columns more. Now you get thirteen tiles across and 117 in total. One full wave, then a tail wave of nine tiles using 9 of the 108 processors. NVIDIA measured this and reports that throughput roughly halves, while duration roughly doubles. Half a percent more work, twice the clock.

Tile counts for m = 2,304 with 256 by 128 tiles. 108 fills one wave exactly. 117 needs a second wave, and only nine of its tiles have work to do.

There is a smaller version of the same trap. With 128 by 128 tiles, two dimensions that miss the boundary by a little make the hardware run 1.5 times the arithmetic the maths needs. The problem itself is 0.39 percent bigger.

None of this shows up in the flop count. All of it shows up in the shape.

Ten times the bill for the same answer

Matrix multiplication is associative, so a chain of them can be bracketed any way you like. The result is the same. The cost is not.

Take three matrices of size 10 by 100, 100 by 5, and 5 by 50. Bracket them as (A × B) × C and you do 5,000 multiplications, get a 10 by 5 result, then spend 2,500 more. Total, 7,500.

Bracket the same three as A × (B × C) and you do 25,000 multiplications to build a 100 by 50 middle, then 50,000 more to finish. Total, 75,000.

Same answer, ten times the bill. Cormen, Leiserson, Rivest and Stein use exactly this chain, and call the first order “10 times faster”.

The cost of a matmul is a function of its shapes and its brackets. The values never enter.

Where a training run’s bill goes

Put enough matmuls together and you get a language model, and the count stays just as simple.

One parameter touches one token in the forward pass and costs two flops. The backward pass needs twice that, which is four. So training costs about six flops per parameter per token, written 6 × N × D. The OpenAI scaling laws paper of 2020 put it as C ≈ 6NBS and named the 6 outright: the forward and backward passes.

Chinchilla is the tidy example. Seventy billion parameters, 1.4 trillion tokens, so 6 × 70e9 × 1.4e12, which is 5.9e23 flops. Gopher was four times larger and burned a near-identical budget. The smaller model won nearly every benchmark.

That rule has a hole, and the people who wrote it said so. Kaplan and his coauthors admit that their estimate “did not include contributions proportional to n_ctx”. They warn that the scaling may be confounded once context length climbs past roughly twelve times the model width.

Here is how slowly that arrives. Attention’s cost grows with the square of the context. The JAX scaling guide derives the ratio of attention flops to the rest as T / 8D, with T the context and D the model width. For a width of 4,096, attention only overtakes the dense layers at 32,768 tokens. At a width of 8,192, it takes 65,536.

The quadratic blowup is real. It arrives at the context lengths people were arguing about in 2026, rather than the ones most of us train at.

What the clock is made of

Go back to the 4,096 by 4,096 matrix and the 128 tokens, and the 17.5 microseconds.

You could rewrite every value in those two arrays and the time would not move by a nanosecond. You could fill them with zeros. You could multiply them in a different language, on a better compiler, and shave a little off. The values are free.

Change the shapes, and the bill moves by a factor of ten. Change the brackets, and it moves again. Round the dimensions up to the tile boundary, and it halves.

A matmul costs what it is shaped like.

How to Split Your Data Without Lying to Yourself

Wed, 28 Oct 2026 03:30:00 GMT

Priya has repaired laptops from the room above the shop for two years. She has 240 enquiries to show for it, from sixty customers, because most people come back.

She wants one thing from those messages: a guess, made before she quotes, about whether a customer will approve the repair or flinch at the price and go quiet.

So she does what every tutorial says. She writes each enquiry down next to what happened next. She holds back every fifth message, trains a small model on the other four fifths, and scores it on the ones it never saw. Then she rotates which fifth is held back, four more times, and averages the five scores.

The number comes back good. It is also wrong, and the model is not what went wrong.

Her sixty customers sent four messages each on average. When she deals the messages out like cards, a held-out message usually has siblings in the training pile. Take one held-out message: three other messages from that customer exist somewhere, and the chance that at least one of them sits in the 192 rows used for training works out to this.

1 - (47 × 46 × 45) / (239 × 238 × 237) = 0.993

Ninety-nine point three percent. The model is almost never asked about a stranger.

The loop, and the one rule inside it

Cross-validation rests on a small idea. You cannot test a model on the data that taught it, so you take turns.

The same data, four times over. Each fold takes its turn as the test set while the other three train, and the four scores get averaged. Image: Joan.domenech91, CC BY-SA 3.0, via Wikimedia Commons.

Five folds means five passes. Forty-eight messages are held out each time, a hundred and ninety-two train, and every message is held out exactly once. The code is four lines.

from sklearn.model_selection import cross_val_score
from sklearn.neighbors import KNeighborsClassifier

scores = cross_val_score(KNeighborsClassifier(3), X, y, cv=5)
print(scores.mean(), scores.std())

The rule that makes those numbers mean anything is smaller than the machinery. Everything the model learns, it has to learn again inside each fold, from the training part only.

Scaling, feature selection, the number of neighbours, the decision threshold, the cleaning steps you chose after looking at the data. All of it belongs inside the loop. Move one step outside and the fold stops being a test.

Why five folds, and not one

A single split gives you one number from one arrangement of the data. Deal the cards again and it moves, often by several points. Averaging five arrangements costs five trainings and buys most of the stability you need.

You could go further and hold out one row at a time. Leave-one-out trains on 239 rows instead of 192, so it squeezes more value from a small dataset, and on paper it is almost unbiased.

It fails in a way worth seeing. Ron Kohavi trained two classifiers over half a million times on real datasets and found that ten-fold stratified cross-validation beat leave-one-out for choosing between models, even when computation was free.

His sharpest example is a trick rather than a benchmark. Take Fisher’s iris, fifty flowers of each of three species, and give it a classifier whose only rule is: predict the most common label in the training set.

On the full dataset that rule scores about 33 percent. Now remove one flower at a time. Remove a setosa and the training set still holds fifty versicolor and fifty virginica, so setosa is no longer the largest class. The classifier predicts the wrong species for that flower, and for every other flower too.

Leave-one-out reports zero percent accuracy for the majority rule on iris.

Then it reports zero variance across the folds, and that is the part that should bother you. The estimate arrives with a confident spread of literally nothing.

When the shuffle lies

One library document says the quiet part out loud. scikit-learn’s guide calls the assumption behind shuffling “Independent and Identically Distributed”, and spells it out: every sample comes from the same process, “and the generative process is assumed to have no memory of past generated samples.”

Priya’s data has memory. Every message remembers the customer who typed it.

The fix is to split on the thing that repeats. GroupKFold takes a column of group ids and keeps each group whole, so a customer’s four messages all land on the same side of the wall.

A version of Priya’s problem built to test this: sixty customers, four messages each, a nearest-neighbour model, two hundred repeats of the whole experiment. Shuffled folds scored 73 percent. Splitting by customer scored 51, and sixty customers the model had never met scored 50.

The gap was recognition doing the work, not knowledge. The model had learned to tell one regular apart from another, and a new customer is nobody it has met.

This is not a corner case. In 2023, Sayash Kapoor and Arvind Narayanan surveyed the published reviews and found data leakage in 294 papers across 17 scientific fields. They sorted it into eight kinds. In the civil war prediction literature, where newer models were reported to beat logistic regression outright, they re-ran the comparisons with the leakage removed. The difference in ranking quality between the complex models and the old regression fell from 0.14 to 0.01.

When the shuffle lies twice: time

Time has the same memory problem, in a form shuffling cannot see.

scikit-learn is blunt about it. Classic folds on time series data “would result in unreasonable correlation between training and testing instances”, producing poor estimates, and the right test is always the future observations “least like those that are used to train the model”.

TimeSeriesSplit does this by growing the training window. The first fold trains on January and tests on February. The second trains on January and February and tests on March. Nothing ever runs backwards, so the model never gets to read next week’s news and then predict this week.

The number under the number

Suppose the split is right, the groups are whole and the order is honest. The score is still an estimate from a sample, and a sample has a width.

Gaël Varoquaux measured that width directly. He ran cross-validation on simulated data of known difficulty, then compared the estimate against the accuracy on ten thousand fresh points. At thirty samples, ninety percent of the errors fell within about nineteen percentage points of the truth. At a hundred samples the spread narrowed to nine. At a thousand, to three.

How far a cross-validation estimate misses, from Varoquaux’s simulations: about nineteen points at thirty samples, nine at a hundred, seven at two hundred, six at three hundred and three at a thousand.

The spread you get by averaging across folds does not show this. Varoquaux’s summary is that the standard error across folds “strongly underestimates” the real error, because the folds share a training set and their mistakes travel together.

Some of the width is pure arithmetic, and it belongs to nobody’s model. Predicting a class at 75 percent accuracy, measured on a hundred test items, has a standard deviation of 4.3 points from sampling alone. A 95 percent interval around that 75 percent is 17 points wide. On thirty test items it is 31 points wide.

Real datasets behave the same way. In a Kaggle competition on brain scans of schizophrenia patients, the gap between a method’s score on the public and private test sets ran to about 15 points. And in the civil war re-analysis, one reported ranking score of 0.85 came with a bootstrapped 95 percent interval of 0.66 to 0.95. Nine of the twelve papers they could fully check reported no significance test or uncertainty measure for the comparison.

The folds you tuned on

Here is a subtler leak, and a common one. You run five-fold cross-validation across a grid of settings, pick the setting with the best score, and report that score.

That best score is the maximum of several noisy numbers. Maxima sit above average by construction.

Sudhir Varma and Richard Simon tested this on data where the two classes were identical by design. There was nothing to learn, so a fair estimate had to come out at chance. Tuning by cross-validation and reporting the winning score gave an error under 30 percent on 18.5 percent of the simulated datasets for one classifier, and on 38 percent for another. On genuinely independent data, both did no better than a coin.

Their fix is a loop inside the loop. The inner folds choose the settings, the outer folds measure. That nested estimate came out close to the independent test set for both classifiers. The plain version had been answering a different question: how well does the best of these settings do on the data that chose it.

What to do on Monday

Decide what the score is a promise to, then make the split keep it.

If this repeats in your data	Split with	The score then describes
Nothing	`KFold`, or `StratifiedKFold` for classes	a new row like these
A person, patient, writer or shop	`GroupKFold`	a new person
Time	`TimeSeriesSplit`	next month
Nothing, but you tuned settings	nested cross-validation	a new dataset

Then hold back one set you touch once, at the end, and report an interval rather than a point.

Back to Priya

Her number was never wrong about her own records. It was answering a question she had not asked: how well can you recognise a customer you have already met, from the messages they already sent you.

The customer at the counter is not that customer. He has never written to her before, and the only thing the model can use is the sentence in front of her.

Keep him out of the training data from the start, and the number you get back is small, unflattering, and about him.

The Word Everyone Uses Is Worth Nothing

Tue, 27 Oct 2026 03:30:00 GMT

Priya repairs laptops from a room above a shop in Thane. Customers send one-line complaints on WhatsApp, and by Friday the messages pile up faster than she can read them.

She keeps four of them in a text file to test a search box she is building.

my laptop battery drains in an hour.

laptop battery drains fast after the update and now the battery dies by noon.

the update killed my laptop speaker.

my laptop screen flickers when it is warm.

Then she types the three words she always types: laptop battery drains.

The first of those words cannot help her. Every message in the file is about a laptop. A word that appears in all four messages tells you nothing about which one to open.

That silence is the design.

A bag holds words, not sentences

Every search system, old or new, begins by turning text into numbers. The oldest way is to tip each message into a bag and count what falls out.

Word order goes first. In the bag, battery drains laptop and laptop battery drains hold the same three words. The bag cannot tell a question from an answer, or a complaint from its fix.

Grammar goes next. The bag does not know that drains is a verb in message one and a noun in a plumbing manual.

What is left is a small table. One row per message, one column per word that appears anywhere in the file, a number in each cell. Most cells hold zero.

That table alone ranks badly, because a long message beats a short one just by having more cells filled. Two corrections fix it, and the second is where the famous weight comes from.

What a word is worth

Start by asking how rare each word is. Count the messages that contain it. That count is the document frequency, and the first correction is to divide the number of messages by it, then take a logarithm.

weight = ln(messages in the file / messages containing the word)

With four messages the arithmetic comes out like this.

Word	Messages containing it	Weight
laptop	4 of 4	0
my	3 of 4	0.288
battery, drains, update, the	2 of 4	0.693
fast, speaker, screen, hour	1 of 4	1.386

Read the bottom row, then the top. A word in one message weighs 1.386. A word in half the messages weighs 0.693, exactly half. A word in three of four weighs 0.288. The word in all four weighs nothing, because the logarithm of one is zero.

Every time a word’s reach doubles, its weight halves, and a word that appears in every message is worth exactly nothing.

Nobody wrote a stopword list for that to happen. Nothing punished “laptop” for being boring. What punished it was sheer reach, and reach is a number you can count.

Each doubling of a word’s reach costs it half its weight. The rule needs no dictionary of English, only a message count.

Real collections look like this at scale. A small number of words sit in almost every document, and a long tail sits in one.

Every word in War and Peace, ranked by how often it appears. The fitted line is Zipf’s law, and the lower panel is what that line leaves unexplained. Image: Cosmia Nebula, CC BY-SA 4.0, via Wikimedia Commons.

Why take a logarithm

The first reason is the ladder. Reach doubles, weight halves, and the steps stay gentle enough to compare words whose reach differs by a thousandfold.

The second is arithmetic housekeeping. If you assume words appear independently, the chance of seeing two of them together is a product. Take logs and products become sums, so a document score can be the sum of one weight per word. Every ranking system after this one kept that shape. The base of the logarithm never matters, only the ratio inside it.

There is a popular story that this weight measures the information a word carries, in Shannon’s sense. Stephen Robertson went through those arguments in 2004 and found them wanting. His better answer: the weight is what a naive Bayes model hands you when you know nothing about which documents are relevant. It predicts relevance without ever being told what relevance is.

Four messages, scored

Score a message by adding the weights of the query words it contains.

Message one contains laptop, battery and drains. That is 0 plus 0.693 plus 0.693, so 1.386.

Message two contains the same three words, so 1.386 as well.

Messages three and four contain only laptop. Both score zero, correctly. One is about a speaker, the other about a screen, and Priya did not ask about either.

So the arithmetic returns a tie, and it shouldn’t. Message two is the one that says the battery dies by noon. It is also the only message that names the battery twice.

Count mentions as well as presence and the tie breaks. Inside message two, battery contributes 0.693 twice, so the message scores 2.079 against 1.386.

Message two wins, which is right. It wins by 50 percent, which is too much. It earned that lead by repeating a word, and it is twice as long as message one, fourteen words against seven. Two sloppy reasons to be confident.

The experiment that failed

In 1972 a researcher at Cambridge named Karen Spärck Jones published a short paper in the Journal of Documentation. She was trying to explain a failure of her own.

She had been grouping terms into clusters to help a search find more of what it should. The clusters kept getting taken over by terms that appear in many documents, and anything that made those terms match more documents pulled in noise with it.

The obvious fix was to delete them, and she tested that. On the Cranfield collection one term appears in 144 of 200 documents. Cutting frequent terms from INSPEC meant deleting 73 words of 1,341, and it lowered the ceiling on recall by at least 30 percent. On the Keen collection, recall fell from 75 percent to 25 percent.

People write requests in common words. In the Cranfield collection the average vocabulary term pointed at nine documents, while the average term used in a request pointed at 31.6.

So delete nothing, and pay the frequent words less. Her own version was cruder than the formula above: a whole number built by rounding to powers of two, with logarithms to base 2. On Cranfield’s 200 documents a term in 90 of them scored 2, and a term in 3 scored 7. She credits the smooth version we use today to Roger Needham, a Cambridge colleague.

She also notes how long it took to matter: AltaVista used this weighting from the start, about twenty-five years after the paper.

The paper justified its weights with a nod to Zipf’s law, and Robertson’s later analysis finds that argument weak. The measure outlived every explanation offered for it, including the one printed with it.

Where the other half of the name came from

The 1972 paper has no term frequency in it. That half came from Gerard Salton’s group at Cornell, who worked with abstracts and short texts, where how often a word appears inside one document carried signal. Salton and Yang published the combination experiments in 1973.

Two researchers, two kinds of data. Spärck Jones read presence and absence in word lists, so the number of documents was her variable. Salton read prose, so the number of mentions was his.

Multiply the two and you get tf-idf, though the multiplication was never derived from anything. It was assembled and then tested.

The ceiling

The last correction made this family of scores survive contact with real search engines. It is called BM25, published in 1994, and it adds one thing to tf-idf: a curve.

Counting mentions pays a flat rate forever. Six mentions are worth six times one. BM25 pays a lot for the first mention, less for the second, and approaches a ceiling it can never cross.

credit = (k1 + 1) × count / (k1 + count)

With k1 at 1.2, the value Lucene uses by default, one mention is worth 1.0, two mentions are worth 1.375, and ten are worth 1.964. The ceiling is 2.2 no matter how many times the word appears.

Raw counting never stops paying. BM25 gives the first mention most of the credit and then creeps towards a ceiling of 2.2.

Run Priya’s message two through that curve. Its doubled battery drops from 1.386 to 0.953, the message total falls from 2.079 to 1.646, and its lead over message one shrinks from 50 percent to 19 percent. The ranking does not change. The confidence behind the ranking does, and that is the honest thing to adjust.

There is a second dial for length, b = 0.75 by default, which discounts the length of a message against the average length of the collection. Message two is twice as long as message one, and that extra length earns it almost nothing.

Both dials were tuned by experiment. Robertson describes them as global parameters that are generally unknown and may be fitted to evaluation data, which is a polite way of saying the field guessed, measured, and kept what worked.

Still the baseline to beat

In 2021 five researchers at Darmstadt published BEIR, a benchmark of 18 retrieval datasets across nine tasks, and ran ten systems on them with no in-domain training. The average change in ranking quality against plain BM25:

System	Average change vs BM25
BM25 plus a cross-encoder re-ranker	+11.0%
ColBERT	+2.5%
docT5query	+1.6%
TAS-B	−2.8%
GenQ	−3.6%
ANCE	−7.4%
SPARTA	−20.3%
DeepCT	−27.9%
DPR	−47.7%

Read the bottom row twice. DPR, a BERT-based dense retriever, averaged 47.7 percent below a formula with no neural network in it at all.

The cost is worse for the newcomers. BM25 answers a query over a million documents in 20 milliseconds on a CPU, with a 0.4 GB index. The winning combination needs 450 milliseconds on a GPU, or 6.1 seconds on CPU. On the 15-million-document BioASQ collection, ColBERT’s index runs to about 900 GB against BM25’s 18 GB.

The benchmark also flatters the old method. On the TREC-COVID dataset, 6.4 percent of BM25’s top ten had never been judged relevant or not, against 31.8 percent of TAS-B’s. When the researchers annotated 980 of those missing pairs by hand, ANCE climbed from 0.654 to 0.735, which put it 6.7 points above BM25 on the same data.

Both things hold at once. A formula from 1994, sitting on top of one from 1972, is a serious baseline, and the test sets that make it look unbeatable were built with formula-shaped tools.

What the bag cannot see

Order. battery drains and drains battery are one bag. Adding no to a complaint raises its score, because no is another word in the bag.
Synonyms. A customer who writes power instead of battery matches nothing. Zero overlap, zero score. This gap is the entire reason dense retrieval exists.
Word relationships. Every word is scored on its own, so nothing in the model knows that screen and flickers belong to the same fault.

The library you will probably use gives numbers that do not match mine, and not because I made a mistake.

scikit-learn adds one to the top and bottom of the fraction before taking the log, as if it had seen an extra message containing every word. Its documentation says the effect is that words occurring in all documents “will not be entirely ignored”. So laptop comes out at 1.0 instead of zero, my at 1.223, a word in half the messages at 1.511, and a word in one at 1.916. The ladder survives. Only the zero disappears.

Robertson also notes that the older formulas, which keep the collection size on top, give a word in more than half the documents a negative weight, meaning its presence counts against you.

Try it on your own messages

Export a hundred of your own messages and count how many contain each word. Sort the list and look at the top twenty. You will find the words that describe your situation in general and none of your actual problem.

Then put two search words in reverse order and watch what your own search tool does with them.

Back to Priya

Her search box has never read a sentence of English. It has no idea what a laptop is, or that a battery dies by noon. It knows which words sit in which messages, and that in this particular file, this particular week, battery is worth paying for and laptop is worth nothing.

She types the three words she always types. Message two comes back first, and it should.

Relevance turns out to be mostly a question of what is rare in the room you are searching.

The Machine That Only Asks Yes or No

Mon, 26 Oct 2026 14:30:00 GMT

You have played this game. Somebody thinks of a person, a place or an object, and you get twenty questions to find it. Every question has to be answerable with a yes or a no.

Watch someone who is good at it. They do not guess. Their first question splits the world roughly in half, is it alive, and their second halves whatever is left. Someone who is bad at it asks whether the thing is smaller than a bus, rules out almost nothing, and has spent a question for free.

That instinct, the one you already have, is the whole algorithm behind a decision tree.

The only move it knows

A decision tree predicts by walking you down a chain of questions. Each one compares a single column of your table against a number: is the distance above twenty kilometres, is the hours-studied figure above four. You answer, you go left or right, you answer the next one, and wherever the chain runs out, the prediction is waiting.

A column here means one fact recorded for every row. Hours studied is a column. Whether a student turned up to lectures is another. The tree can ask about hours, and it can ask about lectures. It cannot ask whether the cost is more than twice the distance, because that question mixes two columns at once.

That restriction sounds like a weakness. In practice it is where most of the tree’s strengths come from, and most of its limits too.

Ten students walk into an exam

Ten students sat a mock exam. We know how many hours each of them studied, whether they attended lectures, and whether they passed.

Hours studied	Went to lectures	Result
1	no	fail
2	no	fail
3	yes	fail
4	no	fail
5	yes	pass
6	yes	pass
7	no	fail
8	yes	pass
9	yes	pass
10	yes	pass

Five passed and five failed. Student seven is the awkward one: seven hours of study, no lectures, and a fail anyway. Tables are full of rows like that, and the whole lesson of this article lives in what a tree does with them.

How mixed is this pile?

The tree builds itself by asking a question of its own at every step: which cut leaves the two new piles less mixed than the pile I started with?

So it needs a number for mixed. The usual one is the Gini impurity. Before any symbols, here it is in a sentence. If you picked two students out of a pile at random, how likely is it that one passed and the other failed?

Our ten students are five and five. Pick two at random and they disagree about half the time, so the pile scores 0.5. A pile of four students who all failed can never disagree with itself, so it scores 0. Everything else lands somewhere in between.

A pile where everyone passed is perfectly clean, scoring 0. A fifty-fifty pile is as mixed as it gets, scoring 0.5. Every cut is an attempt to move left along this curve.

The formula is Gini = 1 - (p1² + p2²), where p1 and p2 are the shares of each class in the pile. Take the share of passes, square it, take the share of fails, square it, add the two together, and subtract the total from one. Half and half gives 1 minus 0.5, which is 0.5. A single class gives 1 minus 1, which is 0.

The squaring is the useful part. It punishes small shares, so a lopsided pile scores low and a balanced one scores high.

The first cut

Now the tree tries every column at every threshold, and keeps whichever cut lowers that number the most.

Cut on hours, at four. The left pile is students one to four, all four of whom failed, so it scores 0. The right pile has one fail among five passes, which scores about 0.278. Combining them means weighting by size, four students left and six right, so 0.4 × 0 + 0.6 × 0.278 = 0.167.

The pile began at 0.5 and is now 0.167. That is a drop of 0.333.

Cut on lectures instead. The students who attended split into one fail and five passes, scoring 0.278. Those who skipped split into three fails and one pass, scoring 0.375. Weighted: 0.6 × 0.278 + 0.4 × 0.375 = 0.317. A drop of 0.183.

Hours wins, 0.333 against 0.183, so hours becomes the first question. Neither cut is clean, and that is fine. The tree only ever needs the cut that is best available.

The first question does most of the work. The second buys a little more, and the box on the right is still mixed. One failed student is hiding in there.

The rule that exists to be wrong about one person

The leftover box holds four students, three passes and one fail. It is student seven again.

The tree can fix this. Ask one more question, hours at seven, and student seven drops into a leaf of his own: perfectly pure, impurity zero. The score improves. The model gets worse.

This is overfitting, and that single-row leaf is the disease in miniature. A rule whose only job is to be wrong about one student fewer on an exam you have already marked teaches you nothing about the next exam. Leave a tree to grow without limits and it will keep splitting until every leaf is pure, which on real data means roughly one rule per training row.

Every library gives you the same three brakes, and the defaults are loose enough that you get the overfitted tree unless you intervene. Cap the depth. Demand a minimum number of rows in each leaf. Or prune afterwards, charging the tree a fee for every leaf it keeps. If your leaves hold one or two rows, you did not fit a model. You indexed a spreadsheet.

What the tree is really doing to your data

Step back from the students and look at the shape of the thing.

Plot two columns as a flat sheet of paper. A question about hours is a vertical line. A question about lectures is a horizontal line. Every cut is a wall parallel to one axis or the other, and the leaves are the rectangular rooms those walls enclose.

A real tree, trained on the iris dataset: 150 flowers, four measurements, three species. Do not bother reading every number. Look at the shape. A handful of yes-or-no questions is enough to separate three species almost perfectly. Image: Dvd8719, CC BY-SA 4.0, via Wikimedia Commons.

That picture explains what a tree cannot do. Suppose the rule is “pass if the hours and the attendance agree, fail if they disagree”, which gives you two diagonal stripes. No vertical or horizontal wall separates diagonal stripes, so the tree has to cut again and again, building a staircase to imitate a diagonal.

scikit-learn’s own documentation says so plainly: the algorithm is greedy, it makes the best-looking choice at each node, it cannot promise the best possible tree, and it struggles with concepts like XOR and parity. Greed is what keeps a tree fast and readable.

Why it still beats neural networks on spreadsheets

In 2022, three researchers assembled 45 datasets shaped like spreadsheets and tuned every model properly, neural networks included. Tree-based models still came out on top on medium-sized tables of around ten thousand rows, before anyone counted how much faster they run.

The explanation comes down to three things, and all three follow from the staircase.

Neural networks expect smoothness: nearby inputs should give nearby outputs. That is a gift for images. Tables are not smooth. A credit score of 699 is fine and 700 is refused. Trees expect nothing smooth, because they are flat by construction.

Neural networks also suffer when a table carries useless columns, since they have to spend capacity learning that a column is noise. A tree never picks it, because a useless column produces no gain.

And tables are built by people, one column at a time, so the useful directions already line up with the axes. Trees cut along those axes. Neural networks mix the columns together first and lose that alignment.

Quinlan’s 1986 paper on the ID3 algorithm has been cited around fourteen thousand times, which tells you how long this idea has been earning its keep.

Three things it cannot do

It cannot extrapolate. Every prediction is the average of the rows that landed in that leaf, so it never returns a value beyond the range it saw. Show it a house twice as large as anything in training and it hands back the most expensive box’s average.
It is twitchy. Change a few rows and the top question can flip, reshuffling everything below it. Two trees fitted on different halves of the same data can look unrelated. Averaging many of them into a random forest is the standard fix.
Its probabilities are coarse. A leaf with three passes and one fail reports 0.75, which is a frequency taken from four students. Treat it as a rough ranking rather than a risk score.

Where to go next

A few good places, roughly in the order I would take them.

Watch it drawn by hand. StatQuest: Decision Trees walks through impurity and splitting on a whiteboard in about seventeen minutes. If the Gini paragraph above felt slippery, this fixes it.
Play with a real tree. explained.ai’s tree visualiser lets you change a split and watch the predictions move. It is the fastest way to feel what a cut does.
Take the free short course. Kaggle’s Intro to Machine Learning is about four hours and ends with you training your own tree in a browser.
Run the iris example yourself. scikit-learn’s decision tree plot is the picture from this article, generated by a short script you can copy.
Read a book, if you want the maths. An Introduction to Statistical Learning is free to download. Chapter 8 covers trees and forests properly, and it assumes less than the title suggests.

The last question

Every model in the family, the forest and the boosted ensemble that wins most Kaggle tables and the ones with names you have not met yet, is a variation on one move: where do I cut?

Learn the tree as a staircase and the rest of the family reads as refinements of the same question.

So the next time an insurance form asks you five yes or no questions and then quotes you a price, you will know what just happened. You walked down a decision tree, and somebody chose where the walls go.