In 2025, Stanford researchers did something that would have sounded almost heretical a decade ago.

They published a paper at ACL titled “Drop Dropout.”

Yes: drop dropout.

Dropout was introduced by Geoffrey Hinton and his colleagues in 2012 as a way to prevent overfitting in deep neural networks. It quickly became one of the foundational techniques of the deep learning era.

For years, it was everywhere. BERT used it. GPT-2 used it. Countless neural networks treated it as standard equipment.

And yet, if you look at today’s large-scale language models, dropout has largely disappeared.

So what happened?

Where did dropout go?

And perhaps more interestingly: did it really become obsolete—or did it simply find a new job?

1. Why Large Models Are Dropping Dropout

At its core, overfitting is a simple problem:

The model sees too little data, too many times, and starts memorizing instead of learning.

Imagine a student with a tiny question bank. They keep practicing the same problems over and over. Eventually, they may memorize not only the answers, but also the quirks and noise of the training set.

Neural networks can do something similar.

They can also develop what we might call “collusive learning.”

A few neurons may repeatedly specialize together: one learns one part of the pattern, another learns something else, and the network relies on the same internal partnerships again and again. The result can work beautifully on the training data—but fall apart when the input changes.

Dropout was designed to disrupt exactly this kind of behavior.

During training, dropout randomly turns off a fraction of neurons on each pass. No neuron knows who will be available next time, or which partners it will have.

The effect is similar to forcing a team to reshuffle itself for every practice session.

Instead of relying on a fixed group of neurons to solve a problem, each neuron has to become more robust and useful on its own.

But here is the catch:

Modern large-scale pretraining is a very different environment from the one dropout was originally designed for.

Large language models are trained on enormous datasets. In many large-scale pretraining setups, the model may see each training example only once—or only a small number of times.

There is simply much less opportunity to repeatedly memorize a small training set and develop the kind of brittle co-adaptations dropout was designed to prevent.

In other words, the original problem that dropout was meant to solve becomes much less severe.

And when the problem changes, the regularizer may no longer be worth the cost.

That is essentially what the Stanford paper Drop Dropout investigates.

The researchers compare different dropout rates across models ranging from BERT to Pythia, including models from roughly 160 million to 1.4 billion parameters. Their experiments compare dropout rates such as 0, 0.1, and 0.3.

Across the main metrics they evaluate, removing dropout generally performs better than keeping it.

And the larger the dropout rate, the more pronounced the degradation becomes in their experiments.

This does not mean dropout suddenly became a bad technique.

It means something more subtle:

The problem dropout was designed to solve is no longer equally important in large-scale language-model pretraining.

Dropout didn’t become weaker.

The environment changed.

2. In Physical AI, Dropout Is Still Doing Its Old Job

If dropout has largely disappeared from large language-model pretraining, that does not mean it has disappeared from machine learning.

Far from it.

In physical AI, dropout is still very much alive.

Consider computational fluid dynamics and automotive aerodynamics.

The DrivAerNet++ benchmark, introduced at NeurIPS 2024, is a large-scale dataset for learning aerodynamic quantities from 3D vehicle geometries. Its reference implementation, RegDGCNN, uses 0.4 dropout in its fully connected layers as part of its standard configuration.

NVIDIA’s PhysicsNeMo ecosystem also continues to support dropout. Its GeoTransolver model for automotive external aerodynamics, for example, incorporates concrete dropout, a formulation that allows the network to learn an appropriate dropout probability rather than relying entirely on a manually chosen value.

Why does dropout remain useful in physical AI?

A big part of the answer is simple:

The data regime is completely different.

Small datasets. Many epochs.

Robotics datasets can be painfully small.

A particular task may have only dozens—or a few hundred—demonstrations. Imitation-learning systems such as ACT can train for thousands of iterations or epochs.

That means the model can see the same examples again and again.

The situation is almost the opposite of large-scale language-model pretraining:

• LLMs: enormous datasets, relatively few passes over each example.
• Physical AI: expensive data, relatively few examples, repeated training.

When a model repeatedly sees the same small collection of demonstrations, memorization becomes a real concern again.

That is exactly the environment where dropout starts making sense.

The same is true for physics simulation.

A single high-fidelity CFD simulation can require enormous amounts of computational resources. Generating enough simulation data to simply “scale the dataset” is often prohibitively expensive.

You cannot always solve the generalization problem by throwing another trillion examples at it.

So the old problem comes back:

When data is scarce and training is repeated, overfitting is still alive.

And when overfitting is alive, dropout still has a job.

3. The More Interesting Story: Dropout Reinvents Itself

But if you think dropout survived merely by continuing to prevent overfitting, you are missing the more interesting part of the story.

Dropout found a second career.

Uncertainty quantification.

In industrial applications, knowing the model’s prediction is only half the problem.

The other half is knowing how much you should trust it.

Suppose an AI model tells an engineer:

“The drag coefficient is 0.25.”

The next question is obvious:

“How certain are you?”

For a chatbot generating a paragraph of marketing copy, uncertainty may be an inconvenience.

For an engineering model making a design decision, uncertainty can be critical.

A wrong prediction can lead to the wrong geometry, the wrong simulation, or the wrong engineering decision—with potentially very expensive consequences.

This is where dropout can take on a completely different role.

MC Dropout: Turning Randomness into a Signal

Normally, dropout is disabled during inference.

MC Dropout does something unusual:

It keeps dropout active at inference time.

The model is run multiple times on the same input. Because a different random subset of neurons is dropped each time, the model produces slightly different predictions.

For example:

Run 1 → 0.25
Run 2 → 0.24
Run 3 → 0.26
Run 4 → 0.25
…

If the predictions stay tightly clustered, the model is relatively consistent.

If they spread out:

0.18 → 0.31 → 0.22 → 0.37 → …

the model is signaling that its prediction is unstable.

The variation across these stochastic forward passes can therefore be used as a proxy for predictive uncertainty.

This is Monte Carlo Dropout, or MC Dropout.

The underlying idea is surprisingly elegant:

Instead of asking the model only what it predicts, ask it the same question many times—and pay attention to how much its answers change.

For physical AI, that distinction matters.

An engineer may not only want to know:

“What is the predicted drag coefficient?”

They may also want to know:

“How uncertain is that prediction?”

But Dropout Is Not a Magic Uncertainty Meter

There is an important caveat.

MC Dropout is useful, but it is not a perfect solution to uncertainty estimation.

For example, dropout-based uncertainty does not automatically give the model a strong sense of distance from the training distribution.

A model can encounter an input that looks very different from anything it saw during training and still produce a seemingly confident prediction.

In other words, the model can be confidently wrong.

There are also situations where uncertainty estimates can collapse toward something much closer to a deterministic prediction, reducing the usefulness of the stochastic signal.

That is why modern physical-AI systems often combine dropout-based uncertainty with other techniques for distribution-shift detection, calibration, ensembles, Bayesian approximations, or uncertainty-aware modeling.

Dropout may have found a new job.

But it is not the only tool in the toolbox.

4. From Deep Learning’s Past to Physical AI’s Future

The story of dropout is therefore not really a story about a technique dying.

It is a story about changing data regimes.

In 2012, dropout was born into an era where neural networks often faced limited datasets and severe overfitting.

As the field moved toward foundation models and large-scale pretraining, the equation changed.

More data.

Larger models.

Longer training runs—but often with vastly more unique examples.

The old bottleneck became less important.

And dropout became less central to the architectures that dominate modern language-model pretraining.

But in physical AI, the constraints are different.

Data is expensive.

Simulations are expensive.

Real-world interactions are expensive.

Robotic demonstrations are limited.

And models may repeatedly train on the same relatively small datasets.

So dropout comes back—not because the field moved backward, but because the underlying problem came back in a different form.

And then dropout finds another use altogether:

uncertainty.

That may be the most interesting part of its story.

5. Write Off the Technique, Not the Idea

There is a broader lesson here.

Machine-learning techniques rarely exist in isolation from the environment that created them.

A method can be extremely valuable under one set of constraints and far less useful under another.

That does not necessarily make the method obsolete.

It may simply mean the problem has changed.

Dropout was created to fight overfitting.

Then, as the field moved toward massive datasets and large-scale pretraining, that particular problem became less dominant in some settings.

Now, in data-constrained physical AI, dropout is finding familiar territory again.

And beyond regularization, it has found a new role in uncertainty estimation.

The technique did not disappear. It adapted.

Technology, Like People, Has Its Own Timing

There is something almost poetic about this.

In 2012, dropout was born into the era of deep learning’s emergence. It became one of the techniques that helped make modern neural networks practical.

Then the field changed.

And for large language models, dropout gradually became less central.

But it did not become worthless.

It simply found another place where its original strengths—and some entirely new ones—still matter.

The same pattern can be seen in the career of Geoffrey Hinton.

For decades, he continued working on neural networks while much of the research community remained skeptical of the approach. Neural networks went through long periods when they were far from the center of attention.

Then the tide turned.

AlexNet arrived in 2012.

The Turing Award followed in 2018.

And in 2024, Hinton received the Nobel Prize in Physics.

The point is not that every old idea will eventually be vindicated.

The point is simpler:

Technology moves in waves.

A technique can disappear from the mainstream without becoming irrelevant.

A researcher can work for decades before the world catches up.

Sometimes an idea is not wrong.

The world simply isn’t ready for the problem it solves.

And sometimes, when the landscape changes again, an old idea finds a new life.

Dropout is a good example.

It was born to fight overfitting.

It survived the age of foundation models.

And in physical AI, it may be becoming something more:

not just a way to regularize a model, but a way to ask a machine how much we should trust it.