12 June 2025 · 5 min
There is an extremely naive model of science and innovation called the linear model: The model postulated that innovation starts with basic research, is followed by applied research and development, and ends with production and diffusion. According to Godin (2006), the model has been falsely attributed to Bush and its dominance is derived rather from its remarkable simplicity. It aligns with a technocratic worldview where bureaucrats get to steer science and innovation.
Yet it has been long documented that innovations arise from practical experience and are driven by user demand. My favorite example is the industrial revolution which was driven by textile which, in turn, served to provide comfortable underwear for women. We invented modern-day keyboards after musicians perfected the keyboard for musical instruments.
Kline and Rosenberg are eloquent on this topic:
« Even more important, from the viewpoint of understanding innovation, is the recognition that when the science is inadequate, or even totally lacking, we still can, do, and often have created important innovations, and innumerable smaller, but cumulatively important evolutionary changes. Recently, a member of the National Academy of Engineering, highly versed in dynamics and control, attempted to analyze the stability of an ordinary bicycle with a rider-and failed. No acceptable analysis is known. But this lack of theory did not prevent the invention of the bicycle a century ago, nor did it prevent a multitude of beta-phase improvements in bicycle design that have cumulatively created a reliable, relatively inexpensive, and useful machine. Nor does the absence of understanding of the theory of stability prevent a 5-year-old child from mounting a bicycle and with a few tries learning to stabilize the human-machine system. Had the idea been true that science is the initiating step in innovation, we would never have invented the bicycle. »
There are countless examples of important innovations that we can consider:
The development of the steam engine, a cornerstone of the Industrial Revolution, did not begin with basic scientific research but with practical needs in mining. Miners faced the challenge of pumping water out of deep coal mines, and early inventors like Thomas Savery and Thomas Newcomen developed steam-powered pumps to address this issue. James Watt’s later improvements, often seen as a scientific leap, were driven by practical experimentation and feedback from users, such as mine operators, rather than purely theoretical research.
The Wright brothers’ invention of powered flight is a classic case of innovation rooted in practical experience. Neither Wilbur nor Orville Wright had formal scientific training; their breakthroughs came from hands-on experimentation with gliders, kites, and wind tunnels they built themselves. Their motivation stemmed from the growing public and commercial interest in aviation, coupled with their own observations as bicycle mechanics, where they learned about balance and control. The iterative process of testing and refining their designs was driven by practical problem-solving, not a linear progression from basic research to application.
Fleming’s initial observation of penicillin’s antibacterial properties in 1928 was not immediately pursued due to a lack of practical application. It was only when the urgent demand for effective treatments for wounded soldiers emerged that scientists like Howard Florey and Ernst Chain, working with industrial partners, developed methods to produce penicillin at scale. This process was driven by the immediate needs of medical practitioners and the military, not a linear path from basic science.
The rise of personal computers, exemplified by companies like Apple and IBM, showcases innovation driven by user demand and practical experimentation. Early computers were developed not by academic researchers but by hobbyists and entrepreneurs, such as Steve Wozniak, who built the Apple I to meet the needs of tech enthusiasts in the Homebrew Computer Club. The rapid diffusion of PCs was fueled by user demand for accessible computing in homes and businesses, with iterative improvements based on feedback from users rather than a top-down research pipeline. Software development, like Microsoft’s operating systems, similarly responded to practical needs for user-friendly interfaces.
Early mobile phones, like Motorola’s DynaTAC, were developed to meet the practical needs of business professionals for portable communication. The transition to smartphones, particularly with the iPhone in 2007, was driven by consumer demand for multifunctional devices that combined communication, entertainment, and productivity. Apple’s success came from observing user behaviors and integrating existing technologies (touchscreens, internet connectivity) into a user-friendly package, rather than groundbreaking basic research.
The rapid development and adoption of generative AI chatbots, such as OpenAI’s ChatGPT, Google’s Gemini, and Anthropic’s Claude, showcase innovation propelled by user demand and practical applications. ChatGPT would not exist without the massive amount of content produced online by users and the powerful graphics processors developed for gamers.
It is high time we bury the linear model of innovation. Researchers are not in the driving seat of technological innovation, nor are government bureaucrats.
Daniel Lemire, "Let us bury the linear model of innovation," in Daniel Lemire's blog, June 12, 2025, https://lemire.me/blog/2025/06/12/let-us-bury-the-linear-model-of-innovation/.
[BibTeX]
The stability of bicycle is quite well understood, it is just complex – there are if I remember correctly at least three physics phenomena that take role in stabilizing riding the bicycle.
https://ciechanow.ski/bicycle/
https://www.youtube.com/watch?v=9cNmUNHSBac
The TV series “Connections” shows perfectly how non-linear innovation is. It is only in games that you have nice linear tree (or directed graph) of inventions.
Terence Kealey has written definitively on this very topic in the two books “The Economic Laws of Scientific Research” and “Sex, Science, and Profits: How People Evolved to Make Money”. I highly recommend both.
What about all the uncorrected chains of assumption?
Assumptions being passed on generation to generation without anyone doing a re-check.
The worst I have seen is with neural network research, where there are quite a number of angles that have not been checked out. If you gave a novice engineer who had never seen or heard of neural networks an example system they would point out many things. For example, one neuron is projecting a pattern onto the next layer through the forward connected weights with intensity x, x being the output value of the neuron. You could go on forever about things that have not been checked out due to inherited bias and assumptions.
Also with a MCG random number generator, if you shift the seed left you can reduce randomness while not introducing any additional computational operations. Useful if you need sub-random (quasi-random) numbers, for which there are some applications.
Yes. Unassumed assumptions can harm us.
Elon Musk, an entrepreneur with no aerospace background, founded SpaceX and challenged the norm of his industry. His team’s iterative testing led to the Falcon 9’s reusable design, disrupting the industry.
Looking through my old code I see that you can implement ReLU by viewing the float as an integer, doing a signed shift right to spread the sign bit over the full integer and then And that with the initial integer. Well it is negative ReLU. Let’s not mention John David McAfee RIP.
If researchers are not doing innovation, what do you think is a better way to understand the role they (should) play? I’d say at least part of the answer should be expanding and solidifying humanity’s collective knowledge. Knowledge doesn’t inevitably lead to innovation, but it can contribute.
Knowledge doesn’t inevitably lead to innovation, but it can contribute.
Unlike power which can be concentrated in the hands of the few, it is difficult (and not necessarily useful) to concentrate knowledge.
Current deep neural networks seem quite good at concentrating knowledge. I don’t know if associative memory might not be an alternative.
-vecLen must be a power of 2 like 32,64…
public final class AM {
final int vecLen;
final int density;
float rate;
final float[] wts;
final float[] surface;
final float[] workA;
final float[] workB;
final FlipMCG flip;
public AM(int vecLen,int density,float rate) {
this.vecLen=vecLen;
this.density=density;
this.rate=rate/density;
wts=new float[2*vecLen*density];
surface=new float[vecLen*density];
workA=new float[vecLen];
workB=new float[vecLen];
flip=new FlipMCG();
}
public void recall(float[] result,float[] input) {
java.util.Arrays.fill(result,0f);
flip.reset();
flip.flipNext(workA,input);
WHT.whtN(workA);
for(int i=0,idx=0; i>1,vecLen);
for(int j=0; j<vecLen; j++,idx+=2) {
float x=workA[j];
result[j]+=x*wts[idx+(x<0f? 0:1)];
}
}
WHT.whtN(result);
flip.flipNext(result,result);
}
public void train(float[] target,float[] input) {
recall(workB,input);
for(int i=0;i<vecLen;i++) {
workB[i]=(target[i]-workB[i])*rate;
}
flip.flipPrevious(workB,workB);
WHT.whtN(workB);
for(int i=0,surIdx=0; i<density; i++) {
for(int j=0; j<vecLen; j++,surIdx++) {
float x=surface[surIdx];
wts[2*surIdx+(x<0f? 0:1)]+=x*workB[j];
}
}
}
}
public class FlipMCG {
private int r;
private int reset;
public FlipMCG(int seed,int quasi) {
r=(seed+seed+1)<<quasi;
reset=r;
}
public FlipMCG(int quasi) {
this((int)System.nanoTime(),quasi);
}
public FlipMCG(){
this(0);
}
public void flipNext(float[] result,float[] x){
flipNext(result,x,result.length);
}
public void flipNext(float[] result,float[] x,int n) {
for(int i=0; i=0; i–) {
result[i]=Float.intBitsToFloat(Float.floatToRawIntBits(x[i])^(r&0x80000000));
r*=0x7AAE1A75;
}
}
public void reset() {
r=reset;
}
public int getState(){
return r;
}
public void setState(int x){
r=x;
}
}
public class WHT {
static void whtN(float[] vec) {
int n = vec.length;
float scale = 1f / (float) Math.sqrt(n);
for (int i = 0; i < n; i += 2) {
float a = vec[i] * scale;
float b = vec[i + 1] * scale;
vec[i] = a + b;
vec[i + 1] = a – b;
}
int hs = 2;
while (hs < n) {
int i = 0;
while (i < n) {
final int j = i + hs;
while (i < j) {
float a = vec[i];
float b = vec[i + hs];
vec[i] = a + b;
vec[i + hs] = a – b;
i += 1;
}
i += hs;
}
hs += hs;
}
}
}
public class Test {
public static void main(String[] args) {
AM net=new AM(256,32,.85f);
float[][] cos=new float[256][256];
float[][] sin=new float[256][256];
float[] work=new float[256];
for(int i=0; i<256; i++) {
for(int j=0; j<256; j++) {
double t=j*(0.03+0.0002*i);
cos[i][j]=(float)Math.cos(t);
sin[i][j]=(float)Math.sin(t);
}
}
for(int i=0; i<1000; i++) {
float e=0f;
for(int k=0; k<256; k++) {
net.train(cos[k],sin[k]);
}
for(int k=0; k<256; k++) {
net.recall(work,sin[k]);
for(int j=0; j<256; j++) {
float diff=work[j]-cos[k][j];
e+=diff*diff;
}
}
System.out.println("Error: "+e);
}
}
}