Large language models (e.g., ChatGPT) as research assistants

Software can beat human beings at most games… from Chess to Go, and even poker. Large language models like GPT-4 offered through services such as ChatGPT allow us to solve a new breed of problems. GPT-4 can beat 90% of human beings at the bar exam. Artificial intelligence can match math Olympians.

The primary skills of academics are language-related: synthesis, analogy, extrapolation, etc. Academics analyze the literature, identify gaps, and formulate research questions. They review and synthesize existing research. They write research papers, grant proposals, and reports. Being able to produce well-structured and grammatically correct prose is a vital skill for academics.

Unsurprisingly, software and artificial intelligence can help academics, and maybe replace them in some cases. Liang et al. found that an increasing number of research papers are written with tools like GPT-4 (up to 18% in some fields). It is quite certain that in the near future, a majority of all research papers will be written with the help of artificial intelligence. I suspect that they will be reviewed with artificial intelligence as well. We might soon face a closed loop where software writes papers while other software reviews it.

I encourage scholars to apply artificial intelligence immediately for tasks such as…

  1. Querying a document. A tool like BingChat from Microsoft allows you to open a PDF document and query it. You may ask “what are the main findings of this study?” or “are there any practical applications for this work?”.
  2. Improve text. Many academics, like myself, use English as a second language. Of course, large language models can translate, but they can also improve your wording. It is more than a mere grammar checker: it can rewrite part of your text, correcting bad usages as it goes.
  3. Idea generation. I used to spend a lot of time chatting with colleagues about a vague idea I had. “How could we check whether X is true?” A tool like ChatGPT can help you get started. If you ask how to design an experiment to check a given hypothesis, it can often do a surprisingly good job.
  4. Grant applications. You can use tools like ChatGTP to help you with grant applications. Ask it to make up short-term and long-term objectives, sketch a methodology and discuss the impact of your work… it will come up with something credible right away. It is likely that thousands of grant applications have been written with artificial intelligence.
  5. Writing code. You are not much of a programmer, but you want an R script that will load data from your Excel spreadsheet and do some statistical analysis? ChatGPT will do it for you.
  6. Find reviewers and journals. Sometimes you have done some work and you would like help picking the right journal, a tool like ChatGPT can help. If a student of yours finished their thesis, ChatGPT can help you identify prospective referees.

I suspect that much academic work will soon greatly benefit from artificial intelligence to the point where a few academics will be able to do the work that required an entire research institute in the past.

And this new technology should make mediocre academics even less useful, relatively speaking. If artificial intelligence can write credible papers and grant applications, what is the worth of someone who can barely do these things?

You would think that these technological advances should accelerate progress. But, as argued by Patrick Collison and Michael Nielsen, science productivity has been falling despite all our technological progress. Physics is not advancing faster today than it did in the first half of the XXth century. It may even be stagnant in relative terms. I do not think that we should hastily conclude that ChatGPT will somehow accelerate the rate of progress in Physics. As Clusmann et al. point out:  it may simply ease scientific misconduct. We could soon be drowning in a sea of automatically generated documents. Messeri and Crockett put it elegantly:

AI tools in science risks introducing a phase of scientific enquiry in which we produce more but understand less

Yet there are reasons to be optimistic. By allowing a small group of researchers to be highly productive, by freeing them to explore further with less funding, we could be on the verge of entering into a new era of scientific progress. However, it may not be directly measurable using our conventional tools. It may not appear as more highly cited papers or through large grants. A good illustration is Hugging Face, a site where thousands of engineers from all over the world explore new artificial-intelligence models. This type of work is undeniably scientific research: we have metrics, hypotheses, testing, reproducibility, etc. However, it does not look like ‘academic work’.

In any case, conventional academics will be increasingly challenged. Ironically, plumbers and electricians won’t be so easily replaced, a fact sometimes attributed to the Moravec paradox. Steven Pinker wrote in 1994 that cooks and gardeners are secured in their jobs for decades to come, unlike stock market analysis and engineers. But I suspect that the principle even extends within the academy: some work, like conducting actual experiments, is harder to automate than producing and running models. The theoretical work is likely more impacted by intelligence artificial than more applied, concrete work.

Note: This blog post was not written with artificial intelligence. Expect typos and grammatical mistakes.

Daniel Lemire, "Large language models (e.g., ChatGPT) as research assistants," in Daniel Lemire's blog, April 27, 2024.

Published by

Daniel Lemire

A computer science professor at the University of Quebec (TELUQ).

One thought on “Large language models (e.g., ChatGPT) as research assistants”

  1. One could argue that software engineering is concrete applied work. Yet, it is probably one of the first to be fully automated, when considering systems such as Devin, etc.

Leave a Reply

Your email address will not be published.

You may subscribe to this blog by email.