My thoughts on how research funding is attributed in Canada to Computer Science

In Canada, most computer science professors seek funding with NSERC, the main Canadian funding agency for science and engineering. It is more or less the equivalent of the American NSF. The core NSERC program is called “discovery” and it funds 5-year research programs. So, roughly speaking, currently funded professors apply for funding about once every 5 years. Once funded, you do not have to pursue the program you proposed: we recognize that you cannot be expected to stupidly stay on the same course for five years in a fast moving field.

Applicants get a rating regarding the excellence of the researcher, the merit of their proposal and on how well they are training students. It is an all-around evaluation. It is quite stressful.

Not all professors are funded or seek funding. However, it is common for computer science departments to expect their new faculty to seek funding. At many places, getting funding is a necessary condition to get tenure. I would expect it to be the case at all top departments. In effect, the NSERC discovery program act as a Canada-wide peer review process.

The grants are modest: from about 20k$ a year to 100k$. Very few people get 100k$ a year, you basically have to be a Turing award recipient. So it is not a lot of money compared to what American professors get. In most cases, all the money goes to students. Professors cannot pay themselves. So getting a grant does not increase your salary.

In computer science, applications go to a committee made of between 40 to 50 people. Most are from Canadian universities, but there are some people from industry and from other countries. Each application is reviewed by a team of five committee member, supervised by a chair (who is also a member of the committee) as well as at least one program officer. There are also external reviews which are taken seriously, but are just one element among many. The applicants must provide samples of their research; they committee members browse and discuss these papers. And there is a 5-page proposal describing the science that the applicant wants to pursue.

I just finished a term as co-president of the committee. It is a lot of work. I could probably have written five more papers these last three years without this service responsibility. Let me add that it is unpaid.

Here are my take-away from the experience:

  1. We often hear that science is all about publishing lots and lots of papers. That is definitively not true. Once you put a bunch of professional computer scientists in a room and you tell them to judge someone… they quickly dismiss sheer volume. They seek quality. They seek actual science. They also tend to go with proxies, as in “they published in the best venue”. Yet, even there, it is not so much the volume that matters as the fact that specific journals and conferences are especially selective. Committee members are eager for explanations as to why the research is great; it is often the applicants themselves who are not forthcoming about details. If you wrote about a breakthrough in an online note or presented it at a local workshop, your peers will be happy to consider it, though you have a bit more explaining to do than if it appeared in a prestigious venue. And it is definitively possible to get the highest ratings without a pursuit of prestigious venues.
  2. We take services to the community and to society very seriously. It is not all about papers.
  3. I don’t think bibliometrics like citations get discussed much at all: again, it is not about numbers. Being highly cited can be good, but it is not the end game.
  4. It is surprising how even the most experienced researchers sometimes cannot describe competently a research proposal. Sometimes there are goals, but no means to achieve them. Sometimes the objectives are all over the map and incoherent. People will mix and match ideas in the most absurd manner.
  5. The committee is quite open to new and bold ideas. In fact, it rewards bold ideas if they are presented in a competent manner.
  6. Years after years, I have seen “old ideas” being praised when put into a solid program. Not everything good has to be about bitcoins and deep learning. That is, it is not required that you work on fashionable ideas. The committee has a lot of tolerance for unfashionable ideas.
  7. People who try to “pad” their resume to look impressive take risks. Committee members get annoyed very quickly at people gaming the system.
  8. Everyone gets assessed on the same grid. It does not matter whether you are at the best or worst university. It does not matter if you are 20 years old or 70 years old. Evidently, it does not matter whether you are man or not. So it is a system that is hard on younger, less experienced professors. It is also hard for people from small institutions. If you are both inexperienced and from a small institution, it is especially difficult. The counterpart is that if you do well while being at a small institution, you should be especially proud of yourself.
  9. Unfortunately, as with every year, many professors will get bad news. They will have failed to get funding. This may mean that they will soon lose their job. Some people are just bad at what they do or they have bad ideas: it is a service to tell them loud and clear. But most people who fail to receive funding at not bad. In many cases, the merit of their work was clear. This is not a perfect system, nor can it be. Some people simply have not yet had the means to reach the thresholds set by the grid. Some people do work that just do not fit well with the grid. This makes me sad and slightly depressed, but there is no easy fix. In my department, we specifically do not require that new professors get a discovery grant to receive tenure. We do our own assessment.
  10. It is definitively not all about politics. I have heard rumours about people from a given school of thought trying to sink people from another school, or people trying to be advocate for their own speciality. I am sure it happens. However, when it is observed, it does get some attention. Nothing is ever perfect, but we don’t let politics take over.

All in all, I feel better about my peers and about computer science after this experience at NSERC. I am generally very concerned about quality in research. There is a lot of junk out there. Yet there is also a lot of good people try to do good work. My expectation is that the Canadian system is probably one of the best in the world because it puts quality and good science first.

 

Further reading: Ebrahim Bagheri was a fellow committee member and he wrote about his experience on twitter.

Published by

Daniel Lemire

A computer science professor at the University of Quebec (TELUQ).

10 thoughts on “My thoughts on how research funding is attributed in Canada to Computer Science”

  1. Regarding point 4: Different people think in different ways. Some people prefer making concrete long-term plans, even if the plans keep changing all the time. For others, the absurd mix and match of ideas is the real plan. Some people in the latter group eventually learn how to write convincing fake plans, but it can be a harder task than the actual research.

    1. It is definitively harder to write a proposal than to write a simple research paper. I don’t know if testing people on their ability to write proposals is the right thing to do. Yet this is the rule of the game and not just in science. Try to get funding from your bank without a business plan that is understandable…

  2. This sounds a lot like the Cravath system–the “up or out” system where young lawyers start at a law firm and either keep getting promoted until they either “make partner”, or are washed out.

    In its way it’s merciful to the less-than-capable junior faculty members where they are told, “OK, sorry, you’re not a good enough fit for academia, go find something else”. Rather than grad students/postdocs being strung along in the field forever, increasing competition for low-level/entry academic jobs.

    1. It is an interesting paper, but I don’t agree with the model. There may be stagnation, but I doubt it is because of the pursuit of citations.

      There is a lot of junk research, and I am sure that this has much to do with the stagnation if stagnation there is, but I am quite certain we would not cure the problem if we stopped counting citations.

      Note that you barely hear talk of citations during NSERC reviews, in part because it is policy to avoid simple metrics, but also, I think, because people do not take them seriously.

      1. Do you think it is possible to measure, or at least roughly estimate, research stagnation? Do you think the Comp Sci committee should have a lever to adjust the funding budget in lean and fruitful periods? Do any of the NSERC committees perform periodic comparative analysis of other programs?

  3. Do Canadian universities do more to assist professors with funding graduate students and post docs compared to universities in the US? The funding amounts you mentioned seem to be 10 times smaller than the typical NSF grant.

    1. It is undeniable that NSF has large grants given out to relatively few professors while the Canadian model is more grants given out to more (relatively speaking) professors.

      It is difficult to compare the systems beyond this point, however. For example, if a funding agency gives me 100k$, then these 100k$ are likely to go entirely to the students (or nearly so). That is, the school won’t take a chunk out of it for its own purposes, as is common in the US. In Canada, the school does get money as a proportion of the grants the professors receive, but it is given out through a separate process invisible to the professor. Also, the money cannot be used to pay the professor’s salary (typically) in Canada.

      Furthermore, what makes the comparison difficult is that the US has a far more elitist system in general. In the US, you have these tiny elite universities getting much funding… and lots of large and relatively poorer universities. Canadian universities are far more uniform. There are, obviously, better off schools… but the differences are not so drastic.

      So do you compare Canadian universities with the much smaller elite American universities, or do you compare Canadian universities against large and relatively poorer schools?

Leave a Reply

Your email address will not be published. Required fields are marked *

To create code blocks or other preformatted text, indent by four spaces:

    This will be displayed in a monospaced font. The first four 
    spaces will be stripped off, but all other whitespace
    will be preserved.
    
    Markdown is turned off in code blocks:
     [This is not a link](http://example.com)

To create not a block, but an inline code span, use backticks:

Here is some inline `code`.

For more help see http://daringfireball.net/projects/markdown/syntax