The Public Referee Reports Debate

I think the wave was started by Seb this time as he hints we should consider publishing reviews when an academic paper is submitted.

A reply comes from Lance who says we should kill this idea quick. I think his counterargument is badly flawed. For example, he describes the review process as iterative:

The referees read the paper and write a report on whether to recommend the paper for publication and give suggested changes. They send this report to the editor who relays this information to the authors without revealing the identities of the referees. The authors will comment on the referee reports and revise the paper. The paper often goes back to the referees and the process repeats until the editor is happy with the paper or decides the paper is not worthy of the journal.

Here’s what I replied:


The back in forth you refer to is not present in 99% of conferences I know. You submit paper, random reviewers write a review, you get the result, end of story. Because the selection process is based on a percentage of acceptance, even a good review is not sufficient: you need to have the random reviewers be delighted at your work. Now, I claim that more than half the reviewers don’t even read the papers. So, what do you think happens? Papers who are sufficiently fashionable get through, others get canned. The only way I can imagine to get this process fixed is to publish reviews.

Then he actually explains why he thinks public reviews are a bad idea:

The process of refereeing requires considerable back and forth conversation between the three parties: the authors, the editor and the referees. Posting the original reports will give a misleading view of the process and will cause referees to act far too cautiously about pointing out problems in a paper.

I’m sorry but how is having your name as the referee of a paper going to entice you to be lenient? How many people wants to go down in history as having accepted a paper that’s flawed?

Quite the opposite from Lance, I believe that reviewers, in a public review system, have a strong incentive at being extremely hard. Nobody will care about the reviews a rejected paper got… nobody but the author… or if the review is really terrible… But people will read the reviews an accepted bad paper got and if they see that a certain individual is letting bad papers pass, his reputation will go down.

I argue quite the opposite: a public review system would make it very hard to get bad papers through.

As for the possible argument that it would make it harder to find reviewers. This argument (which Lance doesn’t push) is probably valid. First of all, you’ll have to work much harder to find capable reviewers willing to invest the time needed to produce thoughtful reviews. However, these people would get rewarded for their work since their review becomes public and hence, contribute to their status in the community.

Being a Nice Researcher and the Real World: pure, applied, and industrial research

This morning, I am deeply upset. Some of you who know me will know why. No, I don’t care so much that Buch was elected, though it does puzzle me. Read through, you might find out why I’m upset.

I did my Ph.D. with the intent of getting into “industrial research”. Yes, there are different types of research. You have “pure” research where people have no idea why they do the research except that it looks nice. For example, solving for all algebra having property X is pure research. You have “applied” research which is often closely related to “pure” research except that the topic is a bit closer to “real world” concerns. For example, finding a new way to solve Navier-Stokes equations is applied research. It should be noted that “applied” research doesn’t equate with “useful in the real world” research. In fact, it could very well be that some “pure” research is more applicable in the real world. Finally, you have “industrial” research. Industrial research is meant to be useful in the real world. It is the primary purpose of such research. The excitement comes not from elegance alone, but mostly by solving real problems people have. You might say that some people working on the linux kernel are industrial researchers, at least when they innovate. A given researcher may cover all three research types, but he may switch hat depending on the project he is working on.

All types of research are equally worthy, but they are not equal in all things. For example, “pure” research might give you a lot of prestige is some universities. Pure researchers have “pure” concerns and it is often thought that only the smarter researchers can be pure researchers. Applied researcher have often an edge when it comes time to get some research funding. That’s because the applied researcher can easily justify that his work might be applicable in the real world. Finally, the industrial researcher might be looked down upon by some people: because industrial research must be tied closely to the real world, it will sometimes trade fashion or elegance for convenience or applicability. Often, “industrial” research may appear to be simpler, maybe easier. Believe me, it is not easier. On the other hand, the “industrial” researcher can really license technology or even start a company. This, in turn, may bring tremendous leverage to such a researcher. Unfortunately, these things take time, and in a university setting where tenure should be first on your mind, the industrial researcher might have a harder time. Hence, in many schools, most professors are pure or applied researchers. Fortunately, the industrial researcher has a broader choice of employment.

This is actually a very interesting topic. Many questions may come to mind… For example, is there any such thing as industrial research in mathematics? You bet! See SIAM.

Now, that you see what I mean by an industrial researcher, you might understand that one of his strength is when dealing with companies. He is able to understand the concerns of their engineers and his research accounted for many of these concerns already. However, for him, it is crucial that people do not get in the way between him and a company. He needs, he wants technology transfert.

That’s why I’m upset today: I feel someone pushed me aside and got in the way. People are always eager to take someone else’s work and then ignore this person when money is involved and the author is not being difficult. I hope this will get fixed, but either way, I’ve been reminded that the industrial researcher must keep a very close eye and much control on the technology transfert process. And not be nice.

At Cross Purposes: What the experiences of today’s doctoral students reveal about doctoral education

Here’s a report released in 2001 on the state of doctoral education. It looks like a serious study and the conclusion is scary:

What we learned may not be entirely surprising because our findings confirm many of the concerns that have been raised in the last 10 years. However, our data provide detailed, confirmatory evidence of particular tension points. We found that:

  • The training doctoral students receive is not what they want, nor does it prepare them for the jobs they take.
  • Many students do not clearly understand what doctoral study entails, how the process works and how to navigate it effectively.

So, you want to do a Ph.D.?

Seb sent me this extract of a book. The extract is called So, you want to do a Ph.D.? As usual with this sort of book, it is delightful.

Here’s a fun quote:

One thing which is seldom mentioned is what happens to you after you finish the PhD. A classic story is as follows. A student focuses clearly, submits the thesis and starts looking for a lecturing job, only to discover that they need two years of lecturing experience and preferably a journal publication as well if they are to be appointable for a job in a good department in their field. If they had known this two years previously, they could have started doing some part-time lecturing and submitted a paper or two to a journal.

I haven’t read the entire book, of course, and I’m somewhat worried that the book might not be sufficiently focused on why one does a Ph.D. and might be a tad too cynical. Learning the rules is very nice and very important, and I wished I had learned them when it was time. However, there is also the issue of figuring out whether these rules make sense, and knowing when to break them. Well, I guess that learning the rules to begin with is a very good start.

Graduate student/faculty relations

Sharleen talks about how evil junior faculty can be in their approach with grad students:

(…) in academia, (…), there are limited options, and a poor grad student may have to work with the asshole who has naive, unethical, or objectionable approaches to working with grad students. Now, we could simply say, the ones who survive are the ones who deserve to get jobs/get the PhD. We could point out that the market is much tougher. But if we respond this way, we’re not critiquing the culture of academia (a culture which, if I may point out, is largely responsible for the other problems that we all bitch about); we’re justifying it.

I’m unsure why she points at junior faculty as the source of the problem. She’s probably got some personal experience going.

However, I agree with her criticism of the tough love approach to supervising graduate students. I don’t think it can be justified from a pedagogical point of view, it is not justified from a management point of view, and so, indeed, it might be some kind of power trip.

On the other hand, I disagree with her implication that there are no choices. In most cases, the graduate student can go with another supervisor. It might costly, but it is almost always an option. Or else, you can simply go out there and find a job and be happy.

Repeat after me: the world is big and there are almost always options. Unless you are a slave stranded somewhere, you can almost certainly find another job, another graduate program, another project… it might be costly, it might imply extra work, but it is most often possible.

The reason why these professors are getting away with treating graduate students badly is that graduate students allow it. If they chose not to go with this “evil” supervisor, there wouldn’t be any problems any more.

That’s how the real world works. Evil employers will have trouble finding good employees. The good employees will leave for a better employer. That’s the market at work.

The day when the employees stop leaving, because they are scared or tired, the market stops working and the trouble starts.

Generally speaking, academia doesn’t have so much a culture problem as it has a market problem: too many potential candidates for some positions leading to a general degradation of the working conditions for everyone involved.

The art of supervising students

I had an off-line discussion with a collaborator about student supervision and how frustrating it can be. As a professor, you have, from time to time, to supervise students. It could be a graduate student you are supervising as part of their studies, it could be an undergraduate project, it could an assistant you’ve hired.

You know you have a bad student if the student

  • cannot keep track of tasks assigned to him and be responsible for such tasks;
  • lies to you about what has been done and what hasn’t been done;
  • repeatedly ignores some of your phone calls or emails.

In my experience, a bad student is a drain on your resources and a professor simply has to drop such a student as soon as possible. Even if you have funding or need of a student, you are better off with no student than a bad student.

So, what about my title? The art of supervising students?

My experience has been that there is no need to be tough or strict with the students. There is nothing magical you can do: forcefully organizing many meetings with the student often won’t help. If you have a bad student (see above), cut your losses as early as possible. Otherwise, trust the student.

Here are a few rules based on my experience:

  • Be clear about the tasks you expect the student to perform and the time it should take.
  • Be available to the student in a personalized way: some students benefit from frequent meetings, others do not.
  • Get to know and leverage the student strengths and know his weaknesses: you are better off doing some of the tasks yourself.
  • Trust the student: most students have tremendous potential and will deliver greatness given a chance.

If we taught you to memorize, we failed you

Tall, Dark, and Mysterious wrote about this student she has in her class who is actually a fairly typical student:

“I memorized how to do the problem you did in class, but then on the test you put a DIFFERENT problem, and you never showed us how to do THAT one, and it’s not fair! My method of doing math by memorizing formulas and then blindly applying them to problems that are identical to the ones I’ve seen has gotten me A’s until now, so what gives?”

Repeat after me: memorization is not learning. Learning has to be a higher level task.

More on the CS enrollment drop

I’ve written on this blog about the recent drop in enrollment for Computer Science degrees in North America: I gave an estimate of a drop by 25%. Looks like it is worse:

The number of new undergraduate majors in U.S. computer science programs has fallen 28 percent since 2000, reports the Computing Research Association, a group of more than 200 North American computer science, computer engineering and related academic departments.

The explanation would be that students do not want a Dilbertesque life:

One reason, say those in the field, is that technology jobs appear less lucrative than they did during the dot-com boom. Then, students thought a computer science degree would lead to riches and a quick retirement. Many took on the major.

Another reason might be that Business Schools are now competing with Computer Science departments for students:

Colleges have also begun to integrate computer instruction into other majors such as e-commerce programs in business schools. A computer science degree, therefore, can be unnecessary.

How Technology Will Destroy Schools

Through Downes’, I found an article by David Wiley’s with the provocative title How Technology Will Destroy Schools (he actually is being needlessly provocative, he means “schools as they exist now”). The gist of his argument goes as follows:

The development of (…) technology will obviate the need for certain types of instruction — like the teaching of facts. Why spend time memorizing when the same information is available just as quickly from the network as it is from your own memory? But never fear, schools! The technology will create the need for new types of instruction — in higher level information literacy skills. Perhaps this will finally force some change through the public schools.

Well, I must admit. I have a Ph.D. in mathematics and I never learned my multiplication tables. There you go. I never saw the point of learning these tables, so I didn’t. Instead, I learned a few tricks to do multiplications… like 9 times 8 is almost 10 times 8, you have to subtract 1 times 8.

Mathematics is not about learning facts. I suspect that all disciplines have a component above learning the facts. You can’t be an expert in something if you only know the facts… because I can easily input the facts into a piece of software and compete with you, but we all know that software can’t compete (yet) against human experts. I’m not very good at memorizing facts, I’ve never been good at it. In fact, I’m not good at memorizing anything and that’s why I have a PDA always with me. Yet, I’m in expert at some things.

It is the difference between real knowledge and shallow knowledge. Most of our education system is based on acquiring and testing shallow knowledge. Most but not all.

How are you going to get past shallow knowledge through technology as Wiley predicts we will? I think that blogs, games, and simulations are good examples. Yes, we can role play without technology, but it becomes so much cheaper to deploy gaming scenarios through technology (because you only have to do it once) that it might become more common place in the future.

Maybe my son Lohan, by the time he makes it to school, will have “gaming instruction” where he will enter a gaming universe to learn basic mathematics. Who knows.

I’m not holding my breath though, I think we lack the human power to do pull it off in the next 5 years.

On tools for academic writting and a shameless plug

First, the shameless plug: my long-time friend, Jean-François Racine published a book available both as hardcover and paperback. The title is “The Text of Matthew in the Writings of Basil of Caesarea”.

More seriously, and maybe he had told me about this, but he told me about this specialized word processor he uses, called Nota Bene. More interesting is a component of this word processor called Orbis. Specifically, Orbis generates vocabulary lists, as well as frequency of occurrence; and it allows you to define synonym lists to expand search capabilities.

Academic life: a balancing act

Today, I realized that the life of a researcher/professor is really a balancing act. A professor…

  • has a rich personal life;
  • gives great courses;
  • gets a lot of funding;
  • has many students;
  • publishes a lot papers each year;
  • consults on industrial/governmental projects;
  • manages something (departement, project, program).

It is no surprise that many professors end up being overworked. I think you simply cannot pull all these things at once. Maybe 2 or 3 from the list. You have to choose or life will choose for you.

Does your university think that “Jobs are for the little people”?

Tall, Dark, and Mysterious is a Math. professor somewhere in Canada, possibly in British Columbia. She graduated from a big school and now teaches at a smaller (lesser?) school.

Well, is it a lesser school? That’s where her tale becomes interesting. Myself, I attended UofT. I don’t know if the rule is true, probably not, but it seem that the larger the school, the more it suffers from the jobs-are-for-little-people syndrome as documented in a post by Tall, Dark, and Mysterious. Here is an insightful quote:

University isn’t job training, because universities are adamant about university not being job training. And it’s not because they’re too busy enriching students’ lives and fostering a love of learning. Underneath all of the cheap idealism – trumpeted by gainfully employed people, many of whom haven’t learned how to play a musical insturment, how to speak a foreign language, or how to play a new sport because none of those things are related to their jobs and because they’re too old to be doing that sort of thing – about learning for the sake of learning is a willful inability to confront the fact that students are not at universities to learn for the sake of learning.

Some insight from John Travolta

This morning, I was chatting with a colleague, Richard Hotte, and we were discussing research, funding, and the relation between the two. I’ve also had these discussions with Martin Brooks. Richard pointed out that John Travolta has figured it out all some years ago when he received a prize at Cannes for the movie Pulp Fiction. According to Richard, Travolta was asked then about how it felt to finally win a prize that he surely coveted for many years. John Travolta answered that his goal was never to win a prize, but rather to grow as an actor. The prize was nice, but not his goal.

This echoes my own feeling about funding for research. If your goal is funding, you’ll probably get some, maybe a lot, but unfortunately, you may never become a great researcher.

Use of blogs in higher education

Through Downes’, I found this great paper on Exploring the user of blogs as learning spaces in the higher education sector.

Looks like a great paper:

This paper explores the potential of blogs as learning spaces for students in the higher education sector. It refers to the nascent literature on the subject, explores methods for using blogs for educational purposes in university courses (eg. Harvard Law School), and records the experience of the Brisbane Graduate School of Business at Queensland University of Technology, with its ‘MBA blog’. The paper concludes that blogging has the potential to be a transformational technology for teaching and learning.

eLearning live!

A few years ago, I remember hearing the word eLearning for the first time. I had accepted a job with NRC circa 2001. I knew there was an eLearning team in Moncton. I remember thinking they were lucky because Moncton is a relatively cool city.

I vaguely remember meeting Stephen Downes for the first time. He probably doesn’t remember me though. I was in a basement and had nothing close to an office. It wasn’t so bad though, but because very quickly we moved to a beautiful office where I had a gigantic office. In any case, I see this tall angry man come in and try to plug is laptop. Can’t remember what wasn’t working, but I remember he was quite angry. This was eLearning for me at the time.

Fast-forward a few years. I’ve grown convinced that eLearning is there to stay. As Stephen might put it: we’ve now integrated technology without changing our ways in any fundamental manner. Next step is to change our ways. Gone will be the lecture halls. Campuses will be lifestyle choices.

It is easy to predict such revolutions, but you need evidence to back your statements. Well, blogging is one such sign. I read many exciting blogs by students, but two come to mind right now Claire’s and Didier’s. Claire is struggling to finish her Ph.D. while Didier is probably a top 1% undergraduate student. Of course, there are many examples of exciting blogs by students… but I pick these two because they are great examples. There is a tight integration between the learning process and the content of the blog. The blog is part of the learning process. You can see it live. In Claire’s case, it is not so much the content of her Ph.D. that is integrated with the blog, but rather the process of writting the damn thing and the usual Ph.D. versus employment struggle. Though Claire might have another blog strictly about the content of her Ph.D. As for Didier, he writes about the content of his classes and textbooks.

In many ways, I feel like Didier could be a student at my school. A relatively close student. Claire could be down the hall some place and we could chat about academia, industry and all the usual stuff. But this is happening on-line. This is all happening without a building. There is no brick-and-mortar involved. I suspect I might know a bit more about Didier and Claire than some of their professors do… This is what physical campuses are up against.

Technology in the classroom is not what eLearning is about. eLearning is about abolishing the classroom just like libraries have been abolished. I still go to libraries, but for the lifestyle effect… not to buy books. If I want to buy a book, I do it on-line.

In many ways, on-line learning is more human, it has more soul. It is about real people communicating, becoming part of a rich networking. Learning and growing together.

(Oh! And my blog is a student’s blog as well. I just happen to be on the other side of the fence, the side charging tuitions…)

Chronic lack of time in academia

Yes, I know, everybody runs out of time. All employees in the world have too much work…

But academia is kind of special because you have one of the most complex job description you can imagine. You are a teacher, a researcher, sometimes an engineer, sometimes a manager, sometimes a public speaker, sometimes a consultant and many other things yet, all wrapped in one job. And you are supposed to be very good at all those jobs. You do spend your typical day wearing many hats. I don’t think there are many jobs where you are expected to wear so many hats.

So, what happens? You run out of time. Which means you don’t do certain things. At some point, you learn to say “No”. Everybody has to say no, but I think that professors have to say no far more often than others. I think. But I’m currently wearing my blogger hat, and I have to quickly go back to my must-work-on-funding-application hat so don’t mind me.

Journal of Algorithms is no longer accepting submissions

We just submited an article to the Journal of Algorithms and we were told that starting in 2003, the editors have stopped accepting papers. One alternative appears to be ACM Transactions on Algorithms.

It seems like the entire board of the Journal of Algorithms had resigned some time ago. I had no idea that Elsevier and other big publishers were in such troubles. I had heard about the Journal of Machine Learning

It feels like soon, all the big journals will have moved to an open or semi-open setup. Very scary for big publishers. Very scary. Yes, they’ve been making ever larger profits, but it may all come down to a stop really soon. Tipping point coming!

Anonymous Academic Bloggers

Ernie’s 3D Pancakes has a post on anonymous academic bloggers. To me, this is an interesting question. I use my own name everywhere on this blog. You can easily figure out where I work, what I teach and to whom, where I publish and so on. You can even find who my son is and so on. I think that Jeff correctly points out that feeling you need to be anonymous is probably misguided. The likelyhood that a colleague is going to come to my blog, read it, be insulted, and try to hurt me on the job, is very, very slim. One reason for that is that I would never bad mouth a colleague on my blog: it just wouldn’t be fun and interesting for my target audience. The likelyhood that a reviewer of a paper I submitted would come on my blog and be insulted and reject my paper is also very slim. However, reviewers have many more reasons to wrongly reject a paper and if you start worrying about this sort of thing, you are not out of the woods!

So, I use my own name. There.

23% Fewer Computer Science Majors This Year!

Slashdot reports on a USA Today article saying that there fewer Computer Science Majors. They cite a 23% decline in enrollment in North America. Here’s one comment about the article:

Most engineering schools are reporting declines in enrollment. This is hardly surprising since most engineering curriculums, including CS, are difficult compared to other fields of study. Without the prospect of a good job waiting for them, many college students are veering away from these majors.

Update: Yuhong correctly points out that this is mostly at the undegraduate level. Graduate schools are finding enough students, at least according to Yuhong. I think this is expected: if job prospects are bad, people won’t enter the system but once they’ve entered it, they will stay in it longer if jobs are scarse.

Michael Nielsen: Principles of Effective Research

Michael just finished his essay: Principles of Effective Research. I think it is a must read for all Ph.D. students, young researchers, and even idiots like me who always get it wrong. Michael takes a very refreshing view to what research is all about. He is not cynical yet he is true to what research really is. You may never win the Nobel prize if you follow his guidelines, you may never be a guru researcher, but I think you’ll be a good or even excellent researcher. As he explains, being an influent researcher is not a subset of being a good researcher, and that’s a very important statement. In any case, Michael did all of us a favor and I hope that he essay is read by a lot of people. (Power of the network?) I implore you all: link to his essay!!!