Science and Technology links (February 24th, 2019)

  1. Dooley interviews Barabási (a famous researcher) on his book The Formula: The Universal Laws of Success. Here is an interesting quote:

    It is true that major breakthroughs are typically connected to young individual’s name, but that’s not because the older folks are not creative, but because the young people are trying more often. That is that the productivity in youth is higher, and with age productivity drops down. People are simply not trying to build new companies, not trying to write new research papers, not trying to paint new paintings, and when we correct for the core productivity, it turns out that innovation has no age. In other terms, as long as we keep trying and putting projects on the table, your chances of breaking through at 80, is exactly the same as breaking through in your twenties.

    (Credit: Part of this quote appeared on

    I am reading through Barabási’s book right now. He lays out several laws of success:

    1. Performance drives success, but when performance can’t be measured, networks drive success.
    2. Performance is bounded, but success is unbounded.
    3. Previous success + fitness = future success.
    4. While team success requires diversity and balance, a single individual will receive credit for the group’s achievements.
    5. With persistence success can come at any time.

    The book is well written and I recommend it.

  2. We know that naked mole rats essentially do not age in the way most mammals do: their mortality does not increase with age as far as we can tell. We do not know why that is. Munro et al. argue that it might due to their mitonchondria (the power plants of our cells) that seem remarkably resilient and well protected.
  3. Men who are to complete more than 40 push-ups are associated with a significantly lower risk of incident cardiovascular disease (e.g., heart attack) compared with those completing fewer than 10 push-ups. The difference is massive (20x). However, few men could do fewer than 10 push-ups (5%), so we can assume that they are at the very lower end of the fitness spectrum. And men who could do more than 40 push-ups are part of an elite group of high fitness (only 15% of all men). What the article suggests is that instead of measuring blood pressure or weight, doctors would be better off asking patients to do push-ups. (Update: Tom Naughton points out that men who can do more pushups are younger and less likely to be smokers. If we compensate for these effects, the difference might be less significant.)
  4. Intense exercise might improve your protein maintenance via short-lived hormonal changes.
  5. Improving teachers’ effectiveness through training based on science is a good idea, but it is unclear whether it is done right currently: “Teachers have a tough job, and we waste enough of their time with education courses and professional development sessions that provide them with little useful information.”
  6. In Can Medicine Be Cured?, O’Mahony points out that medical research is failing patients. Work done in laboratories fails to translate into better health. When there is translation in the clinic, it is often costly and incremental. Large investments translate into mediocre gains. The book rambles on about gluten intolerance but is otherwise excellent.
    A case can be made based on this book that medical research is overfunded: in computer science, we receive a tiny fraction of what the medical researchers receive, but we seem to have a much greater impact on society. I am unsure why medical research is so unproductive, but it seems obvious that the incentives to improve clinical outcomes are too small. The focus seems to be on writing research articles. Research articles are a means of communication, not the final outcome of a research project.
  7. There is a clinical trial aiming to use gene therapy (via a virus) to cure and prevent macular degeneration, a disease that lead many older people to become blind.
  8. The surface of Mars is probably too harsh for even bacteria, that is, it is likely maintained sterile.
  9. An Israeli company just launched a craft called Beresheet at the Moon, at the modest cost of $100 million. They used Elon Musk’s SpaceX services. Right now, the only resident on the Moon is a Chinese robot that landed last Month.

More fun with fast remainders when the divisor is a constant

In software, compilers can often optimize away integer divisions, and replace them with cheaper instructions, especially when the divisor is a constant. I recently wrote about some work on faster remainders when the divisor is a constant. I reported that it can be fruitful to compute the remainder directly, instead of first computing the quotient (as compilers are doing when the divisor is a constant).

To get good results, we can use an important insight that is not documented anywhere at any length: we can use 64-bit processor instructions to do 32-bit arithmetic. This is fair game and compilers could use this insight, but they do not do it systematically. Using this trick alone is enough to get substantial gains in some instances, if the algorithmic issues are just right.

So it is a bit complicated. Using 64-bit processor instructions for 32-bit arithmetic is sometimes useful. In addition, computing the remainder directly without first computing the quotient is sometimes useful. Let us collect a data point for fun and to motivate further work.

First let us consider how you might compute the remainder by leaving it up to the compiler to do the heavy lifting (D is a constant known to the compiler). I expect that the compiler will turn this code into a sequence of instructions over 32-bit registers:

uint32_t compilermod32(uint32_t a) {
  return a % D;

Then we can compute the remainder directly, using some magical mathematics and 64-bit instructions:

#define M ((uint64_t)(UINT64_C(0xFFFFFFFFFFFFFFFF) / (D) + 1))

uint32_t directmod64(uint32_t a) {
  uint64_t lowbits = M * a;
  return ((__uint128_t)lowbits * D) >> 64;

Finally, you can compute the remainder “indirectly” (by first computing the quotient) but using 64-bit processor instructions.

uint32_t indirectmod64(uint32_t a) {
  uint64_t quotient = ( (__uint128_t) M * a ) >> 64;
  return a - quotient * D;

As a benchmark, I am going to compute a linear congruential generator (basically a recursive linear function with a remainder thrown in), using these three approaches, plus the naive one. I use as a divisor the constant number 22, a skylake processor and the GNU GCC 8.1 compiler. For each generated number I measure the following number of CPU cycles (on average):

slow (division instruction) 29 cycles
compiler (32-bit) 12 cycles
direct (64-bit) 10 cycles
indirect (64-bit) 11 cycles

My source code is available.

Depending on your exact platform, all three approaches (compiler, direct, indirect) could be a contender for best results. In fact, it is even possible that the division instruction could win out in some cases. For example, on ARM and POWER processors, the division instruction does beat some compilers.

Where does this leave us? There is no silver bullet but a simple C function can beat a state-of-the-art optimizing compiler. In many cases, we found that a direct computation of the 32-bit remainder using 64-bit instructions was best.

Science and Technology links (February 16th, 2019)

    1. In their new book Empty Planet, Bricker and Ibbitson argue that within the next 30 years, Earth’s population will start to rapidly decline. They believe that official population predictions overestimate future populations because they fail to fully take into account accelerating cultural changes.
    2. It is believed that senescent cells are a major driver of age-related conditions. Senescent cells often occur when cells are the results of too many divisions (beyond the Hayflick limit). Our hearts age, but their cells do not divide very much. That is a problem because our hearts have a limited ability to repair themselves (by creating new cells) but this should protect our hearts from senescent cells… Yet Anderson et al. found that there are senescent cells in the heart: basically cells can become senescent due to damage. What is more exciting is that they found that by clearing these cells in old mice, they could effectively rejuvenate their hearts. Furthermore, there is a growing number of therapies for removing senescent cells. Furthermore, there are ongoing (early) clinical trials to measure the effect of removing senescent cells in human beings. Initial results are encouraging:

      The doctors found that nine doses of the two pills over three weeks did seem to improve patients’ ability to walk a bit farther in the same amount of time, and several other measures of well-being.

      More trials will start this year.

    3. Goldacre et al. looked at how well the most prestigious journals handle the agreed upon set of standards for reporting scientific trials:

      All five journals were listed as endorsing CONSORT, but all exhibited extensive breaches of this guidance, and most rejected correction letters documenting shortcomings. Readers are likely to be misled by this discrepancy.

      (Source: A. Badia)

    4. A new drug appears to reverse age-related memory loss, in mice.

My iPad Pro experiment: almost two years later

Soon after the first iPad Pro came out, I bought one and started using it daily. The experiment is ongoing and I thought it was time to reflect upon it further.

Before I begin, I need to clarify my goals. When I started this experiment, some people objected that I could get a hybrid (laptop/tablet). That is definitively true, and it would be more “practical”. However, it would not be much of an experiment. I am deliberately trying to push the envelope, to do something that few do. So I am not trying to be “practical”.

And, indeed, using an iPad Pro for work is still an oddity. Relying solely on an iPad Pro for serious work is even rarer. I am currently in Ottawa reviewing grant applications. There are a few dozens computer-science researchers (mostly professors) around me. The only other person with an iPad is a Facebook researcher, and he seems to be using the iPad only for reading applications, otherwise he appears to be using a laptop.

In my department, other faculty members have iPad Pros, but I think only one of my colleagues use it seriously. Other colleagues do not appear to use these tablets for work when they have them. I am not sure.

  1. The main impact on my work at relying mostly on a tablet is that I am always focusing on one or two applications at a time. I recall finding it really cool, back in the days, when a Unix box would allow me to have 50 windows open at a time. I think having many windows open is akin to have many different physical files opened on your desk. It is distracting. For example, on a laptop, I would write this blog post while having an email window open, probably a couple of text editors with code. Yes, you can work in full screen mode with a laptop, and I try to do it, but I tend to unavoidably revert back to the having dozens of applications on my screen. Laptops just make it too convenient to do multiple things at once. If you need to concentrate on one thing for a long time, you really want to have just one clean window, and a tablet is great at that. On this note, it is also why I prefer to program in a text editor that has as few distractions as possible. I can write code just fine in Eclipse or Visual Studio, and for some tasks it is really the best setup, but it often leaves me distracted compared to when I work with single full-screen editor with just one file opened.
  2. Though I could not prove it, I feel that using a tablet makes me a better “reader”. Much of my work as a university professor and researcher involves reading and commenting on what other people are doing. The fact that I am entice to concentrate on one document, one task, at a time forces me to be more thorough, I think.
  3. As far as I can tell, programming seriously on a tablet like an IPad Pro is still not practical. However, there are decent ssh clients (I use Shelly) so that if you master Unix tools like vim, emacs, make, and the like, you can get some work done.
  4. I’d really want to push the experiment to the point where I no longer use a keyboard. That’s not possible at this time. I like the keyboard that Apple sells for the iPad Pro 2018. There is a major upside: the keyboard is entirely covered so it is not going to stop working because you spilled some coffee on it.
  5. Generally, most web applications work on a tablet, as you would expect. However, it is quite obvious that some of them were not properly tested. For example, I write research papers using a tool called Overleaf. However, I cannot make the shortcuts work. At the same time, it is really surprising how few problems I have. I think that the most common issues could be quickly fixed if web developers did a bit more testing on mobile devices. Evidently the fact that developers rely on laptops and desktops explains why things work better on laptops and desktops.
  6. At least on Apple’s ios, working with text is still unacceptably difficult at times. Pasting text without the accompanying formatting is a major challenge. Selecting large blocks of text is too hard.

My final point is that working with an iPad is more fun than working with a laptop. I cannot tell exact why that is. I’d be really interested in exploring this “fun” angle further. Maybe it is simply because it is different, but it is maybe not so simple. My smartphone is “fun” even if it is old and familiar.

Science and Technology links (February 9th, 2019)

  1. Though deep learning has proven remarkably capable in many tasks like image classification, it is possible that the problems they are solving remarquably well are just simpler than we think:

    At its core our work shows that [neural networks] use the many weak statistical regularities present in natural images for classification and don’t make the jump towards object-level integration of image parts like humans.

    This challenges the view that deep learning is going to bring us much closer to human-level intelligence in the near future.

  2. Though we age, it is unclear how our bodies keep track of the time (assuming they do). Researchers claim that our blood cells could act as time keepers. When you transplant organs from a donor, they typically behave according to the age of the recipient. However, blood cells are an exception: they keep the same age as the donor. What would happen if we were to replace all blood cells in your body with younger or older ones?
  3. A tenth of all coal is used to make steel. This suggests that it might be harder than people expect to close coal mines and do away with fossil fuels entirely in the short or medium term.
  4. Elite powerlifters have suprising low testosterone (male homone) levels. This puts a dent in the theory that strong men have high testosterone levels.
  5. Chimpanzees learn to crack nuts faster than human beings. This challenges the model that human beings are cognitively superior.
  6. It seems that the male brain ages more rapidly than the female brain.
  7. Grant argues that vitamin D supplements reduce cancer rates, but that medicine is slow to accept it.
  8. Women prefer more masculine looking men in richer countries. I do not have any intuition as to why this might be.
  9. Geographers claim that the arrival of Europeans to America, and the subsequent reduction of population (due mostly to diseases) lead to a global cooling of worldwide temperatures. It seems highly speculative to me that there was any measurable effect.
  10. The New York Times has a piece of a billionnaire called Brutoco who says that “he spends much of his time flying around the world lecturing on climate change” and lives in a gorgeous villa surrounded by a golf course. There is no talk of his personal carbon footprint.

Faster remainders when the divisor is a constant: beating compilers and libdivide

Not all instructions on modern processors cost the same. Additions and subtractions are cheaper than multiplications which are themselves cheaper than divisions. For this reason, compilers frequently replace division instructions by multiplications. Roughly speaking, it works in this manner. Suppose that you want to divide a variable n by a constant d. You have that n/d = n * (2N/d) / (2N). The division by a power of two (/ (2N)) can be implemented as a right shift if we are working with unsigned integers, which compiles to single instruction: that is possible because the underlying hardware uses a base 2. Thus if 2N/d has been precomputed, you can compute the division n/d as a multiplication and a shift. Of course, if d is not a power of two, 2N/d cannot be represented as an integer. Yet for N large enoughfootnote, we can approximate 2N/d by an integer and have the exact computation of the remainder for all possible n within a range. I believe that all optimizing C/C++ compilers know how to pull this trick and it is generally beneficial irrespective of the processor’s architecture.

The idea is not novel and goes back to at least 1973 (Jacobsohn). However, engineering matters because computer registers have finite number of bits, and multiplications can overflow. I believe that, historically, this was first introduced into a major compiler (the GNU GCC compiler) by Granlund and Montgomery (1994). While GNU GCC and the Go compiler still rely on the approach developed by Granlund and Montgomery, other compilers like LLVM’s clang use a slightly improved version described by Warren in his book Hacker’s Delight.

What if d is a constant, but not known to the compiler? Then you can use a library like libdivide. In some instances, libdivide can even be more efficient than compilers because it uses an approach introduced by Robison (2005) where we not only use multiplications and shifts, but also an addition to avoid arithmetic overflows.

Can we do better? It turns out that in some instances, we can beat both the compilers and a library like libdivide.

Everything I have described so far has to do with the computation of the quotient (n/d) but quite often, we are looking for the remainder (noted n % d). How do compilers compute the remainder? They first compute the quotient n/d and then they multiply it by the divisor, and subtract all of that from the original value (using the identity n % d = n - (n/d) * d).

Can we take a more direct route? We can.

Let us go back to the intuitive formula n/d = n * (2N/d) / (2N). Notice how we compute the multiplication and then drop the least significant N bits? It turns out that if, instead, we keep these least significant bits, and multiply them by the divisor, we get the remainder, directly without first computing the quotient.

The intuition is as follows. To divide by four, you might choose to multiply by 0.25 instead. Take 5 * 0.25, you get 1.25. The integer part (1) gives you the quotient, and the decimal part (0.25) is indicative of the remainder: multiply 0.25 by 4 and you get 1, which is the remainder. Not only is this more direct and potential useful in itself, it also gives us a way to check quickly whether the remainder is zero. That is, it gives us a way to check that we have an integer that is divisible by another: do x * 0.25, the decimal part is less than 0.25 if and only if x is a multiple of 4.

This approach was known to Jacobsohn in 1973, but as far as I can tell, he did not derive the mathematics. Vowels in 1994 worked it out for the case where the divisor is 10, but (to my knowledge), nobody worked out the general case. It has now been worked out in a paper to appear in Software: Practice and Experience called Faster Remainder by Direct Computation.

In concrete terms, here is the C code to compute the remainder of the division by some fixed divisor d:

uint32_t d = ...;// your divisor > 0

uint64_t c = UINT64_C(0xFFFFFFFFFFFFFFFF) / d + 1;

// fastmod computes (n mod d) given precomputed c
uint32_t fastmod(uint32_t n ) {
  uint64_t lowbits = c * n;
  return ((__uint128_t)lowbits * d) >> 64; 

The divisibility test is similar…

uint64_t c = 1 + UINT64_C(0xffffffffffffffff) / d;

// given precomputed c, checks whether n % d == 0
bool is_divisible(uint32_t n) {
  return n * c <= c - 1; 

To test it out, we did many things, but in one particular tests, we used a hashing function that depends on the computation of the remainder. We vary the divisor and compute many random values. In one instance, we make sure that the compiler cannot assume that the divisor is known (so that the division instruction is used), in another case we let the compiler do its work, and finally we plug in our function. On a recent Intel processor (Skylake), we beat state-of-the-art compilers (e.g., LLVM’s clang, GNU GCC).

The computation of the remainder is nice, but I really like better the divisibility test. Compilers generally don’t optimize divisibility tests very well. A line of code like (n % d) = 0 is typically compiled to the computation of the remainder ((n % d)) and a test to see whether it is zero. Granlund and Montgomery have a better approach if d is known ahead of time and it involves computing the inverse of an odd integer using Newton’s method. Our approach is simpler and faster (on all tested platforms) in our tests. It is a multiplication by a constant followed by a comparison of the result with said constant: it does not get much cheaper than that. It seems that compilers could easily apply such an approach.

We packaged the functions as part of a header-only library which works with all major C/C++ compilers (GNU GCC, LLVM’s clang, Visual Studio). We also published our benchmarks for research purposes.

I feel that the paper is short and to the point. There is some mathematics, but we worked hard so that it is as easy to understand as possible. And don’t skip the introduction! It tells a nice story.

The paper contains carefully crafted benchmarks, but I came up with a fun one for this blog post which I call “fizzbuzz”. Let us go through all integers in sequence and count how many are divisible by 3 and how many are divisible by 5. There are far more efficient ways to do that, but here is the programming 101 approach in C:

  for (uint32_t i = 0; i < N; i++) {
    if ((i % 3) == 0)
      count3 += 1;
    if ((i % 5) == 0)
      count5 += 1;

Here is the version with our approach:

static inline bool is_divisible(uint32_t n, uint64_t M) {
  return n * M <= M - 1;


  uint64_t M3 = UINT64_C(0xFFFFFFFFFFFFFFFF) / 3 + 1;
  uint64_t M5 = UINT64_C(0xFFFFFFFFFFFFFFFF) / 5 + 1;
  for (uint32_t i = 0; i < N; i++) {
    if (is_divisible(i, M3))
      count3 += 1;
    if (is_divisible(i, M5))
      count5 += 1;

Here is the number of CPU cycles spent on each integer checked (average):

Compiler 4.5 cycles per integer
Fast approach 1.9 cycles per integer

I make my benchmarking code available. For this test, I am using an Intel (skylake) processing and GCC 8.1.

Your results will vary. Our proposed approach may not always be faster. However, we can claim that some of time, it is advantageous.

Update: There is a Go library implementing this technique.

Further reading: Faster Remainder by Direct Computation: Applications to Compilers and Software Libraries, Software: Practice and Experience (to appear)

Footnote: What is N? If both the numerator n and the divisor d are 32-bit unsigned integers, then you can pick N=64. This is not the smallest possible value. The smallest possible value is given by Algorithm 2 in our paper and it involves a bit of mathematics (note: the notation in my blog post differs from the paper, N becomes F).

Follow-up blog post: More fun with fast remainders when the divisor is a constant (where I discuss finer points)

Science and Technology links (February 3rd, 2019)

  1. A Canadian startup built around electric taxis failed. One of their core findings is that electric cars must be recharged several times a day, especially during the winter months. Evidently, the need to constantly recharge the cars increases the costs. I think that this need to constantly recharge cars challenges the view that once we have electric self-driving cars, we can just send our cars roaming the streets, looking for new passengers, at least in the cold of winter in Canada.
  2. If you are going to train computers to emulate medical doctors, you need objective assessments of medical conditions. That may prove more difficult than it sounds. Human experts rarely agree on medical diagnosis and therapies. How can you assess an artificial intelligence in these conditions? Somewhat ironically, the first step might be to learn to assess better the medical doctors themselves. This may not prove popular.
  3. Researchers are making progress toward reconstructing speech from neural patterns (Nature paper). The accuracy is still low but it is getting credible. One day, we may be able to speak through a brain implant.
  4. The US spy agency (NSA) is the largest single employer of mathematicians in the world?
  5. Neanderthals are often believed to have vanished because they could not hunt as efficiently as homo sapiens: they needed to get close to the prey to kill it unlike us. However, it seems that neanderthals could throw their spears far away.
  6. A highly cited Canadian medical researcher (Sophie Jamal) has been banned from getting further research funding in Canada because of how she fabricated data. When caught, she put the blame on her assistant. She is cited about 1000 times a year and she is the author of about 50 research articles. She lost her job at the University of Toronto where she was a professor and her medical license. She was the research director of the Centre for Osteoporosis & Bone Health.
    As I am fond of saying, it is almost trivial in research to fabricate results. Thus, while it is hard to know for sure how frequently results are just made up, it is probably more frequent than most people expect. And before you object that work is peer reviewed: when reviewing a manuscript, you are not going to redo the work to check that it works. Even if you wanted to check the work, it is often impossible to do it in an economical fashion. That’s why I argue that we have to take into account the reputation of the authors when reviewing a science paper. If you have found someone’s results to consistently be reliable in the past, it is reasonable to give them more credibility in the future. Reputation matters.
  7. According to an article in the Guardian, aspirin prevents cancers. (To my knowledge, this has not been robustly demonstrated yet.)
  8. Party balloons were invented by scientist Micheal Faraday.

New Web host

Following my recent blogging problems, the majority advice I received was to move to a different host.

My blog is now hosted by SiteGround. Moving my content over was far easier than I anticipated, so far.

I am no longer using Cloudflare to cache everything so your queries should hit directly my blog engine. In particular, this means that comments should work properly (comment and then see your comment). I will still rely on Cloudflare for performance, but hopefully not to keep the blog alive.

You should also be able to comment on all posts, including older ones.

Please report any problems.Going straight to production without further testing would be insane if I were running a business, but this is a non-profit blog.

Several people offered to help out with the move. Given that it took me less than one hour, it made no sense to outsource that task. I spent the bulk of the time reverse engineering tiny settings. I also have several minor domains that I needed to move, and finding the content was the hard part.

However, if I have further performance problems, I will seek help (either paid or unpaid). Thanks everyone!

Web caching: what is the right time-to-live for cached pages?

I have been having performance problems with my blog and this forced me to spend time digging into the issue. Some friends of mine advocate that I should just “pay someone” and they are no doubt right that it would be the economical and strategic choice. Sadly, though I am eager to pay for experts, I am also a nerd and I like to find out why things fail.

My blog uses something called WordPress. It is a popular blog platform written in PHP. WordPress is fast enough for small sites, but the minute your site gets some serious traffic, it tends to struggle. To speed things up, I tried using a WordPress plugin called WP Super Cache. What the plugin does is to materialize pages as precomputed HTML. It should make sites super fast.

There is a caveat to such plugins: by the time you blog is under so much stress that PHP scripts can’t run without crashing, no plugin is likely to save you.

I also use an external service called Cloudflare. Cloudflare acts as a distinct cache, possibly serving pre-rendered pages to people worldwide. Cloudflare is what is keeping my blog alive right now.

After I reported that by default (without forceful rules) Cloudflare did very little caching, a Cloudflare engineer got in touch. He told me that my pages were served to Cloudflare with a time-to-live delay of 3 seconds. That is, my server instructs Cloudflare to throw away cached pages after three seconds.

I traced back the problem to what is called an htaccess file on my server:

<IfModule mod_expires.c>
  ExpiresActive On
  ExpiresByType text/html A3

The mysterious “A3” means “expires after 3 seconds”.

How did this instruction get there? It gets written by WP Super Cache. I checked WP Super Cache.

I work on software performance, but I am not an expert on Web performance. However, this feels like a very short time-to-live.

I am puzzled by this decisions by the authors of WP Super Cache.

My blog can’t keep up: 500 errors all over

My blog is relatively minor enterprise. It is strictly non-profit (no ad). I have been posting one or two blog posts a week for about fifteen years. I have been using the same provider in all this time ( They charge me about $50 a month. I also subscribe to Cloudflare services, which costs me some extra money.

I use wordpress. If I had to do things over, I would probably choose something else, but that is what I have today. I have thousands of posts, comments, pages, and lots of personalization: I’d rather not risk breaking or losing all this content.

I use php version 7.0. My host provides version 7.3, but 7.0 is the latest they support with something called mbstring. Without mbstring (whatever that is), my blog simply won’t run.

I estimate that I get somewhere between 30,000 and 50,000 unique visitors a month. Despite my efforts, my blog keeps on failing under the load. It becomes unavailable for hours.

I have given up on writing new blog post using the online editor. It is brittle. The old wordpress editor worked relatively well, but since upgrading wordpress, they have now pushed something called the Gutenberg editor. It tries to be clever, but half the time it just fails with a 500 error (meaning that the server just failed). So I use a client called MarsEdit. It seems to work well enough. (Update: after the blog stopped throwing 500 errors constantly, I was able to switch back to default editor which works better for me.)

Several times a week, someone emails me to report that they tried to leave a comment and they got a 500 error.

I seem to get about one spam comment per minute or so. I have just now decided to close comments on posts older than 30 days, in the hope that it will relax the load on my server.

My error logs are filled with “End of script output before headers” (a few every minutes).

I used to rely on WP Super Cache, hoping that it made things better, but I have since disabled most plugins. I am hoping Cloudflare can do the work. (Update: I have since re-enabled WP Super Cache, now that I have fewer php failures, in the hope that it might work. I do not think that it can do its work if your php scripts can’t run to completion.)

I had the following line in the .htaccess file at the root of my blog:

Header set Cache-Control "max-age=600, public"

The intention was that it would entice Cloudflare to cache everything. I do not think it worked.

Because my error logs showed that wp-cron.php was failing every few minutes, I added the following in my wp-config.php file:

define('DISABLE_WP_CRON', true);

I setup a separate cron job to call wp-cron.php every hour.

I now use Cloudflare with the following settings: “Caching Level: Ignore query string”, “Respect Existing Headers” and “Cache: Everything”. I pay for Argo, whatever that is, in the hope that it might improve things. With these settings, I would expect Cloudflare to cache pretty much everything. It apparently does not. My blog gets hammered. Cloudfare reports 45,000 uncached requests for the day, and most of them are in the last couple of hours. (Update: I managed to get cloudflare to cache everything by going to page rules, and setting cache everything, and waiting a few hours. I had to make sure my rules were applied correctly.)

I have asked my host provider ( to give me more memory, but they seem unwilling to do it transparently. Though is neither cheap nor particularly modern, they have been professional. I have purchased a service with SiteGround, as I am considering moving there because it seems more popular than csoft. I am not super excited about tuning my PHP/Wordpress setup, however. I fear that it is wrong-headed optimization.

What am I missing? How can I be in so much trouble in 2019 with such a relatively modest task?

Note 1: I am aware that there are centralized platform like Medium. This blog is an independent blog on purpose.

Note 2: Many people suggest that I move away from WordPress to something like static generation (e.g., Hugo). I am sympathetic to this point of view, but it is a much easier choice to make when you are starting out and don’t have thousands of articles to carry over.

Credit: I am grateful to Travis Downs and Nathan Kurz for an email exchange regarding my problems.

Update: My blog is now hosted with SiteGround.