My blog can’t keep up: 500 errors all over

My blog is relatively minor enterprise. It is strictly non-profit (no ad). I have been posting one or two blog posts a week for about fifteen years. I have been using the same provider in all this time (csoft.net). They charge me about $50 a month. I also subscribe to Cloudflare services, which costs me some extra money.

I use wordpress. If I had to do things over, I would probably choose something else, but that is what I have today. I have thousands of posts, comments, pages, and lots of personalization: I’d rather not risk breaking or losing all this content.

I use php version 7.0. My host provides version 7.3, but 7.0 is the latest they support with something called mbstring. Without mbstring (whatever that is), my blog simply won’t run.

I estimate that I get somewhere between 30,000 and 50,000 unique visitors a month. Despite my efforts, my blog keeps on failing under the load. It becomes unavailable for hours.

I have given up on writing new blog post using the online editor. It is brittle. The old wordpress editor worked relatively well, but since upgrading wordpress, they have now pushed something called the Gutenberg editor. It tries to be clever, but half the time it just fails with a 500 error (meaning that the server just failed). So I use a client called MarsEdit. It seems to work well enough. (Update: after the blog stopped throwing 500 errors constantly, I was able to switch back to default editor which works better for me.)

Several times a week, someone emails me to report that they tried to leave a comment and they got a 500 error.

I seem to get about one spam comment per minute or so. I have just now decided to close comments on posts older than 30 days, in the hope that it will relax the load on my server.

My error logs are filled with “End of script output before headers” (a few every minutes).

I used to rely on WP Super Cache, hoping that it made things better, but I have since disabled most plugins. I am hoping Cloudflare can do the work. (Update: I have since re-enabled WP Super Cache, now that I have fewer php failures, in the hope that it might work. I do not think that it can do its work if your php scripts can’t run to completion.)

I had the following line in the .htaccess file at the root of my blog:

Header set Cache-Control "max-age=600, public"

The intention was that it would entice Cloudflare to cache everything. I do not think it worked.

Because my error logs showed that wp-cron.php was failing every few minutes, I added the following in my wp-config.php file:

define('DISABLE_WP_CRON', true);

I setup a separate cron job to call wp-cron.php every hour.

I now use Cloudflare with the following settings: “Caching Level: Ignore query string”, “Respect Existing Headers” and “Cache: Everything”. I pay for Argo, whatever that is, in the hope that it might improve things. With these settings, I would expect Cloudflare to cache pretty much everything. It apparently does not. My blog gets hammered. Cloudfare reports 45,000 uncached requests for the day, and most of them are in the last couple of hours. (Update: I managed to get cloudflare to cache everything by going to page rules, and setting cache everything, and waiting a few hours. I had to make sure my rules were applied correctly.)

I have asked my host provider (csoft.net) to give me more memory, but they seem unwilling to do it transparently. Though csoft.net is neither cheap nor particularly modern, they have been professional. I have purchased a service with SiteGround, as I am considering moving there because it seems more popular than csoft. I am not super excited about tuning my PHP/Wordpress setup, however. I fear that it is wrong-headed optimization.

What am I missing? How can I be in so much trouble in 2019 with such a relatively modest task?

Note 1: I am aware that there are centralized platform like Medium. This blog is an independent blog on purpose.

Note 2: Many people suggest that I move away from WordPress to something like static generation (e.g., Hugo). I am sympathetic to this point of view, but it is a much easier choice to make when you are starting out and don’t have thousands of articles to carry over.

Credit: I am grateful to Travis Downs and Nathan Kurz for an email exchange regarding my problems.

Update: My blog is now hosted with SiteGround.

Published by

Daniel Lemire

A computer science professor at the University of Quebec (TELUQ).

53 thoughts on “My blog can’t keep up: 500 errors all over”

  1. I don’t know if this comment will show up, but one thing you could consider is to take one or two years worth of the $50 WP hosting fee (so $600 or $1200) and offer it on contract to someone who knows what they are doing with WP, for them to:

    1) Move to a hosting provider that costs say only $10 or $20 per month maximum.
    2) Fix the performance problems through caching and/or identifying the offending code.

    Since step one means you save $30 to $40 a month, this would have a very high return on investment. Of course, you face the challenge of verifying what they’ve done, etc.

    1. Your comment did make it through.

      1) Move to a hosting provider that costs say only $10 or $20 per month maximum.

      I think that the $10 to $20 price range is for unmanaged instances. I would need to handle security, SSL, updates, configuration, backups. I like that someone other than me is doing backups and securing Apache/PHP.

      When I write that csoft.net has been professional, I mean that I haven’t lost any data in 15 years. They also locked things down for me when I was being hacked.

      2) Fix the performance problems through caching and/or identifying the offending code.

      I’d be willing to pay someone for that, but given that I have a managed instance right now, this person could only do so much. You can’t reconfigure Apache, for example, or switch to another server. It seems to come down to installing and configure wordpress plugins, but I am skeptical that it is much help when things go to hell, because at that point, anything php will randomly fail. The .htaccess things is more interesting, of course. There might be some magic hacking there. But by the time you hit php, it is all over.

      Moving to an unmanaged instance and paying someone to help me manage is certainly an option. I think it will cost more in the end, but that might be the solution.

      1. I’d be willing to pay someone for that, but given that I have a
        managed instance right now, this person could only do so much. You
        can’t reconfigure Apache, for example, or switch to another server

        To be clear, I meant that (1) and (2) have to come together. Basically get someone who knows what they are doing with WP, and that comes with a recommended hosting solution in that price range (yes, I suspect they start at way under $50). This person would migrate your existing WP setup and ensure it all works great in a “pushbutton” fashion before handing it off.

      2. … based on that response this whole post seems like an exercise for venting.

        You don’t seem to want change or improvement or lower costs.

        There are definitely better & at the same less expensive hosting options.

        There are definitely worthwhile plugins (WP Super Cache, S3 integrations, etc).

        15 years is a lot of posts, fortunately WordPress it self makes it very easy to export all that data, you could very very easily do that and set up a second website where you could validate data integrity and start exploring ways to save money, harrden WordPress (for security), and take measures necessary to improve performance. You could also offload Commenting to a 3rd party like Disqus that exists solely to make commenting easy from a spam & performance perspective.

        But it seems like with all these options you’re content to be malcontent.

        1. Paul: If you read the next post after this one, you will learn that I have moved to a different host (SiteGround). This happened within a week of writing this post. It is a managed instance, so I get server security, backups and all that. It is also quite fast. It is not, generally speaking, cheaper, but “high cost” was not my concern. (SiteGround is “affordable”, I’d say.)

          The mistake I made was to underestimate how bad csoft.net hosting was. Simply switching host solved my problems.

          Some specific comments:

          • Once your PHP scripts start failing more or less randomly, using plugins like WP Super Cache is not going to save you. These plugins rely on PHP working properly. I had used WP Super Cache for many years, but it did not help the stability of the setup in my experience. In a follow-up blog post I describe how WP Super Cache relies on 3 seconds caching… So it is not going to do a good job, even when it works well, at preventing your wordpress scripts from being hit (hard?).

          • Disqus is not appealing to me, as an independent blogger. I want to own and control my platform. I don’t want to offload comment data and the corresponding user information to a third party that may then possibly resell this data. Reading about disqus (which has, effectively, a monopoly on this type of service) brings many complains having to do with bloat and low performance, sudden changes as to how things work and as to how they are displayed. And, of course, the service may cost money (reasonably enough): I see $9 a month.

          • Moving software to a different setup is not free. The move was much less painful that I expected (took 1h to 2h), but weeks later I am still solving problems related to the move. Some tiny problems that came up took hours to solve, as I needed to research the problems. Avoiding an unnecessary move is rational.

          • Several commenters pointed out that I could just drop WordPress and use something else. I fear that they greatly underestimate how hard this would be. Yes, I know about things like Hugo. My relatively simple home page is built using Hugo… and it took me nearly took weeks of hacking to get it to be how I want. Porting my blog to something like Hugo would be a major disruption, might imply moving to disqus (see point above) and so forth.

          • In my comment above I was answering Travis who was basically recommending that “I pay someone” and go to an unmanaged instance (if I understand correctly). Given that it took me less than one hour to get a clue that performance would be better at SiteGround… It is unclear whether paying someone would have worked. There are transaction costs: you need to setup a job description, you need to agree on the terms, you need to select the person. Then if you go to an unmanaged instance, there are going to be ongoing support costs to make sure things keep on working. It is possible that Travis was right and that I have been wrong, but I think that my own approach to stick with managed instances and mostly do it myself is probably reasonable.

          As someone said, geeks like change, as long as it is change that they cause, otherwise they hate it. Reasonably, I try to maintain stability in my tools and workflow. In this instance, I admit that I was too conservative, I should have moved off csoft.net years go. However, I think that wordpress is probably going to serve me well for years to come. It may be old tech, but the benefits that come from its maturity are probably underrated.

          1. To be clear I was never recommending to go unmanaged. I was recommending that you pay someone who was familiar with the ecosystem to solve your WordPress headache. Since csoft gives you almost zero knobs to turn, this would most likely involve the selection of and migration to another managed instance.

            If you are willing to do it yourself, that could be a good option and in this case it turned out to be a great one since the effort migrate was small (a fraction of what I would have guessed – but that’s why I’m not a WP expert) and you were willing to attempt it yourself.

            I doubt paying someone permanently to handle an unmanaged WP instance makes much economic sense except in unusual scenarios so I wasn’t suggesting that.

  2. Consider moving away from a CMS like WordPress. It’s great at what it does but I think you’ve out grown it. Static content generators might help here. They let you edit off line and compile to a static version of the web content you currently serve. I’d suggest looking at Hugo or one of the many specialised for blogging.
    https://opensource.com/article/18/3/start-blog-30-minutes-hugo

    You can then push the content to something Amazon’s s3 and put cloudfront in front to speed things up.

    You’ll have to then offload the comments to a service but that is relatively easy and cheap. A service like discus is only $9 a month. Moving to s3 you’ll save that in hosting. (I run two sites like this and pay $0.07 on average per month). Not to mention those services are pretty good at filtering spam.

    There are some other options around too. https://darekkay.com/blog/static-site-comments/

    WordPress is nice but it will never out perform static content.

  3. You would get a free hosting with google app engine. And never any 500.
    I host appblit.com on it since 2010 and never worry about a thing. Python, NodeJS or Java. I use python.

  4. With just 30K user, and $50 hosting I don’t think your blog should fail. It means your hosting is way too expensive for quality they are providing. I am using VPS from digital ocean and any $40 server from them if done write will give much better performance to your blog. You need to find another host for sure and possibly with VPS. wordpress only hosting are good too, but they get too complex at time.

  5. Host a static site on GitHub or Amazon S3 (+Cloudfront/flare). Comments can be with Disqus or just link to a post in your own sub reddit / HN.

        1. you can try with Publii static CMS at getpublii.com
          it’s a great piece of software, it’s FREE, runs on Windows, Mac and Linux as desktop app, so you don’t need to login in order to post, you can write while offline and when online just push the update. also has a very nice WordPress importer option, so you can import your WP posts etc…
          and finally you can upload directly from the app to Github pages, Gitlab, Netlify, Amazon S3, or where ever you want thru built-in FTP, SFTP

        2. As someone who’s been slowly trying to move his much smaller blog (lot fewer articles, couple of orders of magnitude less traffic) from WordPress to Jekyll, I can confirm that it’s not as easy as it’s sometimes made out.

          Transferring the blog posts is relatively easy, migrating the comments is a little harder – especially if you want to self host them like I do. Getting the details right like RSS feeds working transparently so your subscribers only notice minimal, if any, disruption is a surprising amount of work.

          That said, I personally still see benefit in moving away from WordPress. WP has a big target on its back security-wise as it’s running so many sites, and I’ve had to recreate a few too many blog posts because the online editor ate them.

  6. Regarding:

    Cloudfare reports 45,000 uncached requests for the day, and most of them are in the last couple of hours.

    I believe the issue is that when CloudFlare caches your page(s), it does so “locally” – whether that means only at that particular CloudFlare datacenter or, more likely, narrowly than that. At least I am pretty sure that Cloudflare does not take the cached response and widely distribute it among its servers.

  7. There are WordPress plugins to convert your existing posts to static site generators (to name a few: Jekyll and Hugo). Jekyll can be hosted by GitHub Pages.
    As said above, you then have to convert every comment to Disqus.

  8. FWIW my blog gets roughly the same amount of traffic, and I don’t think it’s ever been down. I pay $0 for this, it’s hosted using GitHub Pages – they use Cloudflare internally I believe.

    I’ve migrated away from WordPress after realizing I really hate using their editor and this makes me hate the process of writing blog posts. Unfortunately it took some time to migrate and preserve all contents, but I’ve been very happy with the setup since. Some people I know use Hugo, I use vanilla Jekyll that GitHub Pages generate for me.

    1. Thanks. When I started my blog, GitHub did not exist. Facebook did not exist. So there is path dependence involved. I now have to deal with things as they are. I can start anew, of course… but that’s a choice that would cost me.

    1. Thanks. In my case, the php scripts themselves fail, so I am doubtful that adding more plugins can help. Plugins tend to hurt more than they help when they can’t be expected to run without failure.

      Over the years, I tried dozens and dozens of plugins, and I have been consistently disappointed.

  9. Bonjour Daniel,

    I write on Medium and the editor is wonderful. Some articles get featured for extra distribution.

    If you stick with WordPress install the classic editor plugin to remove Gutenberg.

    GitHub pages is also a great free option as mentioned.

    Bonne chance,

    —Martin

    1. I write on Medium and the editor is wonderful.

      A blog on Medium is certainly nice, but it is no longer an independent blog. Someone else decides how it looks and how it is organized, and monetized.

  10. First, Gutenberg is terrible. Install the WordPress Classic Editor (https://wordpress.org/plugins/classic-editor/) as soon as possible and disable Gutenberg.

    For your hosting problems, it seems you’re paying an awful lot just to host a WP blog. Normally a WP blog, with that kind of traffic should be fine
    on a 5-10$ month VPS without any special tinkering. There’s something wrong here.

    If you don’t want to waste your time by fixing your WordPress install (have you tried to export your data and do a clean reinstall ?), you could go with https://wordpress.com/pricing/ (note the .com) the Premium plan should work fine. You’ll still need to migrate your data.

    1. If you don’t want to waste your time by fixing your WordPress install

      I don’t think that the problem is with my installation.

      you could go with wordpress.com

      I expect I would not be able to bring my blog over without breaking things, and I would do away with my independence.

  11. AWS Lightsail offers bitnami LAMPs for 10$/mo. WordPress ready with caching and all. I migrated many WordPress blogs there and its fast. I get about 50,000 monthly visits and no performance issues at all. Check it out.

  12. Give any static site generator a try, blog posts are markdown document which you commit. Statically generated, you’ll never get 500 if you front it with CloudFlare. As Arseny said, hosting it on github pages will be almost free. The true downside is maybe comments, but then you can move to Discuss and never worry about spam again.

    My personnal preference would be Hugo because the installation process is trivial compared to the other.

    1. Here the issue is that there is path dependence. My home page is built using Hugo (https://lemire.me/en/), but it was redone entirely (I threw away my old page). It was a major undertaking to make it look the way I want. I fear it would be weeks of work to port my blog over. Certainly, it would be doable and more efficient, but the cost might be substantial.

  13. From my notes:

    Changed all MaxClients settings (prefork MPM, worker MPM, event MPM) in /etc/apache2/apache2.conf to 40 from 150, on the theory that, with 2GB of physical RAM, and WP_MAX_MEMORY_LIMIT (default) 40M, it’ll use a max 1.6G of RAM which will stop the thrashing “fall over” behavior we’re seeing. (free -m gives 1.7G free when apache’s stopped.)

    The main thing to be sure is that Apache (if you have any control over that at your ISP) isn’t forking more processes than will fit in memory, or else you’ll start thrashing.

    1. I have two concerns with that. One is that moving to a blogging platform breaks my independence. The second one is that I don’t think I can realistically port over my existing blog. I will unavoidably break things in the process.

      1. You’re probably right. I’ve used WordPress.com throughout for my blogs and have not moved to a self-hosted site as yet.
        I’ve exported posts to new blogs so that each site has its own focused niche. But obviously I couldn’t take the existing number of impressions to the new blog. Since the blogs aren’t commercial, it didn’t matter to me. Comments were easily ported as well. This, of course, is only for the free blogs WP.com offers. I haven’t tried any of the professional packages WP.com offers. Ads are displayed on free blogs irrespective if you’ve signed up for WordAds or not.

  14. Just a note on cloudflare: it didn’t save your blog from the “hug of death” because by default it only caches static resources. In the case of this blog that’s only a few mostly inconsequential files. The main page, rendered by WordPress (which is the thing that falls over) isn’t cached cloudflare more or less does nothing.

    1. Yes. It seems that you are entirely correct. To make matters worse, wordpress deliberately defeats caching by appending a query string to static ressources like css or js files. And even when WordPress does not do it, sites like Facebook append their own query strings, again defeating caching. Thankfully, you can tell Cloudflare to ignore that.

  15. I would also advise moving to a static site generator like Hugo but the traffic we’re talking about is not something that should cause even a RaspberryPi to break a sweat.

    Turn PHP’s opcache on, or APC if it’s an older version of PHP. The MySQL configuration probably needs tuning as well. Some plugins may need to be disabled.

    A good tool to figure out what is going on is the Xdebug profiler https://xdebug.org/index.php

  16. Daniel, I could help you in any way possible if you like. Feel free to email me so we could start a thorough investigation. I don’t plan to charge you or anything, so don’t you worry about it.

Leave a Reply

Your email address will not be published. Required fields are marked *

To create code blocks or other preformatted text, indent by four spaces:

    This will be displayed in a monospaced font. The first four 
    spaces will be stripped off, but all other whitespace
    will be preserved.
    
    Markdown is turned off in code blocks:
     [This is not a link](http://example.com)

To create not a block, but an inline code span, use backticks:

Here is some inline `code`.

For more help see http://daringfireball.net/projects/markdown/syntax