At my university, one of our internal software systems allows a professor to submit a revision to a course. The professor might change the content or the objectives of the course.
In a university, professors have extensive freedom regarding course content. As long as you reasonably meet the course objectives, you can do whatever you like. You can pick the textbook you prefer, write your own and so forth.
However, the staff that built our course revision system decided to make it so that every single change should go through all layers of approvals. So if I want to change the title of an assignment, according to this tool, I need the department to approve.
When I first encountered this new tool, I immediately started to work around it. And because I am department chair, I brought my colleagues along for the ride. So we ‘pretend’ to get approval, submitting fake documents. The good thing in a bureaucracy is that most people are too bored to check up on the fine prints.
Surprisingly, it appears that no other department has been routing around the damage that is this new tool. I should point out that I am not doing anything illegal or against the rules. I am a good soldier. I just route around the software system. But it makes people uneasy.
And there lies the scary point. People are easily manipulated by computing.
People seem to think that if the software requires some document, then surely the rules require the document in question. That is, human beings believe that the software must be an accurate embodiment of the law.
In some sense, software does the policing. It enforces the rules. But like the actual police, software can go far beyond the law… and most people won’t notice.
An actual policeman can be intimidating. However, it is a human being. If they ask something that does not make sense, you are likely to question them. You are also maybe more likely to think that a policeman could be mistaken. Software is like a deaf policeman. And people want software to be correct.
Suppose you ran a university and you wanted all professors to include a section on religion in all their courses. You could not easily achieve such a result by the means of law. Changing the university regulations to add such a requirement would be difficult at a secular institution. However, if you simply make it that all professors must fill out a section on religion when registering a course, then professors would probably do it without question.
Of course, you can achieve the same result with bureaucracy. You just change the forms and the rules. But it takes much effort. Changing software is comparatively easier. There is no need to document the change very much. There is no need to train the staff.
I think that there is great danger in some of the recent ‘digit ID’ initiatives that various governments are pushing. Suppose, for example, that your driver’s license is on your mobile phone. It seems reasonable, at first, for the government to be able to activate it and deactivate it remotely. You no longer need to go to a government office to get your new driver’s license. However, it now makes it possible for a civil servant to decide that you cannot drive your car on Tuesdays. They do not need a new law, they do not need your consent, they can just switch a flag inside a server.
You may assume then that people would complain loudly, and they may. However, they are much less likely to complain than if it is a policeman that comes to their door on Tuesdays to take away their driver’s license. We have a bias as human being to accept without question software enforcement.
It can be used for good. For example, the right software can probably help you lose weight. However, software can enable arbitrary enforcement. For crazy people like myself, it will fail. Sadly, not everyone is as crazy as I am.
7 thoughts on “Enforcement by software”
“All our inventions are but improved means to an unimproved end.” (Thoreau)
My other primary concern in the same vein is the ease with which digital bodies of text can be edited after publishing. There are many cases already of online posts being removed, but also of news articles being edited without acknowledging the change. Thus, what is true 5 minutes ago for one reader can be false for the next reader with a few button clicks.
In the past, publishers would need to publish retractions in the following issue, but those previous issues did not simply disappear.
It’s not just that: think of the well-known stance “release early, release often”!
Being aware that a published “work” can easily be edited, completed, reviewed, corrected, patched, … any later time, someone might be (tempted to) release unfinished “work”
An excellent point.
You can comment to support a point of view in an article and later the article can be changed while your name still supports it.
The editors/platforms should update all the commenters about the change with a link to edit their comments.
All of healthcare software is exactly this: regulating to the finest point what providers do, and why: if their justifications in the form of required tests, etc. are insufficient, their action is blocked.
People don’t realize the extend to which computer systems in healthcare have been used to usurp the power of professional organizations and judgment. Try spending a day or a week where 80% of your time is documenting via arcane pull-down menus that don’t have the option you want. Smart, committed people who invested a decade in training are giving up and leaving the field.
There’s little evidence for improved patient outcomes or lower costs, and lots of evidence for higher administrative costs.
Another sad incarnation of “Computer says no”:
For french speaking people, Benjamin Bayart describes this as
“L’ordinateur est fatal”:
It starts at about 3:00 and IMHO, it’s quite funny excepted when you’re confronted to it.
You may subscribe to this blog by email.