Pick one: determinism or fairness

Computers changed our life drastically in the last few decades. Correspondingly, I view the world in terms of algorithms. When I think of how the government works, for example, I see over-engineered heuristics. Bureaucracies are akin to spaghetti code.

Can you tell me how my tax dollars are spent? Surely, you can come up with histograms showing how the money is distributed. But beyond such a rough analysis, nobody knows exactly how money flow within government.

I believe this complexity is beneficial. People need to feel that the system is fair. We achieve the appearance of fairness through obscurity. If I knew that my wealthy neighbor, the guy with a larger house and a larger car, makes all this money through government contracts because he has a friend at the right place, I would get upset. But right now, who knows what is happening?

Effectively, to people like me, government appears like a randomness generator. I don’t quite know how much money the government is charging me, and I don’t know what I am getting back in funding and services. I trust that most people are similarly in the dark. I therefore feel that it is somewhat fair… because it is illegible.

But this works mostly because the data is not available. Of course, if I had access to all government data, it would be too much to take in on my own… but I could analyze it with a computer. Decades of work in business intelligence has taught us a few tricks. We can represent very complex systems in a way that lets the user navigate the information efficiently. Couple this capacity with a social approach, where many citizens could collaborate during the analysis, and we can effectively break the secrets of government. Soon, government could become legible. The illusion of fairness might be broken.

How likely is it that we will have such open governments? It is difficult to tell. In many countries like Canada, we had arrived at a compromise where the press can invoke the law to access some documents. More recently, at least in Canada, even the press is routinely frustrated by its attempts to open up the government. And Wikileaks might have unpredictable effects on governments: some governments will surely tighten security.

Yet I don’t believe governments will be able to hide forever: the long-term trend is that we have access to more data, not less. But maybe there is a saner alternative to obscurity?

One could object that governments could tend toward an ideal of deterministic fairness. Alas this is doomed. For example, Arrow’s theorem states that no deterministic voting is fair. It is one of several mathematical results showing that determinism and fairness do not mix. You get one or the other. In Demarchy and probabilistic algorithms, I promoted that idea that elections themselves should rely on randomness. But I also think that the inner workings of government should embrace randomness:

  1. To ensure that they get the lowest prices, governments ask for vendors to bid on jobs or orders.  Current bidding systems are easily broken by insiders. If you know how others are bidding, you can easily outbid them. To compensate, governments complicate the process to the point where hardly anyone wants to participate anymore.  But there are many random selection algorithms that are much harder to game. By bidding lower, you only increase the probability that you will selected, you can never be sure to get the jobs. Effectively, it can be fairer by being harder to game.
  2. Hiring for public jobs is often a big problem. It is tempting for a civil servant to favor his friends and his family when recruiting. To get around this problem, governments create complicated rules and lengthy processes. But what about selecting randomly among the qualified candidates? This would be fair and I bet that the selected candidates would be just as good as the ones governments hire right now.
  3. In my town, we have some services, such as a community garden, that are provided on a first-come, first served basis. Hardly anyone knows of these services, and that is good because the local government could  not provide them to everyone. So these programs remain obscure. It seems like a random allocation would be a lot fairer.

Conclusion: Governments often insist on determinist methods. These are often unfair. We can better tend toward an ideal of fairness by embracing randomness. It seems counter-intuitive at first, but it is certainly preferable to obscurity.

21 thoughts on “Pick one: determinism or fairness”

  1. @Babcock

    Very good point. People like to think that they are in charge of their life. Yet you are never really in control. Things that you can’t even begin to model always impact your life. (A financial crisis, cancer, death of a child, abuse by a teacher and so on.) You have to accept it.

  2. @Bilar

    Thanks for the great references.

    As you may have guessed, I provided a simplified interpretation of Arrow’s theorem on purpose. I do link to Wikipedia for a more complete discussion.

  3. @Paul

    True. No system would be perfect. We can just alleviate the problems.

    About first-come, first-serve. It is a test, but a test of how well informed you are, not of how badly you need something.

  4. @Andry

    the bidder with the best “value” wins the contract

    This is efficient, but not resilient. What you want is both good value, and also competition to provide such value.

    (…) in the UK at least it is possible under the law to ask for any information about selecting candidates for a contract as well as the bids themselves (…)

    On the long run, that is where we are going. Right now, I see several people frustrated by how complicated it is to get such data.

  5. I agree with you, provided we embrace fairness as a laudable goal in the first place. Which I do, but sometimes I wonder how many people share that premise.

    Deficiencies in education aside, I think one of the reasons it’s difficult for people in the Western world, particularly the United States, to accept that random methods could be superior is that it undermines the premise of the “American dream”. The myth that anyone can pull oneself up “by the bootstraps” and go from lower- or middle-class to upper-class in a single generation, provided one just works hard enough, is steeped in a determinist perspective. Luck doesn’t enter into it: if you aren’t rich yet, then you just aren’t working hard enough!

    In that way, obscurity is also a way of perpetuating the myth. People can continue to lie to themselves and say that only skill and ambition matter, when it’s just as much a matter of luck and connections—as long as we don’t know, it allows them to continue hoping and believing in the myth.

  6. But if you blindfold everyone and just focus on the price, you’re neglecting the secondary issues (…)

    We should favor cheaper solutions, but not necessary pick them all the time. We also want other vendors to participate in the market and be given a chance to prove themselves.

    We use the same idea when picking stocks. Nobody picks a single stock. We always diversify for better resilience.

    If you insist that some metric like ‘years of experience’ is all that matters,(…)

    Precisely. We should not do this. So how do we pick the best candidate out of a set of equally qualified candidates?

  7. Great article. What I liked best was your recommendation for introducing randomness into bidding systems.

    Governments often think that insiders are the only problem with bidding processes, and hence usually try to make the system more “fair” by introducing online tendering and other mechanisms. While this may reduce insider advantage, it does nothing to prevent collusion among contractors.

    Funnily enough, given small sample sizes, randomness can also be perceived as unfair, further preventing collusion among vendors and contractors.

  8. Hi Daniel

    I hope this note finds you well. I enjoyed your raising the issues of democratic governance and randomness.
    I;d like to point out a few things

    1) Your statement of Arrow’s theorem is too strong: no *ranked* voting system with *3 or more options* is fair. For neat overview of voting, see Poundstone’s book and this SIAM review: http://www.siam.org/news/news.php?id=1443

    2) Randomness-as-fairness ties into the “Veil of Ignorance” concept by Harsanyi (and later Rawls). Have a look at a deeper discussion http://emlab.berkeley.edu/~kariv/KZ_I.pdf

    Best regards

  9. @Andy

    Introducing randomness does not destroy competition. You can taylor it so that the better the deal, the more likely one is to win the contract, but you also make it so that nobody can ever crush his competitors completely just by being marginally cheaper.

  10. @Paul

    When I worked in the government, we were never consulted during the hiring of a colleague. In my experience, managers and HR do the recruiting in government. Not colleagues.

    And the managers often don’t have to directly work with the people they hire. You may never see the people who have hired you.

  11. Daniel, if you know about the subtleties, then why do you go on to write

    > Governments often insist of determinist methods. These are *never* fair.

    ?

  12. I would continue a bit on your claim that determinism and fairness do not mix. I really have no idea what “fairness” would mean in a democracy. No matter what the state of the world is, there will always be some for whom their welfare is greater than the welfare of others. No matter who is elected president, some will happy with that result, and others won’t be. The only way you could really have fairness would be to give enough money to those unhappy with the result to cause their utility to be equal to that of everyone else. Totally infeasible with present technology (no hedonimeter has been invented).

    Randomness does not fix this one iota. Random results will still cause some people to be less happy than others. So what you must be talking about is fairness in terms of “size of effect on the result”. Wit randomness, we all have zero effect on the result. However, I fail to see the relevance of that.

    In any case, it is axiomatically proved that the “best” result for the group is the result that maximizes the sum of individual utilities.
    http://ScoreVoting.net/UtilFoundns.html

    That is why Bayesian Regret is “the one right metric” of voting method performance, and why we must focus on improving that, rather than declaring the endeavor futile. Arrow’s Theorem has done an enormous amount of damage to the field of social choice, by wrongly convincing people that the quest for a more ideal system was futile. The worst part is that this theorem is not nearly as important or interesting as the Gibbard-Satterthwaite Theorem

    http://ScoreVoting.net/GibbSat.html

    or the Simmons-Smith Theorem.
    http://scorevoting.net/SimmonsSmithPf.html

  13. While I do generally agree that randomization is underutilized, here are some counterpoints:

    1) Insider bidding would be alleviated but not eliminated (you don’t get to bid $1 less than the previous lowest bid, but you can still optimize your profit/probability ratio with insider information. Collusion may actually be made worse, though. In a binary system, ten companies might agree to double their bids. If one company defects and only multiples by 1.5, they profit. Thus you’ve got a prisoner’s dilemma. The more randomness you add, the more defections colluders can absorb while still profiting.

    2) Again, we alleviate but don’t eliminate. Already this issue is gotten around by tailoring qualifications to an individual. There’s also a related issue of promotions. Here, qualifications can more directly be influenced by a nepotistic individual. Preferred candidates can be commended, competitors demerited. The alternative of not taking performance at a current job into account doesn’t seem promising either.

    3) In a sense, don’t first-come-first-serve and obscure programs serve as need/want testing? Individuals who’d benefit most seem most likely to learn about and show up early for a program. I feel better subsidizing somebody who cared enough about getting veggies to find out about this obscure program and showed up early, then someone who just won a lottery.

    I’m also curious what effect this would have on public morality. When you introduce new safety features to cars, you often see a jump in accidents (people reacting to the improved safety by driving more aggressively). I wonder if the same thing would happen here, where an individual wouldn’t use insider information to win a bid, but might justify using it to increase his odds slightly. Do we have different moral standards for definitely stealing something, versus increasing the odds of a win?

  14. I think that the fundamental point is that “fairness” has different meanings. In the context of government procurement, this should surely mean that the bidder with the best “value” wins the contract – a random process would surely be worse than a deterministic method at achieving this (unless the government is extremely corrupt!)

    In practice, it is not true to say that “hardly anyone participates in the government bidding process” – in fact, a rather large number of companies and individuals thrive on exactly that. And having run several interview processes in government, I can tell you that it isn’t a particularly complicated process; that it doesn’t allow civil servants to interview people they know; and that it is extremely unlikely that a randomly chosen qualified candidate would be equally good (I’ve had some candidates who were technically qualified but clearly incapable of doing the job).

    Government also isn’t really obscure – it publishes all the rules on procurement, for example, and in the UK at least it is possible under the law to ask for any information about selecting candidates for a contract as well as the bids themselves. This is also audited for any substantial purchase.

    This extends further – actually, people DO know how money is spent in government. This might be a good opportunity to extend the randomness principle to something slightly ridiculous – imagine if government funding was randomly allocated, rather than based on a deterministic system of need. Your article suggests that randomness could be fairer than the determinstic system… but is it really “fair” to pay every penny of taxpayer’s money to say, pave the streets with gold? Surely it is “fairer” to exclude at least some possibilities – and so, create a deterministic system?

    Finally, I think you are right to suggest that deterministic systems can never be completely fair. But compared to the alternatives, they are pretty good.

  15. I’m not so sure I buy into the initial premise. If “fairness” (which is subjective) matches some projection function of all the appropriate variables, when a system is optimized in regard to that, it would maintain its determinism.

    What I think you mean is that the current systems are unfair because they are considering the wrong variables. If so, your three examples only shift the problems, but do not eliminate them.

    For example, random bids. The current process elicits favoritism and is easily gamed by the larger organizations. But if you blindfold everyone and just focus on the price, you’re neglecting the secondary issues like quality, consistency and long-term support. You may end up with the cheapest solutions, but if the quality is so poor that they have to be replaced more often, you’ll end up spending more money (thus cheap in the short-run does not equal cheap in the long-run).

    For employees, the right person for the job is the one that fits best. You’re friends with people because there is a good fit. If you insist that some metric like ‘years of experience’ is all that matters, the consequence is that you’ll get a lot more bad fits (and the employees will be having a lot more bad fits 🙂

    On you’re initial point, yes, “ignorance is bliss”. Knowledge is a double edged sword. You may want to know, but you’ll be unhappy when you do…

    Paul.

  16. For solutions, if we were being rigorous I would expect that there would be a number of different metrics on which each vendor was rated. Then one could score them within some set of priorities/trade-offs. It would never be fair (and it would be time-consuming), but gradually as it evolved, it might offer the ‘best deal’.

    With employees, I think it’s always best to leave it up to the people who have to work with the new person. Sure they’ll have irrational ideas for why one is better than the others, they’ll clump by relationships and culture, but at least they’ll be empowered to some degree.

    Fairness in employment is doubled sided. From a prospective employee’s view, if they meet the basic requirements they want the job. From the existing employees view, even if someone is exemplary, if they don’t get along with the existing people on a personal level, they’re not the best qualified. You get to choose between the best people, or the people that work together best. Myself, I’d rather have the latter.

    (Some day my boss is going to wonder why I always have latin numerals on my screen 🙂 )

    Paul.

  17. “This is efficient, but not resilient. What you want is both good value, and also competition to provide such value.”

    Yes, I agree – but randomisation destroys competition, because there are no criteria on which to compete. If you set out your requirements clearly, there is at least a basis to compete (the equivalent of consumer intelligence).

    This goes for recruiting someone, or bidding for a contract. With no incentive to improve particular desirable attributes, an efficient contractor would put in a huge bid on the basis that since it won’t affect the likelihood of getting a contract, they might as well make a huge profit if they do get it!

  18. Where I am is sort of similar. Sometimes people show up, and I’m asked if I can use them or not. But I’m not shy about saying “NO”.

    It would be fine if everyone was truly independent of each other, but of course they’re not. One bad seed, even if they are the most qualified, can reek havoc on morale, structure, etc.

    If we were looking for some ‘fair’ way, perhaps it’s a two stage deal. 1) you get hired, and 2) a team recruits you to their work (if you’re not recruited, and there are no one-man projects, you’re eventually laid off). Thus the recruiting process, which is inherently biased is mitigated by the control at the door. I’ve heard that Google works roughly like this…

    Paul.

Leave a Reply

Your email address will not be published. Required fields are marked *