The Machine Doesn't Decide. A Human Does.

In 2016, journalists at ProPublica pulled the risk scores of more than 7,000 people arrested in one Florida county. The scores came from software called COMPAS, which predicts how likely someone is to commit another crime. Judges saw the number when setting bail and sentences.

Here is the thing to hold onto: COMPAS never sent anyone to jail. It produced a number. A judge — a human — read that number and decided. But judges are busy, the number looks objective, and "the computer said high-risk" is a hard thing to argue against. So in practice, the machine's prediction became the decision.

That is the whole game, and it runs the same way in every AI system you will ever meet. Three questions.

One: name the move. Every system does exactly one of four things:

• Generate — makes new content that didn't exist before (the Coca-Cola ad).
• Rank — orders or filters things that already exist (the Netflix posters).
• Predict — estimates a future you can't see yet (COMPAS).
• Detect — decides whether a thing matches a category (the face-match).

Two: find the decision boundary. A machine either recommends — a human can still say no — or it decides, acting on its own while a human only finds out later. The most dangerous systems are the ones in between: where everyone assumes a human is checking, but nobody really is. A judge who rubber-stamps the score. A detective who trusts the match. A spam filter you never open.

Three: name who pays. When the machine is wrong, the cost almost never lands on the machine, and rarely on the company that sold it. It lands on the person with the least power to argue: a defendant who doesn't get bail, Robert Williams arrested on his lawn, the artists who weren't hired.

One last thing. Northpointe said COMPAS was "equally accurate" for Black and white defendants — and that was true. ProPublica said it was biased — also true. They were using two different definitions of "fair", and both are mathematically reasonable. You usually cannot have both at once. Remember that argument. Next week, you'll have to judge it yourself.