Panopticons, CV, rasism, and the architecture of the AI gaze

#AInthropology

We’ve built a world completely obsessed with teaching computers how to see, but for how deeply sensitive we are to how others see us, it's amazing how rarely we stop to figure out if computers see us the way we WANT them to see us.

Here we are, full control over our image, and we do nothing.

When we look at a CV pipeline—Computer Vision, not the vitae—it’s not just an array of matrix multiplications detecting edges and contours. We are dealing with an institutionalized, electronic gaze, the dots and lines and all the fancy stuff from SF movies that show us we've been "read".

Now, we’ve always adjusted our behavior when we know we're being watched; it's a foundational human trait that varies very little, but in very specific ways (I'll nerd about this some other time. It's gobsmacking).

In 1791, J. Bentham designed a circular prison layout called the Panopticon, where a single unseen guard could be watching any prisoner at any given time; the prisoners slowly and effectively became their own wardens.

Foucault later obsessed over this- how power becomes a machine independent of the person who runs it. In both cases, the signifier—the camera lens—becomes detached from any actual human, leaving us to literally perform for an empty eye.

So here we are in 2026, and we've managed to scale that prison architecture into a cloud-native service, with an automated guard on every street corner, every phone lock screen, and every hiring platform.

The twist: this automated eye doesn't see us equally. The legendary Gender Shades study showed that while commercial facial analysis software had an error rate of just 0.8% for light-skinned men, the ID error rate went to 34.7% for darker-skinned women. Turns out the machine’s gaze is predictably localized, trained heavily on the default faces of a few programmers, and pushed globally as a universal standard just like safety belts were. It’s phrenology with an API token.

When a system only recognizes specific bodies and flags others as anomalies, it’s an asymmetric distribution of vulnerability => the stakes are entirely different depending on who you are. If I dress corporate, a glitch means a minor delay; my relative, who's part of an over-policed minority, gets pulled over for "random" checks quite often when we travel; for her, a high-frequency algorithmic blip can mean an unprompted, armed arrest based on a false match. We’re all trapped in the same glass house, but some of us are sitting much closer to where the stones are being thrown. We've traveled together over 30 times. I've seen it.

We’ve essentially built an optimization engine that asks we adapt to the machine's (narrow) vision, rather than the other way around.

Maybe instead of just trying to make the panopticon more accurate at labeling us, we should start questioning if it's the only transparency model that works.