With Anders Munk and Asger Olesen we have been experimenting with machine learning with the help of a physical device built by Asger. It looks like an arcade game and for reasons that will appear, we call it the “thick machine”. We presented it at the Machine Anthropology Workshop at Copenhagen University today (2020-01-27). It is a story about ambiguity, and the materiality of the digital.
The “thick machine” is a custom-built computer. It is based on a refurbished screen, a Raspberry Pi, some arcade buttons, and a custom-made case inspired of arcade machines. It contains a software coded in Python using a simple game engine, where you can play a guessing game. In this game, you try to guess which of 5 emojis is the right one; but you actually compete with an algorithm. This algorithm is a specially trained classifier, and it tries to guess the right emoji, exactly like you. There is no score or advanced game mechanics: you commit to a choice, then you see both the right answer, and the guess of the algorithm. That’s is all. And it is already interesting, but let me explain what the guess is about.
At the TANT Lab, we have what we call “The Atlas of Danish Facebook Culture”, which is basically a harvesting of the whole Danish Facebook from before the APIcalypse (the Facebook API closed these accesses). That is our starting material.
On Facebook, when you “like” a content, you also have the choice of five different emotional reactions, represented by an emoji. You can react with an emoji, write a comment, or both.
In our corpus we have 128 million comments, 700 million emoji reactions, and 23 million emoji-reactions (both at the same time). The game is about those: given a commentary, can you guess which emoji was used by the author?
We also trained a classifier for that task. We tinkered with different algorithms, but that is not the most interesting part. We landed on a simple neural network from a popular Python library, SciKit Learn.
It turns out this classifier is neither better nor worse than humans (i.e. lab people having played the game on the thick machine). Not only is it as accurate, but the structure of the results is also the same. This is something you can see in a confusion matrix.
A confusion matrix shows which emoji is guessed depending on which emoji is the right one. An algo that guesses right all the time would have only the diagonal filled, and an accuracy of 100%. An algo that guesses at random would have every cell filled equally, and an accuracy of 20% (one chance over five to guess right). In our case, we are somewhere in-between.
We have two confusion matrices. On the left, the results of human guesses. On the right, the results of the algorithm. Both achieve about 50% accuracy, which is not so bad, but also not so great. Perhaps surprisingly, humans do not achieve a great score (note: our numbers are still low, it’s a work in progress). But the most interesting is the similarity of the results:
- There is some degree of confusion between ❤️LOVE and 😲WOW
- There is some degree of confusion between 😢SAD and 😡ANGRY
- The [❤️LOVE+😲WOW] group is rarely confused with the
- The 😆HAHA reaction is rarely confused with others
If we only had the classifier we may presume that something, in the black box of the algorithm, produces this confusion. But now that we know that humans are similarly confused, we think that the confusion is a feature of the data. This observation leads to a productive way to repurpose the algorithm: to find the ambiguous cases.
Indeed, the way people on Facebook is not consistent, what we consider “the right answer” is not really the right answer, because there is no “right answer”. And where this ambiguity lies, we can find the most interesting cultural effects.
For instance, in the two examples below, we suppose that the person uses 😆HAHA to distance themselves from their slightly childish reaction (case 1) or the socially awkward mention of sexual practices (case 2).
Contrary to most cases, which are rather obvious, these reactions are deep. They are more interesting for analysis, and finding them has a scientific application.
Tracking confusion in emoji reactions is interesting because we already know that they are ambiguous. Indeed, emoji have different meanings to different people. The funniest case we observed was this grandma who expresses her sadness with the “tears of joy” emoji. Of course, they have tears, but… judge by yourself.
For us, the accuracy of the classifier is not the productive output (anymore). It is not really a problem that it fails at predicting; we do not want to improve its accuracy. The interesting output lies in the cases where confusion happens.
The machine is thick in the sense of “dumb”, because its accuracy is not great – and humans can be thick too. But it is also thick in the sense of “deep”, in the sense of ethnography, in the sense of Geertz. It is as thick as the wink, something that you cannot understand unless you are yourself involved in the culture.
The claim to attention of an ethnographic account does not rest on its author’s ability to capture primitive facts in faraway places and carry them home like a mask or a carving, but on the degree to which he is able to clarify what goes on in such places (…) This raises some serious problems of verification, all right–or, if “verification” is too strong a word for so soft a science (I, myself, would prefer “appraisal”), of how you can tell a better account from a worse one. But that is precisely the virtue of it. If ethnography is thick description and ethnographers those who are doing the describing, then the determining question for any given example of it (…) is whether it sorts winks from twitches and real winks from mimicked ones.Geertz, C. (1973). The Interpretation of Cultures, p.16
“Great!, will you say, but why building the arcade machine?” Indeed we could have achieved the same result by classifying the cases in an Excel spreadsheet. What does the materiality of the machine change?
Being able to build the machine and actually building it are actually not the same. Imagining that you play the game also differs from actually playing it. But in the physical absence of the device it is more difficult to account for the difference it makes.
When you play the game, you realize how weirdly aligned you are with the classifier. You get the feeling that you and the classifier are, somehow, right, and the “ground truth” is wrong. You feel a logic to reactions, that you share with the classifier, and you can also understand why certain actual reactions deviate from that logic.
This feeling is quite dependent of the situation where you are. The context of Facebook must be removed, so that your guess is based only on the text reaction. You also must not see the actual reaction beforehand, or you would be biased. You must have a simple way to input an answer, as each guess requires focus and attention. The materiality of the arcade machine has a number of effects.
But beyond these details, the arcade machine actively repurposes the classifier. Its symbolic effect is, I think, the most important. By conveying the idea of a game, it puts people in a different position towards the algorithm, and this is actually hard to achieve. We have seen it.
Although our main point was to say that the accuracy of the classifier was not how it was productive for us, multiple people came to us telling how to improve its accuracy. We observed that it is actually hard for researchers to think of their algorithms in a different way. Yes, in theory, anyone can do the same in Excel. But at the same time, our habitual toolkit anaesthetizes our ability to recontextualize algorithms – it is an aesthetic matter! We cannot repurpose without imagination. Breaking the charm requires a strong re-dramatization, and I think that the thick machine accomplished that for us.
By its materiality, the thick machine forced a different setting on us; and that setting allowed us to find a new productive way to repurpose the algorithm.