The register effect: lists, regimes of absence, and the design of discreteness

This is a follow-up to this blog post where I call bullshit on the claim that computer are radically incapable of certain things because they are discrete, while real life is continuous.

The circulation of such claims matters because it shifts accountability. Indeed, we use computations for analyzing social life, computations relying on discrete data. But social life is continuous, right? So we seem doomed to misrepresent real life with computers, to miss something key. Whose fault is it? If computers just cannot deal with continuity, it is our fault: let’s preserve analog sociology. But if computers could deal with continuity, if alternatives existed, then the responsibility would lie with whoever steered us towards discreteness. Did someone decide, at some point, that computers would renounce the continuous? Of course no single being ever had such power, because our digital environment is produced by a myriad of actors with heterogeneous practices and oppositive agendas. But at the same time, computers seem sentenced to discreteness.

General claims about computers shift responsibility away from the architects of the digital infrastructure. Statements uniting the Digital archipelago into a single territory, as if its inhabitants were bound by a shared destiny, renounce to see the won and lost battles that decided its geography. These narratives sever the chain of events that produced our digital environment as it is, rather than as it might have been, echoing the voice of winners (eg. Google and Facebook) and muffling the dissonance of losers (eg. free open source software). Although the latter keep resisting – count me in.

But there is more to it. Hearing attentively, the hymn of computational modernity has its own dissonance. Algorithms are too dumb to recognize a black face but clever enough to power mass surveillance in China. We normalized the remarkable achievements of weather forecast. We normalized the stupidity of human-replacing algorithms, such as the “computerized system” supposed to determine if your emergency call is an actual emergency. Computers are both threateningly efficient and frustratingly inefficient. Of course, those are different computers. The systems thrown at our faces in the name of cost-reduction have a clear motive for dumbness. But military and industrial systems are a different thing entirely. On that side of the balance of power, the supposed inaptitude to continuity sounds more like a wish than an evidence.

Lists

The first brick of the Google empire was just a search engine. Name a thing, and you get a list of results curated from the entire web. We are in 1998, this engine’s speed is mind-blowing, its quality competitor-crushing, it works like magic. There is a whole story to tell, but we will leave it aside and question the premises of the problem. Why a list?

The search engine problem is to order the list of results by relevance, in a sufficiently short time to satisfy a human user: seconds, milliseconds if you can. Sergey Brin and Larry Page solved it by just using the hyperlinks to sort the list (the famous PageRank), blaspheming against the semantic web by stating that relevance has nothing to do with the content but just with the links, making many enemies in the process, and crushing them under the weight of success. The problem preexisted, they did not invent it, but they killed it by challenging its own terms – relevance. But not that it would be a list. So once again: why a list? Where does it come from?

Lists are as old as writing, and in some ways older: the human species started listing stuff before writing sentences. But the Mesopotamian list has a radical difference with the Google list: the former is inscribed on a continuous substrate while the latter is not. On this literal matter, the discrete and the continuous differ. But in subtle ways, changing seemingly minor details about what we can or cannot do. A mundane example will help us visualize: the shopping list.

You write your shopping list on paper: apples, oranges, milk, cheese, shampoo. Later on, you realize you also need butter. What do you do, just append it to the end? No, because you do not want to go in the dairy products aisle to pick up milk and cheese, then to the hygiene aisle, and then back for butter. The order matters in multiple and subtle ways. Firstly, there are multiple reasons why you grouped the items. It is more convenient once in the shop. But writing is also thinking: adding an item makes you think of other items to add, and this groups them. Secondly, there are multiple reasons why this order matters. It is a convenience, in the sense that grouping items helps you pick multiple things at once. But it is more: a program. Because you trust the list to help you. You expect items to be grouped, so if you break the order by putting the butter at the end of the list, you will surely get the unnecessary trip. So you plan ahead. You know the list is a program, and you write it accordingly. You insert the butter between the milk and the cheese. That way, even if you are tired when you shop, you will not waste your time traveling between yogurts and shower gels.

The insertion of an item between two others is a graphical operation made possible by the properties of the paper sheet – or the clay tablet. The sheet has its own planar space, distinct from the intrinsic space of the list. You can find room in between precisely because the sheet is continuous. And even if there is no room in between, your options are as endless as doodling can be. These spatial relations are natural to our cognition, and we naturally excel at graphic manipulation.

The sheet also renders ordering possible, in the sense that we can for instance decide to read the inscriptions top-down. It also allows groupings by proximity. Different supports have different properties.

The list of Google results is also an inscription on a support. The support is a memory structure made of 0/1 bits, somewhere on the digital infrastructure. It is actually replicated in multiple places and under multiple forms, and many sophisticated layers are involved. It is nevertheless an inscription on a material substrate. And that support has properties, the most important of which is discreteness.

Wait, did I just prove the radical difference of the digital? Inscriptions are determined by the substrate, and the digital has a radically different substrate, QED. Oops. The argument is superficial but reasonable. Unfortunately, this is not where the journey ends. This is just a checkpoint. Still, I want to acknowledge here that discreteness as a radical difference of the digital is a sound hypothesis. I am arguing against it for other reasons: I contest the implications it has in practice. A discrete substrate does not prevent all forms of continuity.

Computers deal with graphic space two ways: pixels and vectors. The Gimp versus Inkscape. Photoshop versus Illustrator. With pixels, discreteness seems quite irreducible. With vectors, not so much.

In the world of pixels, space is quite directly memory bits. You cannot zoom infinitely. At some point, you see the pixels, and they are the limit. In that use of the computer, it seems that indeed the discreteness of the substrate translates to graphic discreteness. In part to overcome this limit, we also have the alternative of vector graphics.

In the world of vector graphics, we have no pixels but the coordinates of points and curves to create objects. This produces a strong impression of continuity. The limit is not the size of the space, but the number of objects you can represent. The potential continuity of the graphic space is, in practice, well respected. The discreteness of the digital substrate translates here to the discreteness of represented objects. It is a very different trade-off. For all practical purpose, it allows the same kind of graphic manipulations as the paper sheet.

Vector graphics show how the flexibility of computers can be leveraged to subvert the limitations of discreteness. There are many ways to build our digital infrastructure, and not all of them are stupidly stuck in discreteness. For similar reasons, the list of Google results did not have to be a simple list. In fact, other designs have existed, such as the “visual” search engine Kartoo.

Kartoo results were displayed as a map. Like with vector graphics, they were discrete elements in a continuous space. The continuity of the underlying space suggested that some results might occupy intermediate positions, while in Google the space between elements does not mean anything. Kartoo did not feel as discrete as the Google list.

So at the end of the day, why are Google results a list? By design.

The answer is unspectacular, because we overlook its dissonance with the popular beliefs on computers. Indeed, there is a much stronger narrative out there: that Google is a big database of some sort, and that a list is what you get when you query that. In other words, because of how computers work. This narrative is uncritical and short-sighted. The Google search engine is on par with a weather forecast system in terms of scale and sophistication. It runs on a huge distributed infrastructure. It draws information from other sophisticated systems such as the Google Knowledge Graph or Google Maps. It does not work like a regular database. It does not have to look like one. Results are a list because it works better for Google that way, whatever that means.

We tend to think of computers with the pixel mindset, where continuous things are implemented by a straightforward strategy. Discreteness kicks in at a certain granularity level. In some ways this thought is comforting because the limitation is visible if you look close enough. The digital is a fake that we can unmask with a simple act: zooming in. Our everyday computer confirms this belief. But at the other end of the spectrum, we forget invisible systems that we can only conceive with the vector mindset. Strategies that implement continuous things in oblique ways, blackboxing discreteness in contingent constraints. In such systems, agency is near impossible to retrace, and we never find the corner from where to grasp and pull away the veil of the Matrix. For good reason: these systems are efficient at producing continuity in practice.

We are right to assume that there must be a validity domain outside of which the illusion of continuity breaks. The discreteness of computers requires it. But we are wrong to believe that we will necessarily face the walls of that domain. The vast capabilities of industrial devices allow them blending into the background of our lives. They work well enough that we have stopped paying attention. Not every digital system can be thought of on the basis of our everyday computer.

What does it change that Google makes lists? I have a few ideas, but a more important question first: can we know the difference it makes? These systems have already blended into the background, and I would not presume of our ability to discern them. I do not trust my own powers of observation. But the few I can see is all the more important, because I am probably missing more. It is not so impressive, though: the discrete and continuous lists institute different regimes of absence. They produce distinct kinds of holes, and thus distinct kinds of wholes. Absence and exhaustiveness are instituted differently.

In a nutshell, there are no holes in the Google list, it always seems exhaustive. You never have a clue that something is missing. Absence is invisible, which progressively leads you to believe that Google is exhaustive. Whereas in the world of planar representation, absence leaves a blank, a negative space, a hole with visible borders. Absence is visible, which alerts you. It jumps at your face. This argument will be more understandable with a few graphics.

A visual journey through regimes of absence

In images, absence is generally a question of background and contours. Let us start with the notion of negative space. To put it simply, it is a space occupied by the background, an empty space, a hole. But because it is surrounded by nonempty space, because as a hole it has borders, you can see it. You can, but because it is background you also overlook it. The typical example is the FedEx logo.

You probably know it contains an arrow. If not, enjoy a little epiphany. Its designer, Lindon Leader, set up that aha moment intentionally. The arrow is hidden in plain sight: some people see it right away while others never realize. Our visual system works in such way that the background color of the arrow prevents it from being processed as an object, as a shape – until we get aware of it. The negative space is on the edge between visible and invisible absence.

Removing something often leaves a hole – but not always, as we will see. The marks it leaves can be used to communicate absence. I recently stumbled upon a nice example in the famous French newspaper Le Canard Enchaîné, published on 2019-06-19. In a context where the New York Times decided to stop publishing cartoons, the Canard published an empty image as a way to raise awareness on the issue. The negative space conveys the message of absence, because the hole is visible. It leaves a scar in the page, as you can see below.

Conversely, if you were owning a newspaper and wanted to censor an article, you would not leave a blank as a piece of evidence. You would cover your tracks by putting something else instead. Or by having the embarrassing article not written in the first place. Censorship has a long story of avoiding traces of absence, predating the digital era. Soviet Union did not wait for Photoshop to paint backgrounds over embarrassing people.

In the example above, most of the work has been done by reframing the original picture, another efficient way to make absence invisible. Still, you can note how some shadows had to be repainted to avoid suspect negative spaces. This technique has become so commoditized that we now erase things for fun.

Keep in mind that despite what the result looks like, photoshopping out stuff is not erasing but painting. Extending the background over the shape you want to remove, but also fixing surrounding shadows, reflections, etc. Doing it well requires an effort, and digital artists frequently make mistakes, like forgetting to remove a hand on a shoulder. Spotting these “Photoshop fails” in magazines has become an equally fun practice.

In photographs and video, manipulation is so accessible that we cannot reliably track absence except with specialized tools. We cannot really tell if something is missing with the naked eye.

Visual media allow to make absence visible or invisible depending on how we use it. Not everyone wants to hide absence. In the sciences and humanities, highlighting it is key to assessing authenticity. In archaeology, missing pieces are usually blanked in an explicit way, leaving stunning negative spaces that add to the story of the object.

On the question of absence, writing is better understood as a visual medium. Handwritten manuscripts display how important is the practice of editing texts by crossing out words, writing between lines and in the margins, and using arrows and lines to reorder paragraphs. On a medium where you cannot erase, editing leaves marks. This has been essential to the study of literature, and to teaching: remember how your teachers provided feedback by correcting your works, and the humble but powerful technology supporting it. Visible absence is not a problem, it is a feature.

Erasure technologies show how vast the spectrum from visible absence to invisible absence is. Crossing out a word keeps it readable. Blacking it out makes it unreadable, but its absence is still visible – redacted text has become iconic of censorship. Whiting it out with correction fluid leaves a mark of absence, but if you write over it, there is no hole: absence becomes less visible. Erasing pencil, if well done, can leave almost no trace of editing – invisible absence. This spectrum also tells us that technologies allowing hiding absence often allow showing it as well. There is a choice; then let us pay attention to who makes that choice.

Important remark on censorship in popular culture. Representing censorship requires to make absence visible. For example the blacked-out text above is the sign of censorship, the black blocks embody censorship. Only a visible mark of absence can convey the idea of censorship, the idea that something was there and has been removed. Which leads us to misconceive it. Indeed the most dangerous form of censorship is where absence is invisible. By definition, invisible absence cannot represent censorship – it cannot represent anything because it is invisible. As a consequence, the popular conception of censorship is dangerously naive; it assumes censorship always leaves marks, which underestimates and misdirects the efforts necessary to oppose it.

Digital inscriptions can be rewritten – except in a few edge cases. So digital media can invisibilize absence, which does not mean they always do. Visible absence is so useful to human practices that many devices reproduce the behavior of writing on paper. For instance you can strikethrough text, a feature that only makes sense on a computer as an extension of analog practices. More generally, the revision system in text editors like Open Office Writer, MS Word, and Google Docs, have multiple ways to display absence to optimize the user’s reading and writing experience. Edit marks are useful to track revisions, but they get in the way of normal reading. So you can hide edits if you want. Once aware of the edits, hiding them is desirable and useful – because you can make them reappear. But of course, invisible absence becomes problematic if you have no clue that something is missing.

The discrete list’s regime of absence

You query Google for “Tiananmen square protest”, in China, in 2010. You get nice tourism pictures (see below on the left). The event itself is missing (see the same query from the UK, on the right). You can certainly tell, but not everyone in the world can. That is why it matters that in this list, the missing knowledge leaves no hole, no mark, no scar. Results look whole, complete.

I must now be fair with Google, which is neither the only search engine nor the first. Others (Bing, Ask, Duck Duck Go…) work the same. Since the beginning of this text I referred to Google because everyone knows how it works, not because Google is the worst. And to be fair, in this case Google mentioned that some results were missing. But precisely because the missing results left no marks, they had to make their absence visible another way. Here, a textual mention that some results were removed. Not as efficient, but better than nothing. Note that Google stopped displaying this mention in China in 2012, this time invisibilizing absence completely – actual censorship takes care to cover its tracks. On this topic Wikipedia is a good start, and there is a rich discussion on the web and in the academia.

You might expect ethical search engines like Duck Duck Go to oppose censorship (this one is of course blocked in China). But in fact, we cannot tell by just looking at the results, because absence is still invisible by design. We have to rely on trust, which is fine, but they might still have their own unintentional biases, and those are invisible. Duck Duck Go sets up no different regime of absence.

Years ago I had the occasion to discuss with some of the architects of a search engine on how its design was decided. They had different metrics related to user satisfaction (query speed etc.) and conducted their own market research. They had for instance observed that for many simple queries (eg. “gravity”), most users expected a definition or general knowledge on that topic. Wikipedia provides that kind of answer, and people like it. So Wikipedia was often artificially pushed to the top, which measurably improved user satisfaction. But exhaustiveness was not part of their goals and surprisingly, quite the contrary. I asked explicitly. If one result makes many users happy (like Wikipedia) then similar results will not make the same users more happy. Being exhaustive is not worth it. It is more profitable to satisfy other users, for instance those who had the movie “Gravity” in mind. The top of the list is a very limited space and there, search engines favor variety over exhaustiveness, since it satisfies more users – make your own mind! This strategy requires to remove some results to make room for others. It actively curates the results order – this is no surprise. But it also actively blackboxes those manipulations as a design choice. Discreteness and the regime of absence it enacts make this design choice possible.

I am not saying here that Google and other search engines use the discrete list and its regime of absence in order to hide censorship. Just that it performs something invisible, which is both dangerous and efficient. It works like a charm because it performs something invisible. Indeed the complicated functioning of a search engine is fruitfully blackboxed into something reminiscent of the mundane shopping list – a browsing list. The trick has proven its relevance. This design is accessible, simple and clear. It has these qualities because it is blackboxed. Those are excellent reasons to use the discrete list – as I argued in another post, blackboxing is a resource for design. So good reasons exist, rationales supporting a public good agenda (eg. access to knowledge). But we cannot deny that Google et al. have their own twists on the notion of public good. Hiding the absence of removed results might not primarily a support to censorship, its benefits meet various nuances of morality. Being invisibly absent from search engines can deny a fragment of knowledge its very existence.

At this point you might ask: “but absent with respect to what?” Indeed, absence is relative. It implies comparing two states, a reference state where something could potentially be present, and a state where it is absent. In the Tiananmen example we compared two versions of Google, which conveniently provides a reference state – we implicitly assumed that Google should work in China like in the UK. But most of the time, there is no reference state. When Google works like it is supposed to, what is the comparison point? Which reference could allow us to assess what is missing? This is connected to the question of algorithm bias: biased compared to what? Even implemented straight from the paper, algorithms can be subject to bias because they reproduce undesirable discriminations present in our society, in our data. Bias has to be addressed from outside. The reference point cannot be produced by the algorithm and must be curated externally. What could it be for search engines? On Tiananmen, we can think of the historical consensus about the massacre, for instance. It seems more grounded than how Google works in the UK. But even that notion is situated, hard to implement, and it covers only a tiny bit of knowledge. This thought exercise gives us a glimpse of the difficulty: mankind has no reference point for something as vast as the web, as generic. We have encyclopedias, Wikipedia, but those are not a valid reference points for the web because they contain a different kind of information, a different distribution of knowledge. We have no good reference to assess how a search engine should work, which makes it difficult to deal with the question bias and censorship in practice.

As a society, our incapacity to evaluate a search engine, to make it accountable for censorship or disinformation, shows how much power we have delegated to the digital infrastructure. It shows how far we are from framing the political dimension of computational systems in a democratic setting. The question of absence is the same as the question of accountability. Without a background against which assessing absence, we cannot enforce accountability. Without a reference opposable to algorithms, we cannot evaluate them. As long as our strategy to unveil disinformation is to ask Mark Zuckerberg if Facebook knowingly implemented evil algorithms, we implicitly admit powerlessness. This strategy boils down to asking architects of our digital environment to be nice with us please. Engaging with this question makes us realize that in this game, we have almost no cards in hand. All the cards are in the opponent’s hand. It feels like we arrived late in a game against an opponent who did not wait for us and started playing without resistance a few turns ago. Why did we miss the beginning of the game? Because we believed digital stuff was not political, at least not as much, not like that. We did not see from where would the trouble come. We were blind. And here I would refrain from paranoia and complotism: I am inclined to believe that no one knew, that actors in position to design our digital environment – architects – had no sinister agenda but the usual greed-related goals (user satisfaction, profitability…). I would not presume that architects of the digital knowingly trapped us, the public, in this powerlessness. No one knew we were playing that game, and these actors pushed their agenda the way everyone does. I can believe that our blindness was shared. We all realized quite late the strange places where power had accumulated. Delegating to the digital infrastructure seemed innocuous in the absence of any particular danger. This is where I think regimes of absence, for instance, matter. Invisible absence is precisely the kind of pipe for power we tend to overlook. Once again, I do not trust my own powers of observation: I am not sure of how important it is, or which other sociotechnical assemblages allowed power to circulate and reconfigure. My goal here is to help making these reconfigurations visible by contextualizing overlooked details of the digital.

I am now reaching the limit of my perspicacity. There is certainly more to search engine design, but I find myself without much landmarks to go further – and I am poorly educated in this academic literature. But we traveled far enough for me to draft an argument, to show a direction. I want to wrap up this journey and articulate more tightly my two points: (1) computers are not bound to discreteness in practice and (2) different supports establish different regimes of absence. In this wider story, the list of search engines is just an example among others.

The register effect

To situate discreteness I will draw from an analogy with chemistry. Diamond is much more similar to quartz than to coal, even though its constitutive element is, like coal, carbon. It turns out that constitutive elements matter less than how they are arranged. Diamond and quartz are alike because they are both crystals. Seemingly essential properties like hardness or color are not determined by the elements but by their arrangement. Similarly, even though the digital is discrete by nature, depending on how the code is arranged, the resulting system is rendered discrete or continuous to the human experience. I evoked a few examples but there is more to it. Think for instance of how much infrastructure we dedicate to dealing with technical failures: those are occurrences of the analog bleeding on the digital. Think of webcams, sensors, etc. Computers are not stuck in a bubble of discreteness, they constantly deal with the continuous. We cannot just assume that “digital is discreteness”. We must inquire why and how discreteness matters to the digital. Comparing discrete-looking and continuous-looking systems can help us clarify the situation.

Google Maps makes a strong impression of continuity, while the Google search engine looks discrete. Both enact different regimes of absence. Removing a location in Google Maps leaves marks due to the nature of Earth’s surface (see above), and hiding or minimizing these marks requires an effort. Conversely, in a search engine, removing results is invisible. Without an underlying space like Earth’s surface, there is no background against which checking what is missing. In a discrete list, absence is hidden by default. This has consequences for individual users, critical thinking of our digital environment, and ultimately how such devices are framed in a democratic setting.

Search engines work like registers of the web. They seem to list its entire content. Of course they do not exactly, first of all because they did not crawl everything, but also because they have various reasons to promote or demote certain resources. In this they fulfill their function of offering an entry point to the vast space of the web. This role comes with responsibilities and curating results is, to a certain extent, necessary to face them. Search engines have to distort and reduce web information. But this distortion and reduction is not visible. Unlike looking at the world through a piece of glass, nothing seems distorted, nothing seems missing. The list of results always looks complete. Because search engines are the principal entry point to the web, what they omit is rendered invisible. Users have no clues they might be missing something. Hence my choice of the word register. Search engines produce the impression that whatever is not in the list is simply not there. Like metaphorical screens on the surface of which the web is projected, they hide it by showing a reduced version, and the audience ends up taking the projection for what is projected, conflating the search engine and the web. This is what I call the register effect: the ability of certain lists to produce the impression they register all there is.

Many actors have actively tried to become such screen, and their main asset has always been technology – and more precisely, its design. Microsoft tried to become the entry point to the web by conflating the browser and the file system explorer. AOL tried to become an OS inside the OS and conflate its services with the rest of the web. Facebook tries to conflate its own space with web browsing in order to keep you inside its walls. On that topic, design choices, including a myriad of seemingly minor details, are much more revealing than “algorithms”.

This is where all the pieces come together. The discreteness of search engine results is not a question of technological essence, but of design. Being the entry point to the web is a critical issue. To that end, the register effect is a major asset. And its active principle is the regime of absence enacted by discreteness. Discreteness allows to cover the tracks of manipulation (from legit tuning to upfront censorship), which in turn preserves the impression that the search engine is a neutral interface. A technical commodity. A simple doorway to the web. As it blends into the background of our digital experience, we forget that this “simple doorway” has the potential to hide the existence of entire areas of knowledge. That is how we lose the ability to reflect on a piece of technology, when it fades into the very fabric of our experience, as a mediation.

---

---