The problem with network maps

6 minutes read

The main problem, with network maps, is to figure out what we can trust or not. Not everyone is even convinced that there is something we can trust, and that is also part of the problem.

This is a network map. It comes from Divided They Blog, a paper by Lada Adamic and Nathalie Glance. Check this previous post for context.

After I engaged with network maps intensively and for a long time, I became convinced that there is a reproducible way to interpret them. But like most expertise skills, it is embodied in you; it is reliable despite being subjective; possibly because it is subjective. Yet I believe that I can translate my intuitions into objective reading principles. That has been part of what we call Visual Network Analysis (VNA). Note: I do not mean that we are the first to use the term; we are not.

Beyond spelling out an interpretation method, VNA is about justifying it. It is one thing to observe and document that it works, but quite another to be able to explain why it works, and yet another one to actually prove that it works. These three things are what I am trying to do at the moment.

The current state of interpreting network maps is basically an accepted but criticized practice. Some do it well, some do it wrong, but no one can actually provide a precise reason why it works (or seems to). This kind of situation is quite common, in fact; many scientific methods get normalized because they are useful in practice, even though we do not know exactly why. For another example, p-values have been heavily criticized for a long time and are known to be widely misused, yet they remain popular.

And to be fair, it is not the process of interpretation that we misunderstand in the first place. After all, the field of information design has been extensively theorizing how we interpret data visualizations. No, what we mostly fail to understand is graph embeddings aka force-driven layouts (there is a nuance but I overlook it here), the algorithmic process of placing nodes in a Euclidean space according to their topology.

One may find it surprising that scientists widely use a technique that no one really understands (myself included). There is a specific reason to that: network visualization is not used to produce evidence, but hypotheses. The point has been voiced by Tukey, as he theorized Exploratory Data Analysis. It is not a problem to science that interpreting networks visually is uncertain, because certainty is provided by other means, e.g. confirmatory statistical analysis. For this analysis to be performed, one needs hypotheses, and that is what network maps provide.

An uncertain reading is a false problem, but there are other issues:

  • The lack of reproducibility impairs the legitimacy of VNA.
  • The knowledge being implicit makes it more difficult to teach and disseminate.
  • People might misconstrue the apparent lack of a methodological ground as a form of self-evidence, conflating the data and their representation.
  • More importantly, it makes it difficult to discuss and criticize the process of visualizing networks, and to improve it.

Indeed, even though VNA is not used to provide evidence, it can still shape the scientific outcome. By influencing the hypotheses considered, it impacts the findings. Something might be missed. Any visualization comes with biases that one must be aware of.

I offer here the bias that I find the most dangerous. It is easy to understand, but also to overlook. In a network map, your reading is mostly concerned with the distance between the nodes. That is how one can figure out clusters, structural holes, etc. And of course, distance is mutual, symmetrical. If A is close to B, B is close to A. Unfortunately, in most directed network, the all-important hierarchical effects come from the asymmetry of edges. You retweeting Donald Trump is not Donald Trump retweeting you. I know, you would not do that; yet it has more chances to happen than the opposite. When you look at a network map, by nature of the inscription support, you miss the asymmetries of the links. Of course, one can get some indirect clues: centrality, mapping indegree to node size… It helps, but this is just mitigation. The real protection against that bias is to acknowledge it, and to represent asymmetries with another visualization. For instance a matrix.

A word on matrices versus node-link diagrams. On the surface, these two strategies are complementary. Both have pros and cons, and there is a debate on which is the best. Yet at the core they suffer from the exact same problem: they depend on a node placement algorithm. It might not be obvious for matrices, so let me explain. The patterns you see in a matrix strongly depend on the order of rows and columns, and those represent the nodes. The clusters or blocks appear in the matrix if similar nodes are next to each other, so it depends on an algorithm sorting the nodes. It is a one-dimensional (discrete) placement instead of a two-dimensional one, but the algorithmic strategy is exactly the same.

This algorithmic strategy has one notable feature: it is non-deterministic. If you run the algorithm twice, you get two different results. The fact that force-driven node placement algorithms are non-deterministic leads some critiques to conclude that the reading is not reproducible. The argument sounds reasonable, but the situation is slightly more complicated.

Indeed, it is possible to “seed” randomness so that it becomes deterministic, and it is possible to randomize a procedure to make it non-deterministic. One can easily add or remove determinism, but the operation is only superficial; it does not change the deeper nature of an algorithm. Determinism, here, indicates something else.

The node placement algorithm is asked to place the nodes in a space where their distances have a meaning, but not the coordinates. Of course, the distances, in this Euclidean projection, derive from the coordinates. Let’s say that node positions have a meaning insofar as they provide distances. That is a difference with other visualizations such as scatter plots (more on that in this post). The important here is that the algorithm is forced to take a number of arbitrary decisions. Sometimes, an action requires a choice that bears no signification, but must still be done. This arbitrary decision being taken at random makes the algorithm non-deterministic. In some way, it is more honest to do so. It does not conceal the arbitrariness of the procedure.

The problem with network maps (and matrices) is that their node placement procedures involve arbitrary decisions, that introduce arbitrariness in the visualization. From there, interpreting the image requires distinguishing what is arbitrary (i.e. node positions) from what is not (i.e. node distances), even though those are most often two sides of a same coin.

Hence VNA’s main takeaway: interpret the features that one can describe in terms of distances regardless of the position (clusters, structural holes, visual density, centrality to a cluster, bridges) and disregard the features that also depend on the position (what is on top/bottom/left/right, rotation, mirror inversion). This is also valid at a local level, i.e. within a given cluster, which is less intuitive.

A number of questions that arise from there:

  • Can we translate visual features into statistical metrics and vice-versa? This would solidify the division of tasks where network maps generate hypotheses and metrics evaluate them.
  • Can we remove the arbitrariness of the algorithmic strategy? If not, why is that so? Maybe the strategy is bad in the first place.
  • Can we prove, or at least measure empirically, that distances are trustworthy even though positions are not? Indeed it does not seem that distances are that reproducible.
  • Couldn’t we frame arbitrariness as a form of imprecision brought by numeric calculations? If we had the equivalent of a margin of uncertainty, we could at least quantify the trust, the quality that the interpretation can afford.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.