Digital resources in the Social Sciences and Humanities OpenEdition Our platforms OpenEdition Books OpenEdition Journals Hypotheses Calenda Libraries OpenEdition Freemium Follow us

Close reading Wikipedia from Pareto to Network Science, part 3

This is part 3: Statistical law or distribution?

In this part we focus on a nice issue. As we have seen previously, there is a Wikipedia article for the Pareto distribution. There is also a page for Pareto’s law, which tells you that it refers either to the Pareto distribution or to the Pareto principle. In this case like many others, a distribution is also called a law, but both concepts are not exactly the same. Can we establish the nuance between what is called a distribution and what is called a law? Which leads to two sub-questions: What does Wikipedia state on that matter? And what is the Wikipedia practice when it comes to name a thing law or distribution? Note: here we only study that issue where it relates to network science and the power law, since the corpus of analyzed articles is defined this way (see part 1 for the method).

As a post on my research blog, it exposes more than a typical paper would: honest on what is actually done, at the cost of not-so-relevant material. I also follow the open source software mantra “release early, release often”: style might be raw.

Findings

There is a confusion about law and distribution, and it is entirely the fault of the concept of statistical law. A distribution is just an equation. But law can refer to two clearly different things:

  • The statement that given rule or distribution is empirically pervasive (with the power to describe) and/or mathematically proven (with the power to explain). Wikipedia explicitly states such definition.
  • A given statistical distribution. This usage of law as a synonym for distribution is implicit but widely used in Wikipedia.

These two usages coexist in many articles, and we also find multiple ways the two concepts of law and distribution are presented as equivalent. The situation results in a confusion on the empirical grounding of statistical laws. The confusion creates a fallacy where mathematical validity may seem to cause and justify empirical validity. The claims to pervasiveness in particular deserve some attention. For most laws, the pervasiveness of observations is a defining characteristic without which they would not be called law in the first place. But pervasiveness is not established by the sole virtue of something being called “law”. We will investigate these questions in an upcoming part.

The overlapping definitions of distribution and law

Distribution has a precise definition, stated in its dedicated article. It is clear and operational, and is not challenged in any article of our corpus.

“In probability theory and statistics, a probability distribution is a mathematical function that provides the probabilities of occurrence of different possible outcomes in an experiment.”

Probability distribution

There are two relevant articles about laws, Empirical statistical law and Scientific law. Without any surprise, statistical laws are presented as a special case of scientific laws in the field of statistics. Remarkably, both articles acknowledge multiple meanings for the concept.

“The term law has diverse usage in many cases.”

Scientific law

Despite a reference definition that seems to frame the concept of law as a kind of empirical model (see citation below), the usage does not follow that definition because as we will see, law can also mean distribution.

“The laws of science, also called scientific laws or scientific principles, are statements that describe or predict a range of natural phenomena. Each scientific law is a statement based on repeated experimental observations that describes some aspect of the Universe.”

Scientific law

If you asked a metaphorical Wikipedia whether a statistical law and a distribution are the same thing, you would not get a clear-cut answer. Not only is Wikipedia ambivalent, sometimes pretending they are the same thing, sometimes not, but the confusion is even official. Indeed Wikipedia acknowledges that in the academic literature, different definitions of statistical law coexist, some of which just mean distribution, some of which do not.

The article on empirical statistical laws provides explanations on those multiple layers of meaning.

“An empirical statistical law or (in popular terminology) a law of statistics represents a type of behaviour that has been found across a number of datasets and, indeed, across a range of types of data sets.”

Empirical statistical laws

In other terms, a law can be
(1) a pervasive behavior
…except that:

“Many of these observances have been formulated and proved as statistical or probabilistic theorems and the term “law” has been carried over to these theorems.”

Empirical statistical laws

So a law can also be
(2) a pervasive behavior explained by a mathematical theorem
…but wait, there is more:

“There are other statistical and probabilistic theorems that also have “law” as a part of their names that have not obviously derived from empirical observations. However, both types of “law” may be considered instances of a scientific law in the field of statistics.”

Empirical statistical laws

At last, a law can also be
(3) a mathematical theorem

In a nutshell, it seems that there are two different things than can give birth to a law: empirical observation, and/or math theory. Each is sufficient to justify the name, none is necessary. This is according to what Wikipedia explicitly states on that matter.

Wikipedia states that a statistical law can be three things

As we will see the situation is a bit more complicated, but as a starting point we can already clarify that:

  1. Multiple meanings of law coexist
  2. When a law is mentioned, we should elucidate whether it is for empirical reasons, for theoretical reasons, or for both.

With this picture in mind, aware of the lack of a clear divide between law and distribution, we can focus on the usage and justification of these concepts.

Statements articulating law and distribution

Now that we have an idea of how Wikipedia frames law and distribution, we look into the usage of those term to see when they are differentiated or, on the contrary, presented as equivalent. In addition, we account for a specific kind of claim where law is framed as empirically grounded.

Demarcation

In our Wikipedia corpus we found no statements about a law|distribution divide, but we find demarcations between law and theory. As we have seen a distribution is a mathematical entity, but it is a function and not a theorem. As it turns out, the only demarcations we find are internal to the notion of law, and leave the concept distribution unaffected. For instance from the article about scientific law:

“Laws differ from scientific theories in that they do not posit a mechanism or explanation of phenomena: they are merely distillations of the results of repeated observation. As such, a law is limited in applicability to circumstances resembling those already observed, and may be found false when extrapolated.”

Scientific law
A first type of demarcation is about the power to explain, and is more exclusive

The following picture, from the same article, is supposed to illustrate the point, though it makes an important difference:

Illustration from the article Scientific law

The picture shows the three cases that can be called law, but once again demarcates between on the one side meaning (3) (on the left) and on the other side the meanings (2) and (1) (center and right respectively).

The picture makes an important difference however, insofar as it states that repeated successful predictions both describe and explain phenomena, while the text demarcates the law as unable to explain. If we reorder the picture in the same order as our precedent illustrations, we obtain the following:

A second type of demarcation is about the power to describe, and is more inclusive

According to both demarcations, a mathematical theorem that is not based on empirical observation should not be called a law. This perspective implies that despite anomalies existing for historical reasons, those are framed as the exceptions that prove the rule.

“What distinguishes an empirical statistical law from a formal statistical theorem is the way these patterns simply appear in natural distributions, without a prior theoretical reasoning about the data.”

Empirical statistical laws

This statement makes the exact same inclusive demarcation, but is interesting for another reason. The use of the concept of “natural distribution”, which here refers to empirical observations. As tempting as it may be, we cannot approach the concept of distribution as the mathematical side of the notion of law. Not only can laws be mathematical, but distributions are sometimes located on the empirical side of the observation/theory divide.

Equivalence

Equivalence has multiple forms, explicit or implicit. We expose two of them:

  1. Indifferent naming of the same thing as law or probability
  2. Statements of multiple names

Indifferent naming. The article on normal distribution provides a good illustration, we will focus on it for a few paragraphs. In the first third of the article, the normal distribution is systematically called a “distribution”. But then a few passages mention the “Gaussian law” and progressively, we see mentions of “normal law”.

The simplest form of indifferent naming is the usage of both “normal law” and “normal distribution” in the same context. We consider it an implicit form of equivalence, provided that both names refer to the same thing. In our example, it is important to remark the absence of nuances between the two terms. For instance, following observed demarcations, we could expect the “law” term to be used to emphasize the empirical dimension, but it is not the case. See how the first occurrence of “normal law” in the article clearly refers to the mathematical function:

“Pearson distribution — a four-parameter family of probability distributions that extend the normal law to include different skewness and kurtosis values.”

Normal distribution

As an additional clue of this equivalence, the article on normal law exists and proposes a redirection to normal distribution (along with two other unrelated meanings of normal law, in the field or aviation and in the field of justice).

A slightly different form of indifferent naming is when a distribution is also called a law but with a different name, as in distribution X and law Y. For instance, the “normal distribution” is also called “Gaussian law”. Once again, we verify that this term is referring to the exact same thing as the distribution, as in this passage:

“One of the main practical uses of the Gaussian law is to model the empirical distributions of many different random variables encountered in practice.”

Normal distribution

Incidentally note that while the “law” term refers to the mathematical function, here “distribution” refers to the empirical, in a complete reversal of the expected demarcation.

Statements of multiple names. Sometimes we find statements on the multiple names, which is the explicit version of indifferent naming. For instance the article on the Poisson distribution has a subsection about the law of rare events which explicitly states synonymy (but only “sometimes”).

“The word law is sometimes used as a synonym of probability distribution, and convergence in law means convergence in distribution. Accordingly, the Poisson distribution is sometimes called the law of small numbers because it is the probability distribution of the number of occurrences of an event that happens rarely but has very many opportunities to happen. The Law of Small Numbers is a book by Ladislaus Bortkiewicz (Bortkevitch) about the Poisson distribution, published in 1898.”

Poisson distribution

The source of these multiple names is often implied or stated as historical, as in the article about the normal distribution where there is a History section with a “naming” subsection where it appears that the distribution was first named as a law:

“Since its introduction, the normal distribution has been known by many different names: the law of error, the law of facility of errors, Laplace’s second law, Gaussian law, etc. Gauss himself apparently coined the term with reference to the “normal equations” involved in its applications, with normal having its technical meaning of orthogonal rather than “usual”. However, by the end of the 19th century some authors had started using the name normal distribution”

Normal distribution

Other articulations

Like the Deleuzian fold which separates and joins at the same time, some statements imply demarcation and equivalence at the same time. Take for instance this comment following a list of mathematical extensions to the normal distribution:

“All these extensions are also called normal or Gaussian laws, so a certain ambiguity in names exists.”

Normal distribution

Acknowledging ambiguity is a passive-agressive way to argue for demarcation, pretending that a difference exists while not contributing to situating it. What the sentence actually does is stating that the multiple variations of the distribution share a common name, which is a statement of equivalence, although quite weak.

The law framed as pervasiveness. As we have seen, the reference definition of scientific law refers to its pervasiveness. Similarly the article on Zipf’s law argues that its status of “empirical law” refers to the pervasiveness of the Zipfian distribution. Note the “many” in the following sentence:

“[Zipf’s law] refers to the fact that many types of data studied in the physical and social sciences can be approximated with a Zipfian distribution, one of a family of related discrete power law probability distributions.”

Zipf’s law

In this perspective the law is not exactly the distribution, it is the fact that we find the distribution in multiple empirical situations. This is a possible key to understand the usage of our two terms. Not only is it aligned with the demarcation (the law carrying the empirical dimension of the distribution) but it also suggests an interpretation of the usage of law as a metonymy: “law X” would stand for “law of the pervasiveness of distribution X”. However this interpretation only stands for certain cases where we can observe such nuance, since we have seen with indifferent naming that in some situations there is a strict synonymy between law and distribution. Anyhow, the question of pervasiveness is central to our research question and we will dedicate some specific attention to it.

Claims to empiricism

As we have seen, when the term law is not used as a synonym of distribution, it has an empirical dimension as hinted by the title of the corresponding article, “Empirical statistical laws”. But we will first return to the reference definition of scientific law, once again quoted below. It states the key elements of the empirical dimension of a law, which we will discuss.

“The laws of science, also called scientific laws or scientific principles, are statements that describe or predict a range of natural phenomena. Each scientific law is a statement based on repeated experimental observations that describes some aspect of the Universe.”

Scientific law

Firstly, a law can describe or predict. It states two distinct agencies, that can both be qualified of empirical. The power to describe is the weakest, it only means that the law is a valid reduction of a phenomenon. The power to predict is stronger, since it allows applications. Both are empirical because they related to the observation of natural phenomena, but in different ways.

Secondly, a law is pervasive (“repeated observations”). Implicitly, laws can have validity conditions that prevent them from describing certain phenomena and/or predict them – laws are not realistically expected to apply strictly everywhere all the time. But a law is not something that happens once or twice. It earns its status of law by applying to enough observations, even though no precise threshold is specified.

The same article also states a third kind of agency, the power to explain. It seems to derive from the combination of the other agencies and pervasiveness. The power to describe (or “summarize”) gives the power to potentially explain, while the power to predict and the pervasiveness validate the explanation.

“Scientific laws summarize and explain a large collection of facts determined by experiment, and are tested based on their ability to predict the results of future experiments.”

Scientific law

We also find an alternative way to empirically ground the concept of law, purely based on pervasiveness, and sometimes on a more explicitly structuralist alternative, independence to details. This argument is called “universality” and is stated in a dedicated article.

“In network dynamics, universality refers to the fact that despite the diversity of nonlinear dynamic models, which differ in many details, the observed behavior of many different systems adheres to a set of universal laws. These laws are independent of the specific details of each system.”

Universality (dynamical systems)

We see here how universality refers to the fact itself that there are empirical laws. Note that independence to details is more than observed pervasiveness insofar as it adds an explanation to the phenomenon. In that sense, universality implies the power to explain, while the statements from the article on scientific law describe where the power to explain comes from. Universality only assumes the power to explain.

The article on the log-normal distribution makes a precise universalist point which embodies multiple aspects of the Wikipedia discourse on law and distribution.

“The log-normal distribution is important in the description of natural phenomena. This follows, because many natural growth processes are driven by the accumulation of many small percentage changes. These become additive on a log scale. If the effect of any one change is negligible, the central limit theorem says that the distribution of their sum is more nearly normal than that of the summands. When back-transformed onto the original scale, it makes the distribution of sizes approximately log-normal (though if the standard deviation is sufficiently small, the normal distribution can be an adequate approximation). This multiplicative version of the central limit theorem is also known as Gibrat’s law, after Robert Gibrat (1904–1980) who formulated it for companies. If the rate of accumulation of these small changes does not vary over time, growth becomes independent of size. Even if that’s not true, the size distributions at any age of things that grow over time tends to be log-normal.”

Log-normal distribution

We remark that:

  • It starts as a claim to empiricism (“important in the description of natural phenomena”)
  • It grounds the claim on pervasiveness first (“because many natural […] processes”…)
  • The term law (“Gibrat’s law”) does not refer to the distribution itself (log-normal) but to a “theorem” involving it (central-limit).
  • The passage argues that the theorem explains the observed pervasiveness, which is the universalist stance (regardless of how convincing you may find the point)
  • All of which frames Gibrat’s law as our definition (2), a pervasive behavior explained by a mathematical theorem.

An epistemic typology of statistical laws

In coherence with the three basic understanding of the concept of law that we have highlighted in the start of this section, the article on empirical statistical law proposes a list of examples formulated as a typology. It features various degrees of theoretical grounding, empirical grounding, and combination.

“Examples of empirically inspired statistical laws that have a firm theoretical basis include:
• Statistical regularity
• Law of large numbers
• Law of truly large numbers
• Central limit theorem
• Regression towards the mean

Examples of “laws” with a weaker foundation include:
• Safety in numbers
• Benford’s law

Examples of “laws” which are more general observations than having a theoretical background:
• Rank-size distribution

Examples of supposed “laws” which are incorrect include:
• Law of averages”

Empirical statistical laws

Note that one of those laws is called theorem, and another one distribution, but as we have seen the equivalence between law and distribution has multiple facets.

This typology however does not include some of the cases we have observed, and in particular when law is employed as a strict synonym of distribution. It also distinguishes the truth of a law by qualifying it as “supposed”, suggesting that a falsified law might no longer be law. We will propose our own typology by including all the cases we have observed and ignoring the truth status of the law. In other words, we will recognize a law if it is stated as a pervasive behavior, for instance, regardless of whether this statement is true or false. We see no problem with the concept of a false law, and consider convenient to still call it a law for epistemic reasons. We will also precise some observed characteristics of the different types. Note: this typology extends and enriches our first basic typology (numbering is consistent).

Observed types of usage of the term law

(0) Law strictly refers to a distribution (a mathematical function).
This definition is never stated but is observed in Wikipedia articles.
It maybe interpreted as a metonymy (see below).

(1) Law refers to a pervasively observed distribution/behavior.
Often considered to just describe a phenomenon.
Pervasiveness is required to be called a law.
Some validity conditions may apply.

(2) Law refers to an empirically observed theorem.
Often considered to explain and predict a phenomenon.
The theorem is often about a phenomenon following a certain distribution.
Pervasiveness is not longer required since the theorem is formal.
Some validity conditions may still apply.

(3) Law refers to a theorem (not observed empirically).
Often considered an improper use of the “law” term.

These different types correspond to two different perspectives: descriptive and explanatory. In the descriptive perspective, the law describes a phenomenon, possibly with a distribution. The fact that the distribution itself can also be called a law may cause some confusion. This perspective is purely grounded on empirical observations, which must have some pervasiveness to be called a law.

Descriptive perspective: based on pervasive observations, the law describes but does not explain

The explanatory perspective is on the contrary grounded on a mathematical theorem which is not necessarily observed in the real world. Because the theorem has a formal proof, a deduction, its validity conditions do not require pervasiveness. However the applicability to real life may still involve validity conditions. The mathematical foundation, as we have seen, are often credited for a power to explain and/or to predict. The fact that the theorem may be about a distribution also allow for a certain degree of confusion.

Explanatory perspective: based on a theorem, it not describes but also explains and/or predicts

The last two diagram show how these different usages overlap. As we will see, this situation generates a certain amount of confusion.

the inconsistent usage of the term law

Our four types of usage are not exclusive, as the type (0) is compatible with each of the three others. Indeed though usages (1), (2) and (3) differ by the way the law is grounded (empirically and/or theoretically) and are thus mutually exclusive, the usage (0) is possible any time a distribution is involved. What does it mean? Does it matter?

Let us take an example. An article mentions “Zipf’s law”. What do you think it means? There are two possible answers.

  • (0) It means Zipf’s distribution. It is just a mathematical function.
  • (1) It means that Zipf’s distribution is observed everywhere (inside its validity conditions). It is a statement on the empirical reach of a mathematical function, based on its pervasiveness.

Let us look at the Wikipedia article, which starts like this:

“Zipf’s law […] refers to the fact that many types of data studied in the physical and social sciences can be approximated with a Zipfian distribution”

Zipf’s law

This usage is clearly type (1). But later in the same article we find statements such as those:

“Zipf’s law is most easily observed by plotting the data on a log-log graph”

“The simplest case of Zipf’s law is a “1⁄f function.””

“many natural phenomena obey Zipf’s law”

Zipf’s law

And even the following, that I cannot reproduce in text form in this blog:

Zipf’s law as a formula, referring to the distribution

These multiple usages are as clearly of type (0), referring to the distribution and not to the statement of its pervasiveness. In this example like in others, different usages coexist in the same text.

This practice of mixing the usages is akin to a figure-of-speech such as a metonymy. We hypothesize that the Wikipedia writer may favor the shorter “Zipf’s law” to the longer “the mathematical equation involved in Zipf’s law”. It is however intriguing that for the sake of brevity, the perfectly sound concept of “Zipfian distribution” is not used instead. Distributions never have the meaning of an empirical statement, they always refer to the equation. We found no other reason than the persistance of a historical confusion to explain why the term “distribution” is not used in place of “law” when referring to the mathematical formula.

Does it matter? It brings confusion to statements where we cannot easily determine which of the two usages is intended, which has consequences on the evaluation of those statements. For instance one may wonder: is Zipf’s law true? Understood as type (0), it is as true as a mathematical equation can be, absolutely unfalsifiable inside its validity conditions – though a simple equation makes nothing else than an absolutely abstract statement. But understood as type (1), it entirely depends on the pervasiveness and validity conditions of the law, which are not necessarily robust and might change over time. A law can stop being true while we get more observations.

Fallacies

The main consequence of this confusion is to allow vicious arguments on the validity of laws. By leveraging the confusion to convoke the right perspective at the right moment, we could abuse rhetoric to make a false but convincing point. For instance we could argue that the log-normal law is guaranteed to be true by virtue of the central-limit theorem, then argue that it empirically grounded because of its pervasiveness. This argumentative line is strong because it draws justification from both sides of the empiricism/theory divide. It makes it look like math theory guarantees the law’s empirical reach, but this precisely is a complete fallacy. Indeed the law guaranteed by the central-limit theorem is just the log-normal distribution, not the statement of pervasiveness constitutive of the law understood as something empirical. Mathematical validity does not cause empirical validity, because conditions always apply. Yet the confusion around the concept of law allows a fallacy that shortcuts the gap.

This last example was just for the sake of argumentation, since we did not actually observe such fallacious arguments, and we did not even search for them – for the moment at least. We will return later to the questions of pervasiveness and universality, and we will pay extra attention to the usage of the term law in these situations, as there is a potential for abuse.

The potential for abuse can take the form of more generic and traditional fallacies like a simple circular logic such as:

  1. A law is by definition pervasive
  2. That pervasiveness is what makes it a law

As trivial as it sounds, laws are often evoked in Wikipedia and their status is rarely discussed outside their article, and pervasiveness is not even systematically justified. Wikipedia is not exactly academic literature and has its limits, nevertheless it seems quite dangerous to repeatedly assume that anything named “law” is, by virtue of that name, pervasive.

The Pareto distribution is the favorite law

Our corpus of analyzed articles is focused on Pareto and the power law, so it would only be natural that it is biased in favor of over-representing the Pareto distribution. However very generic articles such as Scientific law and Empirical statistical law seem to have a clear preference for Pareto and Zipf.

The first, and only detailed, examples of Empirical statistical laws, in the corresponding article, are Pareto and Zipf. Both are framed as laws. The latter has its empirical pervasiveness highlighted.

“The Pareto principle is a popular example of such a “law”. It states that roughly 80% of the effects come from 20% of the causes, and is thusly also known as the 80/20 rule. […]

Zipf’s law, described as an “empirical statistical law” of linguistics, is another example. According to the “law”, given some dataset of text, the frequency of a word is inversely proportional to its frequency rank. […] However, what sets Zipf’s law as an “empirical statistical law” rather than just a theorem of linguistics is that it applies to phenomena outside of its field, too. For example, a ranked list of US metropolitan populations also follow Zipf’s law, and even forgetting follows Zipf’s law. This act of summarizing several natural data patterns with simple rules is a defining characteristic of these “empirical statistical laws”.”

Empirical statistical laws

In the article Scientific law, Zipf’s law is the only example.

“The term “scientific law” is traditionally associated with the natural sciences, though the social sciences also contain laws. An example of a scientific law in social sciences is Zipf’s law.”

Scientific law

There is only one mention of “law” in the article on probability distributions and it is about “the prototypical power law distribution” (note the double qualification).

Here we only hypothesize that Zipf’s law and the Pareto principle, which are both sub-species of the power law, are associated with especially strong claims to empiricism.

Anecdote: a law whose name is explained by a law

As a lighter note to this section, we remark that not all laws are about a distribution. In the article on “Cauchy distribution”, inception! the name of the law is itself explained by a law. As the History section states:

“Functions with the form of the density function of the Cauchy distribution were studied by mathematicians in the 17th century, but in a different context and under the title of the witch of Agnesi. Despite its name, the first explicit analysis of the properties of the Cauchy distribution was published by the French mathematician Poisson in 1824, with Cauchy only becoming associated with it during an academic controversy in 1853. As such, the name of the distribution is a case of Stigler’s Law of Eponymy. Poisson noted that if the mean of observations following such a distribution were taken, the mean error did not converge to any finite number. As such, Laplace’s use of the Central Limit Theorem with such a distribution was inappropriate, as it assumed a finite mean and variance. Despite this, Poisson did not regard the issue as important, in contrast to Bienaymé, who was to engage Cauchy in a long dispute over the matter.”

Cauchy distribution

Close reading Wikipedia from Pareto to Network Science, part 2

This is part 2: Concepts of complex network and preferential attachment

As we mentioned in the part 1 of this work, just defining the concepts we need to understand the field requires an effort. Before we move on to analysis, we will clarify two families of concepts, one centered on the complex network, and the other one on preferential attachment.

As this is a post on my research blog, it tends to expose more than a typical paper would. I follow the open source software mantra “release early, release often”.

Findings

Scale-free networks and small-world networks have a different origin and characterization, but it turns out that they are quite the same thing and their differences do not seem so important now that we have studied them further. The more general term of complex network seems appropriate to refer to this family of networks.

Preferential attachment has many names but is precisely defined. The term of “Matthew effect” is not a strict equivalent, though. There is an ambiguity about assortativity, which is an equivalent of preferential attachment only in a specific case, despite how some authors refer to the concept.

Scale-free, small world, and complex networks

Those three types of networks are sometimes presented as the same, sometimes not. Fortunately, a simple distinction makes it possible to define a ground where most articles agree:

  1. The characterizations are not the same. Scale-freeness is not small-worldness is not complexity.
  2. In practice, those three networks might be the same kinds of networks.

Note that the question of the differences the three types, beyond their characterization, is still open. Some articles claim that it is not useful to separate them, while others enforce the demarcations. However all articles seem to agree that the relevant features for this family of networks are richer than just scale-freeness and small-worldness. In that sense, these categories are more historical than an accurate typology of networks.

In the side template on network science present on many articles, all three types of networks appear on an equal foot.

On the topic of typologies, the article on social networks has a one worthy of Borges, where criteria of widely different sorts seem equally relevant at defining their own kind of networks. It is organized in different levels of analysis, and scale-free networks and complex networks appear as items in different levels:

• Micro level:
  o Dyadic level
  o Triadic level
  o Actor level
  o Subset level

• Meso level:
  o Organizations
  o Randomly distributed networks
  o Scale-free networks

• Macro level:
  o Large-scale networks
  o Complex networks

Social network

This article also details these two types of networks, and though the characterization differs, both acknowledge a certain amount of variability in the definition. We also see that the heavy-tail distribution of node degrees is mentioned as a feature of complex networks, which as we have seen is an other way to name the power-law, and thus poses an unnamed equivalence with the characterization of scale-free networks.

“Scale-free networks: A scale-free network is a network whose degree distribution follows a power law, at least asymptotically. […] Specific characteristics of scale-free networks vary with the theories and analytical tools used to create them, however, in general, scale-free networks have some common characteristics. One notable characteristic in a scale-free network is the relative commonness of vertices with a degree that greatly exceeds the average. Another general characteristic of scale-free networks is the clustering coefficient distribution, which decreases as the node degree increases. This distribution also follows a power law. The Barabási model of network evolution shown above is an example of a scale-free network.”

Social network

“Complex networks: Most larger social networks display features of social complexity, which involves substantial non-trivial features of network topology, with patterns of complex connections between elements that are neither purely regular nor purely random […]. Such complex network features include a heavy tail in the degree distribution, a high clustering coefficient, assortativity or disassortativity among vertices, community structure (see stochastic block model), and hierarchical structure.”

Social network

Despite their article, social networks are not a specific kind of network but rather an area of application. The three kinds that interest us have each a specific page where the can state their relations to each other.

The small-world network, does not mention the other kinds, but declares a feature of “fat-tailed distribution” which is an equivalent of the power law, and thus an unnamed link to scale-free networks.

“Networks with a greater than expected number of hubs will have a greater fraction of nodes with high degree, and consequently the degree distribution will be enriched at high degree values. This is known colloquially as a fat-tailed distribution.”

Small-world network

The scale-free network page starts with the characterization by power-law distribution of node degrees. It also mentions “the small world network model” as a sub-category, and makes several confusing references to the concept of “complex network”, like for instance the caption of this image:

From the Scale-free network page

Other example, the mention of “scale-free complex networks” in a paragraph marked as requiring “attention from an expert in Mathematics”.

From the Scale-free network page

The complex network article states explicitly its relations with the two other kinds: it is generalizing them, arguing that the original demarcations have lost relevance:

“Two well-known and much studied classes of complex networks are scale-free networks and small-world networks, whose discovery and definition are canonical case-studies in the field. Both are characterized by specific structural features—power-law degree distributions for the former and short path lengths and high clustering for the latter. However, as the study of complex networks has continued to grow in importance and popularity, many other aspects of network structure have attracted attention as well.”

Complex network

In that spirit, it assumes an explicitly vague definition. It compensates the drawback of not being a mathematical object by declaring itself as an area of research. It then states a form of correspondance with networks defined by their origin rather than their properties, notably “social networks”.

“In the context of network theory, a complex network is a graph (network) with non-trivial topological features—features that do not occur in simple networks such as lattices or random graphs but often occur in graphs modelling of real systems. The study of complex networks is a young and active area of scientific research (since 2000) inspired largely by the empirical study of real-world networks such as computer networks, technological networks, brain networks and social networks”

Complex network

It is interesting to remark that the article on social networks, symmetrically, states an equivalence with complex networks (and as we have seen, with them only).

“Together with other complex networks, it forms part of the nascent field of network science.”

Social network

Preferential attachment, cumulative advantage, rich get richer, Yule process, assortativity, and the Matthew effect

The primary definition of preferential attachment is probabilistic even if we will often find it associated with scale-free networks (in our corpus at least).

“A preferential attachment process is a stochastic urn process, meaning a process in which discrete units of wealth, usually called “balls”, are added in a random or partly random fashion to a set of objects or containers, usually called “urns”. A preferential attachment process is an urn process in which additional balls are added continuously to the system and are distributed among the urns as an increasing function of the number of balls the urns already have.”

Preferential attachment

The following definition (under the name of “the rich get richer”) is similar but more explicit on the implications.

“In statistics, the phrase “the rich get richer” is often used […] where the probability of the next outcome in a series taking on a particular value is proportional to the number of outcomes already having that particular value. This is useful for modeling many real-world processes that are akin to “popularity contests”, where the popularity of a particular choice causes new participants to adopt the same choice (which can lead to the outsized influence of the first few participants).”

The rich get richer and the poor get poorer

This time the equivalence between the different notions is pretty explicit, and the article on preferential attachment explains it. Only the Matthew effect is a little special, and as we have seen it is also considered equivalent to the Pareto distribution. Assortativity is also a special case, as we will see, because it is an equivalent of preferential attachment only in certain cases. Anyway the connection with the power law is central and explicit.

“”Preferential attachment” is only the most recent of many names that have been given to such processes. They are also referred to under the names “Yule process”, “cumulative advantage”, “the rich get richer”, and, less correctly, the “Matthew effect”. They are also related to Gibrat’s law. The principal reason for scientific interest in preferential attachment is that it can, under suitable circumstances, generate power law distributions.”

Preferential attachment

Cumulative advantage is another name of preferential attachment, but it seems it is not much employed anymore.

“In a later paper in 1976, Price also proposed a mechanism to explain the occurrence of power laws in citation networks, which he called “cumulative advantage” but which is today more commonly known under the name preferential attachment. […] Barabási and Albert proposed a generative mechanism to explain the appearance of power-law distributions, which they called “preferential attachment” and which is essentially the same as that proposed by Price.”

Scale-free network

There is also quite a long explanation about the distinction with the Matthew effect, quoted below for reference.

“Preferential attachment is sometimes referred to as the Matthew effect, but the two are not precisely equivalent. The Matthew effect, first discussed by Robert K. Merton, is named for a passage in the biblical Gospel of Matthew: “For everyone who has will be given more, and he will have an abundance. Whoever does not have, even what he has will be taken from him.” (Matthew 25:29, New International Version.) The preferential attachment process does not incorporate the taking away part. This point may be moot, however, since the scientific insight behind the Matthew effect is in any case entirely different. Qualitatively it is intended to describe not a mechanical multiplicative effect like preferential attachment but a specific human behavior in which people are more likely to give credit to the famous than to the little known. The classic example of the Matthew effect is a scientific discovery made simultaneously by two different people, one well known and the other little known. It is claimed that under these circumstances people tend more often to credit the discovery to the well-known scientist. Thus the real-world phenomenon the Matthew effect is intended to describe is quite distinct from (though certainly related to) preferential attachment.”

Preferential attachment

The article on the Matthew effect focuses more on the similitudes than the differences.

“In network science, the Matthew effect is used to describe the preferential attachment of earlier nodes in a network, which explains that these nodes tend to attract more links early on. […] “Because of preferential attachment, a node that acquires more connections than another one will increase its connectivity at a higher rate, and thus an initial difference in the connectivity between two nodes will increase further as the network grows, while the degree of individual nodes will grow proportional with the square root of time.” The Matthew Effect therefore explains the growth of some nodes in vast networks such as the Internet.”

Matthew effect

Preferential attachment is also mentioned on the statistical side of the studied articles, as for instance in the article on Zipf’s law.

“Similarly, preferential attachment (intuitively, “the rich get richer” or “success breeds success”) that results in the Yule–Simon distribution has been shown to fit word frequency versus rank in language and population versus city rank better than Zipf’s law.”

Zipf’s law

As a clue on the cultural side of these notions, the article on “The rich get richer and the poor get poorer” is specifically about the aphorism and not the phenomenon itself, which refers to either “wealth concentration” or “economic inequality”.

“This article is about the catchphrase. For the theoretical process, see wealth concentration. […] In statistics, the phrase “the rich get richer” is often used as an informal description of the behavior of Chinese restaurant processes and other preferential attachment processes, where the probability of the next outcome in a series taking on a particular value is proportional to the number of outcomes already having that particular value.”

The rich get richer and the poor get poorer

Assortativity and homophily

A misunderstanding must be clarified about assortativity, also called homophily.

  1. Assortativity, or homophily, generally means that in a network the probability that two nodes are connected is higher when those nodes have something in common. This is not specifically related to preferential attachment.
  2. However when the point in common is the degree (number of neighbors) then assortativity/homophily becomes an avatar of preferential attachment. Unfortunately, it seems that some authors employ a synecdoche where that specific case is named by the more generic concept, causing some confusion.

“Assortativity, or assortative mixing is a preference for a network’s nodes to attach to others that are similar in some way. Though the specific measure of similarity may vary, network theorists often examine assortativity in terms of a node’s degree.”

Assortativity

On the use of the term “homophily”, we can cite the article on social network analysis: “Homophily (assortativity)”.

The article on social networks is not specific but mentions “assortativity or disassortativity among vertices” as a feature of complex networks.

Note that awkwardly, “Assortative mixing” seems to be the same thing as assortativity but each has its own page, and each refers to the other.

Close reading Wikipedia from Pareto to Network Science, part 1

This exploration searches for traces of the circulation of structuralist claims from statistics to network science, from Pareto to Barabási. Barabási’s claims are indeed reminiscent of the foundations of statistics, when Quetelet or Galton aimed at unveiling the laws of the social. Where does it come from? I saw a possible bridge in the concept of power law, for four reasons:

  1. It is a statistical distribution, comparable to the normal distribution and many others.
  2. It ties to network science insofar as it characterizes scale-free networks (ie. the distribution of node degrees).
  3. As a “statistical law” it carries a structuralist point, a claim to universality.
  4. It also carries a political meaning since it roots in Pareto’s study of inequalities.

Using Wikipedia as a documentary source, I had already observed that the power law and scale-free networks were bridging the domain of statistics and the domain of network science (see my previous blog post on the subject). To challenge my hypothesis and dig deeper, I engaged with the qualitative reading of the relevant Wikipedia pages. From that work I expect to get an overview of the conceptual landscape and to identify hot spots to be leveraged as landmarks for a further study of the academic literature.

Method

From my distant reading of the question, looking at the hyperlinks network, I obtained a list of about 100 Wikipedia pages. This list was too long for the time I had, so I reduced it to the most cited pages in each of my three categories, “Pareto power law”, “Network science”, and the “bridge” between them. I quickly realized that important pages were not listed, so I added them. Of the 45 articles I read, 13 were added in a second time. The distant reading (the network exploration) put me on the right tracks but as expected, the close reading (the qualitative inquiry) was more accurate at delineating the question.

Note: we will not systematically explain the concepts, since it would be make the text heavier for a minor benefit. The general argument is about the relations between the concepts, and should be understandable even without being familiar with them. If not, then of course we suggest to read the corresponding Wikipedia pages.

This is part 1: focusing on the power law

Just defining the concepts we need to understand the field requires an effort. Not because it is complicated, though it might also be, but because each concept appears under different names, with different meanings, and do not always relate consistently. We will focus first on bringing clarity to the main concepts. This part looks only into the power law and a group of closely related concepts that we may call its family – there is quite a lot to write about. The next part will look into other concepts, and then we will inquire further specific kinds of claims and arguments.

As this is a post on my research blog, it tends to expose more than a typical paper would. I publish it early, in the spirit of open research, and following the open source software mantra “release early, release often”. So the text is quite long, but you can the findings summarized below.

Findings

The power law is often referred to as a the Pareto distribution, despite minor differences. The log-normal distribution is often presented as an alternative to the power law, bust also sometimes as an equivalent. There is a controversy about which is better fitting empirical cases, which has failed to reach closure for two decades. Both have crucial similarities but come with widely different narratives. It turns out that the tail of distributions is generally their relevant part, and it does not allow to differentiate our two competitors. As a way out of the controversy, they can be framed as instances of the more general heavy-tail distributions, which also questions the narratives convoked to interpret empirical data.

The power law and its family

  • Power law
  • Pareto distribution / principle
  • Yule–Simon distribution
  • Log-normal distribution
  • Heavy-tailed distribution
  • 80-20 rule
  • Zipf’s law
  • Long tail
  • Matthew effect

A dozen concepts occupy a pretty narrow theoretical space. As we will see, they are close enough that they are sometimes convoked as if they were equivalent. If we had to pick one representative, in the context of Wikipedia it would be the power law, but the Pareto distribution is also very important as we will see. We start by looking at those and from there we give an overview of this space and clarify the relations between these different concepts, and analyze a few key observations.

power law

“In statistics, a power law is a functional relationship between two quantities, where a relative change in one quantity results in a proportional relative change in the other quantity, independent of the initial size of those quantities: one quantity varies as a power of another.”

Power law

In Wikipedia, the power law is posing as the mother of all of its kind. Not only is it defined in a relatively simple and generic manner, independently from any other law or distribution, but it also pretends to subsume all others. In its dedicated page, the section Examples opens with a typical claim about the pervasiveness of the power law, of a sort of claim we will investigate further in a moment: “More than a hundred power-law distributions have been identified in physics (e.g. sandpile avalanches), biology (e.g. species extinction and body mass), and the social sciences (e.g. city sizes and income).” Follows a massive list of 50 cases organized in 7 sections, among which we find several familiar figures: “Pareto distribution and the Pareto principle also called the “80–20 rule””, “Zipf’s law”, “Yule–Simon distribution (discrete)”, and “The scale-free network model”.

Other distributions also acknowledge the generality of the power law. For instance the Pareto distribution, in its own article, is presented as a culturally situated version of the power law.

Pareto distribution

“The Pareto distribution, named after the Italian civil engineer, economist, and sociologist Vilfredo Pareto, is a power-law probability distribution that is used in description of social, scientific, geophysical, actuarial, and many other types of observable phenomena.”

Pareto distribution

So what makes the Pareto distribution something more specific than the power law? The quote above, which is the first sentence of the article, states that the power law takes the “Pareto” name in a certain cultural context. However a second point immediately follows that precises that the power law takes the “Pareto” name for a specific value of a parameter of the power law, producing a specific “80-20” pattern.

“The Pareto distribution has colloquially become known and referred to as the Pareto principle, or “80-20 rule”, and is sometimes called the “Matthew principle”. This rule states that, for example, 80% of the wealth of a society is held by 20% of its population.”

Pareto distribution

Note that the “Pareto principle” and the “Matthew principle” have distinct dedicated articles, and that the “80-20 Rule” redirects not to “Pareto distribution” but to “Pareto principle”. Doublons are not uncommon in Wikipedia and they often make sense insofar as the different pages bring different perspectives on the same thing. Those doublons sometimes coexist ignoring each other, but it is not the case here since the different concepts refer to each other explicitly, and link to each other. However not all variations of the power law state equivalence or relation to all others.

Overview of the conceptual space

The following diagram exposes how each article positions itself vis-à-vis other concepts. Each arrow states how the Wikipedia article at the source of the arrow describes its relation with the concept at the target. The absence of a link means the absence of a position (all pairs have been systematically explored).

How in Wikipedia different concepts similar to the power law state their differences with each other

Download the source file as a CSV matrix
(RIGHT CLICK > SAVE AS or equivalent)

By exploring this landscape we find different types of relations: inclusion (ie. one concept being more generic and the other one more specific), equivalence (ie. two different names for the same thing), and difference. If we omit the details to only look at the type of relations, if we also omit the missing links, and we generalize the relations of inclusion, the network’s structure appears quite close to an ontology. This coherence shows a general consensus, in Wikipedia, about this otherwise tight conceptual space. The diagram below shows the relations of inclusion as boxes containing each other, which greatly simplifies the image. Five relations are not of inclusion, and are thus remarkable:

  • Difference between log-normal distribution and the power law
  • Equivalence between the log-normal and Pareto distributions
  • Equivalence between the Pareto distribution and the Matthew effect
  • Equivalence between the Pareto principle and the 80-20 rule
  • The long tail is a feature of the power law and its more specific versions
Relations between concepts close to the power law, focusing on inclusion represented as boxes containing each other. Other relations are represented as arrow (is a feature of), equal (equivalence) and unequal (difference) signs.

The general landscape seems quite classic, except for a paradoxical relation of equivalence between the Pareto distribution and the log-normal distribution, as in the following quote:

“The Pareto distribution and log-normal distribution are alternative distributions for describing the same types of quantities. One of the connections between the two is that they are both the distributions of the exponential of random variables distributed according to other common distributions, respectively the exponential distribution and normal distribution.”

Pareto distribution

This quote is key because it hints at the specific position of the Pareto distribution in this conceptual landscape. On the one hand, the central concept is the power law (PL), and not the Pareto distribution (PD). The PL is more generic (both concepts agree on that), is claiming to subsume other laws (which PD is not), and has the more open definition. It is also strictly speaking the proper statistical concept to refer to in most situations, for instance to characterize what the long tail is a feature of. But on the other hand the PD is often mentioned along with, or even instead of, the PL. For instance in the following quote, the Zipf distribution uses both the PL and the PD as points of comparison, both times marking its key feature of being discrete (non-continuous).

“Zipf’s law […] can be approximated with a Zipfian distribution, one of a family of related discrete power law probability distributions. […] The Zipf distribution is sometimes called the discrete Pareto distribution because it is analogous to the continuous Pareto distribution in the same way that the discrete uniform distribution is analogous to the continuous uniform distribution.”

Zipf’s law

In this example we can see that the concept of power law is so vague that it raises issues when it is necessary to refer to it. Because of its extremely broad definition (“a functional relationship between two quantities, where […] one quantity varies as a power of another”) the power law plays the role of a group of notions (“family”) or a general principle. It does not have a precise equation and strictly speaking, cannot be called a distribution. This role is better played by the Pareto distribution, which is the archetype of the Power law. We hypothesize that it might actually be the prototype of the power law, insofar as the latter might have been forged as a generalization of the former. Anyway this relation leads to metonymies where the Pareto distribution is invoked in place of the power law when the point requires a precise distribution.

The Pareto distribution and the power law nevertheless have a paradoxical situation. Despite being largely equivalent, they seem in opposite situations towards the log-normal distribution: the power law is mentioned as different, while the Pareto distribution is mentioned as equivalent. How could they be, if they are more or less the same thing? As we will see, this paradox is just an instance of a more general situation where the log-normal controversy is mentioned both as equivalent and as different to the power law. This seems to be the particular form taken in Wikipedia by a controversy about the pervasiveness of these two notions. And as we will see, the Pareto distribution plays a role in it.

log-normal distribution

Taking a look at the concept map above, you can notice that the log-normal distribution only has inbound links, and no outbound links. It does not state any relation with any other concept (of our list), even though other concepts state relations with it. Contrary to the power law and similarly to the Pareto distribution, it is a distribution with a precise equation. It is related to the most central and important of all statistical distribution, the normal distribution, of which it is the logarithmic version. It also relate to the very important central limit theorem. Finally, it also takes the name of an important historical figure of statistics, Francis Galton.

“In probability theory, a log-normal (or lognormal) distribution is a continuous probability distribution of a random variable whose logarithm is normally distributed. […] The distribution is occasionally referred to as the Galton distribution or Galton’s distribution, after Francis Galton. The log-normal distribution also has been associated with other names, such as McAlister, Gibrat and Cobb–Douglas. […] A log-normal process is the statistical realization of the multiplicative product of many independent random variables, each of which is positive. This is justified by considering the central limit theorem in the log domain.”

Log-normal distribution

The log-normal distribution, like the power law, has a claim to pervasiveness. The section “Occurrence and applications” of its article lists 21 cases and starts this way:

“The log-normal distribution is important in the description of natural phenomena. This follows, because many natural growth processes are driven by the accumulation of many small percentage changes. These become additive on a log scale.”

Log-normal distribution

Log-normal/power law controversy

Despite their different origins and characterizations, the log-normal and power law distributions have important similitudes, in particular when used to fit empirical data. A reader could not guess it by just reading the article on the log-normal distribution, since it does not relate to the characterization of the power law and this similitude is not mentioned. However it is extensively evoked in the article about the power law:

“Few empirical distributions fit a power law for all their values, but rather follow a power law in the tail. […] all power laws with a particular scaling exponent are equivalent up to constant factors, since each is simply a scaled version of the others. This behavior is what produces the linear relationship when logarithms are taken of both f(x) and x, and the straight-line on the log–log plot is often called the signature of a power law. With real data, such straightness is a necessary, but not sufficient, condition for the data following a power-law relation. In fact, there are many ways to generate finite amounts of data that mimic this signature behavior, but, in their asymptotic limit, are not true power laws (e.g., if the generating process of some data follows a Log-normal distribution). Thus, accurately fitting and validating power-law models is an active area of research in statistics […] For example, log-normal distributions are often mistaken for power-law distributions”

Power law

To understand what is happening here we need to take a few steps outside Wikipedia. There is indeed a controversy about which of the two distributions better fit empirical data, but unfortunately there is no dedicated Wikipedia page. I realized that a quick investigation in the academic literature was fruitful to understand some Wikipedia statements. A complete study would certainly be worth it but to keep us focused I choose to compromise. I dedicated a few hours to a quick-and-dirty mini literature review, whose findings are not robust but sufficient to help us. In the following paragraphs we will see why a controversy emerged and how it leads to a specific kind of statement in Wikipedia.

On that topic, the Wikipedia article has for reference A Brief History of Generative Models for Power Law and Lognormal Distributions by M. Mitzenmacher, published in 2004. This paper “intended to be accessible to a general audience” provides an excellent overview of the question and summarizes that “lognormal and power law distributions connect quite naturally, and hence, it is not surprising that lognormal distributions have arisen as a possible alternative to power law distributions across many fields.” For a less academic and more dramatic opinion on the controversy, let us cite a 2005 post on the 3 Quark Daily blog where an exchange of emails between
C. Shalizi (champion of the log-normal) and A.-L. Barabási (defender of the power law) sets off the following comment: “This is actually a new trend — take someone’s claim that something is a power law/lognormal and then claim it is actually distributed the other way. Frankly, the two distributions are very close…” Wikipedia does not mention it, but it is important to acknowledge here that there has been a “trend” of claims of this type: “empirical data X, previously considered to follow a power law, are in fact better fit by a log-normal distribution” – or the other way around. At the core of the controversy lies this long lasting dispute about which of the two distributions is a better fit for empirical data (M. Mitzenmacher could trace it back to the 1950s), but the dispute itself is not the controversy.

The log-normal controversy is a dispute about the dispute. On the level of the core dispute itself (power law versus log-normal distribution), at least actors agree on their disagreement. But on the state of the debate, there is no consensus. In particular, multiple authors have successively tried to put an end to the dispute, without any success.

The most common attempt to reach closure states that (1) considering the kinship between the two models, and (2) the inherent gap between any model and the empirical data it fits, then (3) the dispute does not have enough relevance to justify the attention it gets. For commodity, let us call this point the pragmatist closure. In his 2004 paper M. Mitzenmacher already argues for it and suggests that “from a more pragmatic point of view, it might be reasonable to use whichever distribution makes it easier to obtain results.” In 2009, A. Clauset, C. Shalizi and M. E. J. Newman conclude their paper Power-law distributions in empirical data this way “In closing, we echo comments made by Ijiri and Simon more than thirty years ago and similar thoughts expressed more recently by Mitzenmacher. They argue that the characterization of empirical distributions is only a part of the challenge that faces us in explaining the causes and roles of power laws in the sciences. […] We hope that the methods given here will prove useful in all of these endeavors, and that these long-held hopes will at last be fulfilled.” Are their hopes fulfilled now? In a paper published in 2012 and titled Pareto or log-normal? A recursive-truncation approach to the distribution of (all) cities, G. Fazio and M. Modica once again attempted without success to reach a pragmatist closure, observing that “repeating [their analysis] confirms the difficulty of distinguishing a Pareto tail from the tail of a log-normal and, in turn, identifying the city size distribution as a false or a weak Pareto law”. Note in this citation how the key element is the tail of the distribution.

Other authors disagree with the pragmatist closure, presumably because they believe that the core dispute is actually relevant and deserves to be decided. Such authors have tried to reach closure by proposing better models. An often cited 2000 paper by W. J. Reed titled The Pareto Law of Incomes – an Explanation and an Extension argues in favor of the “double Pareto” distribution (followed by other papers of the same author). As M. Mitzenmarcher summarizes, “an appropriate double Pareto distribution can closely match the body of a lognormal distribution and the tail of a Pareto distribution” but obviously, this new model did not close the dispute. A more recent attempt at closure by better model can be found in a 2016 paper by J. Luckstead and S. Devadoss titled Pareto tails and lognormal body of US cities size distribution. “The purpose of [their] study is to propose a distribution […] to model lower and upper tails with Pareto and middle range with lognormal […]. We denote this distribution as Pareto-tails lognormal (PTLN).” M. Bee tried a similar move in 2015 with a paper whose title will suffice: Estimation of the lognormal-Pareto distribution using probability weighted moments and maximum likelihood. There is even a Wikipedia page on the Modified lognormal power-law distribution. Note that in these examples as well, the tail is the key element.

We hypothesize that the inability to reach closure is an issue for Wikipedia contributors who cannot meet a firm ground to settle a statement on the core dispute. It is unclear if the differences between the power law and the log-normal distribution are relevant, and it is unclear if one of the two models, or any other model, is actually better. If a contributor, according to a precautionary or neutrality principle, would undertake to represent the different positions of the controversy, she would corner herself in both (1) mentioning the dispute and (2) contesting its relevance. This double movement to simultaneously acknowledge and undermine the dispute is characteristic to certain statements or sets of statements we find in Wikipedia. For instance the Power law article states that “log-normal distributions are often mistaken for power-law distributions”, while the Pareto distribution article states that “The Pareto distribution and log-normal distribution are alternative distributions for describing the same types of quantities.” The article on Gibrat’s law (a law related to the log-normal distribution even though it does not appear in the diagram presented before) offers another example of this double movement, once again involving distribution tails:

“While the city size distribution is often associated with Zipf’s law, this holds only in the upper tail, because empirically the tail of a log-normal distribution cannot be distinguished from Zipf’s law. A study […] finds that the entire distribution of cities, not just the largest ones, is log-normal. But this last claim that the lognormal distribution cannot be rejected has been shown to be the result of a statistics with little power: the uniformly most powerful unbiased test comparing the lognormal to the power law shows unambiguously that the largest 1000 cities are distinctly in the power law regime.”

Gibrat’s law

Distribution tails

As we have had multiple occasions to note, tails matter. There is indeed a whole science of distribution tails and, for clarity, it is worth noting that even though there is a notional use of “long tail”, there also is a specific statistical meaning. In the articles we have analyzed the notional use is common but not universal. The other important type of tail is the “heavy tail” which appears to be the largest category, subsuming both the power law and the log-normal distribution.

“The term [long tail] is often used loosely, with no definition or arbitrary definition, but precise definitions are possible. […] In statistics, the term long-tailed distribution has a narrow technical meaning, and is a subtype of heavy-tailed distribution. […] Note that there is no sense of the “long tail” of a distribution, but only the property of a distribution being long-tailed.”

Long tail

The multiple mentions of distribution tails we have seen so far come from the fact that the power law and the log-normal distribution have identical tails. The tail is naturally central to the pragmatist closure of the log-normal/power law controversy, since it is where the dispute dissolves.

“Few empirical distributions fit a power law for all their values, but rather follow a power law in the tail.”

Power law

“While the city size distribution is often associated with Zipf’s law, this holds only in the upper tail, because empirically the tail of a log-normal distribution cannot be distinguished from Zipf’s law.”

Gibrat’s law

But the tail is also central to the closure by better model, since many of them achieve to fit empirical data by compositing different power laws. Since the log-normal/power law dispute only happens in the head (of the distributions), this strategy is very efficient. Indeed the head is smaller than the tail and fitting a different law for the head greatly improves the model quality. The “double Pareto” distribution for instance takes its name from being the composite of two different Pareto distributions, separated by a single breaking point. We can find an illustration of such “broken power law” in the article about Zipf’s law.

Example of a “broken power law” from the Zipf’s law page

Compositing power laws displaces the debate to discussing where and how to determine breaking points. Since it keeps using power laws, it allows to retain some of their features and strengthens the corresponding rationale. The log-normal distribution and the power law come with different rationales and whichever better fits empirical data can claim its own explanation for the observed phenomenon. Unfortunately Wikipedia is not rich enough to investigate this aspect of the controversy, so we will just flag this sub-controversy for further inquiry in the academic literature. We can nevertheless mention this passage from the Rank-size distribution article:

“Most simply and commonly, a distribution may be split in two, termed the head and tail. If a distribution is broken into three pieces, the third (middle) piece has several terms, generically middle, also belly, torso, and body. These frequently have some adjectives added, most significantly long tail, also fat belly, chunky middle, etc. In more traditional terms, these may be called top-tier, mid-tier, and bottom-tier. […] Purely quantitatively, a conventional way of splitting a distribution into head and tail is to consider the head to be the first p portion of ranks, which account for 1-p of the overall population, as in the 80:20 Pareto principle, where the top 20% (head) comprises 80% of the overall population. The exact cutoff depends on the distribution – each distribution has a single such cutoff point—and for power laws can be computed from the Pareto index. […] The Yule–Simon distribution that results from preferential attachment (intuitively, “the rich get richer” and “success breeds success”) simulates a broken power law and has been shown to “very well capture” word frequency versus rank distributions. It originated from trying to explain the population verses rank in different species. It has also been show to fit city population versus rank better.”

Rank-size distribution

Finally, we must note that tail issues are also, and possibly mainly, about the size. This is why heavy-tailed distribution is the most generic category of our conceptual landscape. Vilfredo Pareto was “was fascinated by problems of power and wealth”, as mentioned in its own article. The Pareto distribution indeed formalizes a deeply rooted inequality. “It is a social law”, he wrote. The existence of a long tail is enough to characterize this inequality. The log-normal distribution may have a different head, it embeds as much inequality. But the inequality can be evaluated according to the thickness of the tail, hence the relevance of the general group of heavy-tailed distributions, who all embed this relation to inequality.

As a conclusion to this long section about the power law and its related concepts, and as a hint to the next part of our inquiry, network science, this quote from the article on complex networks, shows that the heavy-tailed distribution matters more than the power law:

“Most of these reported “power laws” fail when challenged with rigorous statistical testing, but the more general idea of heavy-tailed degree distributions—which many of these networks do genuinely exhibit (before finite-size effects occur)are very different from what one would expect if edges existed independently and at random”

Complex network

Getting back our imagination about the regulation of algorithms

I disagree with many clever minds when it comes to algorithms. Take for instance the following sentence: “The opacity of the algorithms’ power means that it isn’t easy to determine when algorithmic governance stops serving the common good and instead becomes the servant of the powers that be.” A pretty common claim. I am fine with it, except when it blames “the opacity”. A regrettable misunderstanding is at play, which paralyzes some people’s imagination. I think there are issues with algorithms, and I would like to provide a standpoint from which everyone can be critical, mobilize their political imagination, and step into the debate. My point is dead simple: we do not need to understand how algorithms think as long as we acknowledge that they have agency.

Algorithms, complexity and I have a long story, but here like anyone else I am simply concerned with algorithms impacting my life. They might be hidden and have an indirect influence, their effects are nevertheless real. I am writing this post in reaction to an article written by two Danish thinkers, Jacob Mchangama and Hin-Yan Liu, titled The Welfare State Is Committing Suicide by Artificial Intelligence. It is a short read, and all my quotes come from it. The authors reflect on the recent use of “algorithms to identify children at risk of abuse” in the Danish welfare system. Their main point is that “democratic infrastructures” and “judicial procedures” cannot keep algorithmic power in check, because we “will be largely unable to understand and explain why the algorithm” took its decision, which makes it “impossible for courts to hold [it] accountable.” They locate the source of the problem in the opacity of algorithms which allows, they say, to “take a toll on privacy, family life, and free speech, as individuals will be unsure when their personal actions may come under the radar of the government.” I agree that the situation requires scrutiny from the public but beyond that, I will not waste your time on reading my opinion. I just want to explain why I disagree that opacity prevents us from regulating algorithms. The following quote exposes this precise point.

“Consider the Danish case: the civil servants working to detect child abuse and social fraud will be largely unable to understand and explain why the algorithm identified a family for early intervention or individual for control. As deep learning progresses, algorithmic processes will only become more incomprehensible to human beings, who will be relegated to merely relying on the outcomes of these processes, without having meaningful access to the data or its processing that these algorithmic systems rely upon to produce specific outcomes. But in the absence of government actors making clear and reasoned decisions, it will be impossible for courts to hold them accountable for their actions.”

Indeed algorithms are political beings. Insofar as they take decisions, they produce an effect, hence they have agency. And it is fair to expect them to become “more incomprehensible to human beings.” But concluding that this kind of opacity prevents us from regulating them is misunderstanding what it means to comprehend an algorithm. Contrary to what the authors believe, we have many ways to evaluate an algorithm after its outcomes. We can know it in depth and make many reliable predictions just by analyzing its outputs. This is not free, it comes at an additional cost to developing the algorithm itself, but it does not require to understand how it works, how it thinks. This is sometimes called post-hoc interpretability, to emphasize that interpretation is not relying on the internal mechanics of the algorithm. This is typically the case with deep learning where the algorithm is trained in a way that is “incomprehensible to human beings.” This is nothing special, just new to those who thought we had a divine right to understand everything in this world. As for us who feel the constant pain of being too stupid for what the world has to offer, we are used to have our capabilities exceeded and we find turnarounds to keep going on – when we can. Complexity is a name we use sometimes to talk about that. Post-hoc understanding is a turnaround we use to keep going on with algorithms that are too complex.

To me this whole story feels like there is not much to write about, but I know it is false because so many people feel threatened by opacity. It may come from misplaced confidence in our ability to contain and master all the things we produce, despite the accumulated evidence that it is not the case, culminating in our inability to keep our own habitat, the surface of our planet, in a state that suits our needs. Common misconceptions about what does or does not act are blinding us, for instance when we think that human beings have a power to act that the surface of our planet is lacking – but it provides a hot feedback! Algorithms are in the same situation. Once we acknowledge that they act by themselves (in the sense that they are opaque to us) and consider them accordingly, ways to regulate them in a democratic setting naturally appear. They do because we are surprisingly skilled at post hoc interpretation, something we use everyday without even thinking of it. Except we don’t usually do it for artificial things, only for other human beings.

Regulating the agency of human beings is the point of all politics, even though we barely know how the human mind works. The questions that seem to bother us about algorithms sound surprisingly empty when asked about persons. Let us call our algorithm Donald. What if civil servants working to detect fraud were largely unable to understand and explain why Donald identified a family for early intervention? Well this would be an issue, but not much more than an incompetent employee. Our societies invented many ways to deal with such things. We might stick with Donald until someone complains and then fire him. Or we might evaluate his task after a series of indicators and check if he does his job. We might hire different Donalds and conduct an independent audit. We might ask people to vote. None of these solutions involve looking inside his brain. And we would certainly not conclude that in the absence of clear and reasoned decisions, it is impossible to hold Donald accountable for his actions.

Understanding an algorithm does not even dispense from regulating it. Let us assume that black people are overrepresented in Donald’s targets, and a journalist claims Donald is racist. Are you surprised that Donald could be racist? People are constantly surprised that algorithms can be. Should we assume that Donald is fair? Of course not. What makes him racist, the way he thinks or the way he acts? Imagine that we can look into Donald’s mind and we find a sound rationale, where race is not a factor of the decision but geographical location is, and it turns out that mostly black people live in the targeted locations. Does it make Donald less racist? Algorithms do not dispense us from dealing with such political questions, and our solutions as a society are not so different for algorithms than for people. Even the fact that entire classes of algorithms might be flawed is not a particular problem. #BlackLivesMatter is scrutinizing an entire class of human beings.

Algorithms are problematic, but their problems do not arise from their opacity. They arise from our democratic institutions not acknowledging their agency. We saw the American Congress question Mark Zuckerberg, but it should have questioned Facebook algorithms first. Algorithms are not so mute, they can designate where responsibility flows. Of course the Congress did not have the expertise to question algorithms, but it was also powerless because it had no practical mean to scrutinize them. Why would we leave beings with such powers out of any jurisdiction? We cannot just let their owners have the exclusivity of their scrutiny. That would be an incredibly naïve mistake for a democracy, a mistake we would never do if their agency was more obvious.

I drew a number of conclusions for myself. I share them below as poorly reflected suggestions that might have in fertility what they lack in robustness.

Scrutiny. We do not leave children unattended. We must not leave algorithms unattended. Who is in charge of watching a given algorithm? Our democratic infrastructures could enforce ways that this question always has an answer. No algorithm should be left out of jurisdiction, so no algorithm should be out of scrutiny.

Accountability. Justice succeeds in dealing with the accountability of human beings, which is a difficult question. We can do it for algorithms if we acknowledge their agency. Like for human beings, accountability naturally circulates to others – algorithms, persons… Like for human beings, sometimes no one is guilty. Algorithms can be evil, but can also make honest mistakes, and sometimes both at once. And they have their own disorders.

Disposability. We can dispose of algorithms and we can proliferate them at a low cost, with or without variations. This makes a huge difference with persons and opens additional opportunities to regulate them. In many situations we use a single algorithm that has been declared the best fit for the task. This might be a consequence of an ideological quest for objective efficiency, but it is not very farsighted. Why not employ a swarm of variants so that we have a chance to observe who performs better? It also multiplies scrutiny, because we have more chances to make the distinction between contingent and essential effects.

Understandability. Though understanding algorithms is generally considered difficult, post-hoc understanding can be much simpler. It is an evaluation of the effects produced by the algorithm, and can be described in a simpler language. In the case of the Danish algorithm, it might be written in terms of over-/under-representation of different populations. This information is important anyway. Because it is easier to share, it can also spark the interest of the public and gather more eyes to watch the algorithm.

It is a political fight. Since opacity is not blocking us, we do not have to wait for a better understanding of deep learning. The situation will only get worse over the long term anyway. Regulating algorithms is a political issue, and technology is not holding us back. Culture might be, though, insofar as the modernist vision of the world tends to be blind about the agency of algorithms, which impairs our imagination on that matter. Also for clarification: though political, this fight obviously has to take place (in part) on a scientific ground, in the academic arena. Algorithm scrutiny starts in the papers describing them, and I have a lot to say on that topic but it will be for another time!

Exploring relations between Pareto and Network Science on Wikipedia with Hyphe

I am currently looking into the power law, where it comes from and which role it plays in network science (scale-free networks are often characterized by a power law distribution of node degree). I used the web crawler Hyphe to investigate Wikipedia pages on that topic, and Gephi to analyze the links (I know these tools well). You will find here a report of that small experiment, unfolding the method and discussing it a bit.

In a nutshell I obtained a network of Wikipedia pages where we see two main clusters, one about Pareto and statistics, the other one about network science. We see a bridge between both, and as expected the power law is part of it. It validated my implicit hypothesis, and I learned a few additional things. Here is it (you may want to open it full screen and zoom to read the labels).

Wikipedia pages about Pareto, the power law and network science.

Note: this visualization has features that you may not have seen before, such as the node halos that clarify their links. It is not straight out of Gephi, I used a Javascript device to produce it. It is a prototype and I will talk about it in another post.

Let’s start with the elephant in the room: did I learn anything non trivial from this image? Yes. Nothing big, but useful things in a research context. The image above is the entry point I present you to have a quick idea of what I write about. My findings did not come out of just a quick read of that network. They came out of the whole process, and I provide details below. Now that you are warned against this common misunderstanding, here is what I obtained from this work:

  • My hypothesis about the power law bridging certain statistical concepts and network science is confirmed. No big surprise, but it is a way to establish it.
  • I get oriented in these concepts and I now have a good idea of my next steps. In particular I know which concepts I must prioritize to investigate the relations between the two knowledge areas.
  • I have a well-described and argued set of pages (the “Pareto-to-network-science” corpus) that I can repurpose later in a science context, because the process behind is transparent, reproducible and open to criticism.
  • For the same reason I have a set of pages defined as bridging my two domains, that I can repurpose later (the “bridge” corpus).
  • I also have a better idea of what the two sides are, and in particular the fact that they are asymmetric. I did not expect that (though I should have).
  • I had other surprises, and I value them highly because it is a not-so-common occasion to have a clue about my own biases:
    • I did not expect the “de Solla Price” bridge
    • I did not expect two sub clusters in network science
  • I can show an image that summarizes the situation, which might come in handy in a number of situations. Like this post.

In the next sections I will expose the protocol I used to get that network, and my analysis. This is more or less what I would write in a paper. However in addition I will evoke my exploration, that took place before the final protocol, and that is usually not shared in a paper.

Protocol

1. Starting lists

We start with two manually curated lists of pages related to the two topics we are studying. The two lists have the same amount of pages arbitrary set to 10. Here are the lists:

Pareto and the power law:

  1. https://en.wikipedia.org/wiki/80-20_law
  2. https://en.wikipedia.org/wiki/Long_tail
  3. https://en.wikipedia.org/wiki/Pareto_distribution
  4. https://en.wikipedia.org/wiki/Pareto_index
  5. https://en.wikipedia.org/wiki/Pareto_principle
  6. https://en.wikipedia.org/wiki/Power-law
  7. https://en.wikipedia.org/wiki/Power_law
  8. https://en.wikipedia.org/wiki/Vilfredo_Pareto
  9. https://en.wikipedia.org/wiki/Zipf%27s_law
  10. https://en.wikipedia.org/wiki/Zipf%E2%80%93Mandelbrot_law

Network science:

  1. https://en.wikipedia.org/wiki/Albert-L%C3%A1szl%C3%B3_Barab%C3%A1si
  2. https://en.wikipedia.org/wiki/Complex_network
  3. https://en.wikipedia.org/wiki/Duncan_J._Watts
  4. https://en.wikipedia.org/wiki/Network_science
  5. https://en.wikipedia.org/wiki/Preferential_attachment
  6. https://en.wikipedia.org/wiki/Scale-free_network
  7. https://en.wikipedia.org/wiki/Scale-free_networks
  8. https://en.wikipedia.org/wiki/Small-world_network
  9. https://en.wikipedia.org/wiki/Small-world_phenomenon
  10. https://en.wikipedia.org/wiki/Small_world_network

At this point there is no crawl or corpus, but since we have seen the final result already, let’s visualize where the starting lists will en up in the final corpus. It will make the analysis easier to understand.

In indigo on the left, the “Pareto Power Law” starting pages.
In red on the right, the “Network Science” starting pages.

2. Crawl

Using the web crawler Hyphe, we define all Wikipedia pages as different web entities and we crawl these 20 pages. We obtain a list of 1874 web entities cited by these, most of which are other Wikipedia pages.

3. Corpus cleaning

We filter out all the web entities cited only by 3 or less of the starting pages, and we remove any web entity that was not a Wikipedia page or was a tool page (categories, help, list of links…) and we crawl them to obtain the hyperlinks between them. At this stage we have 201 Wikipedia pages and the hyperlinks between them.

A quick look into the most linked pages in this corpus shows that many of them are not related to our topics. These “high layer” pages are very generic and they are cited by our two topics just because they are generally cited by many Wikipedia pages. We use a simple criterion to rule them out: we remove any page that does not cite back at least one page of the starting 20. This simple procedure removes half of the pages we had.

Our final corpus consists of 106 Wikipedia pages (and the hyperlinks between them) characterized as:

  • Being cited by at least 4 of the starting pages
  • Citing back one or more of those starting pages
  • Not being a “tool” page (categories, lists of links…)

4. Identifying the two topics

We started with two lists of pages corresponding to two different (but related) topics. We assume that once extended to our final corpus, these two topics are still present and distinguishable. Just looking at the resulting network gives a strong clue that it is indeed the case. However we do not have to rely on a visual interpretation.

We define an extended version of each of the starting set. For each list, the extended version contains all pages that are citing or cited by at least 5 pages of the starting list. In other terms, the extended set contains pages that have a link (citing or being cited) with 50% or the starting pages (of that list). Note that this procedure allows some pages to be on both sides, or on neither, but as the table below shows it is a minority of cases (less than 10%).

Visualizing the “Pareto Power Law” extended set, that we will call “PPL” for brevity, shows that it largely overlaps with the visual cluster on the left, but not totally. A few pages have not been captured by our procedure, while a page on the right has been. That page is “Scale-free network”. Note: I do not attribute the power of being true to visual clusters as opposed to our selection metric, or vice versa. I just observe they generally agree while having a few crucial disagreements.

In darker grey, the “Pareto Power Law” (PPL) extended set

We will also shorten “Network Science” in “NS”. Visualizing the extended set shows that despite being bigger, it was well captured by our selection procedure. No nodes were missed from the visual cluster, but a node clearly placed on the left side has been caught, it is “Power law”.

In darker grey, the “Network Science” (NS) extended set

If you are familiar with Gephi and its epistemic culture, you might wonder why I did not use modularity clustering to delineate the clusters. I will discuss this point later and remain focused on describing the protocol.

5. Identifying the bridge(s)

First of all, we must note that two pages belong to both sets, which in itself can be seen a strong form of bridging. These two pages are “Scale-free network” and “Power law”.

We then identified bridges by looking at nodes that have connections with at least 10% of each set in a given direction (citing or being cited). This way we make the distinction between 4 types of bridge: cited by one extended set and citing the other (in both directions), cited by both, or citing both. Each page can have multiple bridging roles.

If we just look at the number of different bridging roles played by each pages, we get the following distribution:

5 bridging roles
Scale-free network

4 bridging roles
Power law

3 bridging roles
none

2 bridging roles
Complex network
Degree distribution
Preferential attachment
Random graph
Scale-free networks
Small-world network
Social network
Sociology
Watts and Strogatz model

1 bridging role
Computer science
Social networks

6. Visualizing results

In Gephi I used a force-driven placement algorithm, Force Atlas 2, to assign node positions as you have seen above. I used the LinLog mode as it emphasizes the clusters, and its drawback (slow convergence) is not really a problem on such a small network. Once the assignment looked to have converged, and only then, I activated the “no overlap” feature to improve readability. As I expected to have to use this “base map” within a text, I chose to rotate it so that it spreads horizontally.

Analysis

Let’s look at how the links are distributed as a function of our sets. A simple way to do it is to look at density, but this metric is biased by the sizes of clusters and the general density of the network. In order to remove these biases, we will normalize the density the same way we would do with modularity. Like modularity, the normalized densities vary from -0.5 to 1 and the higher, the more links there are compared to the number of links there could be in the context of that network.

The PPL set has an internal normalized density of 0.027, versus an external normalized density of -0.014. Normalized densities are generally low and the important fact here is how internal density dominates external density: there are much more links inside the PPL set than between the PPL set and the rest. The PPL is a cluster in that sense.

Similarly, the NS set has an internal norm. density of 0.117 and external of -0.018. NS is also a cluster, and even better defined.

We see it in the visualization, but it does not rely on the visual representation. We have two well defined topological clusters, with different densities and sizes.

The group of pages defined as “Pareto Power Law” are about statistical laws, probabilities and important figures such as Pareto and Zipf. We suspect that it is part of a much bigger group of pages about statistics, but possibly because the power law is a central concept to that field, our strategy might not have been able to capture that whole group. This set is smaller (30 pages) and less dense (0.027 norm. density) than the “Network Science” set (72 pages and 0.117 norm. density). As a conceptual space it is narrower than network science. We hypothesize that it just the fringe of a larger conceptual space about statistics, and it is possibly not so well defined as a subtopic (a different protocol could test this).

The group of pages defined as “Network Sciences” is larger, better defined, and more interconnected. It is well groomed as a conceptual space, with specific concepts (“Preferential attachment”, “Small-world network”…) intertwined with a body of much more generic concepts (“Internet”, “Social network”…). I am confident that the sub-cluster we identify visually (at the bottom of the cluster) and corresponding to the topic of social networks would be confirmed as such by the same kind of density analysis.

The connexions between the two clusters are multiple. Looking at the direction of links in the different kinds of bridges, we see that there are much more pages where links come from NS and go to PPL than the contrary. This indicates that NS cited the concepts of PPL more than the other way around.

Two pages have a more important bridging role: “Scale-free network” and “Power law”. It is not a big surprise, but I am happy to have established the key role of these two concepts in the circulation of concepts from statistics to network science. The rest of this investigation will rely on a more qualitative approach.

The other bridges that we have identified are a priority for my investigation, and more generally now that the corpus has been scrutinized and that we know that it captures the areas it was intended to, it would be a good idea to read all of these 100 pages. A possible follow-up could be a text analysis of the texts of these pages, and/or their Wikipedia history.

Why not using modularity clustering and betweenness centrality?

Because I did not need to, and it was easier to explain my protocol that way.

This is the alternative protocol: curate a corpus manually so that it captures the two topics. Run modularity clustering in Gephi to find clusters. Run betweenness centrality to find the bridges.

The problem with that protocol is how it depends on abstract concepts for a quite simple thing. Modularity clustering is hard to explain. The visual clustering in the visualization, that is known to be coherent with modularity clustering, is hard to explain. Betweenness centrality is hard to explain. We can explain how it works but not what it does.

Betweenness centrality counts the number of shortest paths, so that a node with a high score is on many shortest paths between other nodes. It means that if you remove such a node, you break many shortcuts, you make distances longer in the network. This is how it works. But not what it does. From my point of view what betweenness centrality does is get both “intuitive bridges” and centers. Centers are nodes that are well connected, connected to other well connected nodes, and that you will find with other metrics such as closeness centrality or just the degree. The “intuitive bridges” are the other ones, left over by other metrics, which are often “in between” clusters. How it works does not tell you what it does, and the justification of the method is obscured.

Sometimes there is no other way. But in this experiment, I designed a strategy that would not rely on these hard-to-explain elements and define our corpus and our bridge in a simpler way. It is just about who cites who, and it still works. But of course I knew it would work beforehand, because I had explored the domain. It was not a bet, as the game was rigged from the start.

Exploration

Now that I shared a more finalized product, I will open my kitchen and expose my methodological notes. They are just slightly redacted to be readable from you. They were not written to be exposed in extenso, so you will have to pardon the style.

I just started with the following three pages:

  • https://en.wikipedia.org/wiki/Power_law
  • https://en.wikipedia.org/wiki/Pareto_distribution
  • https://en.wikipedia.org/wiki/Pareto_principle

First round of exploration: among the Wikipedia pages cited by the 3 entry points, if we except special pages related to scientific content (DOI, ISBN…) and specific to the Wikipedia practice (Main page, Help…) we get 3 other pages: Long Tail, Normal Distribution and Zipf Law. We extend the corpus in that direction.

Second round of exploration, pages cited by 3+ of the above 6. We decide to eliminate list pages (eg. Category: Statistical Law). We find tens of pages about the different statistical distributions. In order to avoid a drifting to statistics in general, we rule them OUT except for the Log-Normal distribution because of the controversy about interpreting real data as power law or log-normal. We just get Benford’s Law, Log-normal distribution, exponential distribution, Generalized Pareto distribution and Zipf-Mandelbrot law.

At this point the corpus is hugely skewed towards statistics. We want to expand it towards networks and management or political science, where Pareto and Juran were influent. We search for specific terms in Hyphe’s Prospect to add sufficiently cited pages:

  • “Pareto” gives Vilfredo Pareto, Pareto efficiency and Pareto index
  • “Law” gives Power-law and 80-20 law (aliases of pages we had)
  • “Network” gives Scale-Free Network and Social Network
  • “Watts”and “Barabasi” add Bianconi-Barabasi model, Barabasi-Albert model, Watts and Strogatz model, Albert-Laszlo Barabasi, Duncan J Watts
  • “Small-World” brings Small-world network, Small-world experiment and Small-world phenomenon and Small world network (alias).

The following round of exploration allows to find the links to “network science”. We focus on pages cited by 6+ of the above. A lot of graph theory appears and like previously, we try to stay focused on complex / scale free / small world networks and their specificities. By this mean we add 25+ pages mostly related to network science.

The obtained network is pretty clear: the power-law is bridging statistics and in particular Pareto’s law with network science. (see file “Pareto Law exploration.gexf”).

After this exploration we can do a more understandable and more straightforward protocol. We will start with two lists of pages, one on Pareto and one on Network science, crawl both and see what comes out and how it bridges.