Modes of Information Flow

In the previous post of the series I made an introduction to Information Theory with the concepts of entropy, mutual information and transfer entropy. Now, using the same framework and moving a little deeper in the development of information theory measures I am going to present the Modes of Information Flow developed by James et al. (2018). These modes try to establish a differentiation in the kind of information that is transfered from one variable to another. We can think about the modes of information as a decomposition of the amount of transfer entropy we studied in the previous post of this blog.

Modes of Information Flow

We already described that the measures of information were created based on probabilistic terms. We noticed that the probabilistic nature of information makes it account for different correlations between variables, in fact, the analogous relationship between entropy and variance, and mutual information and covariance were already introduced. These conditions generate a good starting point to look for mechanisms that explain how the information is transferred between variables.

Information flow is the movement of information from one agent or system to another. Historically, this kind of flow have been measured using the time-delayed mutual information:

(1) $I[ X_{-1}:Y_0]$

The equation (1) presents the mutual information between $X$ ‘s past ( $X_{-1}$ ) and $Y$ ‘s present ( $Y_0$ ). Here the information flow is captured by the time delayed structure, however, it fails to make the distinction of exchanged from shared information due to a common history. In other words, how much of the information of $Y_0$ depends on its own history. To overcome the problem of the shared dependence present in the time delayed mutual information Schreiber (2000) proposed the transfer entropy.

(2) $I[ X_{-1}:Y_0 | Y_{-1}]$

Introducing the condition on $Y$ ‘s past transfer entropy excludes the influence of the common history on $Y_0$ . Unfortunately this is not enough too distinguish the different mechanisms for the information flow from $X$ to $Y$ .

Considering the previous limitations James et al. (2018) proposed that information flow from time series $X$ to $Y$ can take three qualitatively different modes. First, intrinsic flow, which is when the past behavior of $X$ directly predicts the present of $Y$ while the past of $Y$ doesn’t give any information about $Y$ . Second, shared flow, occurs when the present behavior of the time series $Y$ can be inferred either by the past of $X$ or the same $Y$ . Third, synergistic flow is when the past of both time series $X$ and $Y$ are each independent of $Y$ ‘s present but if we combine them, they become predictive of $Y$ . In figure 1 it is possible to see examples of the three modes of information flow.

Figure 3: Modes of information flow. “Modes of information flow”, by R. G. James, B. D. Masante Ayala, B. Zakirov, and J. P. Crutchfield, 2018, Arxiv.

To present the elements in a basic example let’s keep using emotions. According to Hareli and Rafaeli (2008), the process of communicated emotions can be described as an evolution of mutual influence between dyads. The dynamic starts when one component of the dyad expresses a particular emotion, then the other component replies expressing its own emotions, from there the process becomes dialectical. In this setting, the evolution of the emotional content in the conversation follows three possible alternatives, contagion when messages express the same emotion, complement when messages express similar emotions, and inference when the receiver tries to predict the mood from the sender. The model proposed by Hareli and Rafaeli clearly shows that there is a flow of emotional information between the dyad. Let’s say that we are interested in analyzing a conversation between Yvonne and Xavier after review each message, determine its emotional content and create the time series we can determine, for example, measure the transfer entropy between them. But what if we want to know how much of the evolution of emotions during the conversation between Yvonne and Xavier was due to contagion, complement or inference following Hareli and Rafaeli’s model? In this case the modes of information flow are useful. Intrinsic flow can measure the concept of contagion which is the reproduction of the same emotion that a person was exposed to. Shared flow works with the idea of prediction by the receiver considering the expression of the sender which is inference of emotion. Finally, synergistic flow can be analogous to complementary emotions. Obviously the overlap between the modes of information and the model proposed by Hareli and Rafaeli is not perfect but it gives a good proxy to understand the main mechanism of emotional flow in the conversation between Yvonne and Xavier. Figure 4 shows a representation of the application of the modes of information to Yvonne and Xavier conversation.

Figure 4

Analysis

Looking at the formula of time-delayed mutual information $I[X_{-1}:Y_0]$ and considering the flows of figure 1 it is possible to notice that time-delayed mutual information captures intrinsic and shared flows. On the other hand with transfer entropy $I[X_{-1}:Y_0|Y_{-1}]$ by conditioning on $Y_{-1}$ the shared flow is excluded but intrinsic and synergistic flows are still captured. Therefore, as long as the intrinsic flow can be computed the shared and synergistic flows can be derived using the time-delayed mutual information and transfer entropy.

The mathematical derivation of intrinsic mutual information escapes to the scope of this blog but if you are interested you can look at the details on the paper (here). Here I am going to give the intuitive idea that is possible to define a slighty different past $\overline{Y_{-1}}$ that I can condition on it to compute the intrinsic flow. The notation for the intrinsic flow is $I[X_{-1}:Y_0 \downarrow Y_{-1}]$ and we obtain the different flow modes as follows:

Instrinsic flow: $I[X_{-1}:Y_0 \downarrow Y_{-1}]$
Shared flow: $I[X_{-1}:Y_0] - I[X_{-1}:Y_0 \downarrow Y_{-1}]$
Synergistic flow: $I[X_{-1}:Y_0 | Y_{-1}] - I[X_{-1}:Y_0 \downarrow Y_{-1}]$

Another representation to clarify how the interaction between time series allows to define the modes of information flow is presented in figure 2. The image shows the interaction between the variables $\overline{Y_{-1}}$ , $Y_{-1}$ , $X_{-1}$ , and $Y_0$ as a markov chain. From the diagram:

Time delayed mutual information: $I[X_{-1}:Y_0] = a+b+c$
Transfer entropy: $I[X_{-1}:Y_0 | Y_{-1}] = a$
Intrinsic flow: $I[X_{-1}:Y_0 \downarrow Y_{-1}]=a+b$
Shared flow: $(a+b+c) - (a+b) = c$
Synergistic flow: $a - (a+b) = -b$

It is important to notice that the intrinsic mutual information is bound from above by transfer entropy (conditional mutual information), therefore we conclude that $b \leq 0$ .

Limitations

The same as the case of transfer entropy, a key asumption is that the time series used to compute the modes of information flow have to be stationary.

The quantification of the intrinsic information flow is based on the cryptographic secret key agreement rate that is approximated with the intrinsic mutual information. There exists better upper bounds on the secret key agreement rate than the intrinsic mutual information.

A Modes of Information Flow Application

For this example I am going to use the same dataset and a slightly different problem from the application I shared in the Granger Causality post. Again the objective is to analyze the relationship between online communicated emotions in the aftermath of a natural disaster. Using the modes of information flow we want to know what emotions can be predicted by intrinsic, shared or synergistic relationships with other emotions. The dataset collected information from Twitter in the aftermath of the earthquake ocurred in Southern California on July 5, 2019. To assess the emotion each tweet was processed using the Natural Language Understanding tool provided by IBM Watson. This tool analyze the text to assign a value from 0 to for the presence of five emotions: sadness, anger, fear, disgust, and joy.

The procedure has the following steps:

Setup: We load the packages needed to work on Python, plus dataset.
Functions: Using Python packages we create the functions to calculate the modes of information flow.
Modes of information flow: We conduct the calculation of the modes to identify the relationships between the five emotions studied
Figures: We create figures to have a visual perspective of the analysis.

You can access to the dataset in the example here.

View this gist on GitHub

References

Hareli, S., & Rafaeli, A. (2008). Emotion cycles: On the social influence of emotion in organizations. Research in Organizational Behavior, 28, 35–59. https://doi.org/10.1016/j.riob.2008.04.007

James, R. G., Ayala, B. D. M., Zakirov, B., & Crutchfield, J. P. (2018). Modes of Information Flow. ArXiv:1808.06723 [Cond-Mat, Physics:Nlin]. http://arxiv.org/abs/1808.06723

Schreiber, T. (2000). Measuring information transfer. Physical Review Letters, 85(2), 461.

Modes of Information Flow

Pablo M. Flores

ε-Machines

Transfer Entropy

Instrumental Variables

Structural Causal Modeling

Granger Causality

Prediction and Explanation in Social Sciences: Introducing a Set of Useful Methods.