Want a new tool for your belt?

What you see above is a historigraph, or a “map of history”.

Wait, what the hell is a map of history?

This example shows the results of US presidential elections from 1788 to 2020. That’s already been done with other heat map-style graphics, but this one has one key difference that might not be immediately obvious: it fits three dimensions onto a 2D graph.

How does it do that?

Look at the ordering of the states. It’s not alphabetical, or in the order of statehood, or anything linear. You might even think it’s random, until you look at the actual data and see it’s very much NOT random. It’s actually maximally structured, and the patterns are obvious.

But, unlike other heat map-style graphics showing the same data, it does one big thing differently. It arranges the states so that any two states that are close on a map will be close on the Y axis. Where that isn’t possible, it ensures that states that are close together on the axis are also similar politically. On a regular map, you have an X axis of longitude and a Y axis of latitude. On a historigraph, latitude and longitude both go onto the Y axis, allowing you to use the X axis to show time. Three dimensions, one map.

What that means is that the patterns of America’s history jump out at you in a way that doesn’t happen in any other form of visualizing history, whether it’s a timeline, or a series of maps, or a heat map where the states are alphabetically laid out. The historigraph is better than any other format at illustrating the complex web of interactions between societies over time that we know as history.

If you want to learn more about the theory of a historigraph and get a walkthrough of how to read it, open my Substack article in a new tab.

This image will make you understand American history

Either way, keep reading because I have a bigger point to make about how the historigraph is just the first step to a form of data literacy that is absolutely make-or-break for the future of civilized society.

It came from data science…

The concept of the historigraph came about as a result of work done for a sports video analytics company. The product is an AI-driven tool for identifying events in soccer footage like passing, shooting, and much more. I identified a key concern among users of the product: they had been analyzing video footage without the help of AI for a long time. The last thing they wanted to do was to throw their years of expertise out the window and blindly trust the AI. What they did want was a partnership with the AI that augmented their capabilities, not replaced them.

My solution was to visualize what they AI was “thinking” as it identified semantic concepts like events and objects. The interface looked like this:

That heat map at the bottom shows a series of variables identified as significant by the AI, as opposed to ones arbitrarily selected by the user. Those variables (represented by rows) are sorted by an algorithm according to how interrelated any two variables are. That is, if two variables go up or down together, they are closer to each other on the Y axis. This allows patterns in the data to appear to the human eye.

Unbeknownst to me, this technique was just starting to be used in neuroscience wherein MRI data would be analyzed, sorted by region of the brain, and displayed over time, using a similar sorting algorithm so that brain regions that were connected to each other appeared closer on the Y axis. The technical name for this is “time-series clustering”.

Here is an example of such brain data. As you can see there are clumps of yellow that span several rows indicating something was happening in multiple parts of the brain, but not all of the brain. Even more exciting is that diagonal line in the bottom right. That means there’s activity moving across the brain over time. Neat, huh?

Ok, but dumb it down a little

If this all seems arcane, that’s the point. This is the type of stuff that has already been done in scientific circles, but there is no reason it has to stay there. Scientists use correlation-sorted heat maps to make sense of complex data and understand what might otherwise be unfathomable. But our everyday world is becoming so complex as to be unfathomable, so what does the everyday person do about it?

Unfortunately, the answer is usually “nothing at all”. When confronted with the overwhelming complexity of the modern world, most people readily surrender their autonomy to algorithms that think for them. Those algorithms decide what news articles they see, what updates from their friends they read, what movies they watch, what food they eat, maybe even whom they marry. And, as you well know, those algorithms are made by bad people who want to harm you. Trusting Big Tech to make your life decisions is like a rabbit trusting an eagle to decide when it’s safe to leave the warren.

Clearly people need to take back the ability to make their decisions. But the world isn’t going to get any simpler. The amount of information you have to bushwhack through to get to the truth is only getting more vast. The tools available to us to sort through all that information are pathetically inadequate. So maybe it’s time that advanced data visualizations went mainstream.

That’s what the historigraph is for. It’s the bridge between the high-dimensional, correlation-sorted heat maps used by the data science world and the dumbed-down world of text and line graphs that most people inhabit. Trying to introduce anything too exotic to John Q. Public isn’t going to go well, so we need something that can offer a sense of power over a vast and complex reality, but also something without a divide-by-zero learning curve.

To the historigraph and beyond

The line between a historigraph like the USA elections example, and a correlation-sorted time-series heat map like the MRI data is a blurry one. It’s more of a spectrum. This is how I would classify the difference, though:

• A historigraph generally works with well-understood categories like states or countries which have a clear upper limit to their number, and whose relationships to each other are readily apparent. The states in the USA can be ordered on a line primarily by their geographic location, with some additional information on cultural and economic ties needed to decide among different ordering schemes (e.g. should California be placed next to Arizona, Nevada, or neither?). The categories don’t have to be geographical, but in practice those are the simplest to understand.
• By contrast, graphics which exceed the scope of the historigraph tend to have a lot of categories. Like, potentially hundreds, with thousands not out of the question. On such a heat map, you wouldn’t necessarily even see the grid; it would be a continuous gradient of colors. And the relationship between these categories would be ineffable. It would be mainly understood through the correlation of its values over time, as in “categories that goes up or down together are placed next to each other”.

What I propose here is a progressive introduction of high-dimensional data starting with a historigraph and evolving in complexity toward more advanced forms of data visualization that might be implemented in user interfaces of the future.

History education

The historigraph is as important to history as a map is to geography. Rather than communicating history as a mass of tangled timelines, you can present it as a clear visual and a mnemonic.

Historigraphs can be read in two ways. The first is that they can surface interesting regions of space-time. In the image below you can identify the Tammany Hall-driven resurgence of Democrats in the post-Civil War Northern states (“A”) as well as the rejection of President Franklin Roosevelt by the Plains states because his policies failed to adequately help small farmers.

They can also be read at the macro-level, showing the overall “shape of history” in which the overwhelming complexity of history is reduced to a simple shape that is easily processed by the visually-oriented human brain. The shape of the map of history can be as iconic in the brain as the image of Europe and Africa on a globe.

I’d like to see these being used in classrooms, books, and documentaries. They could also be made interactive to allow people to more easily connect meaning to the various shapes they see. This would be the most logical first step to build understanding and acceptance of the concept. Remember that maps too require a form of literacy that was not always widespread. Even today, map literacy (or the lack thereof) is a problem.

Journalism

Once the concept of a historigraph is established for slow-moving timescales and relatively limited categories (like states or countries), the next step would be to use it for more rapidly evolving situations. The historigraph format could be used to quickly get people up to speed. Currently, news outlets rely on timelines which are barely better than plain text, or maps which do not communicate change over time.

Below is an example of how a heat map of coronavirus infection by state turns into a meaningful map when the states are organized properly.

Economics and management

The two examples above use geographical regions as categories (rows) since these are the easiest to understand in terms of spatial relationships to each other. But the next level up could start using more abstract categories like companies (and their stocks), currencies, or product SKUs (to show the fluctuation in prices). The example below shows server utilization at Alibaba. Each tiny row is a different server. Notice how the conventional arbitrary sorting hides the interesting patterns in the loads on the servers. This is a pretty arcane example, but the overall approach could certainly be translated to more consumer-oriented applications.

Seeing the “mind” of AI

AI is only going to become more integral to our daily lives. That much is clear. What isn’t clear is how AI even works. Enough has already been written on the black box of AI, and how what we don’t know can hurt us. The biases found in these systems are already causing problems, and it will only cause exponentially more over time.

The era of personal AI that you own and train is coming fast. Trust me when I say you do not want your AI being trained by a bunch of helicopter-parented Silicon Valley adult babies with god complexes. You’re going to need to learn to see under the hood of your robot assistant, and seeing how it processes the information it takes in is going to be a big part of that.

I have seen firsthand the power of this technique and how it can break open the black box. Ultimately this is where we need to go. It starts with the historigraph.

We need a data-literate population or we’re screwed

Despite the arsenal of tools we have at our disposal to make large amounts of information easily consumable — all the advanced data visualization techniques — most communication uses only the most rudimentary ones. It is still mostly text, photos, and videos. These are all poor at communicating the big picture. There’s just too much happening in the world to try and get the big picture by looking at a bunch of small pictures.

I could easily attribute this predicament to the media either being lazy or ill-intentioned. After all, inefficient communications make people ignorant, and ignorant people are much easier to manipulate. But the sad reality is that the average person’s data literacy is so poor that trying to give them advanced data visualizations would amount to pearls before swine. One need only look back to the surfeit of vapid “infographics” so popular in the early 2010s to be reminded of that.

But data literacy has to start somewhere. Thanks to Google Maps, people know how to read a map, and they do it every day. Some people even enjoy looking at maps for fun. The whole purpose of the historigraph is to co-opt that capability, and that affinity to help people read history like they do space. If that’s an overly ambitious goal, so was the Internet. And here you are reading this.

From my Mont Blanc high to your humanity low

This is the Historigraph. It can bring advanced data visualization to the mainstream. was originally published in UX Planet on Medium, where people are continuing the conversation by highlighting and responding to this story.