Real and unreal news (Notes on attention, fake news and noise #7)

What is the opposite of fake news? Is it real news? What, then, would that mean? It seems important to ask that question, since our fight against fake news also needs to be a fight _for_ something. But this quickly becomes an uncomfortable discussion, as evidenced by how people attack the question. When we discuss what the opposite of fake news is we often end up defending facts – and we inevitably end up quoting senator Moynihan, smugly saying that everyone has a right to their opinions, but not to their facts. This is naturally right, but it ducks the key question of what a fact is, and if it can exist on its own.

Let’s offer an alternative view that is more problematic. In this view we argue that facts can only exist in relationship to each-other. They are intrinsically connected in a web of knowledge and probability, and this web exists in a set of ontological premises that we call reality. Fake news – we could then argue – can exist only because we have lost our sense of a shared reality.

We hint at this when we speak of “a baseline of facts” or similar phrases (this phrase was how Obama referred to the challenge when interviewed by David Letterman recently), but we stop shy off admitting that we ultimately are caught up in a discussion about fractured reality. Our inability to share a reality creates the cracks, the fissures and fragments in which truth disappears.

This view has more troubling implications, and immediately should lead us to also question the term “fake news”, since the implication is clear – something can only be fake if there exists a reality against we can share it. The reason the term “fake news” is almost universally shunned by experts and people analyzing the issue is exactly this: it is used by different people to attack what they don’t like. We see leaders labeling news sources as “fake news” as a way to demarcate against a way to render the world that they reject. So “fake” comes to mean “wrong”.

Here is a key to the challenge we are facing. If we see this clearly – that what we are struggling with is not fake vs real news, but right vs wrong news, we also realize that there are no good solutions for the general problem of what is happening with our public discourse today. What we can find are narrow solutions for specific problems that are well-described (such as actions against deliberately misleading information from parties that deliberately mis-represent themselves), but the general challenge is quite different and much more troubling.

We suffer from a lack of shared reality.

This is interesting from a research standpoint, because it forces to ask the question of how a society constitutes a reality, and how it loses it. Such an investigation would need to touch on things like reality TV, the commodification of journalism (a la Adorno’s view of music – it seems clear that journalism has lost its liturgy). One would need to dig into and understand how truth has splintered and think hard about how our coherence theories of truth allow for this splintering.

It is worthwhile to pause on that point a little: when we understand the truth of a proposition to be its coherence with a system of other propositions, and not correspondence with an underlying ontologically more fundamental level, we open up for several different truths as long as you can imagine a set of coherent systems of propositions built on a few basic propositions – the baseline. What we have discovered in the information society is that the natural size of this necessary baseline is much smaller than we thought. The set of propositions we need to create alternate realities but not seem entirely insane is much smaller than we may have believed. And the cost for creating an alternate reality is sinking as you get more and more access to information as well as the creativity of others engaged in the same enterprise.

There is a risk that we underestimate the collaborative nature of the alternative realities that are crafted around us, the way they are the result of a collective creative effort. Just as we have seen the rise of massive open online courses in education, we have seen the rise of what we could call the massive open online conspiracy theories. They are powered by, and partly created in the same way — with the massive open online role playing games in a nice and interesting middle position. In a sense the unleashed creativity of our collaborative storytelling is what is fracturing reality – our narrative capacity has exploded the last decades.

So back to our question. The dichotomy we are looking at here is not one between fake and real news, or right and wrong news (although we do treat it that way sometimes). It is in a sense a difference between real and unreal news, but with a plurality of unrealities that we struggle to tell apart. There is no Archimedes’ point that allows us to lift the real from the fake, not bedrock foundation, as reality itself has been slowly disassembled over the last couple of decades.

A much more difficult question, then, becomes if we believe that we want a shared reality, or if we ever had one? It is a recurring theme in songs, literature and poetry – the shaky nature of our reality – and the courage needed to face it. In the remarkable song “Right Where It Belongs” this is well expressed by Nine Inch Nails (and remarkably rendered in this remix (we remix reality all the time)):

See the animal in his cage that you built
Are you sure what side you’re on?
Better not look him too closely in the eye
Are you sure what side of the glass you are on?
See the safety of the life you have built
Everything where it belongs
Feel the hollowness inside of your heart
And it’s all right where it belongs

What if everything around you
Isn’t quite as it seems?
What if all the world you think you know
Is an elaborate dream?
And if you look at your reflection
Is it all you want it to be?
What if you could look right through the cracks
Would you find yourself find yourself afraid to see?

What if all the world’s inside of your head?
Just creations of your own
Your devils and your gods all the living and the dead
And you really oughta know
You can live in this illusion
You can choose to believe
You keep looking but you can’t find the ones
Are you hiding in the trees?

What if everything around you
Isn’t quite as it seems?
What if all the world you used to know
Is an elaborate dream?
And if you look at your reflection
Is it all you want it to be?
What if you could look right through the cracks
Would you find yourself, find yourself afraid to see?

The central insight in this is one that underlies all of our discussions around information, propaganda, disinformation and misinformation, and that is the role of our identity. We exist – as facts – within the realities we dare to accept and ultimately our flight into alternate realities and shadow worlds is an expression of our relationship to ourselves.

Man / Machine I: conceptual remarks.

How does man relate to machine? There is a series of questions here that I find fascinating and not a little difficult. I think the relationship between these two concepts also are determinative for a large set of issues that we are debating today, and so we would do well to examine this language game here.

There are, of course, many possibilities. Let’s look at a few.

First, there is the worn out “man is a lesser machine”-theme. The idea here is that machine is a perfect man, and that we should be careful with building machines that can replace us. Or that we should strive ourselves to become machines in order to survive. In this language game machine is perfection, eternity and efficiency, man is imperfection, ephemeral and inefficient. The gleaming steel and ultra-rational machine is a better version of biological man. It is curious to me that this is the conceptual picture that seems strongest right now. We worry about machines taking over, machines taking our jobs and machines turning us all into paper clips (or at least Nick Bostrom does). Because we see them as our superiors in every regard.

In many versions of this conceptual landscape evolution is also a sloppy and inefficient process, creating meat machines with many flaws and short comings — and machines are the end point. They are evolution mastered, and instead of being products of evolution machines produce it as they see fit. Nature is haphazard and technology is deliberate. Any advantage that biology has over technology is seen as easy to design in, and any notion of man’s uniqueness is quickly quashed by specific examples of the machine’s superiority: chess, jeopardy, go, driving —

The basis of this conceptual landscape is that there are individual things machines do better than man, and the conclusion is that machines must be generally better. A car drives faster than a man can run, a computer calculates faster than a man can count and so: machine is generally superior to man.

That does not, of course, follow with any logical necessity. A dog’s sense of smell is better than man, and a dog’s hearing is better than ours. Are dogs superior to man? Hardly anyone would argue that, yet still that same argumentative pattern seems to lead us astray when we talk about machines.

There is not, as far as I am concerned, wrong or right here – but I think we would do well to entertain a broad set of conceptual schemas when discussing technology and humanity, and so am wary of any specific frame being mistaken for the truth. Different frames afford us different perspectives and we should use them all.

The second, then, is that machine is imperfect man. This perspective does not come without its own dangers. The really interesting thing about Frankenstein’s monster is that there is a very real question of how we interpret the monster: as a machine or man? As superior or inferior? Clearly superior on strength, the monster is mostly thought to be stupid and inferior intellectually to its creator.
In many ways this is our secret hope. This is the conceptual schema that gives us hope in the Terminator movies: surely the machine can be beat, it has to have weaknesses that allow us to win over it with something distinctly human, like hope. The machine cannot be perfect, so it has to have a fatal flaw, an imperfection that will allow us to beat it?

The third is that machine is man and man just a machine. This is the La Mettrie view. The idea that there is a distinction between man and machine is simply wrong. We are machines and the question is just how we can be gradually upgraded and improved. There is, in this perspective, a whiff of the first perspective but with an out: we can become better machines, but we will still also be men. Augmentation and transcendence, uploading and cyborgs all inhabit this intellectual scheme.

But here we also have another, less often discussed, possibility. That indeed we are machines, but that we are what machines become when they become more advanced. Here, the old dictum from Arthur C Clarke comes back and we paraphrase: any sufficiently advanced technology is indistinguishable from biology. Biology and technology meld, nature and technology were never distinct or different – technology is just slow and less complex nature. As it becomes more complex, technology becomes alive – but not superior.

Fourth, and rarely explored, we could argue simply that machine and man are as different as man and any tool. There is no convergence, no relationship. A hammer is not a stronger hand. A computer is not a stronger mind. They are different and mixing them up is simply ridiculous. Man is of one category, machine of another and they are incommensurable.

Again: it is not a question of choosing one, but recognizing that they all matter in understanding questions of technology and humanity, I think. More to come.

Notes on attention, fake news and noise #4: Jacques Ellul and the rise of polyphonic propaganda part 1

Jacques Ellul is arguably one of the earlier and most consistent technology critics we have. His texts are due for a revival in a time when technology criticism is in demand, and even techno-optimists like myself would probably welcome that, because even if he is fierce and often caustic, he is interesting and thoughtful. Ellul had a lot to say about technology in books like The Technological Society and The Technological Bluff, but he also discussed the effects of technology on social information and news. In his bleak little work Propaganda: The Formation of Men’s Attitudes (New York 1965(1962)) he examines how propaganda draws on technology and how the propaganda apparatus shapes views and opinions in a society. There are many salient points in the book, and quotes that are worth debating.

That said, Ellul is not an easy read or an uncontroversial thinker. Here is how he connects propaganda and democracy, arguing that state propaganda is necessary to maintain democracy:

“I have tried to show elsewhere that propaganda has also become a necessity for the internal life of a democracy. Nowadays the State is forced to define an official truth. This is a change of extreme seriousness. Even when the State is not motivated to do this for reasons of actions or prestige, it is led to it when fulfilling its mission of disseminating information.

We have seen how the growth of information inevitably leads to the need for propaganda. This is truer in a democratic system than in any other.

The public will accept news if it is arranged in a comprehensive system, and if it does not speak only to the intelligence but to the ‘heart’. This means, precisely, that the public wants propaganda, and if the State does not wish to leave it to a party, which will provide explanations for everything (i.e. the truth), it must itself make propaganda. Thus, the democratic State, even if it does not want to, becomes a propagandist State because of trhe need to dispense information. This entails a profound constitutional and ideological transformation. It is, in effect, a State that must proclaim an official, general, and explicit truth. The State can no longer be objective or liberal, but is forced to bring to the overinformed people a corpus intelligentiae.”

Ellul says, in effect that in a noise society there is always propaganda – the question is who is behind it. It is a grim world view in which a State that yields the responsibility to engage in propaganda yields it to someone else.

Ellul comments, partly wryly, that the only way to avoid this is to allow citizens 3-4 hours to engage in becoming better citizens, and reduce the working day to 4 hours. A solution he agrees is simplistic and unrealistic, it seems, and it would require that citizens “master their passions and egotism”.

The view raised here is useful because it clearly states a view that sometime seems to be underlying the debate we are having – that there is a necessity for the State to become an arbiter of truth (or to designate one) or someone else will take that role. The weakness in this view is a weakness that plagues Ellul’s entire analysis, however, and in a sense our problem is worse. Ellul takes, as his object of study, propaganda from the Soviet Union and Nazi-Germany. His view of propaganda is one that is largely monophonic. Yes, technology still pushes information on citizens, but in 1965 it did so unidirectionally. Our challenge is different and perhaps more troubling: we are dealing with polyphonic propaganda. The techniques of propaganda are employed by a multitude of parties, and the net effect is not to produce truth – as Ellul would have it – but eliminate the conditions for truth. Truth no longer become viable in a set of mutually contradictory propaganda systems, it is reduced to mere feelings and emotions: “I feel this”. “This is my truth”. “This is the way I feel about it”.

In this case the idea that the state should speak too is radically different, because the state or any state-appointed arbiter of truth just adds to the polyphony of voices and provides them with another voice to enter into a polemic with. It fractures the debate even more, and allows for a special category of meta-propaganda that targets the way information is interpreted overall: the idea of a corridor of politically correct views that we have to exist within. Our challenge, however, is not the existence of such a corridor, but the fact that it is impossible to establish a coherent, shared model of reality and hence to decide what the facts are.

An epistemological community must rest on a fundamental cognitive contract, an idea about how we arrive at facts and the truth. It must contain mechanisms of arbitration that are institution in themselves, independent of political decision making or commercial interest. The lack of such a foundation means that no complex social cognition is possible. That in itself is devastating to a society, one could argue, and is what we need to think about.

It is no surprise that I take issue with Ellul’s assertion that technology is at the heart of the problem, but let me at least outline the argument I think Ellul would have to deal with if he was revising his book for our age. I would argue that in a globalized society, the only way we can establish that epistemological, basic foundation to build on is through technology and collaboration within new institutions. I have no doubt that the web could carry such institutions, just like it carries the Wikipedia.

There is an interesting observation about the web here, an observation that sometimes puzzles me. The web is simultaneously the most collaborative environment constructed by mankind and the most adversarial. The web and the Internet would not exist but for the protocol agreements that have emerged as its basis (this is examined and studied commendably in David Post’s excellent book Jefferson’s Moose). At the same time the web is a constant arms race around different uses of this collaboratively enabled technology.

Spam is not an aberration or anomaly, but can be seen as an instance of a generalized, platonic pattern in this space. A pattern that recurs through-out many different domains and has started to climb the semantic layers from simple commercial scams to the semiosphere of our societies, where memes compete for attention and propagation. And the question is not how to compete best, but how to continue to engage in institutional, collaborative and, yes, technological innovation to build stronger protections and counter-measures. What is to disinformation as spamfilters are to unwanted commercial emails? It is not mere spamfilters with new keywords, it needs to be something radically new and most likely institutional in the sense that it requires more than just technology.

Ellul’s book provides a fascinating take on propaganda and is required reading for anyone who wants to understand the issues we are working on. More on him soon.

Notes on attention, fake news and noise #1: scratching the surfaces

What is opinion made from? This seems a helpful question start off a discussion about disinformation, fake news and similar challenges that we face as a society. I think the answer is surprisingly simple: opinion is ultimately made from attention. In order to form an opinion we need to pay attention to issues, and to questions we are facing as a society. Opinion should not be equated with emotion, even if it certainly also draws on emotion (to which we also pay attention), but also needs reasoned view in order to become opinion. Our opinions change, also through the allocation of attention, when we decide to review the reasons underlying them and the emotions motivating us to hold them.

You could argue that this is a grossly naive and optimistic view of opinion, and that what forms opinion is fear, greed, ignorance and malice – and that opinions are just complex emotions, nothing more, and that they have become even more so in our modern society. That view, however, leads nowhere. The conclusion for someone believing that is to throw themselves exasperated into intellectual and physical exile. I prefer a view that is plausible and also allows for the strengthening of democracy.

A corollary of the abovementioned is that democracy is also made from attention – from the allocated time we set aside to form our opinions and contribute to democracy. I am, of course, referring to an idealized and ideal version of democracy in which citizenship is an accomplishment and a duty rather than a right, and where there is a distinct difference between ”nationality” and ”citizenship”. The great empires of the world seem to always have had a deep understanding of this – Rome safeguarded its citizens and citizenship was earned. In contrast, some observers note that the clearest sign of American decline is that US citizenship is devolving into US nationality. Be that as it may — I think that there is a great deal of truth in the conception of democracy as made of opinion formed by the paying of attention.

This leads to a series of interesting questions about how we pay attention today, and what challenges we face when we pay attention. Let me outline a few, and suggest a few problems that we need to study closer.

First, the attention we have is consumed by the information available. This is an old observation that Herbert Simon made in a 1969 talk that he wrote on information wealth and attention poverty. His answer, then, remarkably, was that we need to invest in artificial intelligence to augment attention and allow for faster learning (we should examine the relationship between learning and democracy as well at some point: one way to think about learning is that it is when we change our opinions) – but more importantly he noted that there is an acute need to allocate attention efficiently. We could build on that and note that at high degrees of efficiency of allocation of attention democratic discourse is impossible.

Second, we have learnt something very important about information in the last twenty years or so, and that is that the non-linear value of information presents some large challenges for us as a society. Information – at an abundance – collapses into noise, and the value then can quickly become negative; we need to sift through the noise to find meaning and that creates filter costs that we have to internalize. There is, almost, a pollution effect here. The production of information by each and everyone of us comes with a negative externality in the form of noise.

Third, the need for filters raises a lot of interesting questions about the design of such filters. The word ”filter” comes with a negative connotation, but here I only mean something that allows us to turn noise into information over which we can effectively allocate attention.

That attention plays a crucial role in the information society is nothing new, as we mentioned, and it has been helpfully emphasized by people like Tim Wu, Tristan Harris and others. There is often an edge in the commentary here that suggests that there is a harvesting of attention and monetization of it, and that this in some way is detrimental. This is worth a separate debate, but let it suffice for now that we acknowledge that this can certainly be the case, but also that the fact that attention can be monetized can be very helpful. In fact, good technology converts attention to money at a higher exchange rate and ensures that the individual reaps the benefits from that by finding what he or she is looking for faster, for example. But again: this is worth a separate discussion – and perhaps this is one where we need to dig deeper into the question of the social value of advertising as such – a much debated issue.

So, where does this land us? It seems that we need to combat distraction and allocate attention effectively. What, then, is distraction?


Fake news and disinformation are one form of distraction, and certainly a nefarious one in the sense that such distractions detract from efforts to form opinions in a more serious way in many cases. But there are many other distractions as well. Television, games, gambling and everything else that exists in the leisurespace is in a way a distraction. When Justice Brandeis said that leisure time is the time we need to use to become citizens, he attacked the problem of distraction from a much broader perspective than we sometimes do today. His notion was that when we leave work, we have to devote time to our other roles, and one of the key roles we play is that of the citizen. How many of us devote time every day or week to our citizen role? Is there something we can do there?


The tension between distraction and attention forces us to ask a more fundamental question, and that is if the distraction we are consumed by is forced or voluntary. Put in a different way: assume that we are interested in forming an opinion on some matter, can we do so with reasonable effort or are the distractions so detrimental that the formation of informed and reasoned opinion has become impossible?

At some level this is an empirical question. We can try: assume that you are making your mind up on climate change. Can you use the Internet, use search and social networks in order to form a reasoned opinion on whether climate change is anthropogenic? Or are the distractions and the disinformation out there so heavy that it is impossible to form that opinion?

Well, you will rightly note, that will differ from person to person. This is fair, but let’s play with averages: the average citizen who honestly seeks to make up his or her mind – can they on a controversial issue?

A quick search, a look at Wikipedia, a discussions with friends on a social network — could this result in a reasoned opinion? Quite possibly! It seems that anyone who argues that this is impossible today also needs to carry the burden of evidence for that statement. Indeed, it would be extraordinary if we argued that someone who wants to inform themselves no longer can, in the information society.

There are a few caveats to that statement, however. One is about the will itself. How much do we want to form reasoned opinions? This is a question that risks veering into elitism and von oben perspectives (I can already hear the answers along the lines of ”I obviously do, but others…”) so we need to tread carefully. I do think that there are competing scenarios here. Opinions have many uses. We can use them to advance our public debate, but if we are honest a large use case for opinions is the creation of a group and the cohesion of that group. How many of our opinions do we arrive at ourselves, and how many are we accepting as a part of our belonging to a particular group?

Rare is the individual who says that she has arrived, alone, at all of her opinions. Indeed, that would make no sense, as it would violate Simon’s dictum: we need to allocate attention efficiently and we rely on others in a division of attention that is just a mental version of Adam Smith’s division of labor. We should! To arrive at all your own opinions would be so costly that you would have little time to do anything else, especially in a society that is increasingly complex and full of issues. The alternative would be to have very few opinions, and that seems curiously difficult. Not a lot of people offer that they have no opinion on a subject that is brought up in conversation, and indeed it would almost feel asocial to do that!

So group opinions are rational consequences of the allocation of attention, but how do we know if the group arrives at their opinion in a collectively rational way? It depends on the group, and how it operates, obviously, but at the heart of the challenge her is a sense of trust in the judgments of others.

The opinions we hold that are not ours are opinions we hold because we trust the group that arrived at them. Trust matters much more than we may think in the formation of opinion.


If distraction is one challenge for democratic societies, misallocation of attention is another. The difference is clear: distraction is when we try to but cannot form an opinion. Misallocation is when we do not want to form a reasoned opinion but are more interested in the construction of an identity or a sense of belonging, and hence want to confirm an opinion that we have adopted for some reason.

The forming and confirming of opinion are very different things. In the first case we shape and form our opinion and it may change over time, in the second we simply confirm an opinion that we hold without examining it at all. It is well known that we are prone to confirmation bias and that we seek information that confirms what we believe to be true, and this tendency is one that sometimes wins over our willingness to explore alternative views. Especially in controversial and emotional issues. That is unfortunate, but the question is what the relationship is there with disinformation?

One answer could be this: the cost of confirmation bias falls when there is a ready provision of counter facts to all facts. Weinberger notes that the old dictum that you are entitled to your opinions, but not your facts, has become unfashionable in the information society since there is no single truth anymore. For every fact there is a counter-fact.

Can we combat this state of affairs? How do we do that? Can we create a respository and a source of facts and truths? How do you construct such an institution?

Most of us naturally think of the Wikipedia when we think of something like that – but there is naturally much in the Wikipedia that is faulty or incorrect, and this is not a dig against the Wikipedia, but simply a consequence of its fantastic inclusion and collaborative nature. Also – we know that facts have a half-life in science, and the idea of uncontrovertible fact is in fact very unhelpful and has historically been used rather by theologians than by democrats. But yet, still, we need some institutional response to the flattening of the truth.

It is not obvious what that would be, but worth thinking about and certainly worth debating.


So individual will and institutional truth, ways of spending attention wisely and the sense of citizenship. That is a lot of rather vague hand-waving and sketching, but it is a start. We will return to this question in the course of the year, I am sure. For now, this just serves as a few initial thoughts.

What are we talking about when we talk about algorithmic transparency?

The term ”algorithmic transparency”, with variants and variations, has become more and more common in the many conversations I have with decision makers and policy wonks. It remains somewhat unclear what it actually means, however. As a student of philosophy I find that there is often a lot of value in examining concepts closely in order to understand them, and in the following I wanted to open up a coarse-grained view of this concept in order to understand it further.

At a first glance it is not hard to understand what is meant with algorithmic transparency. Imagine that you have a simple piece of code that manipulates numbers, and that when you enter a series it produces an output that is another series. Say you enter 1, 2, 3, 4 and that the output generated is 1, 4, 9, 16. You have no access to the code, but you can infer that the codde probably takes the input and squares it. You can test this with a hypothesis – you decide to see if entering 5 gives you 25 in response. If it does, you are fairly certain that the code is something like ”take input and print input times input” for the length of the series.

Now, you don’t _know_ that this is the case. You merely believe so and for every new number you enter that seems to confirm the hypothesis your belief may be slightly corroborated (depending on what species of theory of science you subscribe to). If you want to know, really know, you need to have a peek at the code. So you want algorithmic transparency – you want to see and verify the code with your own eyes. Let’s clean this up a bit and we have a first definition.

(i) Algorithmic transparency means having access to the code a computer is running as to have a human be able to verify what it is doing.

So far, so good. What is hard about this, then, you may ask? In principle we should be able to do this with any system and so be able to just verify that it does what it is supposed to and check the code, right? Well, this is where the challenges start coming in.


The first challenge is one of complexity. Let’s assume that the system you are studying has a billion lines of code and that to understand what the system does you need to review all of them. Assume, further, that the lines of code refer to each other in different ways and that there are interdependencies and different instantations and so forth – you will then end up with a situation where access to the code is essentially meaningless, because access does not guarantee verifiability or transparency in any meaningful sense.

This is easily realized by simply calculating the time needed to review a billion line piece of software (note that we are assuming her that software is composed of lines of code – not an obvious assumption as we will see later). Say you need one minute to review a line of code – that makes for a billion minutes, and that is a lot. A billion seconds is 31.69 years, so even if you assume that you can verify a line a second the time needed is extraordinary. And remember that we are assuming that _linear verification_ will be exhaustive – a very questionable assumption.
So we seem to have one interesting limitation here, that we should think about.

L1: Complexity limits human verifiability.

This is hardly controversial, but it is important. So we need to amend and change our definition here, and perhaps think about computer-assisted verification. We end up with something like.

(ii) Algorithmic transparency is achieved by access to the code that allows another system to verify the way the system is designed.

There is an obvious problem with this that should not be scooted over. As soon as we start using code to verify code we enter an infinite regress. Using code to verify code means we need to trust the verifying code over the verified. There are ways in which we can be comfortable with that, but it is worth understanding that our verification now is conditional on the verifying code working as intended. This qualifies as another limit.

L2: Computer assisted verification relies on blind trust at some point.

So we are back to blind trust, but the choice we have is what system we have blind trust in. We may trust a system that we have used before, or that we believe we know more about the origins of, but we still need to trust that system, right?


So, our notion of algorithmic transparency is turning out to be quite complicated. Now let’s add another complication. In our proto-example of the series, the input and output were quite simple. Now assume that the input consistens of trillions of documents. Let’s remain in our starkly simplified model: how do you know that the system – complex – is doing the right thing given the data?

This highlights another problem. What exactly is it that we are verifying? There needs to be a criterion here that allows us to state that we have achieved algorithmic transparency or not. In our naive example above this seems obvious, since what we are asking about is how the system is working – we are simply guessing at the manipulation of the series in order to arrive at a rule that will allow us to predict what a certain input will yield in terms of an output. Transparency reveals if our inferred rule is the right one and we can then debate if that is the way the rule should look. The value of such algorithmic transparency lies in figuring out if the system is cheating in any way.

Say that we have a game. I say that if you can guess what the next output will be and I show you the series 1, 2, 3, 4, and then the output 1, 4, 9, 16. Now I ask you to bet on what the next number will be as I enter 5. You guess 25 and I enter 5 and the output is 26. I win the bet. You require to see the code and the code says: ”For every input print input times input except if input is 5, then print input times input _plus one_”.

This would be cheating. I wrote the code. I knew it would do that. I put a trap in the code, and you want algorithmic transparency to be able to see that I have not rigged the code to my advantage. I am verifying two things: the rule I have inferred is the right one AND that rule is applied consistently. So it is the working of the system as well as its consistency or its lack of bias in anyway.

Bias or consistency is easy when you are looking at a simple mathematical series, but how do you determine consistency in a system that contains a trillion data points and uses a system of over, say, a billion lines of code? What does consistency mean? Here is another limitation, then.

L3: Algorithmic transparency needs to define criteria for verification such that they are possible to determine with access to the code and data sets.

I suspect this limitation is not trivial.


Now, let’s complicate things further. Let’s assume that the code we use generates a network of weights that are applied to decisions in different ways, and that this network is trained by repeated exposure to data and its own simulations. The end result of this process is a weighted network with certain values across it, and perhaps they are even arrived at probabilistically. (This is a very simplified model, extremely so).
Here, by design, I know that the network will look different every time I ”train” it. That is just a function of its probabilistic nature. If we now want to verify this, what we are really looking for is a way to determine a range of possible outcomes that seem reasonable. Determining that will be terribly difficult, naturally, but perhaps it is doable. But at this point we start suspecting that maybe we are engaged with the issue at the wrong level. Maybe we are asking a question that is not meaningful.


We need to think about what it is that we want to accomplish here. We want to be able to determine how something works in order to understand if it is rigged in some way. We want to be able explain what a system does, and ensure that what it does is fair, by some notion of fairness.

Our suspicion has been that what we need to do to do this is to verify the code behind the system, but that is turning out to be increasingly difficult. Why is that? Does that mean that we can never explain what these systems do?
Quite the contrary, but we have to choose an explanatory stance – to draw from a notion introduced by DC Dennett. Dennett, loosely, notes that systems can be described in different ways, from different stances. If my car does not start in the morning I can described this problem in a number of different ways.

I can explain it by saying that it dislikes me and is grumpy, assuming an _intentional_ stance, assuming that the system is intentional.
I can explain it by saying I forgot to fill up on gasoline yesterday, and so the tank is empty – this is a _functional_ or mechanical explanation.
I can explain it by saying that the wave functions associated with the care are not collapsing in such a way as to…or use some other _physical_ explanation of the car as a system of atoms or a quantum physical system.

All explanations are possible, but Dennett and others note that we would do well to think about how we choose between the different levels. One possibility is to look at how economical and how predictive an explanation is. While the intentional explanation is shortest, it gives me now way to predict what will allow me to change the system. The mechanical or functional explanation does -and the physical would take pages on pages to do in a detailed manner and so is clearly uneconomical.
Let me suggest something perhaps controversial: the ask for algorithmic transparency is not unlike an attempt at explaining the car’s malfunctioning from a quantum physical stance.
But that just leaves us with the question of how we achieve what arguably is a valuable objective: to ensure that our systems are not cheating in any way.


The answer here is not easy, but one way is to focus on function and outcomes. If we can detect strange outcome patterns, we can assume that something is wrong. Let’s take an easy example. Say that an image search for physicist on a search engine leads to a results page that mostly contains white, middle-aged men. We know that there are certainly physicists that are neither male or white, so the outcome is weird. We then need to understand where that weirdness is located. A quick analysis gives us the hypothesis that maybe there is a deep bias in the input data set where we, as a civilization, have actually assumed that a physicist is a white, middle-aged man. By only looking at outcomes we are able to understand if there is bias or not, and then form hypothesis about where that bias is introduced. The hypothesis can then be confirmed or disproven by looking at separate data sources, like searching in a stock photo database or using another search engine. Nowhere do we need to, or would we indeed benefit from, looking at the code. Here is another potential limitation, then.

L4: Algorithmic transparency is far inferior to outcome analysis in all sufficiently complex cases.

Outcome analysis also has the advantage of being openly available to anyone. The outcomes are necessarily transparent and accessible, and we know this from a fair amount of previous cases – just by looking at the outcomes we can have a view on whether a system is inherently biased or not, and if this bias is pernicious or not (remember that we want systems biased against certain categories of content, to take a simple example).


So, summing up. As we continue to explore the notion of algorithmic transparency, we need to focus on what it is that we want to achieve. There is probably a set of interesting use cases for algorithmic transparency, and more than anything I imagine that the idea of algorithmic transparency actually is an interesting design tool to use when discussing how we want systems to be biased. Debating, in meta code of some kind, just how bias _should be_ introduced in, say, college admission algorithms, would allow us to understand what designs can accomplish that best. So maybe algorithmic transparency is better for the design than the detection of bias?

Reading Notes I: Tegmark and substrate independence

Tegmark (2017:67) writes “This substrate independence of computation implies that AI is possible: intelligence doesn’t require flesh, blood or carbon atoms.”. How should we read this? The background is that he argues that computation is independent of what we use for hardware and software and what is required is only that the matter we compute in fulfills som very simple conditions like sufficient stability (what would intelligence look like if it was based on gases rather than more solid matter, one could ask – remembering the gas giants in Bank’s novels, by the way – sufficiently large gases may be stable enough to support computation?). But what is more interesting here is the quick transition from computation to intelligence. Tegmark does not violate any of his own assumptions here – he is exceptionally clear about what he thinks intelligence is and builds on a Simonesque notion of attaining goals – but there still seems to be a lot of questions that could be asked about the move from computation to intelligence. The questions that this raises for me are the following:

(i) Is computation the same as intelligence (i.e. is intelligence a kind of computation – and if it is not what is it then?)

(ii) It is true that computation is substrate agnostic, but is not substrate independent. Without any substrate there can be no computing at all, so what does this substrate dependence mean for intelligence? Is it not possible that the nature of the matter used for computation matters for the resultant computation? A very simple example seems to be the idea of computation at different temperatures and what extreme temperatures may lead to (but maybe Tegmark here would argue that this violates the stability condition).

(iii) In a way this seems to be assuming what is to be proven. What Chalmers and others argue is that while computation may be substrate agnostic, cognition or consciousness is not. If there was a way to show that intelligence is substrate specific – only certain classes of matter can be intelligent – what would that look like?

(iv) The question of consciousness is deftly avoided in the quoted sentence, but is there an aspect of observation, consciousness and matter somewhere that seems to matter. I know too little about the role of observation in quantum physics to really nail this down right now, but is it not possible that there exists certain kinds of matter that can observe, and others that cannot?

(v) Even if intelligence is substrate agnostic, as computation, may it not be dependent on certain levels of complexity in the organization of the computation and may it not be the case that these levels of complexity can only be achieved in certain classes of matter? That is, is there an additional criterion for intelligence, in addition to the stability criterion laid out by Tegmark, that needs to be taken into account here?

(vi) What would the world have to be like for intelligence NOT to be substrate agnostic? What would we call the quality that some classes of matter has that others lack and that means that those classes can carry intelligence.

(vii) the close connection between computation and intelligence seems to open itself up to a criticism based on Wittgenstein’s notion of an “attitude to a soul”. Is this just a trite linguistic gripe, or a real concern when we speak about intelligence?

(viii) It seems as if we can detect computation in matter, does this mean that we can detect intelligence just by detecting computation? Clearly not. What is it that we detect when we detect intelligence? This brings us back to the question of tests, of the Turing test et cetera. The Turing test has arguably been passed many times, but is not an interesting test at all – but is there a test for intelligence that can be reduced to a physical measurement? There certainly should be a test for computation that can be easily designed, right?

(ix) Intelligence is a concept that applies to action over a longer time than computation. Does the time factor change the possible equivalence between the concepts?

A lot to think about. Fascinating book so far.

Aspect seeing and consciousness I: What Vampires Cannot Do

In the novel Blindsight by Peter Watts mankind has resurrected vampires (no, not a good idea) – found in the book to be real predators that became extinct. One difference between vampires and humans is that vampires can see both aspects of a Necker cube at the same time – they are able to do hyper-threading and think several thoughts at the same time. In other words, vampires are capable of seeing two aspects of something – or more – simultaneously.

Wittgenstein studies this phenomenon in the second part of Philosophical Investigations, and one interpretation of his remarks is that he sees aspect seeing as a way to show how language can confound us. When we see only one aspect of something we forget that it can equally be something else, and that this is how we are confused. The duck-rabbit is not either duck or rabbit, it is ultimately both, it can be seen as both animals.


But maybe we can learn even more from his discussion of aspect seeing by examining the device Watts uses? The duck-rabbit, the Necker-cube and the old woman/young woman are all interesting examples of how we see one or the other aspect of something. But what would it mean to see both? Let’s assume for the moment that there is a being – a vampire as Watts has it – that can see both aspects at the same time. What would that be like?

Trivially we can imagine _two_ people who look at a Necker cube and see both aspects of it. That is not a hard thing to understand or accept. But a single person seeing both aspects at the same time, that seems more challenging, if not impossible. And maybe this is the thing to explore. What if the following holds true:

(i) Consciousness is limited to a single aspect in the world at a time.

We need to dig further, as this is a very imprecise way to put it, we want to find something more general and distinct to say here. When you are looking at a Necker cube you can only see one aspect at a time, and that is a necessary component of being a “you”. Conscious observation collapses the world to a single aspect out of a multitude of aspects.

That seem trivial. What we are now saying is that in order to see the world, you need to see the world in one specific way at a time. You cannot see it in different ways simultaneously. And that hits on something worth dwelling a bit on – the issue of time in aspect seeing. When you see an aspect of something you construct it in your head over time – it is like having lego pieces and assembling as specific lego construction. Just as you cannot assemble two lego constructions out of the same pieces at the same time you need to limit yourself to one single aspect when several are offered.

This idea, that two simultaneous aspects cannot be constructed out of observation at the same time points to consciousness as “single-threading” rather than “hyper-threading” with Watt’s terminology. But there is no way to imagine a world in which you can make two different simultaneous lego constructions out of lego pieces, that simply is a violation of the way the world is. Now, that opens up the following question:

Q1: Is hyper-threading as described by Watts necessarily impossible in the same way that the simultaneously different lego constructions built from the same pieces are?

This in turn is an interesting question, since it seems to imply that we have a boundary condition for consciousness here – it is necessarily single-threaded, or should be treated as two different consciousnesses in the same body as per our early observation that it is easy to imagine two observers seeing different aspects of the same thing.

We can then develop (i) into:

(ii) Consciousness is necessarily single-threaded.

What would this limitation imply, except that we cannot see a Necker cube in both ways at the same time? It would imply that the necessary reduction of several aspects into a single one is a pre-requisite for us to call something individually conscious.

I suspect there is more here, and want to return to this later, perhaps in a more structured fashion.

“Is there a xeno-biology of artificial intelligence?” – draft essay

One of the things that fascinate me is the connections we can make between technology and biology in exploring how technology will develop. It is a field that I enjoy exploring, and where I am slowly focusing some of my research work and writing. Here is a small piece on the possibility of a xeno-biology of artificial intelligence. All comments welcome to nicklas.berildlundblad at

Autonomy, technology and prediction I: some conceptual remarks

“How would you feel if a computer could predict what you would buy, how you would vote and what kinds of music, literature and food you would prefer with an accuracy that was greater than that of your partner?”

Versions of this question has been thrown at me in different fora over the last couple of months. It contains much to be unpacked, and turns out to be a really interesting entry into a philosophical analysis of autonomy. Here are a few initial thoughts.

  1. We don’t want to be predictable. There is something negative about that quality that is curious to me. While we sometimes praise predictability, we then call it reliability, not predictability. Reliability is a relational concept – we feel we can rely on someone, but predictability is something that has nothing to do with relationships, I think. If you are predictable, you are in some sense a thing, a machine, a simple system. Predictable people lose some of their humanity. Take an example from popular culture – the hosts in Westworld. They are caught in loops that make them easy to predict, and in a key scene Dr Ford expresses his dislike for humanity by saying that the same applies to humans: we are also caught in our loops.
  2. The flip side of that, of course, is that noone would want to be completely unpredictable. Someone who at any point may throw themselves out the window, start screaming, steal a car or disappear into the wilderness to write poetry would also be seen as less than human. Humanity is a concept associated with a mix of predictability and unpredictability. To be human is to occasionally surprise others, but also to be relied upon for some things.
  3. To be predictable is often associated with being easy to manipulate. The connection between the two is not entirely clear cut, since it does not automatically follow from someone being predictable that they can be manipulated.
  4. One way to think about this is to think about the role of predictability in game theory. There are two perspectives here: one is that in order to make credible threats, you need to be predictable in the sense that you will enforce those threats under the circumstances you have defined. There are even techniques for this – you can create punishments for yourself, like the man who reputedly gave his friend 10 000 USD to donate to the US national socialist party (a party the man hated) if his friend ever saw him smoking. Commitment to a cause is nothing else than predictability. Following Schelling, however, a certain unpredictable quality is also helpful in a game, when the rational thing to do is what favors the enemy. One apocryphal anecdote about Herman Kahn, who advocated thermo-nuclear war as a possibility – was that he was paid to do this as to keep the Soviets guessing if the US really could be that crazy to entertain the idea of such a complete war. In games it is the shift between predictability and unpredictability – the bluff! – that matters.
  5. But let’s return to the question. How would we feel? Would it matter how much data the computer needed to make its predictions? Would we feel worse or better if it was easier to predict us? Assume it took only 200 likes from a social network to make these predictions – would that be horrifying or calming to you? The first reaction here may be that we would feel bad if it was in some sense easy to predict us. But let’s consider that: if it took only 200 likes to predict us, the predictions would be thin, and we could change easily. The prediction horizon would be short, and the prediction thin. Let’s pause and examine these concepts, as I think they are important.
  6. A prediction horizon is the length of time for which I can predict something. In predicting the weather, one question is for how long we can predict it – for a day? For a few days? For a year? Anyone able to that – predict the weather accurately for a year – would have accomplished something quite amazing. But predicting the weather tomorrow? You can do that with 50% accuracy by saying that tomorrow will be like today. Inertia helps. The same phenomenon applies to the likes. If you are asked to predict what someone will do tomorrow, looking at what they did today is going to give you a pretty good idea. But it is not going to be a very powerful prediction, and it is not one that in any real sense threatens our autonomy.
  7. A prediction is thin if it concentrates on a few aspects of a predicted system. An example is predicted taste in books or music. Predicting what you will like in a new book or a new piece of music is something that can be done fairly well, but the prediction is thin and does not extend beyond its domain. It tells you nothing about who you will marry or if you will ever run for public office. A thick prediction is cross domains and would enable the predictor to ask a broad set of questions about you that would predict the majority of your actions over the prediction horizon.
  8. There is another concept that we need as well. We need to discuss prediction resolution. The resolution of a prediction is about the granularity of the prediction. There is a difference between predicting that you will like Depeche Mode and predicting that you will like their third album more than the fourth, or that your favorite song will be “Love in itself”. As resolution goes down, prediction becomes easier and easier. The extreme case is the Keynesian quip: in the long run we are all dead.
  9. So, let’s do back to the question about the data set. It obviously would be different if a small data set allowed for a thick, granular prediction across a long horizon or if that same data set just allowed for a short horizon, thin prediction with low resolution. When someone says that they can predict you, you need to think about which one it is – and then the next question becomes if it is better if you have a large data set that does the same.
  10. Here is a possibility: maybe we can be relaxed about thin predictions over short horizons with low resolution based on small data sets (let’s call these a-predictions), because these will not affect autonomy in any way. But thick predictions over long horizons with high resolution, based on very large data sets are more worrying (let’s call these b-predictions).
  11. Here are a few possible hypotheses about these two classes of predictions.
    1. The possibility of a-predictions does not imply the possibility of b-predictions.
    2. Autonomy is not threatened by a-predictions, but by b-predictions.
    3. The cost of b-predictions is greater than the cost of a-predictions.
    4. Aggregated a-predictions do not become b-predictions.
    5. a-predictions are necessary in a market economy for aggregated classes of customers.
    6. a-predictions are a social good.
    7. a-predictions shared with the predicted actor change the probability of the a-predictions.
  12. There are many more possible hypotheses worth examining and thinking about here, but this suffices for a first exploration.

(image: Mako)

Agency and autonomy IV: Agency and religion

So, let’s go back to Wittgenstein’s quote. In the second part of the investigations, now called Philosophy of Psychology, a fragment, chapter iv section 22 he writes: “My attitude towards him is an attitude towards a soul. I am not of the opinion that he has a soul.” In section 23 he continues: “Religion teaches us that the soul can exist when the body has disintegrated. Now do I understand what it teaches? – Of course I understand it – I can imagine various things in connection with it. After all, pictures of these things have been painted. And why should such a picture be only an imperfect rendering of the idea expressed? Why should it not do the same service as the spoken doctrine? And it is the service that counts.”

Here, I believe, Wittgenstein is trying to point out that there is after all something fishy about this notion. The “of course” shows how we slip, how we let ourselves be led astray, and he sort of confirms that in section 25 where he simply states: “The human body is the best picture of the human soul” – something can read to mean that we have an attitude towards a soul to bodies.

Now, the importance of these sections to our examination of agency is two-fold. First it shows how agency is determined by an attitude to a soul, that we in many ways ascribe agency not through analytical approaches, but in the adoption of an attitude (which is quite different from the coming to a conclusion). It seems to suggest that we cannot discover agency through reasoning, but reason only about agency after we have indeed adopted that attitude towards a soul that Wittgenstein points to.

The second observation, however, is equally interesting, and that is the strong relationship between agency and religious thought. We already discussed the relationship between agency and punishment, and it now seems obvious, or likely, that agency actually follows the application of a language game based on concepts like soul. Agency is connected with an old language game, and ultimately perhaps connected with the notion of sin – just as it is connected with the notion of will.

Agency follows duties, and establishes rights. If we ever say of a system that it can sin, we have established agency beyond a doubt.