The Attention and Awareness Schema Theories

The attention schema theory was developed in 2012 by Michael Graziano, a neurobiologist from Princeton University [Graziano and W Webb, 2015]. It states that beyond the models of the body (the body schema), the mind (theory of mind), and the world (object permanence), the brain has also evolved to build a model of how the brain itself selects and focuses on certain sensory information. The internal model of this process of attention is called the attention schema (the model of attention). Just like we can use our body schema intuitively (we decide to grab a glass of water instead of thinking about how each individual muscle has to move), we can use the attention schema intuitively to access what we are currently thinking about. In our discussion, we have already encountered the parts making up the attention schema. We referred to them as the working memory, theory of mind, and the stream of consciousness.

Attention schema   The attention schema is a model the brain creates of the process of attention. It allows access to the working memory in order to be able to intervene before an action is taken.

When we are being asked what we are thinking, we cannot reply by listing the biochemical state of every neuron. We can access our model of attention (the working memory), though: what is the current subject or thought on which our brain is focused? This is a brief summary of the results of the selection processes going on in our brain. For example, when we are looking at an apple, our visual system will recognize the concept of “apple.” By accessing the properties connected with the apple (an apple is round, has a certain color, has a stem, tastes sour to sweet, and so on) and the information from our visual system about the apple’s measurements (color, size, location, etc.), our brain can create a model of the apple.

If we are asked to grab the apple and explain our physical relationship to the apple, we access our body schema and the model of the apple. Our working memory contains something along the lines of “I am grasping this red apple with my hand, while my arm is outstretched.” When we are asked what we are currently conscious of, our brain interprets the contents of the working memory as “I am paying attention to the apple.” The “I am paying attention to” part is an interpretation of the brain’s process of selecting thought patterns (the neural competition), the “apple” part is an interpretation of the most prominent thought pattern. To summarize, to answer the simple question “What are you conscious of?” we need to access two models:

• The model of the apple (“What properties does the apple have? Where is the apple in relation to me?”); and
• The model of attention (“What am ‘I’ paying attention to and what is the meaning of ‘I’ in this context?”).

Further self-reflection reveals that we do not have any direct access to any details of how this attention works. We cannot describe intuitively how we decided to look at the apple. Just like we do not intuitively know how we can produce speech with our muscles and tissues, we have no idea about our brain architecture just by thinking about it. Our answer might simply be “Well, I have decided to look at the apple, so I did, and I am perceiving the apple.” But what exactly is the “I”? Depending on our upbringing, we might say that this is our soul, mind, or consciousness, without being able to go into further detail.

And just like we cannot access how this awareness of the self works in other people, we are puzzled by this phenomena in ourselves, too. This intuitive understanding of attention stays with us because our model of attention is built the same way we build models of, for example, the spatial dimensions of objects, our own body schema, or models of our own or other people’s minds (theory of mind). It is not a belief we can explain like a religious belief or scientific theory. Instead, it is on the perceptual level similar to optical illusions. We can know on an abstract level how our brain works, but that has no effect on our feeling of “self.” The way we experience consciousness will stay with us, no matter how much we learn about it—just like we will interpret a combination of red, green, and blue light as “white light,” no matter how much we learn about the electromagnetic spectrum.

The model of attention is not the same as attention, just like the model of a candybar is not sweet. But we are conscious only of that model. It describes something that is actually happening (the process of attention). The actual mechanics of what is happening differ from the simplified model we build intuitively, just like the actual mechanics of our limbs are much more complex than our simplified body schema. We can learn about all the muscles in the arm, but when raising our hand, we will intuitively raise our arm and hold up our hand, not think about activating a dozen muscles that make an arm and hand rise. Or when we want to sing higher, we do not think about the muscles in our throat, we think about singing higher. We intuitively access our simplified body schema.

With the help of the model of attention, we are able to access what is currently going on in the neural competition of the brain. This can help us to make better decisions by enacting corrective measures before actually making a decision and having to face its (potentially negative) consequences. When you feel angry and want to punch someone, your prefrontal cortex can suppress the action and divert it into something more socially acceptable (and less potentially dangerous) like cursing. This is better than seeing yourself acting out those emotions and trying to fix the situation afterwards.

At a neuronal level, the attention schema is very much like the smart thermostat. You can compare the temperature curve of a room with the amount of cognitive resources the brain uses to process an external signal over time. You do not want the room to heat up beyond the desired temperature, just like you do not want to have external signals suppress other brain activity. For example, if you are driving at night, a sudden noise might startle you, but you nonetheless want to keep some resources of your brain reserved for operating your car. You do not want to immediately forget the noise either, but maybe think about it and then take action. For example, maybe you recognize the noise as a burst tire and now decide to find a place to stop the car.

The thermostat-like ability to control the attention to a signal was confirmed in a study comparing the reaction time to visual cues we are aware of with those we are not aware of [Webb et al., 2016]. In the experiment (a so-called “Posner cueing task”), participants look at a computer screen and fixate on the center of the screen. On the screen, there is a box on the left and another box on the right side, with shapes showing up in them. The participant has to press a button once he or she is aware of the shape and the reaction time is measured. By displaying a cue (a dot or small arrow) for a very short time before the shape is shown, the reaction time can be reduced, even though the person did not consciously notice the cue. The study ultimately showed that attention and awareness are two different systems, and that we can better control signals that we are aware of (see Figure 6.29). This in turn means that awareness has a specific function and is not just a byproduct of evolution.

Virtual Prefrontal Cortex

To understand how the attention schema works within the brain, let us imagine that we are playing a computer game in which we control a virtual character. Let us assume that this control is limited insofar as we can make the character do only things that the character’s prefrontal cortex can make the character do. That means we are not steering that person from a first person or even third person perspective. Instead, we have only the ability to suppress or promote actions provided by the working memory by sending signals to the basal ganglia. For example, if we read the entry “I see a candy bar” in the working memory (see Figure 6.30), we can suppress the induced utilization behavior by the parietal lobe and resist eating it. For the decisions about which actions to suppress or to promote, we use our own brain. This allows us to at least to do anything that person is able to do based on what catches his attention. For example, we cannot make the person call a specific number that only we know, but he might see something that reminds him of his mother, leading to a possible desire to call her (which we could promote or suppress). Similarly, we cannot make him speak in a foreign language if he has not learned it before. A passage in a book might catch his attention and we can decide whether or not he will say it out loud, but we (as the prefrontal cortex) cannot make him out of the blue say things of which he is not aware.

In this scenario where we control a virtual character through his prefrontal cortex, what effects would (simulated) neurological conditions like hemispatial neglect have on our interaction with the virtual world? We cannot act upon things that his TPJs did not properly put into the working memory. We might initially not even notice that the character we are controlling in the game has hemispatial neglect as all we are getting in terms of information is the working memory. If we, after a number of encounters with other people and situations, discover that he has hemispatial neglect, we could actively train him (the rest of his brain) to turn his head by rewarding him whenever he does, allowing us to access the option to turn his head more often. Similarly, we could reduce the effort needed to exercise by training him that going for a jog is just part of the normal daily routine. We could not simply tell him to “start to exercise.”

Instead of deciding which element of the working memory to act on in each instance, we can automatize our play by letting the computer decide what to do with the help of models. For example, when we see an apple, we might not want the character to eat it. Maybe we want him to focus on another, more important, task or to realize that he is not really hungry.

We could also expand his prefrontal cortex with social models that contain our ethics to automatize interactions. This could range from abstract rules like “do as you would be done by” to specific rules depending on the situation and people involved. Whether or not a particular social behavior is then promoted or suppressed would depend on that model. Similarly, we will do this with models like risk assessment, long-term planning, and so on. Eventually, he will act within the virtual world as an autonomous avatar.

The Awareness Schema Theory

While a compelling idea, the approach of understanding the brain by programming an artificial prefrontal cortex works well only if the character makes decisions about a present situation. But once he has to take into account the consequences of his actions, he has to imagine the future. The challenge is that each individual prefrontal cortex module cannot do this on its own. For example, when planning to climb up a wall, you have to combine the brain’s resources of the frontal, occipital, and parietal lobes to imagine yourself and your limb positioning for each step of your climb. For this, it is not enough to simply read the working memory and suppress individual entries. Instead, the prefrontal cortex has to initiate the imagination of a future scenario where you are higher up on the wall:

1. Visually identify the first possible step.
2. Check with your parietal lobe whether you can physically reach it.
3. Forward the information to the working memory as a possible next step.
4. Suppress the movement by the prefrontal cortex.
5. Check whether the movement would fit into your plan to reach the top.
6. Return to the working memory the idea of taking this step to reach the top.
7. Imagine a future scenario where you have already taken the step.
8. Visually identify the next possible step.
9. Return to 2.

In other words, the prefrontal cortex recruits other parts of the brain to help with calculation. Those other parts process the data no matter whether they originate from the senses or from a thought-up scenario created by the prefrontal cortex. When you imagine sitting in the grass in the summer and enjoying the view of the mountains, that idea will be just as real at least for some parts of the brain as actually sitting in the grass. This of course depends on your prefrontal cortex’ ability to recruit the visual cortex to create such an image.

Playing chess is another example where you have to “think through” the consequences of your choices before you act. You analyze your current position on the board, have a vague intuition of where to attack, and then calculate move by move how this strategy would turn out for you. Similar to climbing the wall, you would suppress the utilization behavior of just executing the first move that comes to your mind and instead store it in your working memory (“White moves pawn to e4”). But just like not wanting to think only of the first step when climbing, you do not want just to move the chess pieces, you want to think through a particular sequence of moves, one dependent on the other. This means that the output of your analysis of the first move needs to be the input for thinking about the second move, and so on. To achieve this, you would loop through several steps and imagine possible outcomes of the move until you either dismiss it or find that it will improve your situation. Chess players are good at evaluating whether a certain position is generally good, so they can tell relatively easily if another position is better or worse. A simple rule of thumb would be counting the number of pieces you still have on the board versus the number your opponent has on the board. The more chessboard squares you control directly or indirectly with your pieces, the higher your chances are to mount a successful attack.

One difference between climbing a wall and playing a game of chess is that you also have to imagine your opponent’s moves. This makes the game challenging as you also have to use your TPJ and put yourself into the shoes of your opponent and think of possible strategies she might use to win the game. This makes it even more difficult to suppress what your eyes are seeing as you have to mentally rotate the board or remind yourself that her pawns are moving in the opposite direction of yours.

Without the possibility of looping your thoughts through the working memory, playing chess would be much more difficult. Your cortex would have to arrive at a good move in only a single thought-step. While the evolutionary competition among the committees could still come up with a solution to the problem, you would be unable to divide the problem or interrupt thinking about it to return to it at a later time. Learning chess would then require a much more intuitive approach, similar to what you need with the board game “Go” that is far too complicated to calculate deeper than a few moves.

The theory I propose is that the brain cannot just build a model of our attention, but also of our awareness. While attention refers to the neural competition in the brain, awareness refers to the loop of consciousness. Having an awareness schema would allow the brain to modify the contents of the working memory in order to imagine things, recruit other parts of the brain, or think about future scenarios step-by-step. As opposed to the attention schema (“What was put into the working memory?”), the awareness schema addresses the question of how entries in the working memory will stay there.

Modifying the working memory enables the core functions of the prefrontal cortex, namely to focus and pursue goals. Someone without an awareness schema could still use the prefrontal cortex’ abilities to suppress behavior that does not lead to the goal, but he could not pursue goals that are independent of his immediate environment. The utilization behavior of the parietal lobe does not go as far as making use of the models of the prefrontal cortex to think about possible actions in the future; it merely analyzes objects that are immediately available. For example, the function (from our parietal lobe’s perspective) of a piece of chocolate is to be eaten, or the function (from a cat’s parietal lobe’s perspective) of a glass is to be pushed off the table.

Awareness schema   The awareness schema is a model the brain creates of the process of awareness. With the awareness schema, the brain can write to the working memory to influence what the brain will focus on next. It also allows the brain to imagine alternative, past, or future scenarios.

Adding this idea to our diagram results in Figure 6.31. There we see that the prefrontal cortex uses the working memory as an input for its models. The prefrontal cortex returns the results of this calculation to the working memory which in turn feeds it back into the loop of consciousness as an input for the next calculation. This allows the brain to not just suppress external distractions (for example, birds chirping outside), but also to imagine objects in a different condition or scenario. For example, we could imagine an apple in a bowl on a table, or hanging from a tree, or cut into pieces.

Without the awareness schema, you could still pursue strategies by suppressing alternatives, but you could react to the consequences of your actions only after you have executed them. Similar to the thermostat with supervised learning, you may under- or overshoot your desired goal.

In that regard, this ability to modify contents of your working memory is a form of control. To accomplish this modification, the brain uses the awareness schema like the body schema. The latter enables you to intuitively use your body, while the former enables you to intuitively use the loop of consciousness and the mechanisms of focusing on particular thoughts.

Similar to how we can think about the past and future with the help of the awareness schema, we can manipulate the working memory in other ways. For example, when thinking about an alternative scenario (like we did in the example with the climbing), we could imagine not only the external world being different, but also ourselves being different. Try to imagine how people would react to you if you were very muscular or very fragile, and your prefrontal cortex would modify the model you have of yourself. You could even imagine yourself being another person, or you could imagine how other people would react in such a situation. For each case, the underlying principle is always the same: modifying the working memory to make the rest of the brain think about solving this alternative scenario. For example, if Anna thought about Peter joining her climb, she would run through similar steps of thought. But to imagine Peter climbing up the wall, she would need to replace the “Anna” entry with “Peter” in her working memory and think about what steps he would take.

With the awareness schema, we can image how the world might look from another person’s (or animal’s) view. This enables us to have empathy, and to predict other people’s actions. However, damage to the system providing our awareness schema can seriously confuse our thinking process, resulting in, for example, us projecting our own emotions onto others. In contrast, lower brain functions always assume that a perception or emotion is from “self,” but they lack the ability (at least without the neocortex) to imagine what others perceive or feel.

A possible theory of how imagination evolved is that it originated as an emotional coping mechanism. To imagine things, we need to manipulate our working memory which in turn activates, for example, our visual cortex. And the better we can manipulate our working memory, the better we can manipulate our amygdala because the amygdala cannot differentiate very well between “real” sense data and imagined sense data. If we can show emotional restraint and channel our emotions in the right direction, we are better at forming relationships and living peacefully within a tribe—a tribe that can aid us when we are hurt, help us raise our children, or protect us when we are in danger from predators.

Levels of Consciousness

Before we further refine our understanding of consciousness, let us summarize the different models the brain uses:

• Object permanence: a model of the world—“Where are which things in the world that do not belong to my body?”.
• A body schema: a model of our body—“Where are which limbs?” “What belongs to my body and what is not part of my body?”.
• A theory of mind: a model of other people’s minds—“What do other people know?”.
• An attention schema: a model of our own process of attention and that of others—“What am I currently focused on?”.
• An awareness schema of self: a model of our own process of awareness and how we can manipulate our awareness—“What did I focus on, how can I change my focus, and what should I focus on next?”.

We also create an attention schema of other people and animals. The difference compared to the model of attention we create of ourselves is that the sources of information are different. We infer the focus of another person by his gestures, facial expression, eyes, and what he is talking about. For ourselves, our brain can create a richer model of attention because it also has direct access to brain processes.

To improve our understanding of consciousness, we can rank our different abilities and schemas in levels:

1. ‍Sense data: Sense data is registered and pre-processed in the brain.
2. ‍Attention: The ability of the brain to focus on a particular set of sense data using neural competition.
3. ‍Awareness: Something that was put into the working memory including information about “what,” “where,” and “who.”
4. ‍Attention schema: The ability to access the working memory to use it to update the prefrontal cortex’ models and use those models to suppress or promote individual actions.
5. ‍Awareness schema: The ability of the brain to manipulate awareness to think about alternative scenarios. The working memory is used to manipulate future loops of consciousness.
6. ‍Philosophy and science: Abstract knowledge about awareness, enhancing the brain’s ability to create habits and strategies.

Information flows mostly in one direction from level 1 (sense data) down to level 6 (philosophy), meaning that abstract knowledge about a topic will not change your sense perception. Just abstractly thinking about it will not help, but you can develop strategies to put yourself in situations (level 1) to challenge yourself and grow. For example, you can check an optical illusion (for example, the Hering illusion) with a ruler and conclude that the lines are indeed parallel and straight. But this abstract knowledge will not change your sense perception of the optical illusion. Similarly, anything you have learned in this book will not have an immediate effect on your self (or your self-perception). Instead, you first have to put this abstract knowledge into practice by creating new habits in order for your natural behavior to change. Knowledge needs to be integrated not just in your conceptual faculty, but in your entire brain by experiencing cause and effect.

Animals that possess only level 2 of consciousness might have a working memory, too, but they lack the flexibility to take the view of another animal (or person). They have to learn social behavior through trial and error. In animals with level 3 of consciousness, the brain machinery (the TPJ) is required to make the association. If the TPJ is damaged, the information that is unassociated is no longer available to further process in the neocortex through the working memory. It is like saying “Has the key.” But without knowing who has the key, the information is useless. As we have seen with hemispatial neglect, something might still evoke a subconscious reaction even though it is not available in the neocortex: it might still be processed by other brain parts like the amygdala and thalamus. To actually make use of the working memory, we need of course a prefrontal cortex to update our models of the world and use those to suppress other brain parts (for example, resisting a candybar).

Just like the output of any other model, we are not always aware of the output of the model of attention. A moment ago, you were not likely focused on your toes. Now that I have mentioned it, that fact is brought from your body schema to your attention. Your brain is now focusing on that fact and there will be an entry about your toes in your working memory. Similarly, I could ask you the question “What are you currently looking at?” and the output of your model of attention suddenly gets pushed into the working memory: you are currently looking at words in a book. In that regard, it is important to note that we do not have a conscious experience of self all the time. We are not constantly reflecting on ourselves from an observer role. Most of the time, we are engaged in a situation and not very self-aware. When we are self-reflective, we simply ask ourselves repeatedly about our attention schema. This is just like someone else might ask us about how we perceive the world, who we are, what we are currently thinking, or what we are looking at. Similarly, we only become aware of what others are focusing on if we are actively focusing on their behavior. One could call it a “stream of consciousness” if you were to constantly listen to and ask yourself: “How am I doing?” or “What am I thinking at the moment?” We might experience a stream of consciousness when learning something new, especially about ourselves. Whenever we encounter surprising sense data from our environment, it comes into our attention because we need to update our models—which in turn is also something worthwhile to communicate to others to let them know what we plan or what we feel. Being aware of this stream of consciousness requires us to manipulate the working memory, meaning we need to have an awareness schema (level 5). This also allows us to imagine a future and past and to be able to play games like chess, which requires thinking ahead about possible steps our opponent will take and how we could respond.

Finally, philosophy and neurobiology (level 6) can help us to develop better habits, therapies, and even cultural norms (ethics). What stands out in the discussions of level of awareness is the fact that none of the items mentions consciousness. In that regard, it is best to look at consciousness as an umbrella term for all 6 levels mentioned above, combined, with different people and animals having different levels of consciousness. Consciousness also includes the subjective experience which we will discuss in the next section.

Consciousness   Consciousness is an umbrella term for the brain’s abilities to process sense data, focus on something, be aware of something, have the ability to process high-level information (attention schema), be able to control awareness (awareness schema), and reflect on abstract information (philosophy and science).

In terms of the evolution of consciousness, we know that magpies, chimpanzees, elephants, and possibly dolphins hold funeral-like gatherings when one of their group dies. This implies understanding of the past and the future. Likewise, they can also solve complex tasks and use tools. In that regard, more research is necessary. The more we know about other animals with similar brain structures, the more we can uncover about ourselves and our own evolution. Consciousness is not something that suddenly came into existence, but lies on an evolutionary slope from the sponges’ calcium signalling, the Hydras’ processing of basic sense data, the arthropods’ ability to focus, the reptilians’ ability to be aware, the mammals’ (and possibly birds’) attention and awareness schema, and ultimately our ability to conduct neurobiological research and to think about philosophy.

Awareness Schema and Language

A different perspective on our awareness schema is to imagine how our experience of life would be if we did not have it. And it seems that people who have never learned a language presumably never learned to use their awareness schema. After all, if nobody (not even ourselves) asked who we are, what we want to become, and what we are thinking about, why would we feel inclined to use language or think about the past and future? Language could be our “mental limb” which we can “kick” to explore how our mind reacts—just like a baby kicks her feet to build her own body schema. This is supported by the fact that we are using the same parts of our brain to calculate limb positions and to use grammar. Without language, we would still have a working memory that allows other models to access high-level information, but we would be unable to use the information about what we are currently doing or thinking to affect our behavior in the next moment. We would be unable to write instructions into the working memory, our brain would have not been able to build an awareness schema to perceive the past, present, and future. It might even have only a rudimentary attention schema, as a necessary requirement of accessing our working memory is to have a concept of “self” and “other.” Only by putting yourself in relation to your environment can you form a concept of self (and other). The requirement for this is language. As a glimpse into how it might feel to experience the world without an awareness schema, this description from Helen Keller is revealing. She became blind and deaf due to meningitis when she was only 19 months old; she put her experience before she learned a language this way:

Before my teacher came to me, I did not know that I am. I lived in a world that was a no-world. I cannot hope to describe adequately that unconscious, yet conscious time of nothingness. I did not know that I knew aught, or that I lived or acted or desired. I had neither will nor intellect. I was carried along to objects and acts by a certain blind natural impetus. I had a mind which caused me to feel anger, satisfaction, desire. These two facts led those about me to suppose that I willed and thought. I can remember all this, not because I knew that it was so, but because I have tactual memory. It enables me to remember that I never contracted my forehead in the act of thinking. I never viewed anything beforehand or chose it. I also recall tactually the fact that never in a start of the body or a heart-beat did I feel that I loved or cared for anything. My inner life, then, was a blank without past, present, or future, without hope or anticipation, without wonder or joy or faith. —Helen Keller, The World I Live In and Optimism: A Collection of Essays

The Subjective Experience

While we have explored consciousness and its origin, looking at the diagram (Figure 6.31), we still find no evidence for our subjective experience. While the model explains how something can enter subjective experience and how we can think about alternative scenarios, it does not yet explain the subjective experience itself. In philosophy, this question is also called the hard problem of consciousness. If we dismiss the Cartesian theater with a homunculus sitting in our head, then who or what is experiencing what we see, feel, and hear? While we have explained how the data flows through the head with the loop of consciousness, a loop does not magically create a subjective experience.

Hard problem of consciousness   The hard problem of consciousness asks the question where the subjective experience of consciousness comes from. It is “hard” as there are no known ways of detecting this experience objectively without relying on the subjective claims of an individual (or ourselves).

The problem is that the only source that claims that we have an inner experience seems to be us. Even if we assume that our own subjective experience is “real,” the only possibility to inquire about the subjective experience of other people seems to be to ask them and trust that they are telling the truth. Or they could be the victim of a delusion and tell us they have a subjective experience when they do not. After all, we could program a robot to do nothing but answer “I have a subjective experience” and explain in great detail what it is (supposedly) experiencing. In philosophy, such beings without a subjective experience (or more generally, without consciousness) are also called “philosophical zombies.”

Philosophical zombie   A philosophical zombie is a hypothetical being that acts exactly like a human but lacks the inner conscious experience. The existence of a philosophical zombie is used as an argument against consciousness being a mechanistic process: if we are but a mechanistic machine, why would we need a subjective, conscious experience? This argument is addressed by pointing out the necessary evolutionary advantage of having a conscious experience, which is the ability to focus and to explain oneself to others.

Ultimately, the question is if a being that somehow does not have a subjective experience can be distinguished from a being that has one. To examine this, we need to show the evolutionary advantage that subjective experience provides. We have to draw a diagram with one box labeled “subjective experience” and show how this architecture will be able to solve certain problems better than one without subjective experience.

As we have explained all other properties of our subjective experience (attention, awareness, and awareness schema), what is actually left for the subjective experience to do? What are the specific properties of subjective experience that are not yet covered by awareness and attention?

To answer this question, let us take everything we have learned about consciousness and program it into a robot. Then we can discuss whether or not the robot has a subjective experience. One thing is certainly true: if the subjective experience is supernatural, this constructed computer will not have it.

The robot will have the same basic setup as discussed earlier in Figure 6.31. Sense data from the camera eyes arrive at the robot’s visual cortex, are analyzed in its dorsal and ventral streams, and then combined (together with an identification of who is having the perception) in its TPJ. The results are put into its working memory. At any time, the prefrontal cortex of the robot can access the working memory, which contains images, sounds, and abstract information.

You could ask the robot: “Do you see the information in your working memory?” The robot’s temporal lobe will analyze the data, and then it will be processed in the robot’s TPJ as well. The robot can properly interpret the question, recognizing that it should reply whether or not it sees something. Regarding working memory, though, it will inquire what exactly you mean with “working memory” in the context of “seeing.”

You reply: “Let us first agree on what you mean by ‘seeing.”’ You put a red apple on a table. “Here, that is a red apple. If you can identify it as a red apple, you are seeing the apple.”

The robot will connect the idea of “seeing” with information passing through the visual cortex to the working memory. “OK, I am seeing the apple.” What is happening is that the robot is using its attention schema—meaning it can access the working memory, but it does not know that it is accessing its working memory. Just like we are using our body schema but cannot explain what specific muscles we are using when we are, for example, walking.

You reply: “OK, close your eyes and think about an apple. Do you see the apple?”

The question will make the robot deactivate its cameras and activate its memory of an apple, which in turn activates the occipital lobe. Now similar information flows through the robot’s brain as if it were looking at a red apple. Ultimately, the information about the red apple will appear in the working memory and the robot will say “Yes, I see the apple.”

You reply: “But do you actually have a subjective experience of the apple?”

Assuming the robot is philosophically well-versed, it might answer: “What is the difference between seeing, hearing, feeling, touching, tasting, or smelling something and having a subjective experience of it?”

You might respond “Well, when I smell roses, I smell roses. I do not read from my working memory the abstract information that I have just smelled roses. There is more to experiencing something than sense data.”

Indeed, our subjective experience seems to be connected to our sense perception. And indeed, with its interpretation of “seeing,” the robot sees the apple. The challenge we have with this answer is that we usually attribute a much richer experience to our perception of the world than just reading a line like “I see a red apple” from our working memory. Even when accounting for subsequent memories that sense data evoke, our subjective experience still seems to be “richer” than just being abstractly aware of the sense data.

The Limitations that Make Us Human

With our awareness schema, we have the misconception that we are able to be aware of all the details we are perceiving, when in fact, we we can focus only on a small number of details at any given moment. Again, it is like the experience of our body. We think we know what we do when we move the arm, while in reality, our brain orchestrates dozens of muscles to, for example, grab a glass of water.

Outside of our focus, the level of detail decreases rapidly. When focusing on the head of a person in a painting, the rest of the painting becomes a “room” or “garden.” When looking at the flower in the picture, the individual facial features and the facial expression becomes “face.” Ultimately, we should add the source of experience to our working memory as well. A visually imagined apple is something else than if we heard the word “apple.” While pointing to the same thing, the experience is different.

Ourselves, for everything we do not focus on, we drop the measurements and just remember the concept, while a robot might save all the information. The way our brain is optimized, we prefer comparisons (“This smell reminds me of …”) to absolutes (“The molecular composition of this odor is …”). That means we are aware that we have more information. Just because we are aware of something does not necessarily mean we are able to express it. This difference is a significant part of what we call the unconscious or the subjective experience.

Could we create a human-like subjective experience in the robot by limiting its memory?

Following our example, the robot could drop the measurements and process concepts instead of raw sense data. Suddenly, he might seem much more human-like. He will see a small table in a large room, not a 2 by 1 meter wooden table in a 20 by 10 meter room. And when being asked about the table, he can focus in, remove the information of the 20 by 10 meter room and instead load information about the table’s legs, texture, color, etc., and that there is an apple on the table.

When asked about the apple, he will again reduce his focus and move up the tree to more general concepts, removing details about the table, and adding information about the apple (red, juicy, small, …). Hence, we can now ask him:

“Robot, reduce the size of your memory and tell me what you see in this picture.”

“Reducing my memory that describes the world. OK, done. I am seeing a red apple on a table.”

But he is wrong. Below the table, there is also a pair of shoes and they are in his field of view. Let us ask him:

“What about the shoes below the table?”

“Right, now that I am focusing my attention on things below the table, I also see the green shoes. I was not aware that there were shoes there. Previously, that information did not appear in my limited working memory.”

Having only a limited amount of memory, we need to assume that sense data in our brain is not a two-dimensional image or sound recording, but instead represented as a tree-like structure on which we can zoom in or zoom out. We can explore any detail by traversing up the tree to the leaves, or generalize the sense data by moving towards the trunk. Depending on the size of our working memory, we can look at only a limited number of leaves at the same time. The fact that our attention schema can move to any leaf we want at any time provides the illusion that we are looking at the full picture. We are aware that we could examine any part of our working memory in more detail. Evolution has balanced the size and energy costs of the working memory with the benefit. For most tasks, it is enough to focus on one thing in great detail while keeping a rough idea of the rest. For example, you can examine the texture of an apple while also still being aware that you are sitting in a room.

Our thoughts are like a sea with waves where only the largest ones translate into discrete concepts we can communicate.

Even for seemingly simple sense data like the spoken word, we do not just hear the word and then conceptualize it. Spoken words convey emotions, they tell us something about the speaker, his origin (through his dialect), maybe even his health and occupation. All that context information is processed in the brain and is usually available only in your subconscious. Maybe the words themselves remind you of memories. Some words might conjure images in your mind, which again might evoke an emotional reaction.

Brains are not digital computers. Neighboring brain parts can influence each other, despite not being thematically related. For example, some people experience synesthesia, which is a condition in which sense data flows over to other brain parts. They might experience numbers, words, or music also as colors or shapes and vice versa. Even most people without synesthesia associate certain speech sounds with round or spiky shapes (the “bouba-kiki effect”) which points to information overflowing from the temporal lobe to the parietal lobe and the other way around.

Yet another point is that our brain is not just the neocortex. Instead, it is a construct of overlapping responsibilities, usually multiple brain parts dealing in different levels of refinement with similar issues. Perhaps the more primitive systems in our mid- or hindbrain could act in a similar way as the system of attention and awareness in our neocortex that interprets our sense data. In that regard, what we have skipped in our discussion so far is whether or not our subconscious is actually part of our subjective experience. After all, we know from people with blindsight or hemispatial neglect that while they are not consciously aware of what they are seeing, experiments have shown that other parts of the brain produce a “gut feeling.” In that regard, we can assume that any perception produces a gut feeling, even though our conscious experience usually overlaps that feeling. The idea that the subjective perception is equal to reading the working memory ignores the fact that every perception is accompanied by a subconscious feeling we cannot put directly into words. We can be sure that the robot does not have this subconscious feeling—we have never programmed it to have one.

Finally, a lot of information is lost when trying to express what we have in one type of working memory (for example, our visuo-spatial scratchpad) with another (for example, the phonological loop). What we experience as visual sense data might not be fully describable by words, and thus we attribute a richer experience to sense perceptions that we cannot describe as abstract concepts.

In summary, with the attention schema, we have seen that even a simple robot can have a subjective experience, although we would not call it that. For us, beyond the abstract information coming from the working memory, many more (partly subconscious) signals go through our heads which we cannot always describe in words. Our brain is not designed with a clean architecture. Instead, it has parts with overlapping or even duplicate functions.

If we removed our genetic heritage and cleaned up the brain’s architecture to create discrete brain parts, our experience of the world might be similar to that of a robot—reading entries from a limited working memory and organizing sense data according to the attention schema in a knowledge tree. Now, does that mean that the way we experience the world is but a side-effect of evolution? Maybe to a small degree, but we should not forget that the sum of all our parts allowed our ancestors to evolve to become humans. Sometimes, a purely rational choice based on evidence might not have been the choice that led to our survival. As such, how we experience the world is an echo of nearly a billion years of history—something missing in the way we constructed the robot.