Thinking with Your Hands

Introduction

My Journey into Gesture

DURING SEASON 4 OF THE CROWN, LADY DIANA, SOON TO BECOME Princess Diana, gets a quick lesson on how to behave in royal society, including how to use—or not use—her hands when she speaks. Her teacher actually ties up her hands with string, saying, “Gestures reveal us, whether we are anxious, or agitated, or cross. It’s best not to give that away. One should never try to show one’s emotions.” Diana’s teacher believes, as most people do, that your gestures reveal your feelings.1

I agree with Diana’s teacher. There is a vast body of work on nonverbal behavior showing that your gestures can reveal your emotions. But that’s not all gestures can do. They can also reveal your thoughts. They can tell the world not only that you’re angry but also what you might be angry about and why. And the thoughts you express in gesture don’t always appear in your speech, as this example illustrates.

A native speaker of Guugu Yimithirr in Queensland, Australia, was out fishing one day when his boat overturned, rolling to the west. After he got back to shore, he recounted the harrowing experience to a group of onlookers. He talked about the boat overturning, and as he did, he produced a rolling motion away from his body. He happened to be facing west, so his gesture rolled from east to west. On another occasion two years later, he was asked to tell the story again, but this time he was, by chance, facing north rather than west. He produced the rolling-over gesture again, but now his hands rolled from right to left. In other words, despite the awkwardness of the movement, his gesture again moved from east to west. He never explicitly said that the boat rolled from east to west. But he didn’t have to—his hands said it for him.2

This book is about the movements we make with our hands when we speak—our gestures—and what they can tell us about our thinking. Etiquette expert Emily Post tells us that to be good conversationalists, we may use our hands to punctuate a point, but excessive gesturing is distracting. In her view, a proper amount of gesture should accompany speech, dictated by etiquette and not by what you want to say. I think Emily Post is wrong about gesturing: the thoughts you have and want to communicate, not your manners, should dictate your gestures.3

One way to make your thoughts known is to talk about them. Another is to write about them. In fact, most people consider language to be the fundamental substance of thought. Some go so far as to say that language is required to have thoughts in the first place—that prelinguistic children don’t really think; nor do language-less animals. We see language as the medium through which we understand, or misunderstand, one another. If you have ever wondered whether your child is developing apace, whether a student understands what you’re trying to teach them, or whether your coworker really agrees with your proposal, chances are you’re looking for answers in what is said. But, as we will see, children’s gestures can tell you whether they are on track; your student’s gestures can tell you whether you are getting through; and your coworkers’ gestures can reveal thoughts they don’t want to say or don’t even know they have. Language is only one window—and maybe not always the best one—into your thoughts. Languages, both spoken and signed, are rule-governed systems that package information into categories. Gesture assumes a less discrete and more pictorial form and, as such, provides a complementary and, I argue, essential vantage point onto the mind.

I focus here on thoughts that are hidden in your hands. You may not be consciously aware of them, but you’re thinking them. It may surprise (and dismay) you to know that, because they are displayed in your hands, these thoughts are visible to others; everyone is able to read the thoughts your hands express. This means an undercurrent of conversation is taking place when we speak, one that is often unacknowledged. If we want to fully communicate with others, and maybe even with ourselves, we need to understand what’s happening with our hands.

I also argue that focusing exclusively on language as the foundation of communication is wrong. It relies on an incomplete understanding of how the mind works, interfering with our ability to fully understand each other and ourselves. For fifty years, I have been studying how and why people gesture, and I have come to believe that gesture not only reveals our attitudes and feelings about ourselves, our listeners, and the conversation between us, but also contributes to the conversation itself. When Diana’s teacher ties up her hands, she keeps Diana not only from betraying her emotions but also from expressing her ideas.

Let’s take an example from March Madness, the annual basketball tournament of the National Collegiate Athletic Association. On March 20, 2022, Gonzaga was playing Memphis and was behind at the end of the first half when the referees called a shooting foul on Drew Timme, a Gonzaga player. His coach, Mark Few, made his displeasure known by furrowing his brow, a facial gesture that revealed his emotions. But it was his hand gesture—an extended point at the screen on the scoreboard where the presumed foul was being replayed—that got him into trouble. The point connected the coach’s emotion with the foul—it may have been obvious that he felt angry, but the gesture made clear that he thought the call was wrong—and now everyone in the stadium knew it. That gesture also earned him a technical foul, which gave Memphis additional free throws and put Gonzaga further behind. As Gene Steratore, a rules analyst and retired referee himself, put it, “You expect some verbal… but when you start pointing, when you start gesturing, physically, that optic is not good for the game.” Hand gestures say what’s on your mind—even when keeping it to yourself would be wiser.

But why do we gesture at all when we already have language? To answer that question, we need to know a little something about how our minds work. Imagine a world in which all forms of language (spoken, signed, written) have been wiped out, along with everyone’s knowledge of these forms, but everything else stays the same. If you were living in this world, you would continue to think, but obviously not in your language. How would you communicate your thoughts?

It may sound like an impossible test to carry out, but my research explores an even more extreme scenario. Would you communicate if you had never been exposed to language and, if so, what would that communication look like? Of course, we cannot ethically deprive a child of linguistic input. We can, however, take advantage of what we might call an experiment of nature—a situation in which a child, often for complex reasons, is not exposed to linguistic input. Consider, for example, a child whose hearing losses are so profound that he cannot hear, and therefore cannot learn, the spoken language that his hearing parents use to communicate. If this child is also not exposed to a sign language, he will lack usable linguistic input. Will the child communicate?

This question followed me through my early years as an undergraduate at Smith College. Smith is in Northampton, Massachusetts, down the street from the Clark School for the Deaf, which was, and still is, a premier school that focuses on training deaf children to produce and understand spoken language. Over time, the field of deaf education has come to realize that not every deaf child can develop this ability, and the school now tries to identify the students who are likely to succeed at learning spoken language. But when I was an undergraduate nearby, many deaf children who went to the Clark School failed in this effort. The rumor, which I confirmed when watching children out of the teacher’s view, was that even children who were having trouble using spoken language in their classes were able to communicate with one another—using their hands. Children who are not exposed to usable linguistic input can communicate, and they use their hands to do so. The next question is whether this communication shares enough properties with the languages used around the world to be considered a language in its own right.

Propelled by what I’d witnessed as an undergraduate, I decided to focus my graduate studies at the University of Pennsylvania on language and how it is developed. At Penn, I met Professor Lila Gleitman and fellow graduate student Heidi Feldman. Lila and Heidi were interested in the same questions I was, so we began by meeting the deaf community and learning sign language ourselves—until we realized that the children we wanted to study were not part of the deaf community. We wanted to study children born to hearing parents who didn’t know sign language and may never have even met a deaf person before their child was born. The parents wanted their children to learn to speak and did not seek out the deaf community.

We began visiting local oral schools for deaf children (like the Clark School), asking if we could observe some of their students. The hearing parents of six deaf children gave us permission to videotape their children as they interacted naturally at home with them and with us. The parents spoke to their children—language that the children could not hear and therefore could not learn. The parents also did not know a sign language—language that their children could have learned but were not exposed to. Like the hypothetical adults in the far-fetched scenario I asked you to imagine who find themselves suddenly language-less, these children are surrounded by the modern world but lack a way to communicate their thoughts. However, the children in our studies never had a language—the adults I asked you to imagine would have had a language before all language disappeared.

We found that each of these deaf children communicated with the hearing people in their worlds and used their hands to do so. The hand movements are called homesigns (because they were created in the home) and the children homesigners. All animal species communicate in some way—bees, ants, and dolphins use sights, smells, and sounds to communicate with each other—so it’s hardly surprising that a human child will communicate even under challenging circumstances.4

The crucial question is whether the hand movements that the deaf children make to communicate resemble human language. To address this question, we compared the deaf children to other children in the early stages of acquiring a conventional language. At the time, there was relatively little research on deaf children learning sign language from their deaf parents, so we focused on hearing children learning spoken language from their hearing parents. We found striking similarities not only in the topics that the deaf children and the hearing children communicated about but also in how the two groups structured their communications. The deaf children’s homesigns were simple—they were children, after all—but they had many of the features found in human language, signed or spoken. In an important sense, homesign looks and acts more like sign language than like the gestures that accompany speech.

One possibility may be nagging at you, as it nagged at me—maybe the deaf children’s hearing parents fashioned the homesigns to overcome the language barrier, and their children copied the signs from their parents. If so, the parents, not the children, would have invented homesign. The only model the deaf children had for their homesigns was the gestures their hearing family members used as they talked to them—known as co-speech gestures. But, importantly, the deaf children’s homesigns did not look like their parents’ gestures. My collaborators and I have studied deaf children from many countries, including the United States, China, Turkey, and Nicaragua, none of whom know one another. Yet they all do the same thing: they build a language from scratch with their hands, without learning the language from their hearing parents. The disparity between the deaf children’s homesigns and their hearing parents’ gestures underscores two important points. First, the deaf children, not their hearing parents, are inventing their homesign systems. Second, homesign looks different from co-speech gesture.

What does homesign look like? Perhaps it looks like mime. Homesigners could communicate information by enacting events as a mime would. As the example below illustrates, world-famous mime Marcel Marceau’s fluid and continuous movements as he mimes eating an apple capture the experience. Mimes attempt to replicate (and maybe even exaggerate) the actual movements made when grabbing an apple and eating it.

But homesigners don’t mime. They don’t replicate the actual movements of an event. Instead, they break a scene into parts and then combine gestures for each part into a structured string. Their gestures look like a string of discrete movements rather than one unbroken movement. Homesign highlights the aspects of eating that are most informative, leaving out subtle details of the act—a point at an apple, followed by an EAT gesture (fingers touching the thumb while jabbing at the mouth). The picture below depicts a real homesigner who happened to be holding a toy hammer while gesturing and, as a result, produced his gestures with the hammer in hand, making those gestures look even less like the act of eating: point with the hammer at the apple; EAT performed with the hammer in hand; point with the hammer at me to invite me to eat an apple. Homesign looks like beads on a string rather than a picture painted in the air and, in this sense, resembles sign language more closely than mime.

Homesign is created by individual children who aren’t exposed to a conventional language. As a result, it reveals the structures that children impose on their communications when they don’t have a language to learn from. Studying homesign strips away the effect of linguistic traditions, the buildup of language changes that accumulate over generations, which allows us to better see how the mind structures language. Some deaf educators in the past assumed that profoundly deaf children who could not learn a spoken language were not able to think (at the time these educators did not consider sign language a legitimate language). The homesigns that deaf children create make clear that this assumption is wrong. Homesigners do think, and they communicate their thoughts. Homesign provides the best evidence we have for properties of mind that humans bring to language.

But most people use their mouths for language. What do they do with their hands when they’re talking? They gesture. Homesigners use their gestures to take on the full burden of communication—as a language. In contrast, speakers use their gestures along with language and as a supplement to it—as co-speech gesture. It’s easy to understand the need for gesture when you don’t have a language, but that doesn’t explain why you gesture when you do have a language.

As we’ve already established, co-speech gestures are a frequent part of communication—even in speakers who have never seen anyone gesture. Individuals who were born blind move their hands just like sighted people do when they talk. You don’t need to have seen anyone gesture in order to gesture yourself. And gesturing happens all over the world, not only in all speaking cultures but also in signing cultures. Signers use their hands for language, and those sign languages share structural properties with spoken language. Like speakers, signers produce gestures along with their language. These co-sign gestures are distinct in form and function from sign language and share many qualities with co-speech gestures. Facts like this convince us that gesture is a pervasive, although often overlooked, human behavior. The facts also hint that language, on its own, may not be capable of expressing the full range of human thought.

The categories involved in language’s rule-governed systems make it easy, even necessary, to express certain types of information. For example, English requires that you choose a verb that agrees with the number of objects you’re talking about. If you say, “The fish is swimming,” you’re clearly talking about one fish. If you say, “The fish are swimming,” you’re talking about more than one fish. The number of fish that are swimming may not be relevant to your conversation, but that doesn’t matter—English requires you to specify whether it’s one fish or more than one.

Singular versus plural verbs make it easy to convey information like quantity, but they don’t help you convey other types of information. That’s where gesture comes in. If you outline a tight circle with your index finger as you say, “The fish are swimming,” you suggest to your listener that the fish are swimming in a bowl. A more expansive gesture might indicate that the fish are swimming in an outdoor environment, like a pond or lake. Gesture can help you convey thoughts that do not fit neatly into the prepackaged units provided by your language.

The ideas that you embed in your gestures clearly reflect your thinking, but those thoughts are rarely explicitly recognized as such—their communicative power is subtle to both speakers and listeners. Ideas that you don’t want to express in speech, that you don’t yet know how to express in speech, or that you generally don’t want to focus on will often appear in your hands. And you won’t necessarily be held accountable for having expressed those ideas in your hands because we consider language, not gesture, to be our primary vehicle of communication.

Imagine a friend who earnestly tells you that he thinks men and women are equally good leaders. But when he talks about men’s leadership skills, he gestures at eye level, and when he talks about women’s leadership skills, he gestures a bit lower, at mouth level. He may think that he believes in the equality of male and female leadership, but his hands have given him away. This isn’t necessarily an instance of trying to conceal his views. Your friend may really believe that he has an egalitarian view of men and women as leaders. The nonegalitarian view displayed in his hands is an implicit, internalized belief, one that he doesn’t realize he holds. Yet his unspoken and unacknowledged belief was expressed and can be read by all, including his listener, who challenges him on his nonegalitarian views. He is offended because he said he believed men and women are equally good leaders. But his listener swears he heard him say otherwise, not fully realizing why.

In the Watergate hearings, witnesses who testified were convinced that Richard Nixon had said incriminating things. But the incriminating words didn’t always show up on the tapes, raising doubts about the witnesses’ testimony. Anything incriminating that was “said” in gesture would have shown up on a video but not on an audio recording of the conversation. Perceptions of what was or wasn’t said, and therefore what is or isn’t the “truth,” were based on speech and gesture for those present in the room but only on speech for those listening to a recording—and those perceptions were likely to differ. You may have heard that when Nixon ran against John F. Kennedy for the presidency for the first time (and lost), people who saw their debates on television thought Kennedy had won, but those who heard the debates over the radio thought Nixon had won. The nonverbal realm, including gesture, influences what observers take from a speech or conversation.5

Gesture does seem to have a special hold on the truth. One of my former graduate students, Amy Franklin, in her dissertation, told adults to describe a series of vignettes from a Tweety Bird cartoon. In half of the descriptions, they were supposed to describe what they saw. In half, they were supposed to mis-describe the event by, for example, saying the cat jumped to the pole when he actually ran to the pole. The adults did as they were told and mis-described the events—at least in speech. But the truth came out—through their hands. They produced a running gesture while saying that the cat jumped.6

Sometimes the stakes regarding what is said or unsaid can be particularly high. Even if you are a trained lawyer, you may not appreciate the power that gestures give you to read your witnesses’ minds. Imagine a child witness describing the person who allegedly abused him. While talking, he makes a GLASSES gesture—he makes a circle with his right index finger and thumb and another with his left and holds the circles up at his eyes. He doesn’t mention glasses when he speaks, which means that glasses are not part of the transcript. When the lawyer next asks, “Was he wearing glasses?” it reads like he’s asking a leading question. But he isn’t—glasses were introduced into the conversation not by the lawyer but by the child, through his gestures. The child hadn’t even realized he had noticed the glasses; he had unconsciously registered them and depicted them later only with his hands. The lawyer didn’t realize that the child had not actually said the word glasses––if he had recognized that the glasses idea came from the child’s gestures, he would have mentioned the gestures explicitly since only the transcript of what is said counts as legal evidence. The lawyer brought up glasses because he thought he heard the child say glasses, but the child had merely gestured.

Communication goes both ways, and lawyers use gesture not only to see into their witnesses’ minds but, more ominously, to influence those minds. Generally, lawyers are supposed to ask open-ended rather than leading questions: “What else was he wearing?” rather than “What color was the hat he was wearing?” But if you ask an open-ended question (“What else was he wearing?”) while making a HAT gesture (tipping your fist toward your head), witnesses are very likely to mention a hat even if there wasn’t one—just as likely as if the lawyer had asked a leading question in speech (“What color was the hat he was wearing?”). Gesture can cue objects or events and, in so doing, bring them to mind. Gesture is already a powerful tool, but in contexts where speech is highly regulated, it can become even more influential.

Of course, gestures don’t exist simply to give your thoughts away. Gestures help you express ideas that are on the cusp of your understanding—ideas that you are in the process of learning. Imagine two identical tall, thin glasses, each containing the same amount of water, which the child looks at and verifies. The water is poured from one of the tall, thin glasses into a short, wide glass, and the child is asked whether the amount of water in the still-full tall glass is the same as the amount of water in the short glass. You or I would say, “Of course.” But at a certain stage, children are convinced that the amount is different. When asked to justify her misguided belief, the child in the drawing below says that the amount is different “cause that’s down lower than that one.” She focuses on the height of the liquid in her speech. But, at the same time, she tells us with her gestures, and only her gestures, that she has noticed the width—she uses two C-shaped hands to indicate the fat width of the shorter glass (illustration on the left) and a single C-shaped hand to indicate the thin width of the taller glass (illustration on the right).

In order to truly understand that the amount of water does not change when it’s poured from a tall, thin glass into a short, wide one, you have to recognize that the larger width in the short glass compensates for its height. We know that the child in the example is not far from comprehending this concept, because when we later give her a lesson in conservation of quantity, she succeeds on the task. Her gestures tell us that she is ready to learn the principle.

Let’s look at a more challenging example with adults. What would you do if you were asked to prove that two molecules are mirror images of one another and cannot be superimposed? These molecules are called stereoisomers, but you wouldn’t know that if you hadn’t studied organic chemistry. So you might not recognize that to check your solution, you would need to rotate one of the molecules around an axis. Think right and left hands, which are not superimposable—you can only cover your left hand with your right hand and have your thumbs align by rotating your right hand. As a result, you don’t say anything about rotating the molecules when you’re asked to explain your solution. However, you produce a rotation gesture with your hands along with your spoken explanation: you do know rotation is required; you just don’t know that you know it! The student pictured below illustrates this point. He says, “You can’t superimpose this” (while pointing at the drawing of the molecule hidden by his body on his left; the middle panel) “on top of that” (while pointing at the drawing of the molecule on his right; the last panel). He shows that he knows about rotation by circling his pointing finger in the air as he begins speaking (the first panel).