Three Roads to Quantum Gravity

Space and time are hard to think about because they are the backdrop to all human experience. Everything that exists exists somewhere, and nothing happens that does not happen at some time. So, just as one can live without questioning the assumptions in one's native culture, it is possible to live without asking about the nature of space and time. But there is at least a moment in every child's life when they wonder about time. Does it go on for ever? Was there a first moment? Will there be a last moment? If there was a first moment, then how was the universe created? And what happened just a moment before that? If there was no first moment, does that mean that everything has happened before? And the same for space: does it go on and on for ever? If there is an end to space, what is just on the other side of it? If there isn't an end, can one count the things in the universe?

I'm sure people have been asking these questions for as long as there have been people to ask them. I would be surprised if the people who painted the walls of their caves tens of thousands of years ago did not ask them of one another as they sat around their fires after their evening meals.

For the past hundred years or so we have known that matter is made up of atoms, and that these in turn are composed of electrons, protons and neutrons. This teaches us an important lesson - that human perception, amazing as it sometimes is, is too coarse to allow us to see the building blocks of nature directly. We need new tools to see the smallest things. Microscopes let us see the cells that we and other living things are made of, but to see atoms we must look on scales at least a thousand times smaller. We can now do this with electron microscopes. Using other tools, such as particle accelerators, we can see the nucleus of an atom, and we have even seen the quarks that make up the protons and neutrons.

All this is wonderful, but it raises still more questions. Are the electrons and the quarks the smallest possible things? Or are they themselves made up of still smaller entities? As we continue to probe, will we always find smaller things, or is there a smallest possible entity? We may wonder in the same way not only about matter but also about space: space seems continuous, but is it really? Can a volume of space be divided into as many parts as we like, or is there a smallest unit of space? Is there a smallest distance? Similarly, we want to know whether time is infinitely divisible or whether there might be a smallest possible unit of time. Is there a simplest thing that can happen?

Until about a hundred years ago there was an accepted set of answers to these questions. They made up the foundations of Newton's theory of physics. At the beginning of the twentieth century people understood that this edifice, useful as it had been for so many developments in science and engineering, was completely wrong when it came to giving answers to these fundamental questions about space and time. With the overthrow of Newtonian physics came new answers to these questions. They came from new theories: principally from Albert Einstein's theory of relativity, and from the quantum theory, invented by Neils Bohr, Werner Heisenberg, Erwin Schrödinger, and many others. But this was only the starting point of the revolution, because neither of these two theories is complete enough to serve as a new foundation for physics. While very useful, and able to explain many things, each is incomplete and limited.

Quantum theory was invented to explain why atoms are stable, and do not instantly fall apart, as was the case for all attempts to describe the structure of atoms using Newton's physics. Quantum theory also accounts for many of the observed properties of matter and radiation. Its effects differ from those predicted by Newton's theory primarily, although not exclusively, on the scale of molecules and smaller. In contrast, general relativity is a theory of space, time and cosmology. Its predictions differ strongly from Newton's mainly on very large scales, so many of the observations that confirm general relativity come from astronomy. However, general relativity seems to break down when it is confronted by the behaviour of atoms and molecules. Equally, quantum theory seems incompatible with the description of space and time that underlies Einstein's general relativity theory. Thus, one cannot simply bring the two together to construct a single theory that would hold from the atoms up to the solar system and beyond to the whole universe.

It is not difficult to explain why it is hard to bring relativity and quantum theory together. A physical theory must be more than just a catalogue of what particles and forces exist in the world. Before we even begin to describe what we see when we look around us, we must make some assumptions about what it is that we are doing when we do science. We all dream, yet most of us have no problem distinguishing our dreams from our experiences when awake. We all tell stories, but most of us believe there is a difference between fact and fiction. As a consequence, we talk about dreams, fiction and our ordinary experience in different ways which are based on different assumptions about the relation of each to reality. These assumptions can differ slightly from person to person and from culture to culture, and they are also subject to revision by artists of all kinds. If they are not spelled out the result can be confusion and disorientation, either accidental or intended.

Similarly, physical theories differ in the basic assumptions they make about observation and reality. If we are not careful to spell them out, confusion can and will occur when we try to compare descriptions of the world that come out of different theories.

In this book we shall be concerned with two very basic ways in which theories may differ. The first is in the answer they give to the question of what space and time are. Newton's theory was based on one answer to this question, general relativity on quite another. We shall see shortly what these were, but the important fact is that Einstein altered forever our understanding of space and time.

Another way in which theories may differ is in how observers are believed to be related to the system they observe. There must be some relationship, otherwise the observers would not even be aware of the existence of the system. But different theories can and do differ strongly in the assumptions they make about the relationship between observer and observed. In particular, quantum theory makes radically different assumptions from those made by Newton about this question.

The problem is that while quantum theory changed radically the assumptions about the relationship between the observer and the observed, it accepted without alteration Newton's old answer to the question of what space and time are. Just the opposite happened with Einstein's general relativity theory, in which the concept of space and time was radically changed, while Newton's view of the relationship between observer and observed was retained. Each theory seems to be at least partly true, yet each retains assumptions from the old physics that the other contradicts.

Relativity and quantum theory were therefore just the first steps in a revolution that now, a century later, remains unfinished. To complete the revolution, we must find a single theory that brings together the insights gained from relativity and quantum theory. This new theory must somehow merge the new conception of space and time Einstein introduced with the new conception of the relationship between the observer and the observed which the quantum theory teaches us. If that does not prove possible, it must reject both and find new answers to the questions of what space and time are and what the relationship between observer and observed is.

The new theory is not yet complete, but it already has a name: it is called the quantum theory of gravity. This is because a key part of it involves extending the quantum theory, which is the basis of our understanding of atoms and the elementary particles, to a theory of gravity. Gravity is presently understood in the context of general relativity, which teaches us that gravity is actually a manifestation of the structure of space and time. This was Einstein's most surprising and most beautiful insight, and we shall have a great deal to say about it as we go along. The problem we now face is (in the jargon of fundamental physics) to unify Einstein's theory of general relativity with the quantum theory. The product of this unification will be a quantum theory of gravity.

When we have it, the quantum theory of gravity will provide new answers to the questions of what space and time are. But that is not all. The quantum theory of gravity will also have to be a theory of matter. It will have to contain all the insights gained over the last century into the elementary particles and the forces that govern them. It must also be a theory of cosmology. It will, when we have it, answer what now seem very mysterious questions about the origin of the universe, such as whether the big bang was the first moment of time or only a transition from a different world that existed previously. It may even help us to answer the question of whether the universe was fated to contain life, or whether our own existence is merely the consequence of a lucky accident.

As we enter the twenty-first century, there is no more challenging problem in science than the completion of this theory. You may wonder, as many have, whether it is too hard - whether it will remain always unsolved, in the class of impossible problems like certain mathematical problems or the nature of consciousness. It would not be surprising if, once you see the scope of the problem, you were to take this view. Many good physicists have. Twenty-five years ago, when I began to work on the quantum theory of gravity in college, several of my teachers told me that only fools worked on this problem. At that time very few people worked seriously on quantum gravity. I don't know if they ever all got together for a dinner party, but they might have.

My advisor in graduate school, Sidney Coleman, tried to talk me into doing something else. When I persisted he told me he would give me a year to get started and that if, as he expected, I made no progress, he would assign me a more doable project in elementary particle physics. Then he did me a great favour: he asked one of the pioneers of the subject, Stanley Deser, to look after me and share my supervision. Deser had recently been one of the inventors of a new theory of gravity called supergravity, which for a few years seemed to solve many of the problems that had resisted all earlier attempts to solve them. I was also lucky during my first year at graduate school to hear lectures by someone else who had made an important contribution to the search for quantum gravity: Gerard 't Hooft. If I have not always followed either of their directions, I learned a crucial lesson from the example of their work - that it is possible to make progress on a seemingly impossible problem if one just ignores the sceptics and gets on with it. After all, atoms do fall, so the relationship between gravity and the quantum is not a problem for nature. If it is a problem for us it must be because somewhere in our thinking there is at least one, and possibly several, wrong assumptions. At the very least, these assumptions involve our concept of space and time and the connection between the observer and the observed.

It was obvious to me then that before we could find the quantum theory of gravity we first had to isolate these wrong assumptions. This made it possible to push ahead for there is an obvious strategy for rooting out false assumptions: try to construct the theory, and see where it fails. Since all the avenues that had been followed up to that time had, sooner or later, led to a dead end, there was ample work to do. It may not have inspired many people, but it was necessary work and, for a time, it was enough.

The situation now is very different. We are still not quite there, but few who work in the field doubt that we have come a long way towards our goal. The reason is that, beginning in the mid-1980s, we began to find ways of combining quantum theory and relativity that did not fail, as all previous attempts had. As a result, it is possible to say that in the last few years large parts of the puzzle have been solved.

One consequence of our having made progress is that all of a sudden our pursuit has become fashionable. The small number of pioneers who were working on the subject a few decades ago have now grown into a large community of hundreds of people who work full time on some aspect of the problem of quantum gravity. There are, indeed, so many of us that, like the jealous primates we are, we have splintered into different communities pursuing different approaches. These go under different names, such as strings, loops, twistors, non-commutative geometry and topi. This over-specialization has had unfortunate effects. In each community there are people who are sure that their approach is the only key to the problem. Sadly, most of them do not understand in any detail the main results that excite the people working on the other approaches. There are even cases in which someone taking one approach does not seem to realize that a problem they find hard has been completely solved by someone taking another approach. One consequence of this is that many people who work on some aspect of quantum gravity do not have a view of the field that is wide enough to take in all the progress that has recently been made towards its solution.

This is perhaps not so surprising - it seems not very different from the present state of cancer research or evolutionary theory. Because the problem is hard, it might be expected that, like climbers confronting a virgin peak, different people would attempt different approaches. Of course, some of these approaches will turn out to be total failures. But, at least in the case of quantum gravity, several approaches seem recently to have led to genuine discoveries about the nature of space and time.

The most compelling developments, taking place as I write, have to do with bringing together the different lessons that have been learned by following the different approaches, so that they can be incorporated into a single theory - the quantum theory of gravity. Although we do not yet have this single theory in its final form, we do know a lot about it, and this is the basis of what I shall be describing in the chapters to come.

I should warn the reader that I am by temperament a very optimistic person. My own view is that we are only a few years away from having the complete quantum theory of gravity, but I do have friends and colleagues who are more cautious. So I want to emphasize that what follows is a personal view, one that not every scientist or mathematician working on the problem of quantum gravity will endorse. I should also add that there are a few mysteries that have yet to be solved. The final stone that finishes the arch has yet to be found.

Furthermore, I must emphasize that so far it has not been possible to test any of our new theories of quantum gravity experimentally. Until very recently it was even believed that the quantum theory of gravity could not be tested with existing technology, and that it would therefore be many years into the future before the theory could be confronted with data from experimental science. However, it now appears that this pessimism may have been short-sighted. Philosophers of science such as Paul Feyerabend have stressed that new theories often suggest new kinds of experiment which may be used to test them. This is very definitely happening in quantum gravity. Very recently, new experiments have been proposed which it appears will make it possible to test at least some of the theory's predictions in the very near future. These new experiments will employ existing technology, but used in surprising ways, to study phenomena that would not have been thought, on the basis of the old theories, to have anything to do with quantum gravity. This is indeed a sign of real progress. However, we must never forget that until the experiments are performed it will always be possible that, as beautiful and compelling as the new theories may seem, they are simply wrong.

During the past few years there has been a growing sense of excitement and confidence among many of the people working on quantum gravity. It is hard to avoid the feeling that we are indeed closing in on the beast. We may not have it in our net, but it feels as if we have it cornered and we have seen, with our flashlights, a few glimpses of it.

Among the many different paths to quantum gravity, most recent traffic, and most progress, has been along three broad roads. Given that quantum gravity is supposed to arise from a unification of two theories - relativity and quantum theory - two of these paths are perhaps not unexpected. There is the route from quantum theory, in which most of the ideas and methods used were developed first in other parts of quantum theory. Then there is the road from relativity, along which one starts with the essential principles of Einstein's theory of general relativity and seeks to modify them to include quantum phenomena. These two roads have each led to a well worked-out and partly successful theory of quantum gravity. The first road gave birth to string theory, while the second led to a seemingly different theory (although with a similar name) called loop quantum gravity.

Both loop quantum gravity and string theory agree on some of the basics. They agree that there is a physical scale on which the nature of space and time is very different from that which we observe. This scale is extremely small, far out of the reach of experiments done with even the largest particle accelerators. It may in fact be very much smaller than we have so far probed. It is usually thought to be as much as 20 orders of magnitude (i.e. a factor of 10²⁰) smaller than an atomic nucleus. However, we are not really sure at which point it is reached, and recently there have been some very imaginative suggestions that, if they bear fruit, will bring quantum gravity effects within the range of present-day experimental capabilities.

The scale where quantum gravity is necessary to describe space and time is called the Planck scale. Both string theory and loop quantum gravity are theories about what space and time are like on this tiny scale. One of the stories I shall be telling is how the pictures that each theory gives us are converging. Not everyone yet agrees, but there is more and more evidence that these different approaches are different windows into the same very tiny world.

Having said this, I should confess my own situation and bias. I was one of the first people to work on loop quantum gravity. The most exhilarating days of my life (apart from the purely personal) were those when, all of a sudden, after months of hard work, we suddenly understood one of our theory's basic lessons. The friends I did that with are friends for life, and I feel equal affection and hope for the discoveries we made. But before then I worked on string theory and, for the past four years, most of my work has been in the very fertile domain that lies between the two theories. I believe that the essential results of both string theory and loop quantum gravity are true, and the picture of the world I shall be presenting here is one that comes from taking both seriously.

Apart from string theory and loop quantum gravity, there has always been a third road. This has been taken by people who discarded both relativity and quantum theory as being too flawed and incomplete to be proper starting points. Instead, these people wrestle with the fundamental principles and attempt to fashion the new theory directly from them. While they make reference to the older theories, these people are not afraid to invent whole new conceptual worlds and mathematical formalisms. Thus, unlike the other two paths, which are trodden by communities of people each large enough to exhibit the full spectrum of human group behaviour, this third path is followed by just a few individuals, each pursuing his or her own vision, each either a prophet or a fool, who prefers that essential uncertainty to the comfort of travelling with a crowd of like-minded seekers.

The journey along the third path is driven by deep, philosophical questions such as, 'What is time?' or, 'How do we describe a universe in which we are participants?' These are not easy questions, but some of the greatest minds of our time have chosen to attack them head-on, and I believe that there has been great progress along this path too. New and, in some cases, quite surprising ideas have been discovered, which I believe are up to the task of answering these questions. I believe that they provide the conceptual framework that is allowing us to take the next step - to proceed to a quantum theory of gravity.

It has also happened that someone on this third road discovered a mathematical structure which at first seemed unconnected to anything else. Such results are often dismissed by the more conservative members of the field as having no possible connection to reality, but these critics have sometimes had to eat their words when the same structure surprisingly turns up on one of the first two roads as the answer to what seemed an otherwise intractable problem. This of course only proves that fundamental questions are hardly ever solved by accident. The people who discovered these structures are among the true heroes of this story. They include Alain Connes, David Finkelstein, Christopher Isham, Roger Penrose and Raphael Sorkin.

In this book we shall walk down all three roads. We shall discover that they are closer than they seem - linked by paths, little used and perhaps a bit overgrown, but passable nevertheless. I shall argue that, if we put together the key ideas and discoveries from all the roads, a definite picture emerges of what the world is like on the Planck scale. My intention here is to display this picture and, by doing so, to show how close we are to the solution of the problem of quantum gravity.

I have tried to aim this book at the intelligent layperson, interested in knowing what is going on at the frontiers of physics. I have not assumed any previous knowledge of relativity or quantum theory. I believe that the reader who has not read anything previously on these subjects will be able to follow this book. At the same time I have introduced ideas from relativity and quantum theory only when they are needed to explain something. I could have said much more about most of the subjects I mention, even at an introductory level. But to have included a complete introduction to these subjects would have resulted in a very long book, and this would have defeated my main goal. Fortunately, there are many good introductions to these subjects for the layperson. At the end of this book there are some suggestions for further reading for those who want to know more.

I must also emphasize that in most cases I have not given proper credit to the inventors of the ideas and discoveries I present. The knowledge we have about quantum gravity has not come out of the head of two or three neo-Einsteins. Rather, it is the result of several decades of intense effort by a large and growing community of scientists. In most cases to name only a few people would be a disservice to both the community of scientists and to the reader, as it would reinforce the myth that science is done by a few great individuals in isolation. To come anywhere near the truth, even about a small field like quantum gravity, one has to describe the contributions of scores of people. There are many more people to name than could be kept track of by the reader encountering these ideas for the first time.

For a few episodes with which I was involved enough to be confident of knowing what happened, I have told the stories of how the discoveries were made. Because people are most interesting when one tells the truth about them, in these cases I am happy to introduce some very human stories to illustrate how science actually gets done. Otherwise I have stayed away from telling the stories of who did what, for I would inevitably have got some of it wrong, in spite of having been a close observer of the subject for the last two decades.

In taking the liberty of telling a few stories I also take a risk, which is that the reader will get the impression that I believe my own work to be more important than the work of other people in the field. This is not true. Of course, I do believe in the approach I pursue in my own research, otherwise I would not have a point of view worth forming a book around. But I believe that I am also in a position to make a fair appraisal of the strengths and weaknesses of all the different approaches, not only those to which I've contributed. Above all else, I feel very privileged to be part of the community of people working on quantum gravity. If I were a real writer, skilled in the art of conveying character, I would like nothing better than to describe some of the people in this world I most admire, from whom I continue to learn, every chance I get. But given my limited skills I shall stick to a few stories about people and incidents I know very well.

When our task is done, someone will write a good history of the search for quantum gravity. Whether this will be in a few years, as I believe, or in many decades, as some of my more pessimistic colleagues expect, it will be a story in which the best human virtues, of courage, wisdom and vision, are mixed with the most ordinary sort of primate behaviour, expressed through the rituals of academic politics. I hope that story will be written in a style that celebrates both sides of our very human occupation.