The deepest advances in science generally hinge on the discovery of fundamentally new concepts and ways of thinking. It's rare that technical wizardry finally solves stubborn old problems in old terms, and more usual that new concepts transform old problems out of existence. No one ever managed a heroic calculation that finally explained the stability of the classical atom; the problem was simply defined away and disappeared with the advent of quantum theory.

Numerous deep issues confront us today. Can we find a solution to the 'measurement problem' of quantum theory, one that gives a coherent view of the entire world without requiring any shifty split between quantum system and classical observer? Is there any way to unify quantum mechanics and gravity? Can we understand how it is that 'emergent' structures — bacteria, for example — can possess their own autonomy, even though their behaviour depends entirely on the underlying laws of biology and chemistry?

How can we find the right concepts to make these puzzles vanish? There is, I'm sure, no certain recipe. But one promising place to look is clearly pure mathematics, which has a track record of finding or inventing new logical structures far in advance of their use in science.

In particular, one interesting direction is 'topos theory', which offers a path to generalize the basic ideas of analysis in terms of sets and functions, the mathematics on which most current science is based. To be clear, topos theory isn't new — it's been around for about 50 years — but it is new or unknown to most scientists. The basic ideas (see John Baez's website for a little more detail: http://math.ucr.edu/home/baez/topos.html) run as follows.

Set theory is typically developed using postulates that concern elements and set membership. The notion of subsets can be used to delimit closed and open sets and so on, with functions entering later as mappings between sets. Membership is the primary concept. But does it need to be? In the early 1960s, mathematician Bill Lawvere began to explore how set theory might change if functions took a primary role, with the concept of membership being demoted.

The ability to break out of old logical structures makes 'topos theory' potentially valuable.

This leads to another branch of mathematics — so-called category theory — which studies collections of objects and arrows or 'morphisms' between them. A category can be as simple as the objects A, B and C, with two arrows mapping A to B and B to C. It turns out that there are many categories. Take the objects as sets, and the arrows as functions between sets, and you have a category known as Set. Consider the objects as vector spaces, and the arrows as one-to-one mappings (isomorphisms) between such spaces, and you have another distinct category again.

By emphasizing functions over membership, Lawvere and others were ultimately led (after “a bunch of work”, as Baez puts it) to invent the concept of a 'topos', which is also a category but one with some special properties that make it quite similar to the category Set. Describing these properties is itself a fairly technical task; a topos has to have a natural notion of limits as well as structures that play the role of exponentiation, among others.

The end result is a generalization of set theory, and the demotion of that theory to just one of an infinite set of possible mathematical structures that might be used in science. Build theories with ordinary set theory and you're led inexorably to certain conclusions. For example, it's obvious that something either belongs to the set A or does not, in which case it must belong to the set NOT A: if you're not taller than six feet, then your height must be six feet or under. But if this is always true in set theory, it needn't be true in some other topos. Some don't satisfy this 'law of the excluded middle' (that something must be either A or NOT A).

It's this ability to break out of old logical structures — in ways we find hard to imagine because those structures are so familiar — that makes topos theory potentially so valuable. For instance, in a paper from a few years ago, physicists Andreas Doering and Chris Isham argued that topos theory might offer a logical framework permitting a natural wedding of quantum theory and gravity (preprint at http://arxiv.org/abs/0803.0417). Most current efforts to do so work within the confines of familiar set theory and use, among other things, the continuum of real or complex numbers. But it's fair to ask if this makes sense. It is entirely possible, after all, that a proper theory of quantum gravity may ultimately demand a view of spacetime that emphasizes a discrete nature, for which the continuum is ill suited.

Doering and Isham proposed that topos theory offers a natural way to generalize quantum theory in a way that avoids assuming a number continuum, yet also preserves some 'realism' in the theory, describing a world with real properties independent of their observation. In the usual theoretical framework, a system has a state s, from within a set S of possible states, and physical properties — energy, volume and so on — that are functions of that state, E(s). Preserving this structure, but replacing the topos of 'set theory' with another alternative topos, you go directly to other kinds of theories, including some in which properties do not have to be described by the field of continuous numbers. Moreover, the 'states' of the system no longer have to be elements of some set, but can be objects of a more general kind; the properties no longer have to be real or complex numbers.

I don't know whether Doering and Isham's project has so far succeeded as well as they had hoped, but it clearly seems to have mapped out fertile ground for further development. Quantum theory has always played havoc with our intuitions, which are so firmly limited by the notions of set theory. Topos theory may show the way to more general, but equally coherent, kinds of intuition. And the same may also be true in many other areas of science where our basic concepts seem to fail. How can we square our apparent free will with the knowledge that everything we do flows out of inexorably deterministic biological chemistry? Perhaps determinism is an illusion, or our autonomy is an illusion; alternatively, maybe our thinking sees a logical contradiction where there needn't be one.

Mathematicians sometimes refer to arguments in category theory as 'general abstract nonsense', a term invented by the founders of the field in the 1940s. It's not actually meant as a term of derision, but of humorous respect for the abstract power of the theory. Topos theory, which evolved from category theory, seems to be widely viewed as the biggest development in pure mathematics of the past 50 years. Scientists should know more about it.