Introduction

There has been a long-standing emphasis in speciation research on describing conditions that may facilitate the build-up of premating isolation when diverging populations are undergoing gene flow. The basic issue is as follows: when populations are exchanging genes, they will tend both to lose the population-specific characteristics that mark them as distinct and to have a difficult time evolving further population-specific characters to increase the isolation between them (for example, Mayr, 1963; Felsenstein, 1981). Solutions to this problem range from insights about the type of isolating barriers that might evolve, to details of the genetics of incipient species that might facilitate the evolution of isolation under continuing gene flow.

Both the problem that gene flow poses and the potential conditions that might ease divergence are relevant in a number of situations in which gene flow occurs. These include both de novo divergence during sympatry (sympatric speciation; for example, Mayr, 1963) and the further evolution of isolating barriers during secondary contact (reinforcement, driven by selection against hybridization; see for example, Dobzhansky, 1937; Servedio and Noor, 2003). Geographically, these scenarios span complete sympatry or syntopy, to a narrow zone of contact, to the exchange of migrants between populations with distinct ranges. Although not all solutions to the gene flow problem are general to all situations, there are some common recurring threads.

The progression of speciation through the establishment of species-specific genotypes often ultimately relies on the build-up of linkage disequilibrium between genes involved in premating and postzygotic isolation. Here, I discuss solutions that have been proposed to ease the conditions for speciation with gene flow. The purpose of this article is not to thoroughly review these conditions, but to concentrate on underemphasized aspects and their implications.

One-allele versus two-allele mechanisms

Much has been written on a distinction made by Felsenstein (1981) between one-allele versus two-allele modes of speciation. Speciation occurs through a ‘one-allele’ mechanism if isolation between incipient species results from the spread of a single allele in both populations; this allele might, for example, cause individuals not to migrate, or to prefer to mate with individuals that look like themselves. Isolation is caused by a ‘two-allele’ mechanism if it requires two different alleles to be maintained across the species pair, with one becoming characteristic of each of the incipient species. Examples of two-allele mechanisms of isolation include species-specific mating preferences or alleles that cause early versus late flowering times. Felsenstein (1981) pointed out that speciation should be more difficult with a two-allele than a one-allele mechanism, because with a two-allele mechanism recombination between genes from individuals of different incipient species acts against the formation of species-specific genotypes, formed by the build-up of linkage disequilibrium and involving the premating isolation alleles. Because one-allele mechanisms are not in fact species-specific, recombination has no such effect in those cases. Furthermore, two-allele mechanisms require that a polymorphism be maintained at the locus causing premating isolation, across the ‘population’ consisting of all individuals connected by at least partial gene flow (Servedio and Noor, 2003). The conditions for polymorphism maintenance are obviously stricter than those for the fixation of an allele, all that is required under the one-allele mechanism.

Frequent discussion in the literature of the distinction between these mechanisms may, however, have led to a generally exaggerated impression of the severity of the problems associated with two-allele mechanisms. In fact, Felsenstein (1981, p 135) himself argues that ‘in the case of sympatry, speciation would be nearly impossible unless it were based on genetic variation which could lead to one-allele reproductive isolation.’ Yet models of sympatric speciation demonstrate that it can occur even when two-allele reproductive isolation is involved. Kondrashov and Kondrashov (1999), for example, find that sympatric speciation can occur through a matching of female preferences with marker traits carried by males. The models of Dieckmann and Doebeli (1999) and Doebeli (2005, see also discussion therein) also rely in part (see below) on the divergence of male marker traits, a two-allele mechanism. These studies (and several others, for example, model variants in Kawecki, 1997; Higashi et al., 1999) show that although linkage disequilibrium is certainly needed to establish reproductive isolation in these cases, it is not impossible or even unlikely for it to build to sufficient levels, despite recombination.

Reinforcement, as a process, should occur more easily than sympatric speciation because much of the divergence between populations required for long-term persistence has evolved in allopatry (Kirkpatrick and Ravigné, 2002). Given that two-allele models of successful sympatric speciation are not uncommon, similarly successful two-allele models of reinforcement would also be expected. Reinforcement through two-allele mechanisms has indeed been demonstrated repeatedly under wide range of conditions (for example, Liou and Price, 1994; Payne and Krakauer, 1997; Servedio and Kirkpatrick, 1997; Servedio, 2000, 2004; Kirkpatrick, 2001). Servedio (2000) showed that tightening recombination rates between population-specific preference alleles and previously established population-specific mating cues does indeed facilitate reinforcement, but even free recombination does not prevent it when selection against hybrids is sufficiently strong.

In both of these sets of models, recombination does not prevent the build-up of linkage disequilibrium, although it does slow its accumulation by degrading it. This degradation may be enough to allow other evolutionary forces to overcome the build-up of linkage disequilibrium and prevent speciation, but it is not recombination per se that performs this role. Moreover, it should not be forgotten that the development of premating isolation through one-allele, as well as two-allele, mechanisms is ultimately dependent on the build-up of linkage disequilibrium between the premating isolation allele and the population-specific alleles in the system, unless there is a direct viability or fecundity advantage to the premating isolation allele (that is, it is favored by ‘direct’ selection). When premating isolation depends on the spread of a new allele (for example, an allele for reduced migration or for self-referent phenotype matching), this linkage disequilibrium, combined with selection on population-specific alleles, provides the means by which the allele causing premating isolation may spread (see Table 1 in Servedio, 2000).

In certain scenarios where speciation occurs through a two-allele mechanism, recombination may not be a factor reducing the likelihood of speciation. Take, for example, the case in which assortative mating is an ancestral trait, and speciation relies on divergence of a marker trait on which to base assortative mating (for example, sympatric divergence of a sexually selected character when there is parental imprinting). Here, speciation through the two-allele mechanisms of the establishment of a different marker trait in each incipient species faces the difficulty of polymorphism maintenance at this locus (across the ‘population’ of individuals connected by gene flow), but recombination has no effect because there is no other underlying species-specific trait with which the marker must become associated.

In instances similar to this one, the distinction between one-allele and two-allele models of speciation is not as simple as it seems. The model of Dieckmann and Doebeli (1999) serves as an excellent illustration of the fact that a single instance of speciation can include both one-allele and two-allele components of premating isolation (see also Felsenstein, 1981, p 133). In one of the variants of their sympatric speciation model, speciation relies on both divergence of a marker trait and the evolution of assortative mating from nonrandom mating (using the marker trait as a cue). The marker trait, at which a different phenotype must fix in each incipient species, operates through a two-allele mechanism. Without divergence at this trait, assortative mating, if it were to evolve, would have no phenotype to act on. Assortative mating in this model, although controlled by many loci, operates in a manner analogous to a one-allele mechanism; the same phenotype, for assortative versus disassortative or random mating, must fix in both incipient species for speciation to result. Speciation thus relies both on one-allele and two-allele mechanisms in this model, in this case operating simultaneously.

In general, one-allele mechanisms may often rely on the existence of underlying two-allele mechanisms to produce polymorphisms on which assortative mating may act (although in some cases, such as the reduction of migration or assortative mating based on cultural or environmentally based differences, such genetic polymorphisms are unnecessary). Marker traits, for example, must by their very nature be ‘two-allele’ traits. When one is examining speciation a posteriori, it may be impossible to determine whether an instance of speciation such as that in the Dieckmann and Doebeli (1999) model described above ultimately resulted from the evolution of a one-allele mechanism (for example, assortative mating) based on the prior existence of a ‘two-allele’ polymorphism (presumably this would be a relatively favorable circumstance for sympatric speciation) or from the divergence of a two-allele marker trait given the prior existence or coevolution of one-allele assortative mating (presumably more difficult; Servedio and Noor, 2003).

The distinction between one-allele and two-allele models is probably the single most useful insight to date into the categorization of speciation models and their attendant difficulties. For the reasons outlined above, however, it is clear that this distinction is not necessarily a simple one, and that the details of every speciation scenario must be examined very carefully to understand the extent of the application of this categorization to it.

Linkage disequilibrium, recombination and selection against hybrids

Selection against hybrids (postzygotic isolation) is one of the primary driving forces leading to the evolution of premating isolation through the process of reinforcement. Here, I discuss conditions that facilitate progress toward speciation by affecting the action of selection against hybrids and subsequently the links between postzygotic and prezygotic isolation.

Maintaining species identity

Although most researchers studying speciation concentrate on the forms of isolation between species emphasized by the Biological Species Concept (Mayr, 1942), many taxonomists describing species concentrate on whether ‘fixed’ differences can be found between species—whether species are diagnosably distinct (see for example, Sites and Marshall, 2004 for application of this criterion). By this criterion, species must maintain some genetic differences despite potential continued gene flow. These differences may be involved in causing selection against hybrids. This is particularly likely when diagnosable differences are involved in local adaptation, as hybrids may then be ill adapted to the ecological niche of either parent (that is, ‘extrinsic’ isolation). In other cases, genes that cause genetic incompatibilities (‘intrinsic’ isolation) may or may not be associated with genes that cause phenotypic differences used to identify species; but if these phenotypic differences are maintained despite extensive gene flow, such an association is likely.

One mechanism that should aid in the maintenance of species differences is the presence of chromosomal rearrangements (for example, Noor et al., 2001; Rieseberg, 2001). Such rearrangements may reduce recombination in incipient species still undergoing gene flow, allowing the increased persistence (Noor et al., 2001; Rieseberg, 2001) or the build-up (Navarro and Barton, 2003) of incompatibilities, while premating isolation has a chance to evolve. The specific mechanisms by which this may occur have been extensively reviewed elsewhere (for example, Coyne and Orr, 2004; Butlin, 2005). The effects of the reduction in recombination made possible by chromosomal rearrangements may also occur through other mechanisms of recombination reduction as well (see Butlin, 2005).

Recombination may also be reduced between genes causing postmating and premating isolation if these genes are on the same chromosome (for example, in Drosophila; Noor et al., 2001). Further reduction of recombination on sex chromosomes may facilitate speciation even more (see also Lemmon and Kirkpatrick, 2006). Genes for female preferences, male plumage ornamentation and genes causing low hybrid fitness have all been found to be located on the Z chromosome in flycatchers (Sætre et al., 2003; Sæther et al., 2007).

Is selection against hybrids effective in driving speciation?

When there is free recombination, the efficacy of a given strength of selection against hybrids in driving speciation by reinforcement is primarily dependent on the number of levels of linkage disequilibrium that ultimately connect selection against hybrids with the evolving prezygotic isolating mechanism. Although this point may seem obvious, a thorough understanding of it leads to some under-recognized implications regarding the types of isolation that are likely to have more (or less) significant roles in speciation with gene flow.

A very effective type of selection driving the evolution of premating isolation is divergent selection on a trait used as a mating cue (sometimes called a ‘magic’ trait; Gavrilets, 2004). Because of their roles as mating cues, these types of loci will form strong genetic associations with the locus (or loci) that cause premating isolation, and that use these cues as markers. The fact that selection (generally divergent) is based on the phenotypes at these same marker loci causes this selection to be transmitted to the loci that cause premating isolation through only one level of linkage disequilibrium (see Figure 1a). Magic trait loci may sometimes cause selection against hybrids, particularly if hybrids at these loci perform very poorly in the ecological niches of the parents (that is, they are involved in extrinsic isolation). (A semantic issue arises as to whether performance of hybrids at these loci and in these niches must be non-additive for the term ‘selection against hybrids’ to be valid (for example, must hybrids have lower fitness than the average fitness of the parents in a particular niche?).)

Figure 1
figure 1

Schematic diagram of types of disequilibrium involved in the evolution of premating isolation. (a) A ‘magic trait’ mechanism. Selection acts directly on a marker trait (M). Linkage disequilibrium (arrow) is established between the marker loci and loci involved in premating isolation (P). (b) An ‘indicator’ mechanism. Selection acts against hybrids at loci involved in hybrid incompatibilities (H). Linkage disequilibrium (solid arrows) connects the hybrid incompatibility loci and marker loci (M); see text for mechanisms to generate this. Linkage disequilibrium also connects the marker loci with loci for premating isolation (P). Weaker linkage disequilibrium (dashed arrow) forms between the loci for hybrid incompatibilities and the loci for premating isolation due to the presence of the linkage disequilibria represented by the solid arrows. Note that in both of these scenarios H, M and P may represent a single locus or a phenotype controlled by multiple loci.

Full size image

Magic trait models have been criticized as being unrealistic (the name ‘magic’ trait itself is an allusion to this, although I use it here for shorthand). Evidence does exist, however, for the existence of traits that are both under divergent selection, or even cause selection against hybrids, and are used as a cue for mating. Reynolds and Fitzpatrick (2007, see references therein for additional examples), for instance, find that females of different color and pattern morphs of the aposematically colored poison dart frog Dendrobates pumilio mate assortatively. Hybrids between these forms have intermediate phenotypes that are likely to not provide effective warning coloration (aposematic colors are generally under strong stabilizing selection).

Another potential mechanism to generate a magic trait with selection against hybrids exists in that the low fitness generated by intrinsic postzygotic isolation may prevent hybrids from being able to generate condition-dependent traits (Servedio, 2004). This provides a mechanism whereby hybrids, even if they inherit a dominant secondary sexual character or display behavior, may be discriminated against because they express it poorly, thus providing a connection between selection against hybrids and a potential mating cue. The trait in question may not even necessarily be a secondary sexual characteristic; hybrids with low fitness may look unhealthy in general, which may cause females to discriminate against them (but see further discussion of this example below).

Many empirical cases of reinforcement are undoubtedly better described by ‘indicator’ models of reinforcement (Servedio, 2004) than by magic trait models. In an indicator model, a marker trait, either neutral or under selection, serves as an indicator of species identity, whereas hybrid incompatibilities are caused by the action of other loci. The marker and incompatibility loci may be physically linked (for example, on the same chromosome, in flycatchers Sætre et al., 2003) or on different chromosomes—both allow reinforcement to occur (for example, Servedio, 2000).

In an indicator model, two types of linkage disequilibrium must form for selection against hybrids to result in the evolution of premating isolation (Figure 1b). First, linkage disequilibrium must form between the locus for premating isolation and the marker trait. Because the marker trait acts as a mating cue, this linkage disequilibrium forms automatically by nonrandom mating. In a magic trait model, this is the only linkage disequilibrium that is needed, because the marker traits themselves are the source of divergent selection or hybrid incompatibility. In an indicator model, however, for selection against hybrids to drive premating isolation, linkage disequilibrium must also exist between the loci causing selection against hybrids and the marker loci (Figure 1b). This linkage disequilibrium may often be a result of shared history; if both the marker trait and the traits causing hybrid incompatibilities diverge in allopatry, linkage disequilibrium will exist between these loci across the whole system when secondary contact is established. If the marker locus is selectively neutral in the absence of nonrandom mating (for example, a plumage pattern that is not affected by viability selection), this linkage disequilibrium will be transitory, and alleles at the marker locus may eventually shuffle randomly between the incipient species. If, however, genetic variation for premating isolation is present or emerges quickly, reinforcement may occur even with a selectively neutral marker locus as a mating cue; the evolution of premating isolation is driven by selection against hybrids before the association with the marker locus degrades. The linkage disequilibrium between the marker and hybrid incompatibility loci and between the marker and premating isolation loci will, through the principle of transitivity, cause linkage disequilibrium between the loci causing selection against hybrids and the locus causing premating isolation (dashed arrow, Figure 1b). This linkage disequilibrium will be relatively weak, as it is formed by the existence of the two other imperfect genetic associations, so the evolution of premating isolation may tend to be slow when it occurs through an indicator mechanism.

Clear parallels exist between indicator models of reinforcement and some models of sympatric speciation that involve a selectively neutral marker. The model of Dieckmann and Doebeli (1999) discussed above provides an example of such a scenario. In one of the sympatric speciation models in this study, mating is based directly on an ecological trait under divergent selection due to competition; this is in essence a ‘magic trait’ model. In another model variant, however, assortative mating is based on a neutral marker trait; this is the parallel of an indicator model of reinforcement. In this sympatric speciation model, the linkage disequilibrium between the marker trait and the ecologically important trait (analogous to that between the marker trait and the loci causing selection against hybrids in the discussion of reinforcement above) is caused not by shared history, but by the fact that there are a finite number of individuals considered in this model. This causes small genetic associations between loci to form stochastically. These can be large enough to provide the linkage disequilibrium necessary for a sufficiently strong genetic association to ultimately form between the ecologically important locus and the locus (or loci) for nonrandom mating, ultimately driving the evolution of assortative mating in the system.

It is possible to have both a magic trait-style mechanism and reinforcement, through selection against hybrids in an indicator mechanism, occurring simultaneously. This may occur if the marker trait is not selectively neutral, but locally adapted or otherwise under divergent selection (for example, Servedio, 2000, 2004; the indicator model in Servedio, 2004 includes both of these effects). In such a case, both pathways will lead to the evolution of premating isolation. However, selection through the magic trait mechanism would be expected to be much more efficient in driving the evolution of premating isolation than that occurring through the indicator pathway (Servedio, 2004). It should again be noted that unless it is under divergent selection, it may be unlikely that the marker locus will maintain either genetic variation or sufficiently strong genetic associations with hybrid incompatibility loci for long enough to allow the evolution of premating isolation.

The fact that selection is more efficient in driving the evolution of premating isolation when it acts through one, versus two, levels of linkage disequilibrium has clear implications for the types of postzygotic isolation that will be likely to lead to the evolution of prezygotic isolation, and for the types of prezygotic isolation that may evolve. Traits that act as magic traits, such as the examples described above, are obviously especially likely to lead to the evolution of premating isolation. It is also worth thinking about types of traits that are especially unlikely to be magic traits (of course exceptions may always be found in these cases). This includes some traits involved in local adaptation, such as metabolically or physiologically important traits, that may produce an ecologically important phenotype under divergent selection, but may be unlikely to have that phenotype per se be a target of mate choice.

Previously, I discussed how hybrids that are physically in poor shape may not be able to produce secondary sexual characters that are governed by condition dependence. Although this mechanism certainly provides a plausible link between low hybrid fitness and mating cues, it only explains part of the mechanism necessary to drive the evolution of premating isolation. For assortative mating to evolve based on a mating cue, that cue must be favored by sexual selection in a species-specific manner. Imagine, for example, a trait that has a phenotype a in incipient species A and b in incipient species B; recall that if premating isolation has not yet evolved, we are considering a and b as potential mating cues, but they are not yet under sexual selection. It is not enough that hybrids between the two incipient species cannot produce either phenotype a or b, but a nonrandom mating mechanism must evolve whereby, in incipient species A, phenotype a is preferred over both the unattractive hybrid phenotype and over phenotype b (and vice versa in incipient species B). Disruption of a condition-dependent phenotype alone cannot account for the preference for phenotype a over phenotype b in this example.

Other traits that may be unlikely to be potential examples of magic traits driving premating isolation are traits involved in postmating, prezygotic incompatibilities (that is, selection against hybridization rather than selection against hybrids). These incompatibilities include mechanisms that increase mortality in mated pairs of heterospecifics or that lower fertility in heterospecific pairs (see for example, Servedio, 2001). Postmating, prezygotic incompatibilities can lead to reinforcement through an indicator mechanism, potentially just as efficiently as postzygotic incompatibilities can (Servedio, 2001). In many systems, however, loci involved in egg–sperm incompatibilities, sperm storage or other specific features directly affecting fertility may not be likely to also be explicit targets of mate choice (this also applies to loci that directly cause higher mortality between mating and offspring production); remember that it is not enough that they be genetically associated with such targets, to truly be a magic trait these loci must themselves directly act as mating cues. One potential exception, depending on semantics, comes in the VERL/lysin gamete recognition protein system of abalone (and potentially likewise in other gamete recognition systems); here positive selection, most likely driven by some form of sexual conflict such as polyspermy avoidance, has acted directly on the VERL receptor, driving evolution in both components of the system (for example, Aagaard et al., 2006). This example only works, however, if we consider gamete recognition systems not to have a ‘choosing sex’—if we consider ‘choice’ to occur through the female component of the system, then the positive selection in this example is acting on the wrong component of the system to technically be a magic trait (that is, not on lysin, the ‘mating cue’).

The number of levels of disequilibrium between selection against hybrids (or hybridization) and prezygotic isolation may also affect what type of prezygotic isolation will evolve. Marshall et al. (2002) speculate that conspecific gamete precedence, the increased usage of conspecific over heterospecific sperm when a female has mated with both types of males, may evolve through a reinforcement-like mechanism in response to selection against hybrids. Although this evolution is certainly possible through a mechanism of cryptic female choice (Lorch and Servedio, 2007), selection against hybrids is, however, not likely to be an extremely efficient driver of conspecific gamete precedence, as it is hard to imagine a mechanism whereby the loci involved in selection against hybrids would also be the exact same loci used by females to discriminate between conspecific and heterospecific sperm. The loci involved in selection against hybrids would therefore not act as a magic trait in this case. It may be more likely, however, that traits involved in postmating, prezygotic isolation through reduced fertility of heterospecifics would be involved in discrimination between sperm by females (and hence act as a magic trait), because these loci express their phenotype at the same stage of the life cycle in which conspecific gamete precedence acts (although it is certainly possible, if not probable, that neither type of loci would often be a direct cue in conspecific gamete precedence).

Conclusions

The establishment of linkage disequilibrium between loci involved in postzygotic and prezygotic isolation is clearly often critical in the process of speciation, a fact that has rightly been emphasized in comparisons of speciation models and discussions of the role of recombination in speciation. The distinction between one-allele and two-allele models of speciation is a crucial one that emphasizes the importance of recombination and linkage disequilibrium, but it is important not to overgeneralize about the likelihood of speciation in a given scenario because of which of these categories it best fits. A second distinction emphasized in the literature, that between magic trait models and indicator models also ultimately relies on linkage disequilibrium, this time on how many levels of linkage disequilibrium must exist for the evolution of premating isolation to occur. It is not always obvious which of these two factors involving linkage disequilibrium is more important in determining the likelihood of speciation. For example, using the scenarios described in Figure 1, it is unclear whether speciation would be easier in a two-allele version of the magic trait model of Figure 1a (for example, if assortative mating was already established and speciation relied on divergence at the marker locus), or in a one-allele version of the indicator model shown in Figure 1b. Careful consideration of both of these roles of linkage disequilibrium in the particular biological context at hand is critical to a thorough understanding of what types of selection are likely to drive speciation in a given scenario.