Main

Interactions between members of the opposite sex are essential for sexually reproducing animals. Evolutionary benefits have been proposed for homo- and bisexual traits1,2, which exist in many animals2 from American bulls3 to Japanese rhesus monkeys4. Studies of animals with different sexual preferences are essential for understanding the seemingly simple decision of a male to court a female.

Research in Drosophila has uncovered genes required for Drosophila courtship preference, but none of their homologues have been shown to affect mammalian sexual preference. Research in mammals has demonstrated that pheromone sensing in the periphery is important for sexual preference. Male mice lacking Trpc2 (Trpc2−/−), which encodes a channel expressed in the vomeronasal organ, mounted other males, emitted ultrasonic vocalizations (USVs) towards males and were less aggressive towards males5,6. However, understanding of the central mechanisms for sexual preference remains limited.

The neurotransmitter 5-HT has been implicated in male sexual behaviours such as erection, ejaculation and orgasm in mice and humans7,8. Depletion of 5-HT by treating animals with p-chlorophenylalanine (pCPA) or tryptophan-free diets induced male–male mounting9,10,11. However, pCPA treatment was thought to increase sexual activity whereas its effect on sexual preference has not been investigated. Interpretation of pCPA results was complicated further by the lack of specificity: pCPA may affect noradrenaline and dopamine at higher concentrations12.

Almost all serotonergic neurons in the brain were missing from embryogenesis to adulthood in Lmx1b conditional knockout mice in which the floxed Lmx1b allele was deleted by ePet1-Cre13. We compared the behaviours of male mice of different genotypes: ePet1-Cre/Lmx1bflox /Lmx1bflox as homozygous mutants (Lmx1b−/−); their littermates ePet1-Cre/Lmx1bflox/+ as heterozygous mutants (Lmx1b+/−); and Lmx1bflox /Lmx1bflox without ePet1-Cre as the wild type (Lmx1b+/+ ). We also used ePet1-Cre without Lmx1bflox as a control.

We tested first how a male responded in his home cage when a wild-type target C57 male was introduced. Compared to the ePet1-Cre, Lmx1b+/+ and Lmx1b+/− controls, Lmx1b−/− mice showed significantly more mounting of male intruders (Fig. 1 and Supplementary Movie 1; see Supplementary Data 1 for numbers of mice used and statistics for all figures). The percentage of males who mounted target males was significantly higher in Lmx1b−/− males than ePet1-Cre, Lmx1b+/− and Lmx1b+/+ males (Fig. 1a). Lmx1b−/− males mounted with a shorter latency (Fig. 1b), higher frequency (Fig. 1c) and longer duration (Fig. 1d). These results show that the absence of serotonergic neurons in the brain increased male–male mounting.

Figure 1: Male–male mounting and USV by mice lacking central serotonergic neurons.
figure 1

a–g, Numbers of mice used and statistical analysis are all included in Supplementary Data 1. a–d, A test male was presented in its home cage with an adult wild-type male and its behaviour was recorded for 30 min (all data shown as mean ± s.e.m.). Compared with Lmx1b+/+ , Lmx1b+/− or ePet1-Cre, Lmx1b−/− males mounted males at a higher percentage (a), lower latency (b), higher frequency (c) and for a longer duration (d). e, Typical USV patterns emitted by males when presented with female or male intruders. The two left panels show USVs in 2 min, whereas the two right panels show parts of USV graphs at higher magnifications. f, Female intruders elicited USV from almost all ePet1-Cre, Lmx1b−/− , Lmx1b+/+ , or Lmx1b+/− males . Male intruders elicited USVs more from Lmx1b−/− males than from ePet1-Cre, Lmx1b+/+ or Lmx1b+/− males. g, The number of USVs emitted by Lmx1b−/− males towards males is higher than those by ePet1-Cre, Lmx1b+/+ or Lmx1b+/− males, whereas ePet1-Cre, Lmx1b+/+ , Lmx1b+/− and Lmx1b−/− males were similar in USVs towards females. *P < 0.05, **P < 0.01, ***P < 0.001.

PowerPoint slide

Full size image

A sexually dimorphic behavioural response of males is to emit 30–110 kHz USVs when they encounter female mice or pheromones, which may function as love songs to facilitate female receptivity14. Lmx1b+/+ , Lmx1b+/− and Lmx1b−/− males were similar in USV emission towards females (Fig. 1e–g). However, the percentage of Lmx1b−/− males emitting USV towards males was significantly higher than that of ePet1-Cre, Lmx1b+/+ or Lmx1b+/− males (Fig. 1f). Numbers of USV ‘syllables’ emitted towards females were similar among ePet1-Cre, Lmx1b+/+ , Lmx1b+/− and Lmx1b−/− males (Fig. 1g). Lmx1b−/− males emitted more USV ‘syllables’ towards males than ePet1-Cre, Lmx1b+/+ and Lmx1b+/−. The number of USV emissions by Lmx1b−/− males towards males was approximately 720 times higher than that of Lmx1b+/+ males (Fig. 1g).

Although Lmx1b−/− males still emitted more USVs towards females, the preference for females over males was significantly reduced: the ratio of USVs towards females over that for males was only 3 for Lmx1b−/− males, significantly reduced from 1,002 for ePet1-Cre males, 2,438 for Lmx1b+/+ males and 52 for Lmx1b+/−.

In the mating choice assay, an oestrous female C57 target mouse and a sexually naive male C57 target mouse were introduced into the home cage of a test male. Wild-type males preferred to mount female targets (Fig. 2a): a higher percentage of Lmx1b+/+ (or ePet1-Cre, Lmx1b+/−) males mounted female targets than male targets (Supplementary Movie 2). However, the percentage of Lmx1b−/− males mounting females was not significantly different from that mounting males. ePet1-Cre, Lmx1b+/+ and Lmx1b+/− males mounted female targets with a shorter latency, higher frequency and longer duration than male targets (Fig. 2b, d, e), whereas Lmx1b−/− males mounted males and females with similar latencies, frequencies and durations (Supplementary Movies 2 and 3). Thus, elimination of serotonergic neurons led to a loss of sexual preference in mounting.

Figure 2: Lack of sexual preference by mice without central serotonergic neurons.
figure 2

a–f, Each test male was presented with a male and an oestrous female, and its mating choice was analysed for 15 min. a, More ePet1-Cre, Lmx1b+/+ and Lmx1b+/− males mounted female than male targets. A similar percentage of Lmx1b−/− males mounted females and males. b, ePet1-Cre, Lmx1b+/+ and Lmx1b+/− males mounted female targets faster than male targets. Mounting latencies of Lmx1b−/− males for females and males were similar. c, More than 40% of Lmx1b−/− males but none of the ePet1-Cre, Lmx1b+/+ or Lmx1b+/− males chose a male as their first mounting target. d, ePet1-Cre males mounted females significantly more often than males, as did Lmx1b+/+ and Lmx1b+/− males. Lmx1b−/− males mounted females as often as males (P > 0.05, t-test). e, ePet1-Cre males spent more time mounting females than males, as did Lmx1b+/+ and Lmx1b+/− males. Lmx1b−/− males did not show differences in mounting males or females. f, The mounting frequency ratio of Lmx1b−/− was different from that of ePet1-Cre, Lmx1b+/+ and Lmx1b+/−. *P < 0.05, **P < 0.01, ***P < 0.001.

PowerPoint slide

Full size image

Further analyses were carried out to detect a change in sexual preference separate from an increase in sexual drive: (1) in the mating choice assay, all ePet1-Cre, Lmx1b+/+ and Lmx1b+/− males mounted females before males, whereas 46.2% of Lmx1b−/− mounted males first (Fig. 2c); (2) the mounting frequency ratio of Lmx1b−/− males in the mating choice assay (female mounting frequency − male mounting frequency)/(female + male mounting) (that is, (♀ − ♂/♂ + ♀)) was significantly different from ePet1-Cre, Lmx1b+/+ and Lmx1b+/− males (Fig. 2f); and (3) when a test male was presented only with an oestrous female target, Lmx1b−/− males were not statistically significant different from wild-type and heterozygous males in male–female mounting (Supplementary Fig. 1).

We tested male mice for their preference of pheromones present in the genitals or the bedding. In the genital odour preference assay15, a slide with one half smeared with female genitals and the other half with male genitals was presented to a test male. The total time spent sniffing both halves of the slide was reduced in Lmx1b−/− males (Supplementary Fig. 2a). Lmx1b+/+ and Lmx1b+/− littermates spent significantly more time sniffing female than male genital odour, whereas Lmx1b−/− males spent equal time sniffing female and male genital odours (Fig. 3a). Lmx1b+/+ , Lmx1b+/− and Lmx1b−/− were similar in the amount of time spent sniffing male genital odour. Female genital odour sniffing time was less in Lmx1b−/− males than in Lmx1b+/+ and Lmx1b+/− littermates (Fig. 3a). The genital odour preference ratio (♀ − ♂/♂ + ♀) of Lmx1b−/− males was significantly lower than those of Lmx1b+/+ and Lmx1b+/− males (Fig. 3b). Compared with Lmx1b+/+ and Lmx1b+/− males, a significantly higher percentage (62.5%) of Lmx1b−/− males spent more time sniffing male than female genital odour (Fig. 3c).

Figure 3: Loss of sexual preference for genital odour and bedding by males without central serotonergic neurons.
figure 3

a, Lmx1b+/+ males spent more time sniffing female than male genital odour, as did Lmx1b+/− males. Lmx1b−/− males spent a similar amount of time on female and male genital odour. Three groups were not significantly different in male genital odour sniffing time but Lmx1b−/− males spent less time sniffing female genital odour than the other two groups. b, Sniffing ratio of Lmx1b−/− males was significantly different from Lmx1b+/+ and Lmx1b+/− males (P < 0.05 for Lmx1b+/+ versus Lmx1b−/−, P < 0.05 for Lmx1b+/− versus Lmx1b−/−, P > 0.05 for Lmx1b+/+ versus Lmx1b+/−; one-way ANOVA). c, Compared with Lmx1b+/+ and Lmx1b+/−, a higher percentage of Lmx1b−/− males spent more time sniffing male than female genital odour. d, ePet1-Cre males spent more time above female bedding than male bedding, as did Lmx1b+/+ and Lmx1b+/− males. Lmx1b−/− males spent a similar amount of time above female and male bedding. Compared with ePet1-Cre, Lmx1b+/− and Lmx1b+/+ , Lmx1b−/− males spent less time above female bedding but more time above male bedding. e, The bedding time ratio of Lmx1b−/− was different from ePet1-Cre and Lmx1b+/+ . f, Compared with ePet1-Cre, Lmx1b+/+ and Lmx1b+/−, a significantly higher percentage of Lmx1b−/− males spent more time above male bedding. *P < 0.05, **P < 0.01, ***P < 0.001.

PowerPoint slide

Full size image

In the bedding preference assay16, the total time spent over male and female bedding was similar among ePet1-Cre, Lmx1b+/+ , Lmx1b+/− and Lmx1b−/− males (Supplementary Fig. 2b). ePet1-Cre, Lmx1b+/+ and Lmx1b+/− males spent significantly more time above female than male bedding whereas Lmx1b−/− males spent equal time above female and male beddings (Fig. 3d). Compared with ePet1-Cre, Lmx1b+/+ and Lmx1b+/− males, Lmx1b−/− males spent more time above male bedding and less time above female bedding. The bedding preference ratio of Lmx1b−/− males was significantly lower than those of ePet1-Cre, Lmx1b+/+ and Lmx1b+/− males (Fig. 3e). The percentage of males who spent more time above male bedding was significantly higher in Lmx1b−/− males (58.8%) than those in ePet1-Cre (0%), Lmx1b+/+ (6.3%) or Lmx1b+/− (12.5%) males (Fig. 3f).

Thus, in both the genital odour and bedding assays, Lmx1b−/− males had lost preference for female pheromones over male pheromones: in the genital odour preference assay, Lmx1b−/− males showed decreased sniffing time for female genital odour; in the bedding preference assay, Lmx1b−/− males showed increased time spent over male bedding and decreased time over female bedding.

Multiple assays involving odour or pheromone sensing were carried out to test for possible changes in olfaction. In the sesame oil preference assay17, Lmx1b+/+ and Lmx1b−/− males were indistinguishable in spending significantly more time with sesame than air (Supplementary Fig. 3a). In the fox urine avoidance assay18, Lmx1b+/+ and Lmx1b−/− males were also similar (Supplementary Fig. 3b). Thus, Lmx1b−/− males were not defective in either innate attractive or avoidance response.

In the social approach assay19, Lmx1b+/+ and Lmx1b−/− males were similar in spending more time close to a strange male than the empty chamber (Supplementary Fig. 3c).

In the social recognition assay20, Lmx1b+/+ and Lmx1b−/− males spent a similar amount of time exploring the first intruder at initial presentation, displayed social habituation towards the familiar intruder over the next three presentations and displayed dishabituation when a new intruder was introduced (Fig. 4a).

Figure 4: Odour discrimination.
figure 4

a, Both Lmx1b+/+ and Lmx1b−/− males showed habituation and dishabituation in sniffing time. No statistical difference was found between Lmx1b+/+ and Lmx1b−/− males at any point. b, After seven training sessions with male and female urine, no significant difference was detected between Lmx1b+/+ and Lmx1b−/− males at any point.

PowerPoint slide

Full size image

An operant conditioning assay was used to test whether Lmx1b−/− males could distinguish between male and female pheromones21. Two arms of a T maze were supplied with the odour of either female or male urine. Electroshock was applied in such a way that the test mice had to run or stay in the same arm depending on the urine. Over 3 days of training, Lmx1b+/+ and Lmx1b−/− males were similar in learning to avoid punishment (Fig. 4b). Thus, no olfactory defects for general odours or pheromones were detected in Lmx1b−/− males.

Results from Lmx1b−/− mice indicate a role for serotonergic neurons. To study the role of 5-HT, we used mice unable to synthesize 5-HT in the brain. 5-HT is synthesized in two steps: tryptophan is converted by a Tph into 5-HTP, which is converted into 5-HT by 5-hydroxytryptophan decarboxylase and aromatic l-amino-acid decarboxylase.

There are two Tph enzymes: Tph2 is required centrally and Tph1 peripherally. We have generated Tph2−/− mice (J.-Y.K. et al., manuscript in preparation), which were viable22,23,24. High-performance liquid chromatography (HPLC) analysis showed that the 5-HT level was significantly reduced in the brains of Tph2−/− males (Supplementary Fig. 4a). Male–male mounting (Supplementary Movie 4) was significantly higher in Tph2−/− males than either Tph2+/+ or heterozygous Tph2+/− males: the percentage was significantly higher, duration longer, latency shorter and frequency higher (Supplementary Fig. 4b, c and Fig. 5a, b). In the bedding preference assay, both Tph2+/+ and Tph2+/− males preferred female over male bedding, whereas Tph2−/− males showed no preference (Fig. 5c). In the genital odour preference assay, both Tph2+/+ and Tph2+/− males preferred female over male genital odour, but Tph2−/− males showed no preference (Fig. 5d).

Figure 5: Brain chemistry and behaviours of Tph2 knockout males.
figure 5

a, b, Compared with Tph2+/+ and Tph2+/−, Tph2−/− males showed a shorter latency (a) and higher frequency in mounting males (b). c, Both Tph2+/+ and Tph2+/− males significantly preferred female over male bedding, whereas Tph2−/− males did not show a preference between male and female bedding. d, Both Tph2+/+ and Tph2+/− males significantly preferred female over male genital odour, whereas Tph2−/− males did not show a preference between male and female genital odour. *P < 0.05, **P < 0.01, ***P < 0.001.

PowerPoint slide

Full size image

When presented with an oestrous female target, male–female mounting was not significantly changed in Tph2−/− males (Supplementary Fig. 5). In mating choice, Tph2−/− males had lost preference for females over males in percentage, latency, frequency and duration (Supplementary Fig. 6a, b, d, e). No control males mounted target males before females, whereas more than 40% of Tph2−/− males mounted males first (Supplementary Fig. 6c). The mounting frequency ratio of Tph2−/− males was significantly different from those of Tph2+/+ and Tph2+/− males (Supplementary Fig. 6f).

Lmx1b−/− and Tph2−/− mice lack 5-HT from embryogenesis. To study the role of 5-HT in adulthood, we took two complementary approaches: first, we depleted 5-HT from adult mice pharmacologically with pCPA25; then we attempted to rescue the phenotype of adult Tph2−/− mutants.

Adult C57BL/6J males were injected with either pCPA or saline for three consecutive days. 5-HT level was significantly reduced by pCPA (Supplementary Fig. 7). pCPA-treated males showed shorter latency, higher frequency and longer duration than control males in mounting target males (Supplementary Fig. 8a–d), and lost bedding preference (Supplementary Fig. 8e, f).

To test whether 5-HTP injection into adult mice could rescue the Tph2−/− phenotype, we examined first whether 5-HTP could rescue 5-HT synthesis in Tph2−/− males and found that 5-HT levels were restored 35 min after intraperitoneal injection of 5-HTP but not saline (Fig. 6a and Supplementary 9a, b).

Figure 6: 5-HTP rescue of chemical and behavioural deficits in Tph2 knockout mice.
figure 6

a, Levels of 5-HT and 5-hydroxyindoleacetic acid (5-HIAA) were analysed in Tph2+/+ and Tph2−/− males 35 min after injection of either 5-HTP (40 mg kg−1 body weight) or control saline. b, c, Male–male mounting in Tph2−/− mice was significantly rescued by 5-HTP: the latency was lengthened and frequency reduced. d, Bedding preference was monitored between 35 and 40 min after injection. 5-HTP could significantly restore the preference of female over male bedding by Tph2−/− males.

PowerPoint slide

Full size image

5-HTP significantly reduced male–male mounting of Tph2−/− males: the percentage was decreased, latency increased, frequency decreased and duration shortened; all returning to wild-type levels (Fig. 6b, c and Supplementary Fig. 9c, d). 5-HTP rescued the loss of sexual preference in mounting latency, frequency and duration in the mating choice assay (Supplementary Fig. 10a–c) and the bedding preference of Tph2−/− males (Fig. 6d and Supplementary Fig. 9e).

When a test male was presented with a target female, Tph2−/− males were similar to wild-type and heterozygous males in mounting percentage, latency, frequency and duration (Supplementary Figs 5, 11). 5-HTP injection into Tph2−/− males did not affect male–female mounting (Supplementary Fig. 11), although 5-HTP injection into wild-type males reduced male–female mounting. Because 5-HTP injection in wild-type males increased the level of 5-HT beyond the wild-type level (Supplementary Fig. 9a, b), it indicated a dosage-sensitive effect of 5-HT: 5-HT at concentrations above the wild-type level inhibited male–female mounting, but 5-HT concentrations between the wild-type and Tph2−/− levels did not affect male–female mounting.

We conclude that central serotonergic signalling is crucial for male sexual preference in mice. This is the first time, to our knowledge, that a neurotransmitter in the brain has been demonstrated to be important in mammalian sexual preference. Previous studies in mammals have implicated 5-HT and dopamine in male sexual behaviours, but neither has been demonstrated to have any role in sexual preference: dopamine is thought to facilitate male sexual behaviours whereas 5-HT is thought to inhibit sexual behaviours7,8,9,10,11,26. Our studies have established a role for 5-HT in male sexual preference. Multiple results showed a loss in sexual preference beyond or separate from hypersexuality: (1) the ratio of male–male and male–female interactions was repeatedly measured to analyse sexual preference (Figs 2f, 3b, e, 5c, d, 6d and Supplementary Figs 6f, 8f, 9e, 10d); (2) Lmx1b−/− males showed increased USVs towards males but not towards females (Fig. 1g); (3) in mating choice, the latency, frequency and duration of Lmx1b−/− males to mount males, but not to mount females, was changed (Fig. 2a, b, d, e); (4) in bedding preference, Lmx1b−/− (Fig. 3d) and Tph2−/− males (Figs 5c, 6d) showed an increase in time spent over male bedding but a decrease in time over female bedding; (5) wild-type males always mounted females before males but a significant fraction of Lmx1b−/− or Tph2−/− males mounted males first (Fig. 2c and Supplementary Fig. 6c); (6) in the genital odour preference assay, both Lmx1b−/− (Fig. 3a) and Tph2−/− (Supplementary Fig. 5d) males showed a decrease in time on female genital odour, which could not be explained by hypersexuality; and (7) when presented with an oestrous target female, neither Lmx1b−/− males (Supplementary Fig. 1) nor Tph2−/− males (Supplementary Fig. 5) were different from wild-type males.

Increased sexual drive was observed in males lacking 5-HT when they were tested in the presence of live target males and females (Supplementary Fig. 6). This has been noted before in mice defective for Trpc2 and vomeronasal organ olfaction5,6. Trpc2−/− males have been previously reported to have lost male–female preference in mating choice5,6. Trpc2−/− males showed increased mounting of both males and females (figure 2c in ref. 6). The conclusion of a loss in sexual preference in Trpc2−/− males was inferred from a relative change: Trpc2−/− males showed a 2-fold preference for females over males whereas the wild-type showed a 10-fold preference. The phenotypes reported here for Lmx1b−/−, Tph2−/− males and pCPA-treated males were stronger than for Trpc2−/− males in mating choice: these males did not show significant preference for females (Fig. 2 and Supplementary Fig. 6).

At present, it is not known whether 5-HT regulates the vomeronasal organ pathway in pheromone sensing or acts further downstream in behavioural decisions. Differences have been noted between Trpc2 and Lmx1b in the brain: aggression was largely lost in Trpc2−/−, but not Lmx1b−/−, mice (data not shown). It is more likely that 5-HT regulates central decision-making than influencing peripheral olfaction. However, we cannot completely rule out the possibility that 5-HT regulates a specific innate olfactory pathway processing sexual information27. In mice, it will be interesting to identify specific subsets of serotonergic neurons and serotonergic receptors involved in sexual preference.

An unavoidable question raised by our findings is whether 5-HT has a role in sexual preference in other animals. In a positron emission tomography study of humans, the response of heterosexual men to the selective serotonin reuptake inhibitor (SSRI) fluoxetine was found to be different from that of homosexual men28. SSRIs inhibited compulsive sexual behaviours in homosexual and bisexual men29. However, so far, none of these studies has investigated whether 5-HT has a role in sexual preference. Attempts have been made to map genetic loci affecting human sexuality30, although specific genes have not been identified. Our discovery of a role for serotonergic signalling in mouse sexual preference should stimulate further studies into the role of 5-HT in sexual interactions in particular and roles of neurotransmitters in mammalian social relationships in general.

Methods Summary

We used conditional knockout mice for Lmx1b and knockout mice for Tph2. Levels of 5-HT in these mice and their heterozygous and wild-type littermates were measured by HPLC. Most of the behavioural assays were similar to established methods.

Online Methods

Mouse stocks

ePet1-Cre mice were a gift from E. S. Deneris and the floxed Lmx1b mice were a gift from R. Johnson. Tph2 knockout mice were generated by deleting exon 5, which encodes the tryptophan hydroxylase domain (for details see J.-Y.K. et al., manuscript submitted). Mice were weaned at the age of 21 days. Mice were maintained on a 12 h light, 12 h dark schedule and housed initially in groups of five up to the tenth week and then singly housed until the end of experiments. Food and water were provided ad libitum. Room temperature was 23 ± 1 °C. Humidity was 40–60%. All test mice were 12–16 weeks old. The target mice were 11–13 weeks old.

Mouse genotyping

Genomic DNA was extracted from mouse tail tissues at the day of weaning. Mutant mice were generated by crossing ePet1-Cre mice with floxed Lmx1b mice and following intercross within the F1 generation mice. Littermates used in the tests were of the same sex and similar body weight as the knockout mice. The primers were: AGGCTCCATCCATTCTTCTC (floxed Lmx1b1); CCACAATAAGCAAGAGGCAC (floxed Lmx1b2); ATTTGCCTGCATTACCGGTCG (Cre1); CAGCATTGCTGTCACTTGGTC (Cre2).

Immunocytochemical analysis with anti-5-HT antibodies confirmed that 5-HT-positive neurons were absent in Lmxb1 knockout mice (data not shown).

The Tph2 line was maintained by crossing heterozygotes. Littermates included wild-type, heterozygotes and homozygous knockout mice. The primers for genotyping were: GGGCATCTCAGGACGTAGTAG; GGGCCTGCCGATAGTAACAC; GCAGCCAGTAGACGTCTCTTAC.

Measurement of 5-HT

The levels of 5-HT and its metabolites were separated by HPLC and measured by an electrochemical detector in samples from adult male mice. In 5-HTP rescue experiments, mice were injected with 40 mg kg−1 5-HTP or saline (both at the volume of 5 ml kg−1). They were euthanized 35 min later. The brain was dissected and the raphe region was isolated on ice. Samples were weighed before ultrasonication. Monoamines were extracted by perchloric acid. The sample was filtrated by 0.22 μm filter before being injected into RP-HPLC (ESA). Noradrenaline, 3,4-dihydroxyphenylacetic acid (DOPAC), dopamine, HIAA, homovanillic acid (HVA) and 5-HT were measured by an electrochemical detector. Their concentrations were calculated by CoulArray software (ESA) based on standard samples. Values of amine per wet tissue weight are shown in the final figures.

Order of behavioural assays

Male mutant mice and their littermates at 12–13 weeks of age and of similar body weight were sexually naive and group-housed with same-sex mice before 10 weeks of age. After 2 weeks of single housing, mice were tested in the following order: bedding preference, male–male resident–intruder assay, mating choice assay, sexual behaviours with an oestrous female, bedding preference again (no difference was observed with results from the first bedding preference). Mice were given one week of rest between each test. For Lmx1b mice, the same group of mice were used in male–male mounting, mating choice and male–female mounting. For Tph2 mice, a different group were used for male–female mounting. Sexually experienced mice were used for USV, social approach, habituation and olfactory learning assays. Sexually naive mice were used for urine preference and olfactory tests.

Resident–intruder tests

All test mice were sexually naive. The bedding of the test mice had not been changed for at least 4 days. Intruder mice were 11–13 weeks old, sexually naive and group-housed C57Bl/6J males. All activities within a test were recorded by an infrared camera (Sony Video Recorder, DCR-HC26C). Mounting latency, mounting frequency and total duration of mounting within 30 min were measured.

Mating choice assay

Beddings of test mice had not been changed for at least four days. A group-housed sexually naive 11–13 week-old C57Bl/6J male and a sexually naive oestrous 10-week-old female C57Bl/6J female were introduced into the cage of each test male. Each assay lasted 15 min after the target mice were introduced. All activities were recorded by an infrared camera. The latency, frequency and duration of mounting of male or female targets were analysed.

Sexual behaviours with females

An oestrous female was presented to a test male and video was recorded for 30 min using an infrared camera. The latency, frequency and duration of male mounting of the female were analysed.

USVs

Tests were carried out with singly housed adult males during the dark phase in the home cage. UltraSoundGate 116-200 system (Avisoft) was used to record the ultrasound. We recorded the background sound for 1 min before a stimulus mouse of 10–13 weeks old was introduced. The recording lasted for 2 min. Recorded data was analysed with SASLab (Avisoft)5. Sounds over the frequency range of 30–110 kHz were analysed. Profiles of background noise created by mouse movement were very different from USVs. To confirm that the resident mouse was the source of USVs, we recorded from assays in which either the resident or the intruder mouse was devocalized. We were able to record robust USVs (presented in our figures) only when the intruder mouse was devocalized and not when the resident mouse was devocalized.

Genital odour preference assay

This assay was modified from a previously described procedure15. The anogenital area scent from a male was rubbed on the left or right side of a clean glass microscope slide while the anogenital area scent from a female was rubbed on the other side of the slide. Five seconds later, the slide was hung in the middle of the cage by a clamp. The slides were ∼5 cm over the bedding. Activities of the test mice were recorded for 3 min by an infrared camera and the sniff time on the scent portion on either side was analysed as was the amount of time a test male licked the slide or its nose touched the slide.

Bedding preference assay

Bedding from group-housed adult C57BJ/6J males or females was not changed for 4 days. Ten grams of male or female bedding were put in one side on the bottom of a cage in an area of 11.5 × 17 cm2. Male and female beddings were prevented from mixing by a plastic bar of 6 cm. The size of cage was 29 × 17 × 15 cm (length × width × height)16. A grid of plastic bars separated the test mice from the bedding on the bottom of the cage. The bars were 5 mm wide with 5 mm intervals. The test mouse was put into the cage to be familiarized with the cage without bedding for 5 min before the mice were taken out and the bedding and a clean grid was put into the cage. After each assay, the cage was washed with water and then alcohol to remove odour.

Olfactory learning assay

We employed a T maze in which electric shock could be applied to either side of the horizontal chamber as described previously21. Briefly, there was a door at the intersection of the horizontal and vertical chambers. The horizontal chamber of 8 × 8 × 60 cm3 was divided into three parts: a left arm of 8 × 8 × 23 cm, a right arm of 8 × 8 × 23 cm and a middle zone of 8 × 8 × 14 cm. Each test mouse was introduced into the vertical chamber of the T maze. After it entered the horizontal chamber, the door between the vertical and horizontal chambers was closed and the mouse was allowed to walk within the horizontal chamber. The mouse was not allowed to stay in the middle zone for longer than 8 s, otherwise it would be punished with electroshock. The position of the test mouse was monitored by a video recorder. Urine samples were collected from more than 20 C57BL/6J males or females and stored at −20 °C. A 1.5 ml urine sample was used for each test. The odour of male or female urine was puffed into the left or right arm of the horizontal chamber and expirated from the middle zone. Odour was presented for 50 s. We trained the test male mouse with electroshock to stay in the arm with female odour and to avoid the arm with male odour. The mouse had to make a decision to stay in or leave the arm when an odour was presented. Each training session of 18 trials lasted for 30 min. Every mouse was given 6 training sessions over 3 days before the final test. There were 10 trials in the final test. The percentages of correct choices in every training session and the final test were analysed.

Innate behavioural responses to odours

The set-up is the same as that for the olfactory learning assay, except that no electroshock was applied. Sexually naive males (mutants or littermates) of 10–16 weeks old were tested for their choices of fox urine versus air, or sesame oil versus air. Fox urine was used to test the innate avoidance of a predator’s odour. Fox urine was diluted at two concentrations (60× and 20×). The main air flow velocity was 250 l h−1. The air flow through fox urine was 70 ml min−1 or 210 ml min−1, respectively. The time that mice spent in the empty arm or the fox urine arm was recorded by Matlab software. Sesame oil diluted 83× was used to test innate attraction to food. Time spent in the air arm or the sesame oil arm was recorded by Matlab software.

Social approach

The social approach experiment was tested in a modified T-shaped box. There was a small cage separated by wire at each end of the arms in the horizontal chamber. A test mouse was allowed to habituate for five minutes before an unfamiliar target male was randomly placed in one of the small cages. The target mouse could be seen, smelled and heard, but could not be touched. The test mouse was allowed to move in the box for 5 min. Its location was video recorded and analysed by a computer.

Social memory

Singly housed adult males were tested in the dark phase and in the room where they were reared. Ovariectomized C57Bl/6J females were used as stimulus mice20. They were ovariectomized at 6 weeks old and used 2 weeks later. A stimulus mouse was introduced into the cage housing a test mouse for 1 min and then was removed. After an interval of 10 min, the same stimulus female was introduced again for 1 min. The stimulus mouse was presented four times. On the fifth time, a new stimulus mouse was introduced for 1 min. The behaviour of test mice was videotaped and time spent on body sniffing was analysed.

5-HT depletion by pCPA treatment

Male C57Bl/6J mice of 11–13 weeks of age were used. They were injected with either 500 mg kg−1 of pCPA (Sigma, C6506) or saline control for 3 consecutive days after 4 days of being singly housed. Animals were tested with adult C57 female mice. Mice that did not show mounting behaviour in 15 min were discarded. Mice that qualified were then singly housed for 1 week before social behaviour testing and their bedding was not changed. Animals were randomly divided into pCPA or saline treatment groups. pCPA was suspended in 1% Tween saline at a concentration of 50 mg ml−1. The pCPA group were injected intraperitonially with pCPA (10 ml kg−1) at 72, 48 and 24 h before testing. The control group received 1% Tween saline. Resident–intruder and mating choice assays were carried out. Behavioural tests were performed in the dark.