Main

The linear dispersion and energy quantization in field of the Dirac surface state of the archetypal topological insulators Bi2Se3 and Bi2Te3 have been revealed by angle-resolved photoemission spectroscopy and scanning tunnelling spectroscopy respectively4,5,6,7,8,9. However, the transport properties of the surface states, which are arguably their most technologically useful as well as fundamentally interesting characteristics, have proved to be especially difficult to measure owing to the overwhelming effect of bulk conduction channels10,11,12,13,14. The materials challenge is to find a way to cleanly eliminate the bulk conductivity so that the properties of the surface can be observed. By partially substituting15 Sb for Bi in Bi2Se3 we systematically reduce the bulk carrier density to n∼1016 cm−3. We study samples in which the Fermi surface of the Dirac fermion is small enough that at pulsed fields of up to 60 T we can access, for the first time, the 2D quantum limit (see Supplementary Information SA)—a regime where correlation effects are likely to arise16,17,18,19,20. In addition, we observe a distinguishing signature of topological insulators at high field whereby the phase of the oscillations is affected by the Zeeman energy.

In Fig. 1a we illustrate the dependence of the resistivity on temperature for representative single crystals with bulk carrier densities ranging from ∼1019 cm−3 down to ∼1016 cm−3. The carrier density was determined by the (low field) Hall effect, which matches the size of the Fermi surface measured by quantum oscillations, known as Shubnikov–de Haas oscillations (SdHO) (see Fig. 1b–d). These oscillations are periodic in inverse field and their period ϒ can be related to the extremal cross-sectional area Ak of the Fermi surface in momentum space through the Onsager relation 1/ϒ=(ℏ/2π e)Ak. When the field is rotated away from the crystal trigonal axis, the frequency of the SdHO changes according to the morphology of the Fermi surface. In every one of our samples, the bulk Fermi surface is a closed 3D ellipsoid21,22,23. The effective mass m*∼0.12me varies weakly with doping for these carrier densities, as previously reported21,22. For our lowest carrier density sample, the pocket is expected to have an orbitally averaged Fermi wavenumber of kF=0.0095 Å−1. This is small enough that we are able to exceed the bulk (3D) quantum limit with moderate fields ∼4 T. This allows us to directly measure all of the relevant transport properties of the bulk, considerably simplifying our analysis, because there is no need to rely on surface-sensitive probes, which are inadequate for determining the bulk Fermi energy23. The remainder of this study is concerned with the 2D properties of the lowest carrier density samples at high fields.

Figure 1: Transport properties arising from the bulk Fermi surface.
figure 1

a, Progressively reducing the bulk carrier density in Bi2Se3 results in a significantly increased resistivity. Optimizing growth conditions for the binary melt can reduce carrier densities down to n∼3×1017 cm−3. Partial substitution of Sb for Bi allowed further reductions down to n=2.9×1016 cm−3. b–d, Representative magnetoresistance data for samples with indicated n, for fields oriented along the trigonal c axis. The progressive reduction of the Fermi-surface volume is evident from the stretching of the SdHOs. e–g, Carrier densities are extracted for each of these samples from the (low field) Hall effect. These agree with that expected from the size of the Fermi surface (shown schematically in the insets, with Fermi wave vector kF indicated). For the lowest carrier density samples shown in e and f, the Hall signal deviates from linearity as the field approaches the 3D (bulk) quantum limit.

Full size image

In Fig. 2a,b we illustrate the longitudinal and transverse (Hall) resistances, Rx x and Rx y respectively, taken at 1.5 K on sample Σ1 with n∼4×1016 cm−3. Strong features appear in Rx y and Rx x at similar fields in the bulk ultra-quantum limit24. To investigate the dimensionality of these features, we rotate the crystal in the field. For the 2D surface state of a topological insulator, quantum oscillatory phenomena depend only on the perpendicular component of the field and are periodic in 1/B. In Fig. 2c we show the background-subtracted signal, illustrating that the oscillatory features grow with increasing field, as one might expect of SdHOs (for details of the background subtraction for all samples Σ1–3, see Supplementary Information SC). In Fig. 2d,e we have plotted the dRx y/dB and −d2Rx x/dB2 as a function of —the usual procedure to determine whether minima in Rx x (equivalent to minima in the negative second derivative) fall on top of the Hall plateaux (appearing as minima in the first derivative). Pronounced dips that are periodic in 1/B in both the Hall derivative and the magnetoresistance align at all angles, providing unambiguous evidence that the plateau-like features in the Hall and minima in the SdHOs originate from a 2D metallic state.

Figure 2: Transport signature of the 2D surface state of a topological insulator.
figure 2

a,b, Rx x (a) and Rx y (b) data traces as a function of magnetic field for indicated angles for a single crystal with n∼4×1016 cm−3 (sample Σ1). The Rx x traces are offset for clarity. The deviation from linearity of the low-field Rx y indicates the 3D quantum limit (B∼4 T), which does not vary with angle. Beyond this limit, additional features in both Rx y and Rx x move smoothly up in field as the tilt angle θ is increased. θ=0 is defined to be when the field is perpendicular to the surface (parallel to the trigonal c axis). c, A smooth second–third order polynomial can be fitted as background and subtracted from the raw Rx x, revealing oscillations that grow with field (see Supplementary Information SC). d,e, −d2Rx x/dB2 (d) and dRx y/dB (e) as a function of , where aligns all features associated with the 2D surface state. The vertical lines in c–e indicate the first three Landau levels N=1,2,3 of the 2D state.

Full size image

Examining Fig. 2c,d the second and third minima occur at twice and three times the 1/B value of the first, and so we assign the Landau indices N=1,2,3 as shown. These indices connect a straight line through the origin when plotted versus (Fig. 3b). The obvious candidate for the origin of the 2D states is the surface Dirac fermions. The plateau-like features in Rx y are reminiscent of the quantum Hall effect in 2D electron gases24,25, but the quantization of the conductance is not evident (for a topological insulator this is expected to be σN=2(N+1/2)e2/h; ref. 26). The discrepancy can be attributed to the remaining parallel conductance channel from the bulk described in detail below. Nevertheless, the periodicity and angular dependence of these data conclusively show that we are able to get to the quantum limit of the 2D state.

Figure 3: Fermi energy dependence of the Landau intercept.
figure 3

a, Quantum oscillatory phenomena from sample Σ3 from 0.55 to 40 K. The higher Fermi energy leads to a larger slope and a reduced Landau intercept. b, The integer Landau levels from sample Σ1 (black filled circles) and Σ3 (dark blue filled circles) are shown as a function of inverse field. Σ1 and Σ2 have a zero intercept, which when fitted assuming a Landau quantized spectrum of (see Supplementary Information SD) yields a g-factor gs≈50. The larger slope of Σ3 makes the apparent intercept obtained from fitting N=2 to 4 negative, in quantitative agreement with our calculations (see Supplementary Information SD). The calculated values for the Landau intercept are shown in light blue and grey for the two different slopes, from fits to the observed Landau indices. The solid lines correspond to linear best fits to the experimental data. Note the Landau intercept does not correspond to what the filling factor is, which can be determined only by the quantized value of the Hall effect.

Full size image

A crude method to confirm that the effect is surface-related is to briefly expose the sample to atmosphere, which is known to lead to n-type doping of the surface23. After ∼1 h exposure the oscillatory phenomena in Figs 2 and 3 are almost completely absent (see Supplementary Information SB and SH). Furthermore, if the origin of the signal is assumed to be a bulk 2D Fermi surface, the carrier contribution should be ∼kF2/2π c=6×1018 cm−3, where c=28.64 Å is the lattice constant. Given that we can observe SdHO, such a pocket has a mobility large enough that it should dominate the Hall effect. Yet, we observe a carrier density 200 times smaller than this number and so a bulk origin of the SdHO seems unlikely. We conclude that the observed signal does indeed originate from the surface state of the topological insulator.

With this identification, the period ϒ of the oscillations yields kF=0.031 Å−1 for the Dirac fermion, or EF≈90 meV (or 83 meV when the Zeeman term is included, see below) above the Dirac point, using the value for the Fermi velocity23 vF=4.2×105 ms−1 determined by photoemmission. This is consistent with the expected band bending23,27 at the surface for the given bulk carrier density (see Supplementary Information SA). Finally, a conventional Lifshitz–Kosevich analysis of the quantum oscillatory signal28 (see Supplementary Information SG and SE) yields kFl≈10, where kF is the Fermi wave vector and l is the mean free path, and a surface mobility of μ=1,750 cm2 V−1 s−1.

It is important to distinguish this work from previous measurements of the Dirac fermion, which have been mostly done in zero field4,5,6,7. The Zeeman term in the Hamiltonian, which determines spin coupling to the magnetic field, lifts the degeneracy at the Dirac point such that the quasiparticle gains a mass. This may account for the small mass enhancement we observe from the calculated zero-field cyclotron mass (m2D*∼0.089me whereas we measure 0.11me, see Supplementary Information SG). Furthermore, the Zeeman term can perturb the Landau quantization itself for a sufficiently large g-factor gs (see Supplementary Information SD) affecting the intercept of a plot of Landau index N versus 1/B. If gs=0, such a plot would yield an intercept of (0, 1/2). For gs≠0, the Landau index is weakly aperiodic and the last few Landau levels deviate from linearity yielding a smaller intercept20. For the period observed in our data, the observed intercept can be accounted for if EF=83±5 meV and gs≈50±10 (Fig. 3b). This estimation assumes the oscillations are sinusoidal, but a saw-tooth shape could reduce the g-factor considerably (see Supplementary Information SD). Nevertheless, gs≈50 is similar to reported bulk values29.

Notably, for a higher Fermi energy the intercept can seem to be negative if only the last few Landau levels are observed. In Fig. 3a we show data for sample Σ3 with an SdHO frequency F≈100 T. Using a value of gs=50 obtained in the analysis of sample Σ1 and fitting Landau level N=2 to N=4, one would expect an intercept of (0,−0.17), in quantitative agreement with the observed value of (0,−0.2), see Fig. 3b. This dependence of the apparent intercept on the Fermi energy and Landau indices fitted illustrates the peculiar relationship of the quantum oscillatory phase and the Zeeman term in the topological-insulator Hamiltonian, distinguishing this Dirac system from graphene30. The associated field dependence of the N=0 Landau level should in principle be observable through scanning tunnelling spectroscopy measurements. However, the anticipated shift in energy for a field of 10 T is only 16 meV, which is perhaps comparable to the tip drift between constant field traces, and thus eludes detection8,9.

To understand the amplitude of the oscillations we use a simple parallel resistor model to describe the effect of bulk and surface channels (see Supplementary Information SE). This analysis shows that the amplitude of the oscillations for N≥2 (for sample Σ2 and Σ3) is consistent with expectations if the Hall effect is assumed to be quantized in σx y=2(N+1/2)e2/h (see Supplementary Information SE and ref. 26). For sample Σ1, a conservative estimate of kFl∼1 yields a (upper bound) discrepancy of a factor of approximately 20 at N=1. However, a more realistic estimate of kFl will reduce this discrepancy considerably. This discrepancy might be partially due to difficulties in the background subtraction (see Supplementary Information SC), but more likely is related to inadequacies of the parallel resistor model. In particular, this model cannot account for surface–bulk coupling, inhomogeneities in the side surfaces or the anomalous nature of the topological-insulator Hall channel31,32.

Additional minima are visible in the second derivative −d2Rx x/dB2 versus in the range 0<N<1 and 1<N<2, as shown in Fig. 4a,b. These approximately line up at each angle when plotted as a function of , and therefore also evidence 2D physics. These minima are not periodic in inverse field and thus cannot originate from another Fermi surface. They are also reproducible using different sweep rates of the pulsed magnetic field and thus cannot be artefacts of the pulsed magnet environment that are periodic in time (see Supplementary Information SF). Intriguingly, these minima fall very near simple rational fractions of the Landau integers N=2/3, 5/7 and 4/5 (in Fig. 4a) as well as N=6/5, 4/3, 3/2 and 5/3 (in Fig. 4b). In Fig. 4c,d we show the background-subtracted data for Σ1 and a further sample Σ2. The signal in both samples is almost identical and in particular, the first pronounced fraction (4/5) in Σ1 is also pronounced in Σ2. The correspondence of these minima to simple fractions of the Landau indices is striking, and is suggestive of precursors to the fractional quantum Hall effect. Whether these features originate from fractional filling or indeed from other unanticipated physics of topological insulators remains to be seen.

Figure 4: Anomalous transport in the quantum limit of the 2D Fermi surface.
figure 4

a, −d2Rx x/dB2 as a function of in the field range for which N<1 for θ=0°, 9.1°, 19.9° and 26.8° (top to bottom), illustrating three additional features at fields corresponding to fractional N. b, The same signal for the field range for which 1<N<2 for angles θ=0°, 19.9°, 43.1°, 55.7° and 58.7°, illustrating four further features. Vertical lines are drawn in a and b corresponding to the nearest fractional N. c, Background-subtracted data for sample Σ1 at three representative angles (θ=0°,19.9° and 27° bottom to top) illustrating a pronounced feature at a fraction of 4/5. d, Similar background-subtracted data for sample Σ2 shown at four temperatures T=0.6,1.5,4.4 and 6 K (bottom to top). The data in all panels have been offset for clarity.

Full size image

Experimental access to the transport of the surface state of Bi1−xSbx)2Se3 provides a new laboratory for studying topologically non-trivial quantum matter and perhaps the effects of correlations among Dirac fermions too16,17,18. Although the material realization of a topological insulator with a truly insulating bulk remains elusive, the present study illustrates that quantum effects associated with the surface states of these materials can be observed if the bulk conductivity is sufficiently suppressed.

Methods

Crystals with bulk carrier densities of 2,300×1016 cm−3 and 33×1016 cm−3 (red and orange curves in Fig. 1a) were obtained by slow cooling a binary melt with different Bi/Se ratios. The principal origin of these relatively high carrier densities is Se deficiency and antisite defects. Samples with n≤11×1016 cm−3 were obtained by slow cooling a ternary melt containing progressively more Sb. Although Sb is isovalent with Bi, Sb substitution apparently acts to control the defect density in the bulk crystals, reducing the bulk carrier density. Samples with the lowest carrier density used in this study were grown from a melt with a nominal Sb/Bi/Se ratio of 7:52:130. This mixture was sealed in quartz under a partial pressure of argon. The mixture was heated to a temperature of 740 °C in 14 h, held for 4 h then cooled over 50 h to 550°, and then annealed inside the quartz ampule for 80 h. Before measurements were carried out, samples were cleaved on both sides with a scalpel blade in a dry atmosphere, mounted with contacts (using conductive silver epoxy) and pumped to 10−4 mbar within 20 min. The leads made contact along the sides of the crystal to maximize the contact with the bulk. All samples were measured using a standard four-probe configuration. Magnetoresistance and Hall effect measurements were carried out at the National High Magnetic Field Laboratory (Los Alamos) in the short-pulse (∼10 ms rise time) 55 T magnet. Sweeps at both negative and positive field polarities were measured for all temperatures and angular positions. The data were then ‘symmetrized’ by combining the data from each field polarity to extract the components that are even in magnetic field (magnetoresistance, Rx x) and odd in magnetic field (Hall resistance, Rx y). The sample was rotated so that θ=90° corresponded to field parallel to the current. The sample used for the angle dependence study had dimensions ∼0.63×0.36×0.15 mm3.